The Scottish Government

« Previous | Contents | Next »

Listen

CHAPTER 5 EVALUATION OF THE OVERALL PROGRAMME

Introduction

5.1 The purpose of this chapter is to bring together the findings on each of the elements and set them within the context of the overall Programme. This is difficult given what has been a very diverse range of activities and outcomes. In doing this, we draw not only on research on the separate events, but also on consultations with individuals in the Scottish Executive and in other organisations that had involvement with the Scotland in the Netherlands Programme as a whole at a more strategic level.

5.2 The starting point is to bring together the results for each of the events and draw conclusions on their effectiveness. Table 5.1 sets out the benefits reported from each of the individual events.

5.3 In all cases, those involved in an individual event believe that it has achieved its event-level objectives. These were not always formally set out and generally did not include specific target outcomes. However, the analysis indicates that the outcomes delivered have met and often exceeded the expectations of those involved.

5.4 The Brain Science events have helped form a longer term relationship with Dutch partners, have led to consideration of collaborative initiatives and more generally have raised the profile of Scottish science and its academic community.

5.5 The Cancer seminar enabled an exchange of views and information, strengthened relationships and generated new contacts. Specifically it will contribute to the strategic report on cancer patient needs in Scotland. These benefits were achieved without the event having to take place physically in the Netherlands.

5.6 The learning journey on drugs/substance abuse and on prostitution issues has both developed useful contacts and, importantly, helped to inform Scottish Executive thinking on these policy areas. This was primarily about learning from Dutch experience and building relationships rather than promotion of Scotland. In this respect, the initiative has been very successful.

5.7 The study trip and conference on Access to Justice helped improve understanding of the Scottish legal structure in the Netherlands and enabled learning from the Dutch relevant to the strategic review of the Scottish Legal Aid system.

Table 5.1 Summary of benefits

5.8 Among the arts-related activities, organisers also believe that objectives have been met in full. The Wee Stories Theatre visit strengthened existing links with Dutch counterparts and made further appearances at Festivals more likely. The theatre exchange between the ISH and EGYT also strengthened organisational links, provided real inspiration for the young people involved and will provide new ideas for the SAC's Youth Arts Strategy. The NAEN study trip has similarly strengthened relationships and will help contribute to Scottish arts policy making in cultural entitlement.

5.9 The urban planners' study trip was particularly influential in bringing together the Better Cities group and encouraging exchange of practice in waterfront developments. It also generated ideas that will be used in future projects in Scotland. The Landforms Exhibition promoting Scottish architecture attracted just under 300 Dutch visitors.

5.10 Finally, the research undertaken for the Evening of Scottish Culture, the centrepiece of the Scotland in the Netherlands Programme, has demonstrated significant shifts in awareness and perceptions of Scotland among an influential audience. All the research points to a successful and well-run event.

5.11 There are also some important lessons, many of which cut across different events. These can be summarised as:

• Increasing the opportunities for networking - a number of events and seminars did not have sufficient time for participants to develop new relationships
• Building in discussion on follow up - some participants felt that including a session on "what next?" would help to generate more collaboration
• Allowing sufficient planning and lead time - this is always an issue for event organisers and is especially important if potential synergies between events in a programme are to be identified and exploited
• Debriefing meetings - most event organisers would welcome a session, post -event, to share experiences of their events and look at ways in which they could be improved.

5.12 These common themes point to some simple suggestions that would help participants to get the most from their experience. We would strongly recommend that these lessons be considered in the planning of any future programme. For example, it is important in policy-related and academic conference/seminar events that time is built-in for informal networking and, if possible, for participants to discuss opportunities for further related activities. Finally, most of those consulted would welcome a debriefing meeting with the Executive. All these suggestions may not be appropriate when there is such a wide range of activities and events, but with more structured programmes, these suggestions would help.

5.13 In summary, we are confident in concluding that at the individual event level there have been some valuable benefits achieved and that there are sufficient benefits to justify the expenditure made. However, it could be argued that these events could have been carried out independently, assuming that the funding was available. It is therefore appropriate to consider how bringing together the individual events under the Scotland in the Netherlands umbrella has changed or enhanced their delivery and associated benefits.

Review of media coverage

5.14 Overall media coverage in the Netherlands was limited, although not significantly less than appears to have been anticipated by stakeholders based in the Netherlands. In the view of one consultee in the Netherlands: " the Dutch press already covers the UK, including Scotland - Scotland in the Netherlands offered nothing exceptional". Event organisers generally felt that a more co-ordinated approach to the development and implementation of a media strategy would have been beneficial. There appears to have been some uncertainty over who was responsible for what - whether this was matter for individual event organisers or a function to be picked up at overall programme level.

5.15 The evaluation has included a review of media coverage obtained for the Scotland in the Netherlands Programme. In scope, it was agreed to limit this review to press cuttings provided to us by the Scottish Executive, together with information provided by consultees, notably event organisers plus a representative of the press office of the British Embassy in The Hague.

5.16 We received no documentary evidence of media coverage for the Programme in Scotland beyond copies of Executive press releases that we envisage had wide local distribution. We understand that an article on the ISH Theatre street dance activity with the East Glasgow Youth Theatre was published in the Times Educational Supplement Scotland.

5.17 In addition to the Executive's press releases, the British Embassy in The Hague issued a press release in late September, 2004 to launch a new web site (www.i-uk.nl) on information about the UK. This included the following statement: "As part of the Scottish Executive's campaign Scotland in the Netherlands, an entire section on i-uk.nl will be dedicated to Scotland and the campaign events until the end of this year". This was picked up in at least one Dutch language publication, the Journal. Web site visits increased generally after this publicity, but we understand not especially to the parts of the site devoted to Scotland.

5.18 The same press release launched a creative competition to develop a modern image of Scotland that was judged by the British Ambassador. The assignment was to submit a visual expression of the image that Scotland brings to mind, illustrated by completing the sentence 'When I think of Scotland …". We understand this received only a small number of applications: the winning entry 'When I think of Scotland, I remember being far from the madding crowd' was illustrated by a collage of photographs that conveyed the "tranquil and peaceful" image the winner had of Scotland.

5.19 An interview with the Scottish Deputy Minister for Justice was published in English in the Netherlands. This interview provided information about the Scotland in the Netherlands Programme generally as well as focusing on the subject of drug policy.

5.20 The press cuttings provided to us illustrate coverage of UK involvement in the events associated with the 60 ^th anniversary of the Battle of Arnhem that were held around the time of the Scotland in the Netherlands Programme. An Executive press release covered the First Minister's attendance at the commemoration service. An article in the Dutch newspaper De Telegraphe featured a picture of a Scots veteran.

5.21 The cuttings also include articles from the Dutch press on the East Glasgow Youth Theatre/ ISH Theatre activities, one of which features the Executive's Minister for Culture.

5.22 In terms of media coverage for future programmes, our consultees recommend a more focused, better co-ordinated and more targeted campaign. However, we sense from the responses that in locations in which the UK and Scotland are already quite well known limited press coverage is regarded as almost inevitable without an "exceptional" focal point activity.

Programme level issues

5.23 It is more difficult to analyse performance and benefits arising at a Programme level. Aggregate benefits have been categorised under five headings:

• strengthening and initiating organisational relationships
• raising awareness and influencing perceptions of Scotland and specific areas of activity
• strengthening and initiating individual and business level relationships
• encouraging collaboration
• learning and exchanging good practice

5.24 All the elements of the Programme contribute to these to varying degrees, but on balance, we found that much of the Programme had focused on the objectives of strengthening organisational relationships and on learning, and the exchange of good practice. The RSE, SAC, other arts groups, The Lighthouse and the Scottish Executive all benefited from opportunities to renew and develop relationships. The individual events delivered many benefits, particularly in building and reinforcing relationships, and through the potential for follow-up or reciprocal visits to Scotland. There appears to have been a great deal of learning and there seems little doubt that public policy in Scotland will have benefited from these activities.

5.25 Many of the events address the objective within the International Strategy of developing cooperative links and ongoing engagement that can help inform best practice in Scotland and rather less about the promotion of Scotland in the generally understood sense of the word. The Evening of Scottish Cultural evening was arguably the exception.

5.26 One of the possible reasons for this balance was that the catalyst for the Netherlands Programme was the 60th anniversary of the contribution made by Scottish troops to the liberation of Dutch towns during World War Two. In this context, the promotion of Scotland was done in a lower key way: the main benefits appear to have been around relationship building.

5.27 At a Programme level there has been only a limited sense of the events being part of a coherent, inter-related programme. The feedback indicates that the Netherlands activity could perhaps more accurately be described as a collection of events rather than a programme. As a consequence, potential benefits from the synergy of supporting a "programme" were difficult to achieve. The types of benefits that could be generated through a programme would include for example the ability to achieve co-ordinated media coverage; ability to put in place a consistent way to monitor outcomes; the development of more widely recognisable branding. A number of consultees also felt that because the Programme was spread over a longer period than previous Scotland in... initiatives it did not have the same concentrated effect and had generated less momentum.

5.28 In part, this is related to the choice of events that did not offer much potential synergy. More fundamentally, it may reflect the way in which the Programme was assembled. However, it is also important to recognise that the shape of a programme is likely also to be influenced heavily by time pressures and the availability of key people to participate in events. Nonetheless, the opportunities to link activities within a programme, to achieve synergy, are significant and we recommend that these issues are worthy of further attention in future.

5.29 The development of the Programme has been led primarily by proposals from policy leads in the Executive and partner organisations in the International Forum. This gives all these stakeholders an opportunity to use the Scotland in ... programmes as a vehicle to develop events. On this occasion, this produced an interesting and diverse programme, but there is a danger in this approach of the initiative being used opportunistically by others to focus more on their own internal objectives and less explicitly on those of the promotion of Scotland.

5.30 There were limited monitoring or measuring mechanisms in place. This is partly because the objectives are often described as "broad". Many of the outcomes cannot be anticipated, for example groups choosing to work more closely together after an event or further invitations to return to the Netherlands. This is understandable in the first few Programmes, but with more experience of Scotland in... activities it should become easier to anticipate the types of outcomes that are achievable and to build some guidance as to what might be achieved into an options appraisal and monitoring framework.

5.31 One of the issues that the current research has raised is how an evaluation framework can take into account the synergistic effects of the Programme as opposed to evaluating a series of individual events. The evaluation framework being piloted in this research is central to this (see Annex B). The approach we have taken in this pilot will no doubt be refined over time, but we are confident that it will greatly help in structuring the development of future events and in the assessment of their achievements.

Programme costs

5.32 The Scottish Executive provided a detailed breakdown of the Programme costs. This includes the direct expenditures associated with each of the events, including funding for the SAC, RSE and Apple Tree. It also includes all the smaller costs to cover venue hire, performers' fees and gifts for those attending the Evening of Scottish Culture. Although it includes costs of travel and expenses, it does not include the time costs of Scottish Executive staff or any additional costs that organisations such as SAC or RSE may have incurred themselves in organising their events.

5.33 The Scottish Executive allocated a total budget of £230,000. All costs were incurred in the financial year 2004/05.

Value for money

5.34 To assess value for money it is usual to monetarise both the scale of the costs and the benefits. In this case, while the costs are relatively transparent, the benefits are diverse and complex. At an event level, each of the individual events has produced valuable outcomes. We would argue that at this level, the investment has generated links, relationships and learning that represent good value for money, although we would add that this would depend on how these relationships are developed in the future.

5.35 The Evening of Scottish Culture was the most expensive element of the Programme and therefore justifies a closer examination. Our research has produced a quite detailed assessment of the changes in perceptions that were achieved among attendees.

5.36 It is appropriate that the event is seen to contribute to generating impacts on the Scottish economy, for example by encouraging new export sales, business links and collaborations. The survey indicates that the event did have an impact on perceptions and the telephone interviews also identified some new contacts and linkages that may in the future generate benefit for Scotland.

5.37 Had the event attracted more media coverage than it did, or helped to join up the other events in the Programme, the case for having achieved value for money would have been easier to make. However, as a standalone event, without wider coverage, a judgement must be made on whether the changes in perception among those that attended justified the investment. This in turn depends on the influence of those that attended.

5.38 Based on the feedback from the British Embassy and the organisers, we understand that the guests were people who will have significant influence across a range of business and public service interests. Those attending are considered to be opinion formers with a wider circle of influence than would be reached with other traditional forms of promotion. The changes in perceptions recorded are therefore more likely to impact upon policy and business decisions than could be generated through traditional promotional tools. This type of benefit cannot easily be measured and therefore an assessment of value for money comes down to how the goodwill generated translates into decisions and actions that benefit Scotland.

5.39 Overall, our conclusion is that the majority of lower cost, smaller events has delivered value for money, primarily through the development of relationships and new learning that will influence policy. The Evening of Scottish Culture was a cost effective tool for reaching a specific audience. It demonstrated a positive effect on perceptions and the nature of the audience means that there are likely to be longer-term benefits.

5.40 These longer-term benefits include future investment, collaboration, importing Scottish goods and services as well as more generally "enhancing the profile of Scotland post-devolution". The volume of positive feedback received through the various strands of the evaluation suggests that there is a good chance that some of these potential benefits will be realised. It would only take a modest amount of new activity to justify this investment and on this basis we would conclude that the event will prove to be value for money.

5.41 We would conclude that, at a Programme level, Scotland in the Netherlands has delivered reasonable value for money. However, in the case of the Evening of Scottish Culture, the most expensive event, this requires assumptions to be made about the value of influencing a specific audience. We also conclude that the value for money overall could have been greater if potential synergies from a programme-based approach had been achieved. We expand upon both of these issues in the following chapter.

« Previous | Contents | Next »