« Previous | Contents | Next »
Listen
CHAPTER SIX CONCLUSIONS AND RECOMMENDATIONS
Overview
6.1 At the level of individual events, the elements of the Scotland in the Netherlands Programme demonstrate a good "fit" with the higher-level objectives set for the promotion of Scotland in the Netherlands. For each activity, it is possible to point to evidence of exchange of practice, change in perceptions of Scotland, enhanced knowledge of Scotland and/or new or strengthened networks and contacts.
6.2 The evidence from the evaluation supports organisers' views that this was a series of successful events. We would also conclude that the events supported have generally been well delivered and well received by those participating in them. Taken individually they have all delivered value to the parties involved.
6.3 Within the Programme as a whole, many of the events addressed the International Strategy objective of "developing cooperative links and ongoing engagement that can help inform best practice in Scotland". With the exception of the Evening of Scottish Culture, there was less emphasis on the promotion of Scotland in the generally understood sense of the word. With the 60th anniversary of the contribution made by Scottish troops to the liberation of Dutch towns during World War Two as the main catalyst for the Programme, it was perhaps not appropriate to carry out more intensive and explicit promotional and related PR work. For the future, however, we recommend there be a more explicit recognition of the balance of events and relationship with strategic objectives.
6.4 We found that there was only a limited sense of the events being part of a coherent programme and as a consequence, potential benefits from the synergy of supporting a programme of activity may have been lost. Consultees rarely considered their event within the wider context of the Programme. This may not be important for them individually - their aim is to deliver to the objectives of their event - but it is important if a programme is to be more than simply the sum of its parts.
Value for money
6.5 The Scottish Executive allocated a total budget of £230,000. All costs were incurred in the financial year 2004/05. The lower cost, smaller events - the majority in the Programme - strengthened relationships and generated new learning that will influence policy at relatively little cost. The Evening of Scottish Culture demonstrated a positive effect on perceptions. This was supported by consultations with event organisers and the British Embassy. In this sense, the Evening of Scottish Culture has been a cost effective tool for engaging an audience that it would be difficult to engage with and influence through other means.
6.6 The benefits of reaching this audience are that as senior business people and civil servants they can influence future investment, encourage collaboration with Scottish companies and encourage imports of Scottish goods and services. In our view, the volume of positive feedback received indicates that there is a good chance that some of these potential benefits will be realised.
6.7 Overall, our conclusion is that the Programme has delivered value for money, but that this could have been greater if there had been more opportunity during both design and implementation to develop and benefit from potential synergies.
Programme development
6.8 In building the Programme, members of the International Forum and policy leads within the Executive were given the opportunity to contribute events. This led to activities that fitted well with aspects of public policy in which the Dutch are particularly strong - architecture and design, and elements of social policy. All of which will help in Scottish policymaking.
6.9 This approach has strengths and weaknesses. It catalysed and enabled a wide cross-section of interests in Scotland and in the Netherlands to engage in activity which otherwise may not have happened. In terms of weakness, the diversity may have limited the overall profile of the Programme in both Scotland and the Netherlands. It also made links across the Programme more difficult to achieve and some of the activity (notably that undertaken to enhance learning in the Executive) arguably could have taken place without the Scotland in the Netherlands initiative.
6.10 Evidence to justify these claims comes from a number of sources. It comes from event organisers who in retrospect would have wished to know more about what else was happening in the Programme and how their own activity might have benefited. It comes from the assessment of the media strategy and subsequent coverage, which appears to have been piecemeal and limited. It comes from our evaluation of outcomes and impacts, which appear to be event specific, with little additional benefits evident from the programme-based approach.
6.11 Our point is not that there could have been very much more synergy between the events selected and delivered - we recognise that there was limited potential for links between the actual events that were chosen for this Programme - but that designing for and exploiting synergy should be a stronger principle in developing future programmes. We also understand the considerable difficulties in putting together this type of activity where there is a limited timescale, a range of organisations each with their own objectives and the problems of securing ministerial time. The diversity of activities also reflects the fairly general nature of the international strategy and its objectives, within which a very wide range of activities can be fitted.
6.12 The approach to programme development through canvassing for events may also tend to favour activities to gain learning for Scottish proponents from international counterparts. This reasonable "self-interest" in event-level design and selection may lead to less emphasis on "promotion" in the generally accepted sense of the term.
Programme monitoring
6.13 There are presently limited monitoring or measuring mechanisms in place. This is partly because the overarching objectives of the international strategy are fairly broad, but also because many of the outcomes cannot be anticipated, for example groups choosing to work more closely together after an event or further invitations to return to the Netherlands. This is understandable in the first few Scotland in ... programmes, but with more experience it should become easier to anticipate the types of outcome that are achievable and to build in a guide as to what might be achieved into an options appraisal and monitoring framework.
6.14 In terms of scale, although a number of individual events had associated input or activity targets, e.g. numbers of attendees, there was little or no measurable outcome or impact targets set for the events or for the Programme. In reality, this is difficult to do for the kinds of objectives set for the Programme. However, we would recommend in future that during the pre-event phase of programme and events option appraisal and design, a more explicit and more fully documented development of SMART ( Specific, Measurable, Achievable, Realistic and Time-bound) objectives is attempted.
6.15 We recommend that the evaluation framework for the pre-event phase should in future examine more fully the degree to which those leading on programmes and their component events seek explicitly to design for and exploit synergies between individual activities in order to justify a programme-based approach.
6.16 Although there are difficulties in translating the undoubted successes of the Programme's events into tangible, measurable outcome and impacts of benefit to Scotland, time and again during this evaluation we were informed of the "quality" - the highly influential status - of those from the Netherlands who engaged with the events. One well-informed and experienced consultee engaged in promoting UK interests internationally also commented that " the quality of those attracted to international promotional events should be taken as a key measure of their success".
6.17 We stress that our views on the desired approach to programme design, monitoring and evaluation are less concerned about achieving significantly greater levels of quantification - this is difficult given the present broad object objectives of current strategy - but rather about ensuring that design, monitoring and evaluation takes place consistently within an overarching, programme-level structure. This will help inform the design and selection of individual activities and ensure that they are aligned to a more explicit rationale and strategy. The implementation of the evaluation framework as something taken into account explicitly during programme design will assist with this (see below).
Developing the Evaluation Framework
6.18 We consider that the pilot evaluation framework, with its three-phase structure - pre-event, event and post-event - provides an effective basis for evaluating a programme of this type.
6.19 We make three recommendations concerning the future use of the evaluation framework:
- for future programmes with as many, diverse events in terms of content and scale, consideration should be given in the pre-event phase as to the level of evaluation resource appropriate to smaller scale elements
- as indicated elsewhere in this report, we consider the "before" and "after" perceptions of Scotland survey to offer useful, underpinning evidence of effecting change. We would advise more widespread use with in-country participants engaging with programmes in future
- we would propose having a more explicit examination of the attention given at the pre-event appraisal and design stage to inter-relationships, if any, between each event in the overall programme. This should cover both strategic and operational matters. To justify a programme-based approach, we consider that synergies and an additional level of benefit should be sought during the pre-event phase and should be discernable in the ex post evaluation.
6.20 It will be important to have the evaluation framework in place at the outset and firmly in the minds of those leading on future programme and event options appraisal and selection. With these modifications, we believe that the use of the evaluation framework to inform the pre-event phase of future programmes and their associated events is likely to lead to SMARTer ( Specific, Measurable, Achievable, Relevant and Time-bound) objective setting and to more cohesive programme-level approaches).
Other issues raised
6.21 While this evaluation has focused on the achievements of the Scotland in the Netherlands Programme, we feel that it raises issues that are likely to go beyond this programme.
6.22 We would point to the following wider issues emerging during this research:
- clarity of rationale and objectives: there would be benefits from having the Programme-level rationale and strategy more clearly communicated and taken into account in the process of developing and selecting options for events
- significance of changes in perception: much emphasis has been placed on changing perceptions of Scotland and, indeed, we have pointed to evidence of some success here as a result of the Scotland in the Netherlands Programme. However, we consider that more research is required on the significance of whose perceptions are changed, what specifically is changed and crucially what is the desired versus the actual effect of such a change. It is probably self-evident that positive perceptions of Scotland internationally are a "good thing" but what more is required and expected in terms of evident benefit? What is the "theory of change" envisaged which links inputs to the change process such as the Scotland in... series to these desired benefits?
- sustainability: sustainability of benefits from programmes such as Scotland in the Netherlands is a key objective and measure of success. Although we have evidence of intentions to follow-up contacts and/or arrange reciprocal visits, we have also been advised that sustainability should not be an objective of these kinds of promotional activities. Rather they should be viewed as " keeping Scotland on the radar screen" of other countries and regions: sustainability according to this view requires the development of jointly designed, funded and implemented programmes of several years' duration. This emphasises the need to think through the progression from event to desired longer term benefits and to ensure that any resources for post-event follow-up are considered up front
- in-country activity: we consider that a factor in the pre-event appraisal stage of event development should be the balance of advantage of having the proposed event held in Scotland or held in the target country. The Cancer Seminar was held in Edinburgh during the Scotland in the Netherlands Programme without evident disadvantage. We have been given the view that people's perceptions of a country are changed most effectively by a visit to the country. These factors should be examined more thoroughly in the context of future programme design
« Previous | Contents | Next »