« Previous | Contents | Next »
Listen
CHAPTER THREE EVALUATION FRAMEWORK AND RESEARCH METHODOLOGY
3.1 In this Chapter we summarise the methods adopted in the evaluation.
3.2 As well as providing an evaluation of the Scotland in the Netherlands Programme itself, an important additional part of our assignment has been to pilot a new evaluation framework designed by the Scottish Executive specifically for assessing international promotional initiatives in future. In addition to the summary of methods provided here, we provide therefore a fuller account of the pilot and our conclusions and recommendations concerning the efficacy of the framework in Annex B.
3.3 The evaluation has examined each event within the Scotland in The Netherlands Programme in three parts:
- the pre-event or set-up phase
- the event itself
- the post-event phase.
3.4 In addition, we have also examined overall Programme level matters both in terms of aggregated benefits and in terms of other cross-cutting evaluation issues.
Methods adopted in the evaluation
3.5 We have adopted a mix of methods appropriate both to these three phases and also to the time in the process from set-up to post-event at which SQW began its research. When commissioned, most pre-event activity had been completed and indeed a number of the events had already been held. We deployed our research methods therefore during the event phase and/or post-event phase. Although we researched pre-event or set-up issues, we did not conduct any pre-event stage research in real-time.
3.6 Methods deployed have included:
- desk research - on contextual documents pertaining to policy on international promotion of Scotland and on background to individual events
- face-to-face consultations - with officials in the Executive and with event organisers in Scotland
- observational research - conducted by SQW staff at four events - the Evening of Scottish Culture, the Access to Justice Conference and the Brain Science Public Lecture and the associated Seminar
- perceptions survey - conducted by e-mail with invitees to the Evening of Scottish Culture in the days immediately preceding the event and again c. 3 months after the event
- telephone survey - a series of telephone interviews conducted by Dutch staff of our sub-contractor, TNS with a cross-section of Dutch participants in the Access to Justice Conference, the Brain Science Seminar and the Evening of Scottish Culture
- telephone interviews - a series of telephone interviews conducted with Dutch co-organisers of events and with representatives of stakeholders such as the British Embassy in The Hague and The British Council
- review of press coverage in the Netherlands - based on material supplied to us by the Scottish Executive.
3.7 Throughout, we maintained close links with the Steering Group for the study consisting of officials from both Analytical Services and the International Division - Overseas Events team. An interim report was submitted in late December, 2004. A presentation on emerging findings and conclusions was made to the Steering Group on 14 April, 2005.
Figure 3.1: Research methods and sources of evidence used with respect to the three- phase evaluation framework
Findings of the Evaluation Framework Pilot
3.8 Overall, we consider that the three phase approach to framing the evaluation process is both appropriate and effective. It provides a good basis for examining the key linkages and progression between inputs (including importantly time and commitment to initial design and planning), activities, immediate outputs and subsequent outcomes.
3.9 This has been a useful and valid pilot test. However, we consider that greater benefit from the application of the framework will be derived in future from having it available for more explicit consideration during the pre-event phase by all those leading subsequent programmes and their associated events. The key research questions required by the framework will we believe inform and provide useful additional challenge to the process of options appraisal and testable objective setting during the pre-event phase.
3.10 As expanded upon in Chapter 6, we make three recommendations concerning the future use of the evaluation framework:
- consideration should be given in the pre-event phase as to the level of evaluation resource appropriate to smaller scale elements
- "before" and "after" perceptions of Scotland surveys merit more widespread use with in-country participants engaging with programmes
- we propose a more explicit and upfront examination of potential synergy between each event planned for a programme in order to justify a programme-based approach.
« Previous | Contents | Next »