« Previous | Contents | Next »
Listen
ANNEX B EVALUATION FRAMEWORK AND RESEARCH METHODOLOGY
The evaluation framework
B.1 In this Annex we describe the methods adopted in the evaluation. As well as providing an evaluation of the Scotland in the Netherlands Programme itself, an important additional part of our assignment has been to pilot a new evaluation framework designed by the Scottish Executive's Analytical Services Division specifically for assessing international promotional initiatives in future. Therefore we provide here an account of the pilot and our conclusions and recommendations concerning the efficacy of the framework.
B.2 The evaluation framework requires an examination of each event within the Scotland in the Netherlands Programme in three parts:
- the pre-event or set-up phase
- the event itself
- the post-event phase.
B.3 The post-event phase is bounded in time by the span of SQW's assignment.
Methods adopted in the evaluation
B.4 We have adopted a mix of methods appropriate both to evaluating the three phases in the framework and to the time in the process from set-up to post-event at which SQW began its research. These methods are illustrated diagrammatically in the following figure.
B.5 When commissioned, most pre-event activity had been completed and indeed a number of the events had already been held. We deployed our research methods therefore either during the event phase or post-event phase. Although we researched pre-event or set-up issues, we did not conduct any pre-event stage research in real time.
Figure B.1: Research methods and sources of evidence used with respect to the three phase evaluation framework
B.6 Methods deployed have included:
- desk research - on contextual documents pertaining to policy on international promotion of Scotland and on background to individual events
- face-to-face consultations - with officials in the Executive and with event organisers in Scotland
- observational research - conducted by SQW staff at following events - the Evening of Scottish Culture, the Access to Justice Conference and the Brain Science Public Lecture and Seminar
- perceptions survey - conducted by e-mail with invitees to the Evening of Scottish Culture in the days immediately preceding the event and again c. 3 months after the event
- telephone survey - a series of telephone interviews conducted by Dutch staff of our sub-contractor, TNS, with a cross-section of Dutch participants in the Access to Justice Conference, the Brain Sciences Seminar and the Evening of Scottish Culture
- telephone interviews - a series of telephone interviews conducted with Dutch co-organisers of events and with representatives of stakeholders in the British Embassy in The Hague and The British Council
- review of press coverage in the Netherlands - based on material supplied to us by the Scottish Executive.
B.7 Throughout, we maintained close links with the Steering Group for the study consisting of officials from both Analytical Services and the International Division.
B.8 An interim report was submitted in late December, 2004. A presentation on emerging findings and conclusions was made to the Steering Group on 14 April, 2005.
Piloting the evaluation framework
B.9 To pilot the evaluation framework we developed a set of research tools - survey questionnaires and aide memoires to guide our consultations. These were based on our "deconstruction" of strategic aims and objectives associated with the Scotland in the Netherlands Programme and its events, plus the full description of a draft evaluation framework provided by the client.
B.10 The charts at the end of this Annex provide the design template we used in our research. This is presented in the form of charts for each of the three phases in the framework - pre-event, event and post-event - for each event in the programme. More specifically, the template:
- summarises the likely evidence base and requires that documents and consultees are identified. It indicates the methods to be used in each phase
- sets out the key research questions - the basis for our primary research
- sets out the key deliverables from the event-level research
- is intended to provide a template which can be populated in a fairly systematic way for each event.
Strengths and weaknesses of the evaluation framework
B.11 Overall, we consider that the three phase approach to framing the evaluation process is both appropriate and effective. It provides a good basis for examining the key linkages and progression between inputs (including importantly time and commitment to initial design and planning), activities, immediate outputs and subsequent outcomes.
B.12 The evaluation process is time consuming and fairly resource intensive. In the context of the Scotland in the Netherlands Programme evaluation, full application of the framework in terms of type and scale of primary research was directed to only to a sub-set of events - the Evening of Scottish Culture, Brain Science and Access to Justice seminars. If a programme with as many and diverse events is undertaken in future, consideration should be given in the pre-event phase to the level of evaluation resource appropriate to smaller scale elements.
B.13 Although this has been a useful and valid pilot test, we consider that greater benefit from the application of the framework will be derived in future from having it available for more explicit consideration during the pre-event phase by all those leading on subsequent programmes and their events.
B.14 The key research questions required by the framework would, we believe, inform and provide useful additional challenge to the process of options appraisal and testable objective setting during the pre-event phase.
B.15 As indicated elsewhere in this report, we consider the "before" and "after" perceptions of Scotland survey to offer useful, underpinning evidence of effecting change. It proved effective in use with the attendees at the Evening of Scottish Culture. We would advise more widespread use with those engaging with programmes in future.
B.16 In terms of weaknesses, we would point to the limitations in the framework in terms of assessing the programme-level benefits as distinct from individual event-level benefits. We would propose building in a more direct, explicit examination of the inter-relationships, if any, between the contributions of each event in the overall programme at the pre-event appraisal and design stage. The framework at present does demand an evaluation of "fit" with high level promotion of Scotland and country/programme-specific objectives. However, this still allows a portfolio of activities to be delivered, all aligned appropriately with strategic objectives, which nevertheless together do not contribute more than the sum of their parts. To justify a programme approach, we consider this additional level of benefit should be sought during the pre-event phase and should be discernable in the ex post evaluation. This should cover both strategic and operational matters.
Recommendations for the evaluation framework
B.17 For programmes with many and diverse events in future, consideration should be given in the pre-event phase to the level of evaluation resource appropriate to smaller scale elements.
B.18 The framework should include an explicit assessment of how opportunities for developing and exploiting "synergy" between individual events within a programme were considered during the pre-event phase as well as how the benefits of synergy are evident in the ex post evaluation. This would help to justify a programme-based approach.
B.19 With these modifications, we believe that the use of the evaluation framework as a programme design tool at the pre-event phase of future programmes and their events is likely to lead to SMARTer (Specific, Measurable, Achievable, Relevant and Time-bound) objectives and to more cohesive programme-level approaches.
Table B.1: Evaluation Framework Pre-Event Phase Template
Event name: | Evidence base: | Research methods: | Research questions: | Deliverables: |
|---|
EVENT A | SECONDARY SOURCES: Scottish Executive contextual documents and planning and approval documents (give references); others (specify) PRIMARY SOURCES: consultations with named members of the Executive's staff engaged in event proposal appraisal and organisation; consultations with representatives of third party organisations involved in event design and implementation. (Note: consultations with individuals informed about all phases of activity would cover all phases in one interview) | Desk research; consultations with policy leads and organisers | Which higher level objectives/themes were to be addressed by this event? What country-specific objectives were being addressed by this event? What objectives were set specifically for the event - including activities, outputs, short and longer term outcomes/impact etc? What was known at the outset about the country and in particular those aspects of it that relate to the objectives for this event? What baseline evidence was available or subsequently arranged in order to refine objectives and targets, and to enable the subsequent tracking of change - including arrangements with e.g. VisitScotland and Universities Scotland on visitor numbers and student numbers respectively? What was known at the outset about comparable activities pursued by other countries in The Netherlands? Was benchmarking information available to inform objective setting and options appraisal? What options to realise these objectives were considered and why was the option of this event selected? What design and implementation plan was developed and how effectively was this process managed? Who was involved in design and implementation - with what inputs and with what objectives and expectations? What other inputs were committed and from what sources? How were the necessary resources determined? What monitoring and measuring arrangements were envisaged for the event and how were these to be implemented? What were the key strengths and weaknesses of the pre-event phase practices? What would be the three most important things to change next time in the pre-event phase? | Précis of key extracts from documents; transcripts of consultations (anonomised where appropriate); summary of findings for each research question, appropriately supported by references; summary of lessons and recommendations |
Table B.2: Evaluation Framework: Event Phase Template
Event name: | Evidence base: | Research methods: | Research questions: | Deliverables: |
|---|
EVENT A | Secondary sources: documentation on the event; information on media coverage Primary sources (as appropriate to the event) : from among - event phase monitoring and evaluation information obtained by the Executive; similar information obtained by other organising bodies; observations made directly by Scottish Executive staff and by those of other organising bodies; observations by SQW staff; perceptual survey data collected by SQW - strictly obtained immediately prior to attendees "experiencing" the event. (Note: consultations with individuals informed about all three phases of activity would cover all phases in one interview). Also anecdotal feedback provided by attendees to organisers at events. | Desk research; observations; consultations; survey of attendees | How well were the immediate objectives for the event met? Did the event achieve the expected activities and target outputs envisaged in the pre-event phase? Did the event achieve the immediate outcomes envisaged in the pre-event phase - e.g. immediate perceptual change, creation of new linkages? How effective and efficient were the operational processes around the event? How well did they match with what was planned? Did the inputs prove to be appropriate/sufficient to deliver what was intended? What about the event worked well and what did not? How effective were any monitoring and measuring arrangements in place for the event and what have these revealed? What were the key strengths and weaknesses of event activities? What would be the three most important things to change next time in the event phase? | Précis of key extracts from documents; transcripts of consultations (anonomised where appropriate); summary of findings for each research question, appropriately supported by references; summary of lessons and recommendations |
« Previous | Contents | Next »