« Previous | Contents | Next »
Listen
CHAPTER 3: EVALUATION OBJECTIVES AND METHODS
Chapter 1 described how the evaluation of Government consultation activities is by no means routine or systematic, and that a deliberate decision was taken by the commissioners of the consultation on Best Value in Public Services to include a formal evaluation as an element of their consultation. The preceding discussion also highlighted the absence of a preferred model of consultation evaluation within the Scottish Executive, and described a variety of approaches taken by different bodies to develop loose frameworks for evaluation which can be adapted for different contexts.
Against this background, this evaluation was commissioned, following a tendering competition, to provide an independent assessment of the extent to which the consultation achieved its stated aims and to identify lessons to be learned.
This Chapter outlines the nature of the evaluation: its stated objectives, timing, scale and the method it adopted.
3.1 THE EVALUATION OBJECTIVES
The evaluation was commissioned to evaluate the quality and success of the Scottish Executive's consultation process on Best Value in Public Services, and in so doing, to demonstrate and share good practice, or lessons learned.
Specific research objectives for the evaluation were:
- To evaluate whether the consultation exercise met its objectives;
- To evaluate the suitability of the approach chosen to consultation (ie non-written);
- To review the planning of the consultation process, and informal pre-consultation dialogue with external stakeholders;
- To observe and critically evaluate the operation of the 5 stakeholder engagement seminars, and the December Ministerial conference 3 and specifically:
- To assess the success of grouping stakeholders by Ministerial portfolio
- To consider the appropriateness of the venue chosen in terms of accessibility and convenience
- To assess the benefits and problems associated with facilitation of the events
- To evaluate the quality of debate and management of conflict.
- To probe stakeholders' views on the consultation process, their involvement, and their capacity to influence the future direction of Best Value policy through the use of self-completed questionnaires;
- To make suggestions and recommendations for improved engagement as a result of this evaluation.
3.2 TIMING OF THE EVALUATION
The evaluation of the consultation commenced part way through the consultation process, and is now taking stock, largely for reasons of expediency, with the consultation having only recently finished. Thus, in terms of timing, the identification of longer-term outcomes and impacts is not yet possible, but the evaluation can comment on the inputs, processes and outputs to emerge from the consultation.
The individual components of the consultation were described in detail in the previous Chapter. They are listed in Figure 2 below to illustrate the timing of the evaluation in relation to the elements of the consultation exercise.
Figure 2: Timing of the evaluation in relation to the elements of the consultation exercise
3.3 SCALE OF THE EVALUATION
The evaluation was funded with fixed costs including overheads at approximately £13,500 (excluding VAT). Due to the spread of consultation activity across several months (rather than intense activity in a short timescale) it was possible to staff the evaluation with one dedicated external evaluator which provided important consistency across the various components of the exercise.
The evaluation methodology was designed to provide a generalisable, relatively low cost, simple model, aspects of which could be replicated on a low budget by other consultation commissioners across the Scottish Executive.
3.4 EVALUATION METHOD
The method designed for the evaluation took as its starting point the following broad questions and sought to construct research tools which would generate data to address these:
- What were the aims of the consultation?
- What were the inputs to the consultation?
- By what process(es) was the consultation undertaken?
- What were the immediate outputs?
- What were the longer-term outputs?
- What were the predicted outcomes?
- What are the lessons to be learned?
- What were the barriers/what were the successes?
The evaluative approach was largely qualitative in nature although where a few opportunities arose for some quantitative measurement, this was exploited. The evaluation was conducted in the following stages:
3.4.1 Briefing with Consultation Commissioners
An early meeting was held with the policy representatives responsible for overseeing the consultation process. This provided the opportunity for clarification of the scope of the consultation and the proposed evaluation approach.
3.4.2 Stakeholder Seminars: Observation
The evaluator attended all 5 of the stakeholder seminars as an observer. Observation was active and systematic, with observation notes recorded on a dedicated "observation grid" designed specifically for the events. The grid comprised 7 pages headed as follows:
- General Notes
- Entry activity: questions raised by participants; general comments; level of involvement of participants; other observations; value of activity
- Flip Chart Comments: questions raised by participants; general comments; level of involvement of participants; other observations; value of activity
- Small Group Discussions: questions raised by participants; general comments; level of involvement of participants; other observations; value of activity
- Closing Plenary: questions raised by participants; general comments; level of involvement of participants; other observations; value of activity
- Areas of Conflict or Agreement: Issue; Balance of View; Resolution (process/outcome)
- Issues of Format and Facilitation
3.4.3 Stakeholder Seminars: Participant Self-Completion Questionnaire
Every participant at the stakeholder seminars was asked at the end of the event to complete a questionnaire. A copy of the questionnaire is in Annex 1.
The reason for issuing the questionnaire, and in particular, its importance as a vehicle for gathering evidence for the evaluation of the consultation process, was emphasised by the evaluator. A few participants left their respective seminar early and arrangements were made for them to complete the questionnaire elsewhere and post it to the researcher. In addition, participants were encouraged to contact the researcher with any further views they had on their seminar following the event.
In total, 48 completed questionnaires were collected from the seminars and/or posted to the researcher following the events, representing a 98% response rate. In addition, further comments were submitted by one participant via email.
3.4.4 Stakeholder Seminars: De-Brief with Seminar Facilitators
Immediately following each seminar, the evaluator met with the facilitators of the event to ascertain views on:
- What went well
- What did not go as well as expected
- Lessons for the next event
- Unexpected issues
- Whether the event had contributed to meeting the aims of the consultation
Views were recorded by the evaluator for analysis in the evaluation.
3.4.5 Stakeholder Seminars: Follow-up of Non-Attendees
Overall, 27 invited delegates who had originally been expected to attend, failed to participate in a stakeholder seminar. Some of these invitees had indicated that they would not be attending, others failed to show on the day. All 27 were contacted by email by the evaluator and invited, politely, to provide a reason for their non-attendance. The response to the enquiry was relatively high with 14 invitees providing a reason for their non-appearance (52% response rate).
3.4.6 Informal Consultation: Telephone Interviews with Informal Consultees
Telephone interviews were conducted with a sample of consultees who had been involved in the one-to-one in-depth discussions with the Scottish Executive representatives at the start of the consultation process. In order to meet confidentiality agreements, the selection of officers for interview was conducted indirectly by means of the Scottish Executive consultation team inviting participants to volunteer for interview. Thus, interviewees were self-selected. Overall, 4 participants volunteered for a telephone interview. They represented organisations of a range of size and policy field.
The telephone interview schedule is reproduced in Annex 2. The average duration of interview was 16 minutes.
3.4.7 In-depth Group Interview with Consultation Commissioners
An in-depth face-to-face group interview was conducted by the evaluator with 3 members of the consultation commissioning team. Two members of the team were actively involved in the consultation. The remaining member had left the team but had been involved previously in planning the consultation and had taken part in the early stages. The interview was conducted after the stakeholder seminars had been completed, but before the external reference group had commenced its meetings.
The interview schedule was semi-structured and wide-ranging. It aimed to facilitate investigation of the choice of consultation approach, the underlying rationale, views on the success or otherwise of each element of the consultation and lessons learned. A copy of the topics covered and questions asked is at Annex 3.
The group interview took place over 2.5 hours. It was audio-taped and transcribed to aid analysis.
3.3.8 External Reference Group: Observation
Meetings of the external reference group commenced in November 2005. Members met regularly, with meetings still ongoing at the end of the evaluation period (March 2006). The evaluator attended 2 of the group meetings, in December 2005 and in March 2006, as an observer. At these meetings, observation notes were taken regarding the nature of the meeting, the input from members, the role played by the Scottish Executive consultation team delegates, any conflicts or tensions and how these were resolved, and the outputs of the meetings.
3.4.9 External Reference Group: Final Questionnaire
Most of the members of the external reference group had been involved in other elements of the overall consultation process, such as the informal one-to-one discussions, and/or the stakeholder seminars. A final self-completion questionnaire was designed for issue to group members to capture their reflections on the consultation process overall, their individual input, and what they considered had been gained by the process on an individual level and in general. The questionnaire was distributed personally to 11 delegates who attended the March meeting and by post to 3 members who had sent their apologies. Delegates were asked to take the questionnaire away with them and return it by post to the evaluator. The response rate was 57%, with the 8 forms completed containing very comprehensive and thoughtful responses. A copy of the final questionnaire is provided at Annex 4.
3.4.10 Evaluation Report
This evaluation report was drafted by the evaluator of the consultation process and brings together the evidence from the range of methodological approaches applied across the various consultation elements.
The evidence was interpreted within the context of the key evaluation questions on the extent to which the consultation has achieved its aims, in what manner and at what cost.
3.5 EVALUATION MODEL
In the interests of facilitating the design of future formal evaluations of Government consultations, a degree of introspection is useful regarding the nature of the evaluation model deployed for the Best Value consultation.
In particular, although it is clear that the purpose of the research has been to evaluate a participatory exercise, choices have been made along the way about the degree to which the evaluation itself should be "participatory". It has been argued that all evaluations could be labelled as participatory on account of their needing to take account of the views of stakeholders. However, the extent to which stakeholders are involved in, for example, deciding on the key evaluation questions and the values they attach to different outputs and outcomes are variables which may differ between evaluations.
As outlined in Chapter 1, evaluation of consultation is still in its infancy, with no established history of tried and tested evaluation frameworks. The evaluation model adopted for the Best Value in Public Services consultation combines a mix of pre-planned design aimed at capturing the data required to meet the information requirements of a series of standard evaluative questions identified by the evaluator, along with various ad hoc methodological adjustments made in response to changes in the design of the consultation process itself.
Such a combination represents a merger of what Involve (2005, p9) terms second and third generation evaluation, where second generation comprises an evaluator describing patterns of strengths and weaknesses of the consultation and third generation comprises the evaluator also operating as a "judge" (as well as describer). However, in this instance, aspects of the Best Value consultation are also "judged" by participants and commissioners to present a range of perspectives on effectiveness.
3.6 SUMMARY OF KEY POINTS
- A decision was taken by the consultation team to commission a formal, independent evaluation as an element of their consultation.
- The external evaluation commenced soon after the start of the consultation and is reporting with the consultation in its final stages.
- The evaluation was designed to provide a generalisable, low cost, simple model, aspects of which could be replicated by others.
- The evaluative approach was largely qualitative in nature and designed to address key evaluation questions on effectiveness and efficiency.
- Evaluation methods comprised face-to-face and telephone interviews, event observation and participant questionnaires.
- The evaluation model could be described as combining the evaluator's identification of strengths and weaknesses with a judgement on effectiveness by both evaluator and participants.
« Previous | Contents | Next »