The Scottish Government

« Previous | Contents | Next »

Listen

CHAPTER 1: EVALUATING CONSULTATIONS

1.1 STATE OF PLAY: OVERVIEW

The evaluation of the Best Value in Public Services consultation took place at a time of wider development and change in both consultative techniques and consultation evaluative approaches. However, an examination of relevant literature reveals an imbalance between the two, with developments in evaluation design trailing behind the more rapid progress which has been made in developing new and innovative consultative approaches.

The different pace of development between consultation techniques and their evaluation has led to the use of approaches not yet grounded in empirical evidence of what works and why. Indeed, in 2001 the OECD remarked that no OECD country appeared to be conducting any systematic evaluation of government performance in public participation ² in policy-making. By 2004, Rowe and Frewer noted that:

" the merits of participation are difficult to ascertain as there are relatively few cases in which the effectiveness of participation exercises have been studied in a structured (as opposed to highly subjective) manner" (p512).

Many commentators have suggested reasons for the paucity of robust and systematic evaluative models in this context. Nicholson (2005a) summarises a selection of those encountered in her literature review of civic participation in public policy-making across Western democracies:

• Confusion over what is meant by "effectiveness" of consultation
• Difficulties in identifying longer term outcomes when evaluation is often executed at the short term output stage
• Tendency for commissioners to be content with subjective measures of effectiveness
• Difficulties of sensible benchmarking when contextual differences between consultations can be significant
• Confusion over the ultimate purpose of the consultation exercise

In response to this gap in evaluative provision, a variety of attempts are now being made to devise loose frameworks for evaluation which can be adapted to suit particular circumstances and contexts. For example, InterAct (2001) has produced a working paper containing, " a framework for evaluating participatory, deliberative and co-operative processes" (p1). Within the field of environmental policy, Abelson et al (2003) identified what they considered should be the 4 key components of any evaluation of consultation activity:

• Representation
• Structure of the process or procedures
• Information used in the process
• Outcomes and decision arising from the process

Jones and Gammell (2004) were attracted to a Best Value concept to evaluate consultations and suggested criteria based upon the 3 E's and 4 C's "formula":

Various Government Departments are also engaged in devising comprehensive frameworks for evaluating consultation activities. For example, the Department of Health funded advisory group Involve, has developed a framework for use in assessing a variety of different participatory techniques.

However, even with the strides now being taken in devising appropriate frameworks for consultation evaluation, areas of weakness in approach still remain. In particular, Involve (2005) considers that:

" there has been very little evaluation of the costs of participation…… while the number of large scale evaluations of participatory working in major UK public policy areas has grown in the last couple of years… the costs are not covered in any detail at all" (p8).

1.2 state of play: SCOTTISH EXECUTIVE

Reviews undertaken in 2000 and 2005 respectively provide insight into the position of the Scottish Executive in terms of routine evaluation of its consultation exercises. In 2000 the Executive's Policy Unit undertook an investigation into the consultation approaches used by the Government to promote participation in policy-making. In its assessment of consultation evaluation the report states:

" few consultation exercises …appear to have been subject to any evaluation. This makes it impossible to judge whether the exercise was successful. It also stops us from learning lessons about what works" (p11)

The authors go on to recommend that policy divisions, " produce a one or two page evaluation of how far each consultation exercise met the objectives set" (p11)

A more recent examination of consultation activity in the Scottish Executive (Nicholson, 2005b), provides an indication of progress since 2000 in evaluative approaches relating to Government consultation. Nicholson reports on participation activity undertaken by the Scottish Executive in 2004. Reflecting on consultation evaluation, she notes:

" the evaluation of civic participation activities is an issue which requires further consideration and more guidance for Scottish Executive staff on consistent and meaningful approaches" (p1)

Of the 273 Government participation activities identified by Nicholson which took place in 2004, around half (49%) were reported by their commissioners as having been evaluated or were about to be evaluated. Amongst the different activities identified, conferences and written consultations were the most likely to have been evaluated, with working groups amongst the consultative approaches least likely to have been subjected to any formal evaluation.

This review suggests that some progress in the incidence of evaluation of consultation activities has been made since 2000. However, closer examination reveals that in many of the cases where evaluations have been attempted, these have tended to focus on the process of the activity rather than its outcome. Where evaluations did not take place and/or were not planned, many commissioners of participation activities commented that they would only pursue evaluation if they had more time.

Although no formal framework for evaluation of consultation approaches has been prescribed by the Scottish Executive, its document "Consultation Good Practice Guidance" (2003) contains a chapter on evaluating consultations and suggests possible frameworks to adopt. A limited volume of evaluative information relating to written consultation exercises is systematically collected via their Consultation Registration and Evaluation System ( CRES). Introduced in 2003, this electronic database provides a tool for recording information on process and outputs and appears to be well used by commissioners of written consultations.

Against this background, comments made by respondents in Nicholson's (2005b) mapping of Scottish Executive participation activity suggest a common understanding of elements of "success" which could form the basis of any more formal future evaluative tool and these have been used as such in Chapter 11.

• The activity resulted in modifications to policy
• It represented good value
• We got past the usual suspects
• Participants were pleased to have been involved
• The activity was well structured
• We obtained a very wide range of views
• It helped to get "buy in"
• It worked well with our other activities

1.3 why evaluation of consultation is important

The review of Scottish Executive consultation activity concluded that, " evaluation of activities is not a particularly high priority for many commissioners of civic participation" (Nicholson, 2005b, p57). Increasingly, however, commentators in wider circles are stressing the importance of systematic and routine evaluation of consultation activity. With robust evaluation design still in its infancy, but rapid developments being witnessed in consultation techniques, many consultation exercises across the public sector must be taking place without an underlying confidence about what is likely to work and why.

InterAct (2001) provides a comprehensive summary of the benefits likely to accrue from formal evaluation of participatory exercises including:

• Improvements in practice by understanding more about what works and what does not work; identification of success and consolidation of achievement; extending the involvement of participants into the evaluation of the exercise; reducing the isolation of those commissioning the participatory processes; sharing learning.
• Building support for consultation and enabling mainstreaming; producing a body of evidence which can be used to increase confidence of funders as well as practitioners.
• Expanding the criteria for evaluating success by helping to identify new and effective ways of allowing methods to be assessed .
• For funders evaluation can, inter alia, help ensure good use of funds and highlight good practice and inform policy .
• For project organisers and practitioners evaluation can, inter alia, provide an analysis of strengths and weaknesses which can be used to develop future plans; contribute to setting standards and implementing quality control; and help validate new approaches.

The role of evaluation in facilitating the sharing of experience of what works was stressed by DETR (1998) with the observation that without a tradition of widely agreed and tested frameworks for evaluation of participation activity, learning from experience amongst local authorities was rendered, " haphazard and personalised" (p10).

The Scottish Executive mapping study (Nicholson, op cit) also suggested that pockets of staff throughout the Executive were privy to valuable lessons learned from their own consultation experiences, but sharing of such information was very limited. A selection of issues which arose and which are relevant to later discussion in this report include:

• Concern regarding the degree to which participant views represented their organisations
• Concerns regarding the intensity of resourcing the activity
• Lessons learned regarding use of electronic equipment as part of the consultation process
• Ways of handling conflicting views
• Managing the process of moving participant input from the detail to the bigger picture and back again
• Handling participants' suspicion that outputs had already been decided by the Scottish Executive
• Giving thought to the induction of new participants who join the process part way through

1.4 why evaluate the best value consultation?

Against this background of somewhat patchy and inconsistent evaluation of Government consultation exercises, it was decided early on that the consultation on Best Value in Public Services should be subject to formal evaluation. Emerging from interviews with members of the Public Service Reform Group of the Scottish Executive, key reasons for this decision included:

• Good practice to evaluate this initiative and learn lessons which can be shared
• The subject of the consultation - Best Practice - is associated with reflecting and identifying best approaches
• Encouragement from the Scottish Executive's Civic Participation Research Team
• The evaluation is of much interest to others in view of the innovative approach of the consultation. In particular there was no written consultation which was seen as leaving the consultation team exposed to a higher risk than might otherwise have been the case
• The consultation approach could present a cost effective way of involving stakeholders in policy-making, and lessons were also sought on whether the approach offered a speedier way of conducting participation than might be possible with a formal written consultation

1.5 Summary of key points

• The evaluation of the Best Value in Public Services Consultation took place at a time of wider development and change in both consultative techniques and consultation evaluative approaches.
• There is a paucity of robust and systematic evaluative modelling although frameworks for evaluation are gradually being developed.
• Within the Scottish Executive, some progress has been made in the last 6 years in evaluative approaches to Government consultation, but evaluations have tended to focus predominantly on process issues with many commissioners pursuing evaluation only if time permits.
• Commentators have identified a host of benefits to accrue from evaluating consultation exercises, including promoting improvements in practice and building support and confidence in consultation.
• The commissioners of the Best Value in Public Services consultation envisaged that the evaluation would enable lessons to be shared from what was seen as a particularly innovative approach to consulting.

« Previous | Contents | Next »