« Previous | Contents | Next »
Listen
EXECUTIVE SUMMARY
INTRODUCTION
This report documents the evaluation of a Scottish Executive consultation on Best Value in Public Services. The consultation was designed by the Scottish Executive Public Service Reform Group with the overall aim:
"To agree the best possible vehicle for delivering Ministers' vision of refreshed Best Value in public services by April 2006".
Specific aims of the consultation were:
Aim 1: To begin to facilitate a dialogue with public bodies and their sponsor departments on how to be Best Value organisations;
Aim 2: To start to refresh and renew the spirit of Best Value across the public sector by empowering public bodies to help set the direction for future resources;
Aim 3: To initiate the exchange of information between public bodies and the Scottish Executive on good practice in Best Value, helping the policy team to begin to understand and plug the gaps in Best Value knowledge management.
CONTEXT
Best Value principles are intended to inform and reinforce organisations' efforts towards achieving continuous improvement in the performance of their functions. A series of Best Value pilots in local authorities was run during the late 1980s with lessons learned informing the roll-out to all local authorities following the statutory duty placed on them by the Local Government in Scotland Act, 2003.
In parallel with these developments in local government, was the preparation of a non-statutory duty of Best Value to be applied to public service organisations in the Scottish Administration with Accountable Officers. These comprise Scottish Executive Departments, Executive Agencies and Non-Departmental Public Bodies.
Guidance on Best Value was prepared for both local authorities and central government public service organisations, with the local government guidance immediately adopted. The guidance for central government has already been the subject of written consultation, with the findings indicating a need for various changes.
In his letter of August 2005 to government public service organisations, the Minister for Finance and Public Service Reform outlined the need to refresh and renew Best Value activity by consulting on the nature of improved guidance, toolkits and knowledge management arrangements which were requested in responses to the earlier consultation exercise. From mid-2005, the Scottish Executive's Public Service Reform Group has been undertaking this consultation which has comprised a variety of activities:
- 10 bi-lateral meetings with consultees
- Group meeting with consultees
- 5 Stakeholder seminars
- External reference group
It was considered good practice to commission a formal evaluation of the Best Value in Public Services Consultation. This was seen as reflecting the ethos of Best Value and was of particular interest in view of the innovatory techniques used for this consultation. The evaluation was undertaken by an external evaluator and ran alongside the consultation from September 2005.
THE EVALUATION
The evaluation was commissioned to evaluate the quality and success of the Scottish Executive's consultation process on Best Value in Public Services, and in so doing, to demonstrate and share good practice, or lessons learned.
Specific research objectives for the evaluation were:
- To evaluate whether the consultation exercise meets its objectives;
- To evaluate the suitability of the approach chosen to consultation (ie non-written);
- To review the planning of the consultation process, and informal pre-consultation dialogue with external stakeholders;
- To observe and critically evaluate the operation of the 5 stakeholder engagement seminars, and the December Ministerial conference 1 and specifically:
- To assess the success of grouping stakeholders by Ministerial portfolio
- To consider the appropriateness of the venue chosen in terms of accessibility and convenience
- To assess the benefits and problems associated with facilitation of the events
- To evaluate the quality of debate and management of conflict.
- To probe stakeholders' views on the consultation process, their involvement, and their capacity to influence the future direction of Best Value policy through the use of self-completed questionnaires;
- To make suggestions and recommendations for improved engagement as a result of this evaluation.
EVALUATION METHODS
The evaluation adopted a variety of methods in order to assess the extent to which the consultation achieved its aims. The methods comprised: an initial briefing with the consultation commissioners; observation at the stakeholder seminars; seminar participants' self-completion questionnaires; seminar de-briefings with facilitators; follow-up contacts with seminar non-attendees; telephone interviews with consultees; an in-depth group interview with the consultation commissioners; observation at the external reference group meetings; and external reference group members' self-completion questionnaires.
FINDINGS
The evaluation used the evidence collected to assess the degree to which the consultation had met the aims set out above. It concluded that based on this evidence, the consultation has resulted in potentially useful vehicles for delivering Ministers' vision of refreshed Best Value in public services. However, it is too early to state whether the vehicles are the "best possible" and this would require follow-up assessment.
In terms of the specific aims of the consultation, the evidence suggested that all 3 had been achieved to some extent. However, whilst significant progress had been made towards Aims 1 and 2, commentators representing varying perspectives considered that the consultation could have gone further to address Aim 3.
Evidence on process and efficiency collected for the evaluation demonstrated that much attention had been paid by the commissioners to containing the costs of the consultation. The consultation offered significant value by achieving a range of expected and unexpected beneficial outcomes whilst keeping within a limited budget.
RECOMMENDATIONS
Six key overarching recommendations emerged:
1. It is recommended that efforts are made within the Scottish Executive to develop further a core evaluation framework which can be customised for use across different contexts.
2. Further guidance is required to enable commissioners to be more informed regarding the likely costs of different consultation approaches, and help them to estimate input to consultation exercises in financial terms.
3. It is recommended that effort goes into building on the indicators in this evaluation (such as attendance rates; ratings on achievement, and so on) in order to develop a robust "bank" of benchmarks for wider use.
4. It is recommended that priority is given to paving the way for participants to have positive perceptions of the aims of the consultation activities and ensure that their opportunity to influence the outcomes is genuine.
5. An overriding recommendation to emerge from the stakeholder seminars is for future commissioners to consider carefully the balance of different activities within the timescale allocated to an event and err on including too few elements rather than risking an overload of features and possible reduction in the opportunity for participant contributions.
6. It is recommended that future consultations invest time in planning not only the individual parts of the exercise, but also their interface and links between one stage to the next.
GOOD PRACTICE
The consultation demonstrated much good practice which could be customised for future consultations. In particular, it made use of existing expertise within the Scottish Executive at the planning stage. The consultation team prepared effectively for handling the conflicts in views and perspectives which inevitably emerged during the exercise. The consultation framework facilitated useful and productive dialogue not just between consultees and the Scottish Executive, but also amongst participants themselves. It also helped enable many consultees to immerse themselves gradually into the consultation process and helped them to feel more confident about contributing at later stages.
The consultation was effective in using lessons and findings from each stage to inform subsequent stages. Some aspects of the consultation were particularly resource intensive, but the team used techniques to share the resource load between different players.
The decision by the team to host sector-specific stakeholder seminars was effective in terms of helping the discussion to flow amongst people with a shared understanding of common issues.
GENERAL LESSONS LEARNED FOR FUTURE CONSULTATIONS
The evaluation highlighted several lessons to take on board in the future. Amongst the more generalisable of these was:
- there should be encouragement to use Scottish Executive internal facilitators for seminars and discussions following their success in this consultation;
- there is a need to be proactive in promoting attendance at consultation events;
- there is a need to consider carefully the frequency of consultation meetings/activities and minimise the demands placed on consultees in this respect;
- the provision, where appropriate, of a variety of different consultation activities and formats is useful in order to accommodate different consultee needs;
- there should be careful consideration given to the length of consultation events (eg half-day or whole-day);
- the value of small group discussions within an overall consultation framework should be recognised;
- the fine details of planning of events are important and should not be overlooked;
- very clear parameters need to be set for anyone invited to make a presentation at a consultation event;
- in-depth consideration should be given to activities planned within a consultation event in order to ensure that they are entirely suitable for the type of consultee likely to attend;
- there is a need to agree precise roles for any Scottish Executive staff attending consultation events.
SPECIFIC FINDINGS ON INPUTS, PROCESSES AND OUTPUTS
More specific findings also emerged relating to the inputs to the consultation (eg resources), the processes it adopted (eg how events were organised) and outputs (ie the bi-lateral meetings; stakeholder seminars; and external reference group).
Inputs
It has been argued that the weakest element of previous evaluations of consultations has been in ascertaining the inputs, and in particular, the costs. The current evaluation has used costs which were readily available and added new data on time input generated from questionnaires. However, significant gaps in cost data remain.
Inputs to the consultation were identified as: planning and running the consultation; support from Scottish Executive colleagues; Policy Skills Team contribution; the facilities used; participant input; and the cost of the evaluation. The staff input to the consultation appeared to be resource intensive. The involvement of a range of Scottish Executive colleagues in advising on the consultation could be seen as one of the hidden costs of the exercise.
The input of the members of the external reference group was calculated using data provided by these participants. Based on this information, the total time input of the 12 members of the group from their initial involvement up until March 2006 was estimated at approximately 72 days. Lunches and refreshments provided at the consultation events amounted to £1,000.
The external evaluation budget was £13,500 (excluding VAT) with the evaluation constituting another input to the overall consultation.
Processes
Data was collected on 6 process themes: representation; attendance levels; communication of aims; organisation of events; facilitation; and resolution of conflict.
The representativeness of participants was raised as important by the consultation team. In reality, however, the extent to which participants wholly represented their respective organisations varied.
It was possible to identify reasons for unexpected absence from stakeholder seminars with smaller organisations reporting particular resource-related difficulties of attending. The attendance rates produced for this evaluation provide useful process indicators for future benchmarking purposes. Holding a series of consultation meetings in quick succession was highlighted as potentially problematic in terms of resourcing participation.
Overall, the smaller group dialogues between Scottish Executive officials and selected stakeholders demonstrated a higher engagement rate than the larger seminar events.
Participants rated the communication of the aims of the consultation relatively highly. Overall, the one-to-one dialogue and small group settings appeared to be more effective vehicles for communicating strategic aims than the larger, less intimate stakeholder seminars.
Participants reported being content with the standard of facilitation at the stakeholder seminars. The consultation team handled conflicting views in several ways: by being well prepared for conflicts; by providing firm guidance on what could be debated and what was outwith scope; by allowing views to be aired but making no attempt to bring conflicting opinions to a resolution; and by tabling controversial issues for discussion at a later date.
Consultees identified other process issues of importance to them. A recurring theme to emerge from participants was that the consultation process had provided them with valuable networking opportunities. Those involved from an early stage identified benefits from such early inclusion in the process including building their confidence so that they could contribute effectively later on. There were mixed views on whether the consultation had benefited from what was seen as the open and listening approach adopted by the consultation team, or whether this had served to slow up the consultation process.
Outputs - Bi-Laterals
Ten informal bi-lateral discussions were held between representatives of the Scottish Executive Best Value and Performance Team and a range of public bodies of different sizes, covering the spread of Ministerial portfolios.
The bi-laterals produced both tangible and less tangible outputs. In particular the consultation team came away from these with a clear picture of the challenging issues and options for future direction. Less tangible outputs from the bi-laterals were identified by the team and included building rapport between the Scottish Executive and public bodies. A disadvantage of the bi-lateral approach was that it offered no scope for wider discussion amongst other bodies on shared concerns and solutions.
Consultees involved in the bi-laterals appeared to be clear on their purpose and enthusiastic about their meeting. They reported that both they and the Scottish Executive officials involved had gained from it. Indeed, from the comments expressed, it appeared that the bi-laterals were successful in sowing the seeds of mutual co-operation and partnership working to take forward the Best Value agenda. Consultees perceived the bi-laterals to present various advantages: they promoted a more honest dialogue; enabled individuals to speak out; prevented the Scottish Executive from presenting a fait accompli; provided the opportunity for officials to explain unexpected policy decisions; and helped all parties to reach the same level of understanding.
Outputs - Stakeholder Seminars
The stakeholder seminars were seen to have been successful in meeting Aims 1 and 2 of the consultation but less successful in relation to Aim 3.
The most successful elements of the seminars were identified as the break-out groups; their participative focus; the opportunity they provided for networking; and the informal approaches used. However, the least successful aspects included what was seen as the lack of time available for developing discussion; the lack of focus at some seminars; the slow start to some seminars; the mode of identification of topics for discussion; and the disappointing quality of feedback from the break-out groups. Ideas provided by participants for improving the seminars fell into 4 categories: better introductory session; advance background papers; more time allocated to discussion; and greater emphasis on practical Best Value issues.
Facilitators also identified what they thought had not worked well. Amongst these features were: practical aspects of the use of small rooms; lack of focus on practical solutions relating to Best Value; too intense and heavy presentations to start the events; lack of time for discussion; failure of the final plenary session to wrap up the event; and ineffective ground-rules and group vote activities. In addition, some of the seminar activities were perceived to be inappropriate for the senior level of public sector participant present at the events.
In practical terms the consultation team suggested that with hindsight, full-day seminars would have been preferable to the half-day seminars they hosted in order to let the discussion reach a more detailed and productive level.
The external evaluator observed many apparently successful aspects of the seminars including: the sector-specific organisation; depth of discussion at the break-out groups; presence of higher ranking Scottish Executive officials for most of the time; and the clarity of recurring messages which emerged to steer the subsequent stage of the consultation.
The evaluator also identified aspects which did not work well: the complexity of the design of the activities sometimes interfered with the free-flowing ethos of the events; overload of activity instructions and explanations within the limited time available; the process of the activities becoming the focus rather than the outputs and outcomes; the inappropriateness of some of the rooms and layout of seminars; and the lack of clear and consistent roles for Scottish Executive staff present.
Outputs - External Reference Group
As the external reference group is still meeting at the time of writing and has not yet produced its entire final output, its overall effectiveness cannot yet be gauged. However, indicators of process suggest a high degree of commitment and engagement amongst the external reference group members. From observation and examination of meeting papers it is evident that the group is moving towards delivering tangible outputs over the next 5 months.
The external reference group approach appears to have been successful in inputting a strong bottom-up dimension to the consultation aim of refreshing and renewing the spirit of Best Value across the public sector. It is encouraging that all of the consultees who responded stated that they would take part in another similar exercise if invited to participate by the Scottish Executive.
Amongst the few suggestions for improving the process in the future, consultees recommended using a project management approach; greater clarity on expected input and operation of the group; and greater representation on the group of different types of consultee.
SELECTED OUTCOMES
The extent to which longer term outcomes could be identified was curtailed by the timing of the evaluation. A follow-up investigation would be required to ascertain longer term outcomes of the consultation. It was possible, however, to identify some of the wider and possibly longer term benefits of the exercise for organisations and those involved from responses to the external reference group questionnaire.
Consultees identified a variety of best features of the consultation which suggested longer term benefits such as the willingness of participants to engage with future consultation exercises. They anticipated that useful tools would emerge from the consultation process which they had helped to shape.
Overall, external reference group members were consistent in rating the exercise highly on genuine participation and influence. Participants were able to provide examples of their personal input, demonstrating perceptions of personal empowerment and involvement.
CONCLUSIONS AND OVERARCHING RECOMMENDATIONS
The evaluation concluded that based on the evidence collected, the consultation has resulted in potentially useful vehicles for delivering Ministers' vision of refreshed Best Value in public services. However, it is too early to state whether the vehicles are the "best possible" and this would require follow-up assessment.
In terms of the specific aims of the consultation, the evidence suggested that all 3 had been achieved to some extent. However, whilst significant progress had been made towards Aims 1 and 2, commentators from varying perspectives considered that the consultation could have gone further to address Aim 3.
« Previous | Contents | Next »