The Scottish Government

« Previous | Contents | Next »

Listen

CHAPTER 11: DISCUSSION OF FINDINGS AND OVERARCHING RECOMMENDATIONS

This Chapter reflects on the evidence gathered for the evaluation with a view to assessing the extent to which the consultation has met its aims to date. The elements of "success" identified by Government respondents to Nicholson's (2005b) mapping of participation activity provide a useful structure for assessment. The focus on these elements here also tests out their usefulness as part of an evaluation framework for future evaluations of this nature. These elements are reproduced below:

• The activity resulted in modifications to policy
• It represented good value for money
• We got past the "regular participants"
• Participants were pleased to have been involved
• The activity was well structured
• Obtained a very wide range of views
• It helped to get "buy in"
• It worked well with our other activities

The Chapter applies the evidence briefly to each of these elements before re-visiting the overall objective and specific aims of the consultation:

" To agree the best possible vehicle for delivering Ministers' vision of refreshed Best Value in public services by April 2006".

Specific aims were:

• To begin to facilitate a dialogue with public bodies and their sponsor departments on how to be Best Value organisations;
• To start to refresh and renew the spirit of Best Value across the public sector by empowering public bodies to help set the direction for future resources;
• To initiate the exchange of information between public bodies and the Scottish Executive on good practice in Best Value, helping the policy team to begin to understand and plug the gaps in Best Value knowledge management.

Finally, recommendations to emerge from the findings are collated and presented.

11.1 MODIFICATIONS TO POLICY

One commonly understood indicator of success of previous consultations is the extent to which the exercise resulted in a change to policy. In the case of the Best Value consultation a policy was not in need of change, but of assistance with direction. Help was requested to identify and agree the best possible vehicle for delivering the Ministers' vision of refreshed Best Value in the public sector. The findings of the evaluation inform the extent to which direction was provided by consultees.

The consultation team reported coming away from the bi-lateral meetings with a clear picture of challenging issues and options for future direction on refreshing Best Value. The subsequent group meeting was also seen as providing support and enabling agreement to be reached on future direction. Consultees also perceived the bi-laterals to provide the opportunity for them to influence policy direction.

Subsequent stakeholder seminars were viewed as partially successful in influencing direction by commissioners and stakeholders alike. Whilst enabling networking and other unexpected benefits to emerge, the consultation team and participants identified certain limitations to the seminars as vehicles for directing policy at the detailed level necessary for the task in hand - producing specific Best Value tools.

Views on the final element of the consultation, the external reference group, suggest that this has been highly successful in delivering direction and practical guidance to the policy on refreshing and renewing the spirit of Best Value. The first of the 5 Best Value products was published in May 2006, in line with the target set for the consultation. The group's deliberations have been at the detailed and practical level resulting in a strong bottom-up dimension to the consultation. The enthusiasm of the group members for their task and their willingness to engage in another such task if required in the future suggest that their experience has been positive and has lived up to expectations.

11.2 REPRESENTATION OF GOOD VALUE FOR MONEY

Although Scottish Executive staff raised good value for money as relevant in assessing the success of a consultation exercise, it is noted in this report that the weakest element of previous consultation evaluations has tended to be their cost data, with this evaluation no exception. This evaluation identified various direct and hidden inputs to the consultation and also attempted some new costings of various aspects. However, even where detailed financial and other costs were readily to hand, without a database of previous costs against which to benchmark this consultation, the usefulness of one-off costs is curtailed.

Of course, to assess "good value for money" costs require to be seen in relation to the direct, indirect and unexpected benefits of the consultation. The evaluation has identified a range of all of these including both tangible and intangible outputs and longer term organisational and personal benefits of participation. It has been argued that the economic "cost-benefit" model does not sit easily with the more complex consultation processes and outputs and it is recommended here that further work is devoted to developing an appropriate framework for considering "good value" in consultation.

11.3 GETTING PAST THE "REGULAR PARTICIPANTS"

The consultation team gave much thought to who they wanted to participate in the consultation. Amongst their targets were public sector bodies of various sizes and different levels of engagement with Best Value, and which represented the range of Ministerial policy portfolios. In addition, invitations to the stakeholder seminars were pitched at a senior official level with the events restricted to half-days in an effort to draw consultees from this higher management level.

The team adopted 3 different strategies to reaching their target participants: meeting them on their home ground; staging stakeholder seminars; and hosting external reference group meetings. Each resulted in a different level of success with regard to moving beyond "regular participants"

Although the bi-laterals enabled the consultation team to select their participants and visit them on their home territory ensuring 100% attendance, the team reported that the effectiveness of the meetings was dependent in part on the specific personnel fielded by the public sector organisations.

Even though the stakeholder seminars resulted in a slightly disappointing (according to the consultation team) 50% attendance rate, they did result in the involvement of many high ranking officials from public bodies, as had been intended. However, the consultation team expressed concern that smaller public bodies were over-represented amongst the non-attendees.

The external reference group was formed by self-selection of interested parties. However, an examination of the participant list reveals personnel from different Ministerial policy portfolios, sizes of organisation and levels of engagement with Best Value. As was outlined, previously, attendance rates at the group meetings has been consistently high, suggesting that this mode of participation has been very successful in attracting and sustaining participation amongst the participants targeted by the consultation team.

One final aspect of "moving beyond regular participants" applies to the expectations of the external participants themselves and whether the consultation met these in terms of the personnel fielded by the Scottish Executive. As described in the findings, this varied according to the event in which they participated. Whereas some participants were impressed by the seniority of Government officials in attendance and their contribution, others noted what they saw as the lack of input from higher ranking Scottish Executive staff.

11.4 PARTICIPANTS WERE PLEASED TO HAVE BEEN INVOLVED

Participants' views on their involvement were ascertained via questionnaires and interviews. Taking an overview of their responses on their participation, their mood was generally very favourable in terms of their participation with a host of personal and organisational benefits identified. Indeed, even where the participation activity (external reference group) was seen by some to be particularly resource intensive and clashed with their other commitments, they were positive about their involvement with all indicating their willingness to become involved again.

It was interesting that those participants who inputted from the start (in the bi-laterals) were amongst the most enthusiastic about the consultation and their involvement. They highlighted how this early start gave them confidence to contribute in a more useful way in later stages.

Participants described a range of longer lasting benefits which they could take away from the consultation, such as new networks, a better understanding of the Scottish Executive set up and mindset, more confidence in handling Best Value within their organisation, and so on. These benefits, together with the general perception that the consultation process had indeed been two-way, with useful and tangible products to result, most likely contributed to the consultation "feel-good factor" and satisfaction at being part of it.

11.5 THE ACTIVITY WAS WELL STRUCTURED

The consultation fell into 3 key stages each with a different format and a different delivery.

The bi-laterals were semi-structured and delivered by officials on a fact-finding mission. The stakeholder seminars were relatively tightly structured around a number of activities intended to demonstrate explicitly the role of the participants in setting the agenda and to engage them in pursuing this agenda in discussion. The emphasis was on involving larger numbers and harnessing a variety of individual needs within a mutually agreed framework for discussion. The external reference group focused on detailed and practical dialogue with a clear goal of products to deliver.

Overall then, the consultation "shape" moved between activities and gradually shifted from an open and wide-ranging discussion through to a very intense and thorough examination of detail. All of these elements of the consultation had been pre-planned with their respective aims agreed and the consultation activity designed to meet these.

It was noted that such a level of pre-planning and proactivity was unusual in the wider context of participatory approaches. In terms of this consultation, the 3 strands of the exercise fitted together, with one element feeding and informing the next. Some participants, who were involved in more than one activity, commented on and appreciated the way in which their input from one stage could be witnessed in subsequent activities.

11.6 A VERY WIDE RANGE OF VIEWS WAS OBTAINED

To some extent, this indicator of success overlaps with 11.3 in that its achievement depends on the degree to which the consultation attracts participants other than the "usual suspects". As has been seen, the techniques used to target a wide range of participants, and the level of attendance which emerged suggests that, apart from some concern about gathering the views of smaller organisations, the consultation participants certainly represented a wide range of stances. The issue here is whether, in representing these stances, the consultation enabled participants to present a range of views.

The consultation team's description of the bi-laterals producing a range of obvious differences between organisations of varying structures and constitutions suggests that these interviews were successful on this count.

The stakeholder seminars appeared to be less successful in generating such a range of perspectives. All parties involved expressed concern that the events did not provide enough time for full discussion. What was viewed by some as their overly complex entry activities appeared, to some extent, to hinder rather than facilitate discussion of a wide range of topics. There was some evidence of attempts to broaden the discussion failing to move the focus away from a few central issues of concern.

However, there were signs that by structuring the events around sector-specific policy fields, wider views were encouraged by virtue of the common level of understanding brought to the event by participants. In addition, organising the seminars around policy rather than size of organisation meant that the perspectives of public bodies of different sizes and budgets cross-cut each discussion.

The external reference group membership, despite being overly represented by personnel from larger organisations, does contain representation from some of the smaller bodies, with its working influenced significantly by pragmatism arising from differing experiences based on the daily business of these bodies.

Overall then, it could be argued that the consultation has had mixed success in generating a wide range of views. The careful pre-planning of the consultation set the context for a range of views to emerge, but the intricate activities and timing issues associated with the stakeholder seminars may have prevented the seminars from achieving their full potential in terms of producing a wide range of views.

11.7 IT HELPED TO GET "BUY-IN"

The evaluation identified many indicators of level of "buy-in" to the consultation process and its aims. A general feel for the impact of the process on "buy-in" came from the external reference group members, most of whom felt that that they had shifted from a relatively cynical view of the usefulness of the consultation to a confidence in the process and satisfaction that practical outcomes would result. These members also "bought-into" the consultation process as a genuine vehicle for dialogue, with all of them expressing their willingness to participate in future exercises. The relatively high attendance levels at the external group meetings could also be seen as an indicator of "buy-in" to the consultation aims.

It is useful to refer once more to the framework provided by IAP2 Model of Public Participation (previously displayed as Figure 3 in Chapter 4) to illustrate the extent of "buy-in" generated in the consultation. Figure 3 is replicated overleaf.

Figure 3: Application of Findings on Purpose of Activity to IAP2 Model of Public Participation

(Figure 3 was originally produced in Nicholson, 2005b.)

Chapter 9 described how the external reference group displayed a gradual increase in ownership of aspects of the consultation, and in its later meetings began, in turn, to take on the responsibility for promoting wider buy-in and credibility of its outputs. Such a shift provides a clear example of a consultation moving effectively through the stages from involving, collaborating to empowering.

Participants at the stakeholder seminars provided a wide range of rating scores for various aspects of the seminars, suggesting that at this stage, individual experiences of the practical and organisational dimension of the seminars perhaps influenced the degree to which participants felt committed to, and supported by, the consultation.

Those public body representatives involved in the early bi-laterals were open about their understanding that one purpose of these meetings was to commence a process of "buy-in" from the public organisations. The overarching view was that the meetings had served their purpose on this front in a positive manner with the seed sown for mutual co-operation and partnership working to take forward the Best Value agenda.

In summary, the individual bi-laterals and external reference group meetings were successful in generating "buy-in" to the consultation and its aims, whilst the stakeholder seminars produced a wider range of levels of commitment by different participants. However, looking across the consultation process as a whole, the individual activities worked successfully together in gradually fuelling "buy-in" and encouraging and supporting the commitment of participants to an extent that those involved in the latter stage of the consultation displayed levels of empowerment, with an acceptance that aspects of decision-making were now in the group's hands and its responsibility.

11.8 IT WORKED WELL WITH OUR OTHER ACTIVITIES

The Best Value and Performance Team pursued the Best Value agenda within the context of Ministerial plans for wider public service reforms and previous related policy initiatives on Efficient Government, equality, sustainable development and so on.

It was clear from the responses and comments made by participants at the bi-laterals and the stakeholder seminars that they expected the consultation to encompass the positioning of Best Value within this broader context and for it to provide guidance to organisations on how the various initiatives interfaced.

Again, the consultation was partially successful in meeting these expectations and needs. Whereas the participants at the bi-laterals benefited from one-to-one responses to their questions on these topics, comments from various participants at the stakeholder events demonstrated that the relation between this consultation and the wider public service reform activities was important to them with many feeling that the significance of this had not been recognised by the Scottish Executive at their seminar. (Though links to public service reform were flagged up to varying degrees by Scottish Executive Officials in their "scene setting" presentations.) Indeed, some participants considered that representatives of related policy initiatives should have attended the seminars.

It was interesting that 2 of the 9 Best Value principles, sustainable development and equal opportunities, both examples of key Scottish Executive cross-cutting policy agendas, were those for which participants were least able to identify positive examples within their organisation (during the wall chart activity in the stakeholder seminars). The findings from the consultation suggest that whereas the strong focus of any consultation team will be on the aims and objectives of their particular consultation and achieving an agreed outcome from the consultation, participants may approach the subject with a broader perspective, with expectations of linkages to other initiatives and policies.

11.9 DID THE CONSULTATION MEET ITS OBJECTIVES?

The evaluation framework used above is useful in its coverage of many of the input, process, output and outcome dimensions of the consultation. Taking these into account along with the findings generated throughout the consultation and evaluation it is evident that the consultation has resulted in potentially useful vehicles for delivering the Ministers' vision of refreshed Best Value in public services to materialise. The first of these, guidance for use by public bodies, was published in May 2006 with more materials following over the coming months.

Whether these materials are the "best possible" vehicles for delivery remains to be seen and would require follow-up study. However, the analysis above demonstrates that the development of the materials has been grounded in stakeholder experience and input with their "buy-in" and confidence suggesting perhaps a "best possible" launching base for these vehicles.

It could be argued that the specific aims of the consultation, couched as they are in very broad terms: " to begin to…", " to start to..." and " to initiate…", could hardly fail to be met to some degree. From the evidence gathered, all 3 specific aims have been achieved by the consultation. However, although all 3 consultation approaches resulted in significant progress being made towards Aims 1 and 2, commentators from varying perspectives considered that the stakeholder seminars did not go far enough in addressing Aim 3. This aim focused on the exchange of information, helping understanding and plugging gaps in knowledge. The stakeholder seminars as structured and delivered for this consultation were not seen as particularly successful in meeting this Aim.

The evaluation has identified a range of efficiency issues associated with aspects of the consultation. In summary it appears that the consultation team consistently strove to promote efficiency and economy in the design and operation of the activities. Efficiency was taken into account in, for example, decisions on facilitation, location of events, the sharing of duties and so on. Such attention paid off in terms of containing costs, and also in promoting the impression of functional, productive working rather than gloss and spin.

Whilst the development of frameworks for cost-benefit analysis of consultations is still in its infancy, and an economic analysis has not been attempted for this consultation, the evidence gathered is sufficiently robust to suggest that the exercise offered significant value by achieving a range of expected and unexpected beneficial outcomes whilst keeping within a limited budget.

11.10 RECOMMENDATIONS

The evaluation has produced a host of specific lessons and advice for future commissioners of consultations. These are reported in detail within individual Chapters and their summaries, with good practice and key lessons further summarised in Chapter 12. In addition to this good practice and detailed lessons, some broad, generalisable recommendations have emerged from the evidence and are listed below.

• In view of the benefits to accrue from evaluating consultation exercises, it is recommended that efforts are made within the Scottish Executive to develop further a core evaluation framework, which can be customised for use across different contexts. As shown here, the model used above which is based on Government officials' perceptions of consultation "success", provides a useful starting point for the construction of such a framework. ( Chapter 1)
• Further guidance is required to enable commissioners to be more informed regarding the likely costs of different consultation approaches, and help them to estimate input to consultation exercises in financial terms. ( Chapter 5)
• Future evaluations will benefit from being able to benchmark various indicators of process and effectiveness against the findings in this evaluation. It is recommended that effort goes into building on the indicators in this evaluation (such as attendance rates; ratings on achievement, and so on) in order to develop a robust "bank" of benchmarks for wider use. ( Chapter 6)
• The evaluation demonstrated that if participants have positive perceptions of the aims of consultation activities and see the opportunity to influence the outcomes as genuine, then they can become very strongly motivated to take part. It is recommended that creating the conditions for these 2 factors to materialise should be prioritised in future consultation exercises. ( Chapters 8 and 9)
• Many operational and organisational lessons can be learned from the running of the stakeholder seminars. An overriding recommendation to emerge from these events is for future commissioners to consider carefully the balance of different activities within the timescale allocated to an event and err on including too few elements rather than risking overload of features and possible reduction in the opportunity for participant contributions. ( Chapter 8)
• This consultation has demonstrated the added value to be gained from deploying a series of consultation stages and different events within one consultation framework. It is recommended that future consultations invest time in planning not only the individual parts of the exercise, but also their interface and links between one stage to the next. This was undertaken in this consultation (for example, overlaps in the participants between one stage and another; continuation of theme between stages) and reaped dividends in terms of participants' motivation to take part and their perception of the credibility of the consultation.

« Previous | Contents | Next »