Scottish Government

MONITORING AND EVALUATION OF THE SCOTTISH COMPACT Baseline Results 2004

APPENDIX TWO: COMPACT COMPARISONS AND COMPACT AND IMPLEMENTATION STRATEGY ANALYSIS

INTRODUCTION

A2.1 The purpose of this Appendix is to:-

• Analyse the 2003 version of the Compact;
• Look at the detail of the changes between it and the 1998 version; and
• Analyse the Review Group's Implementation Strategy.

The Report starts by outlining the background to the Compact.

THE 2003 COMPACT

A2.2 The Goal of the 2003 Compact is to strengthen the relationship between government and the sector. This is to be done by the attainment of the following Objectives, which are essentially the outcomes of the Compact implementation process:-

• Give more opportunities for the sector to contribute to the development and implementation of policy;
• Make government policies more responsive to the potential and needs of the sector;
• Enable the sector to communicate its needs to the Executive;
• Mobilise the sector behind Executive programmes;
• Increase mutual understanding;
• Generate evidence of mutual value and impact; and
• Inform longer term planning.

In their turn these Objectives are to be attained by government and the sector making, and fulfilling, a number of commitments, structured into 5 Commitment Areas ³.

COMPARING THE COMPACTS

A2.3 In both versions of the Compact these commitments were structured under the 5 headings of: Recognition; Representation; Partnership; Resources; and Implementation. Table A2.1 compares the 2 Compact versions according to the number of commitments made under each heading. The key points are:-

• The number of commitments has increased from 44 to 64;
• The greatest increase has been in Representation, up from a total of 7 in 1998 to 15 in 2003; and
• The Compact has become more symmetrical in that there is now a greater balance between the numbers of commitments made by the 2 parties.

TableA2.1 Comparisons of Commitments in the 1998 and 2003 Compacts

THE DETAILED CHANGES BETWEEN 1998 AND 2003

A2.4 Analysing the changes between the 2 versions of the Compact, it is apparent that there are a number of general (and minor) changes such as:-

• Replacing the word "Government" with "Scottish Executive";
• Replacing "umbrella bodies" with "intermediary bodies"; and
• Grammatical corrections, such as replacing "whom" by "which".

These are essentially updating (to take account of the setting up of the Parliament and Executive) and general "tidying up".

A2.5 A more substantive change is the role given to intermediary bodies. They were always seen as having a key role in fulfilling some of the commitments, particularly Representation and Implementation. However, their role is now more explicitly endorsed in the Section of the 2003 Compact headed "Aim of the Compact", where it is stated that:-

"implementation of the Compact is a key role for intermediary voluntary organisations" (Scottish Executive, 2003a, p.2.)

A2.6 In an attempt to undertake a more systematic analysis of the changes to the varied commitments, the 64 commitments in the 2003 Compact have been classified into the following categories:-

• Commitments that are identical to those in the 1998 document;
• Those that have had minor wording changes, albeit that some of these might have quite significant implications;
• Ones where new text has been inserted into the original commitment; and
• Ones that are totally new.

A2.7 Table A2.2 summarises the differences whilst Annex 1 to this Appendix looks in greater detail at the individual changes. In terms of the overall changes the Table shows that:-

• 24 (38%) of the commitments are identical between the 2 Compact versions;
• 11 (17%) have received minor wording changes;
• 13 (20%) have received new text; and
• 16 (25%) are new.

Table A2.2 Changes in Compact Commitments Between the 1998 and 2003 Versions

A2.8 The key themes that emerge from the analysis in Annex 1 are:-

• Additional commitments have been included to increase the symmetry between government and the sector under the various Areas. This mainly involves including sector commitments that are identical to those of government;
• There is more explicit emphasis on the role of intermediary organisations in areas such as implementation. Related to this, is a realisation that some groups may have difficulty in making their voices heard and that the sector may not always be representative of all groups;
• The "tone" is generally more pro-active. For example words such as: "work to ensure" have been replaced with "put in place"; and "met with" replaced by "work collaboratively with";
• The text is permeated with a greater sense of reality. For example: public resources are said to be limited; the sector's involvement in partnerships may be constrained by resources; the need for full cost recovery in contract bids has to be recognised; and decision making may not always be open and transparent; and
• The text has greater clarity. For example: 1998 commitments have been split into their component parts; the purpose of some commitments is, at times, given; and there is a fuller explanation of terms such as monitoring;

A2.9 However, overall the changes are more in tone than content. This can be explored more fully when the Implementation Strategy is considered, and particularly those parts of it relating to monitoring and evaluation.

THE IMPLEMENTATION STRATEGY

A2.10 The initial 2000-01 monitoring exercise (see Paragraph 1.3) found faults on both sides. For example there was lack of definition of the commitments to the Compact by the sector and the Executive and both parties had problems in meeting the 3 month consultation deadline. One of the main issues (that has been replicated in Compacts elsewhere) was the pre-occupation with funding issues at the expense of broader Compact principles. In part this may have been a reflection of the difficulties that the Executive had in making funding decisions within the 3 month deadline.

A2.11 Against this background the Implementation Strategy was developed to ensure that the Compact was underpinned by clear and consistent practices. It was also to raise awareness, generally promote the Compact and develop implementation capacity. The Strategy is to be implemented over 3 years and is split into 6 themes: strategic aims and vision; leadership; mainstreaming; raising awareness and promoting best practice; communication; resources; and monitoring and evaluation.

A2.12 Table A2.3 categorises the 39 actions that the strategy contains. It will be seen that only 10 (26%) have been classed as Specific Actions: that is commitments to undertake specific initiatives, some of which are time limited. The remainder are generally ongoing, broader, actions the attainment of which will be difficult to be definitive about. Each of the Themes will now be looked at in greater detail.

A2.13 Strategic Aims and Vision are essentially about the Compact and Implementation Strategy being endorsed by the Executive, at Cabinet level, and by SCVO and national intermediaries. As such this is giving the Compact high level endorsement by the key agencies that are responsible for promulgating its principles within government and the sector.

A2.14 The Leadership theme relates to 2 main issues:-

• The co-ordination of ongoing promotion of the Compact. However, beyond specific actions such as a Parliamentary debate, seminars and training events, how this is to be done is not specified; and
• The updating, revising and (in the case of Good Practice Guides) amalgamation of literature such as the Compact and the Guides.

A2.15 Mainstreaming relates to the systematic application of Compact principles throughout the Executive and the sector. Thus the sector should be involved in policy formulation and there should be an awareness by the Executive of the impact of legislation on the sector. The sector also needs to ensure that it involves all parties in consultation, not just national organisations or those receiving Executive funding. Specific actions include:-

• Meetings between various parts of government and the sector;
• The appointment of Voluntary Sector Liaison Officers (VSLOs) in the Executive and publication of their contact details;
• The Voluntary Issues Unit (VIU) developing guidance on the roles and responsibilities of the VSLOs;
• VIU to look at the potential for having VSLOs in agencies and NDPBs;
• Sector intermediaries to identify key contacts in the sector; and
• VIU and SCVO to take the lead in promoting and disseminating information about the Compact, with SCVO paying particular attention to smaller voluntary sector organisations.

A2.16 Awareness Raising and Best Practice are seen as key areas of concern, and are especially an issue for smaller organisations within the sector that may not appreciate that the Compact has relevance for them. Actions relate to such things as using websites for publicity, seminars for Executive, NDPB, Agency and sector staff, encouraging the setting up of local compacts and discussing "models" that could be used to promote awareness raising and good practice dissemination.

A2.17 There are a number of structures that exists to promote Communications, both within and between the Executive and the sector (for example the Voluntary Sector Intermediaries Network). A variety of actions are to be undertaken to refine and revise their roles to promote and mainstream the Compact. A key action is, however, to be the piloting of a compliance system in one Executive department. The aim of this is to research and resolve grievances. When it is not possible to get a resolution, the issue will be logged and used to provide information on the effectiveness of the Compact. The system is to form part of the monitoring and evaluation of the Compact.

Table A2.3 Categorisation of The Compact Implementation Strategy Actions

Notes:-
1. "Specific Actions" are defined as commitments to undertake identifiable actions. They are so specific that it will be possible to give a definitive answer to the question "Has this action been undertaken". Examples include the VIU piloting compliance systems in one Department and the revised Compact being launched.
2. "Broad Actions" are defined as general commitments to undertake actions that are specified only in general terms. Examples include "SCVO and the voluntary sector to continue to develop infrastructure in response to identified needs". The attainment of this can only ever be a judgement and a matter of degree.
3. "Discussion, consideration and agreement" includes such actions as "All partners to discuss potential models….", and "VIU to investigate the potential…". Essentially these are commitments to undertake preliminary work before making a decision.
4. "Encouragement" is essentially awareness raising. For example "All parties to encourage the development of local compacts". By their nature such actions are difficult to evaluate.
5. "Acknowledgement" is the action whereby "Scottish Executive, Departments, Agencies and NDPBs and NHS Boards to acknowledge resources required to implement the Compact". As with "encouragement" actions, this is very difficult to evaluate.

A2.18 Implementation of Compact principles is to be mainstreamed, in part, through the Executive's role in providing Resources to the sector. Intermediaries are to set out their approach to promoting the Compact in their funding agreements with the Executive and in their own business plans.

A2.19 The importance of Monitoring and Evaluation can be seen from the fact that in the Strategy it merits a separate Chapter. The dilemma of monitoring the Compact is well summarised: the Compact is about core principles, about relationship building and the development of mutual trust and confidence. As such monitoring, if it is about identifying a list of actions and getting bogged down in process and documentation, is likely to miss the essence of the Compact. Indeed it may be that it would be possible to conform to a long list of actions and processes and yet make no progress at all in developing mutual trust. However, monitoring actions and processes has the advantage in that they are readily measured. Trying to capture changes in confidence and trust is far more problematical, especially when, as the Strategy states, any monitoring and evaluation system has to ensure that it is not "excessively burdensome" (Paragraph 5.3).

A2.20 Against this background the Monitoring and Evaluation actions cover:-

• Undertaking a monitoring survey at 18 month intervals, with the first survey providing the baseline from which subsequent changes can be measured;
• Devising a number of direct and indirect monitoring indicators from the Direct Funding Database and the Consultation and Registration Evaluation and Compliance Systems. These latter 2 were both in the process of being set up when the Strategy was drawn up;
• Supplementing the earlier 2 information sources with interviews, focus groups and "real-time" information; and
• Publishing the monitoring results every 3 years.

CONCLUSIONS

A2.21 This analysis of the 2003 Compact and the Implementation Strategy has found that the 1998 and 2003 Compacts are broadly similar and any changes are of 2 main types:-

• They increase the Compact's symmetry so that there is a greater balance between the commitments of government and the sector; and
• They give greater recognition to the fact that the sector is so diverse that Compact implementation needs to be more engineered. To this end intermediary organisations are now accorded a greater role in the implementation and dissemination process within the sector.

Annex 1: Detailed Changes Between the 1998 and 2003 Compacts

INTRODUCTION

AN1.1 The purpose of this Annex is to analyse in greater detail the changes between the 1998 and 2003 Compact versions under the 5 headings of Recognition, Representation, Partnership, Resources and Implementation into which Compact commitments are categorised. These are summarised in Table AN1.1. It will be seen that government accounts for 60% of the 40 changes outlined in the Table.

Table AN1.1 Changes in Compact Commitments Between 1998 and 2003 According to Sector

Note:-
1. G is government; and
2. S is the voluntary sector.

AN1.2 The following sections look in greater detail at the changes outlined in the Table.

RECOGNITION

AN1.3 The differences in Government commitments between the 2 documents are generally slight, involving little more than some minor changes of wording. For example "community action" replaces "community development" and "stakeholders", "NDPBs". The only new text addition is to now see community action as also "contributing towards economic development" in addition to active citizenship.

AN1.4 Voluntary Sector commitments have increased from 4 to 6. The 2 new ones are identical to Executive commitments and have therefore been included to make the Compact more symmetrical. They are:-

• "Take positive steps to involve in the partnership, groups which face difficulties in making their voices heard"; and
• "Promote an understanding of the value of voluntary sector activity to all public sector bodies and stakeholders".

REPRESENTATION

AN1.5 There has been a considerable increase in the total number of commitments under this heading: from a 1998 total of 7 to 15.

AN1.6 In terms of the Executive there are 4 new commitments:-

• "Provide access and support, including the provision of information, to enable voluntary organisations to contribute to the policy development process";
• "Ensure that all national infrastructure voluntary organisations are routinely informed of NDPB appointment vacancies that fall within the remit of the Commissioner for Public Appointments, including NHS bodies";
• "Consult as appropriate in a flexible and open way, communicate clearly using appropriate media and ensure feedback is provided to all respondees and consultees"; and
• "Consult fully and to a stated timetable, jointly agreed when possible".

AN1.7 There are then 2 examples where minor wording changes have been made:-

• "Ensure clear and accessible channels of communication" replaces the wordier "provide information about channels of communication which is clear and accessible". Arguably this change also makes the commitment more pro-active; and
• When nominations are to be sought from the sector for such things as public boards and agencies this is now to be done "through open and transparent mechanisms".

AN1.8 A final commitment (clearly stating Executive objectives and priorities) is identical.

AN1.9 In terms of the sector, as in the 1998 Compact, these commitments are addressed to the sector's "representative lead bodies", it being recognised that the diversity of the sector means that no single or group of bodies can speak on behalf of the whole sector.

AN1.10 Of the 8 commitments 3 are identical, albeit that one of these has been separated out from what was formerly a larger "commitment" containing a number of "sub" commitments.

AN1.11 There are 3 new commitments:-

• "Ensure clear and accessible channels of communication to the sector", which is essentially identical to a commitment made by the Executive, thereby adding to the Compact's symmetry;
• "Publicise NDPB appointment vacancies as appropriate to relevant voluntary organisations". Again this promotes symmetry between the sector and government; and
• "Contribute as appropriate to consultation and policy development exercises".

AN1.12 The 2 examples of minor wording changes are:-

• For the representative bodies to represent the views of their constituencies "accurately and honestly"; and
• The addition of "voluntary sector" to the commitment to develop infrastructure to allow groups to express their views to the Executive.

PARTNERSHIP

AN1.13 Government commitments have increased from 6 to 7. There is one new one, 4 are identical to those in the 1998 Compact and there have been 2 minor wording changes. In addition one of the 1998 commitments has been split. The additional commitment is:-

"be aware of the resource constraints on voluntary sector participation in partnerships".

AN1.14 The 2 minor wording changes are:-

• To replace "meet with" by "work collaboratively with". This implies a far more positive form of interaction; and
• To add "using an appropriate format" into the commitment to "promote effective dialogue, using an appropriate format and the efficient use of time".

AN1.15 The commitment that has been split is the one that committed government to "meet with the sector to develop policy and practice and promote effective dialogue and the efficient use of time". This is now 2 commitments: to develop policy and practice and to promote dialogue.

AN1.16 Of the 5 voluntary sector commitments, 2 are identical to those in 1998, one is new, one has had minor wording changes and one textual changes.

AN1.17 The new commitment is again identical to an Executive commitment. Its inclusion is therefore making the Compact more symmetrical. It is:-

"Work with the Executive on issues of concern to either party in the spirit of openness and partnership."

AN1.18 A further commitment has had the text changed to make its meaning clearer:-

"Promote the value of collaborative working with the public sector in the best interest of its (the voluntary sector's) members, users and the wider public".

AN1.19 New wording has been added to one commitment so that the purpose of collaborative working is made clear:-

"to develop policy and practice and to promote effective dialogue".

RESOURCES

AN1.20 Of the 7 Executive commitments 2 are identical to those in the 1998 Compact, 2 are new, 2 have minor wording changes and 1 new text.

AN1.21 The 2 new commitments are:-

• "Recognise the need for full cost recovery in bids for service contracts"; and
• "Proof documents in order that potential impact on the voluntary sector is considered routinely as part of policy development".

AN1.22 The 2 minor wording changes are:-

• The addition of "consistently" when making a commitment to applying best practice in funding and grant administration;
• To support intermediary bodies and infrastructure "as appropriate" across the sector.

AN1.23 The new text is the insertion of:-

"and apply and promote Best Value processes in ways which secure equality of treatment for voluntary organisations".

AN1.24 Of the 4 voluntary sector commitments one is identical, one is new and 2 have new text.

AN1.25 The new commitment is:-

"Recognise that public resources are limited and that public funding is provided on the basis of value for money and its contribution to public priorities".

AN1.26 The new text is:-

• The addition of "financial and non-financial resources" when talking about championing good management; and
• Adding "and apply and promote Best Values processes in ways which secure equality of treatment for voluntary organisations" to the commitment that talks about securing Best Value in resource use.

IMPLEMENTATION

AN1.27 There are a total of 14 implementation commitments as opposed to 15 in the 1998 Compact. However, one of the key changes is the greater symmetry in the 2003 Compact, as Table 1.2 shows, with both the Executive and the sector each now being responsible for 7 commitments.

AN1.28 Of the Executive's 7, one is identical to the 1998 Compact (establishing a monitoring and evaluation framework), 5 have new text whilst one has minor wording changes.

AN1.29 Of those with new text:-

• The commitment to "work to ensure" that implementation arrangements are put in place is now more definitive: "put in place…..arrangements to implement the Compact";
• A commitment to put in place mechanisms to facilitate cross-departmental working now includes, as part of this, a commitment to mainstream Compact principles;
• A commitment to promote the Compact within the various arms of Government now includes other public bodies such as local authorities and CoSLA. In addition, promotion is to include "the spirit and principles" of the Compact. In essence this commitment is an amalgamation of 2 separate ones in the 1998 Compact;
• The 1998 commitment (to having a designated Minister and senior staff in NDPBs and agencies with responsibility for relations with the sector and for Compact review and implementation) has been revised to read: "ensure all Scottish Executive Departments, NDPBs and Agencies" have this responsibility. The responsibility also now includes monitoring and evaluation; and
• 2 1998 commitments (to report annually on the operations of the Compact to the Scottish Office and Voluntary Sector Forum and to contribute to an annual review) have been amalgamated into one commitment which is to now report "regularly on implementation progress".

AN1.30 Minor wording (and updating) changes relate to the commitment to have a dedicated unit within the Executive to promote the sector's interests. In 1998 there was a need to "ensure" that such a unit existed. Now they need is to ensure that it "continues" to exist.

AN1.31 The sector's commitments are again, as with Representation, largely addressed to representative bodies. Of the sector's commitments 2 are identical, 2 are new and 3 have new text.

AN1.32 The 2 new commitments are:-

• " Promote the spirit and principles of the Compact to all public bodies, including local authorities and CoSLA". This is almost identical to one of the Government commitments and is developing symmetry; and
• "Ensure implementation of the Compact is a core function of national infrastructure and intermediary organisations".

AN1.33 New text relates to:-

• Putting in place systems to monitor the "implementation" of the Compact throughout "the voluntary sector";
• Establishing a monitoring and evaluation framework jointly with the Executive, rather than participating in Compact operations monitoring; and
• Reporting "regularly" on Compact implementation, rather than "annually" on its "operations".

CONCLUSION

AN1.34 Overall the 2003 Compact is a more balanced document that its 1998 predecessor. It is now clear that the Compact is a partnership with both parties (government and the sector) having commitments that they need to meet if the Compact is to make a difference to partnership working and, ultimately, to service delivery.