MONITORING AND EVALUATION OF THE SCOTTISH COMPACT Baseline Results 2004
1. This Report on the Monitoring and Evaluation of the Scottish Compact was commissioned from GEN Consulting by the Scottish Executive.
2. The Scottish Compact is a protocol that sets out the key principles whereby the Executive, its agencies, Non-Departmental Public Bodies (NDPBs) and the voluntary sector can work in partnership. The Compact was first published in 1998. A revised version appeared in 2003, updated to reflect the changes that had taken place in Scottish governance.
3. As part of the Compact monitoring process a Compact Review Group, with members drawn from government and the voluntary sector, was established in 2002. Amongst other things it produced an Implementation Strategy in 2004. This proposed that a baseline be established that summarised the extent of knowledge of the Compact and looked at relationships between government and the sector. This could then be repeated at 18 month intervals and the impact of the Compact assessed.
4. To undertake this work a contract was let by selective competitive tender and awarded to GEN Consulting.
NON-SURVEY BASED INDICATORS
5. The first part of the research involved assessing the available data sources that might be of use for Compact monitoring purposes. An earlier study by GEN for the Executive had identified 11 key factors (or Activity Areas) that it was felt any Compact monitoring and evaluation framework needed to capture. Seven of these were covered by 11 indicators that could be derived from 2 sources: the Direct Funding database and the Consultation and Registration and Evaluation System.
6. The Direct Funding database contains details of the grants given by the Executive to voluntary sector organisations. It was proposed to derive 4 indicators from this source, each of which could be easily generated: for example the percentage of applications from the sector that were approved by the Executive.
7. The Consultation, Registration and Evaluation System became operational at the end of 2003, although as yet it has not been used to generate any data. It collects information on consultations undertaken by all Executive departments, covering such things as the time given for responses and what was done as a consequence. It was felt that the System was capable of generating 7 indicators that would be of use for Compact monitoring purposes. These included such things as the percentage of consultations with the sector for which at least 12 weeks were given for responses and the percentage of sector consultations about which complaints were received.
8. In addition there are other systems that are being set up that could generate data that would be useful for monitoring. Chief amongst these is the Compliance System that is to be set up to record and, perhaps, resolve grievances relating to non-compliance with the Compact.
Survey Based Indicators
9. The other 4 Activity Areas were to be monitored by undertaking surveys of government and the sector. Examination of the changes between the 1998 and the 2003 Compacts and the earlier work on indicators undertaken for the Executive by GEN resulted in a decision being made to devise surveys that concentrated upon Compact outcomes, rather than outputs.
10. Accordingly 2 questionnaires were drawn up. Each contained 16 questions and, as far as possible, they were symmetrical: that is they asked identical questions of the sector and government. They were structured as follows:-
- The first 5 questions were Compact specific and asked about awareness of the Compact and the existence of systems for disseminating information about it and its implementation;
- There were then 3 questions on the use of the Compact and the extent to which this had been found to be useful;
- Five statements were then given to which respondents were asked to give graded responses. Each statement was framed so that it captured the essence of the Compact's intended outcomes in such areas as Recognition, Partnership and Implementation. Different statements were put to government and the sector, equating to their different commitments under the Compact;
- There were then 2 questions about co-operation; and
- Finally space was given for any other comments that the respondent might have.
THE SURVEY METHODOLOGY
11. The questionnaires were sent to all Executive Departments, agencies and NDPBs, 323 in total. Ninety six were returned, a response rate of 30%. The Scottish Council for Voluntary Organisations' database was used to generate a stratified random sample of 700 organisations. One hundred and ninety one were returned, a response rate of 27%. The overall response rate was 28%.
12. In distributing the questionnaires both post and email was used. Overall the postal response rate was better (39% as opposed to 17%). Although there may be reasons for the poor email response (for example a failure to send follow-ups) it is suggested that any follow-up surveys use the post.
13. Awareness of the Compact amongst voluntary sector respondents was low, with only a third having heard of it. Awareness was related to organisational size, with the larger organisations being more likely to be aware of the Compact. The level of awareness amongst government was far higher (72%).
14. It was more likely that government had a staff contact point for the compact than the voluntary sector (35% as against 13%) and had a strategy for its implementation (17% as against 4%). There was also a greater probability that Executive respondents were aware of an implementation strategy than those from agencies or NDPBs.
15. Government was also more likely to have made use of the Compact in the last 18 months than was the voluntary sector (38% as against 11%). Of those who had made use of the Compact the majority, in both government and the sector, found that this had been "very" or "quite useful" (62% and 81% respectively). This usage was felt to have resulted in a better service being provided for 81% of government respondents and 43% of those from the sector. However, the small numbers involved mean that these figures need to be treated with a degree of caution.
16. Respondents were asked to give a graded response to a number of statements relating to their (government or the sector) commitments under the Compact. Generally the more Compact specific the statements then the greater the percentage of respondents who gave a "don't know" answer.
17. A significant percentage of sector respondents "neither agreed nor disagreed" with the statements or "did not know". This may reflect a lack of awareness of the Compact, allied to limited contact with central government and its agencies. Between 11% and 27% of sector respondents either "disagreed" or "strongly disagreed" with all of the statements. Given that some of these related to concepts central to the Compact, such as partnership working, this should be a matter of concern.
18. Government responses were far more positive, with generally over half either "strongly agreeing" or "agreeing" to the statements.
19. Respondents were asked for their views on co-operation with government or the sector, both now and over the last 18 months. Government was far more positive than the sector, with a third of respondents feeling that current levels were "very good" or "good", in contrast to 9% of sector respondents. A more positive note was given by those who felt that the level of co-operation had increased over the last 18 months (25% in government and 11% for the sector).
THE FOCUS GROUPS
20. Focus groups were held in 3 contrasting geographical areas: Arbroath, Glasgow and Inverary. The aim was to explore some of the issues related to the Compact in greater depth. Invitations were sent to groups in the local area and to government bodies such as Communities Scotland, and the Scottish Environmental Protection Agency.
21. With the exception of the Arbroath group, the level of awareness of the Compact was limited. A local compact had recently been launched on Tayside and this was responsible for awareness of the National Compact being far greater there.
22. One of the main themes to emerge was the relevance of the National Compact to many local groups, given that their main interface was with their local council rather than central government or its agencies. Overall it was felt that the principles of the National Compact needed to be incorporated into local compacts.
23. Focus Group members felt that one way of increasing awareness of the Compact was to publicise it more widely through "champions": high profile individuals in government and the sector who could promote its values.
24. Compacts, whether national or local, were felt to need "teeth": some form of sanctions to ensure that all parties conformed to their principles. Without this they could become little more than tokenism.
25. Ten Recommendations arising from the research are made. The key ones are that:-
- The Executive's Consultation, Registration and Evaluation System needs to be accessed so that baseline data can be produced for the 7 indicators that are to be derived from it;
- The baseline survey of government and the sector should be repeated every 18 months using the same questions and methodology as outlined in this Report;
- Consideration needs to be given as to how to raise awareness of the Compact amongst smaller voluntary organisations and agencies and NDPBs; and
- The Executive should promote the implementation of the existing Partnership Agreement to establish local compacts to act as an interface between the voluntary sector and local councils.