Scottish Government

MONITORING AND EVALUATION OF THE SCOTTISH COMPACT Baseline Results 2004

CHAPTER TWO NON-SURVEY BASED INDICATORS

INTRODUCTION

2.1 The 2003 Study identified 13 indicators that could be derived from a variety of Executive data sources to monitor the Compact. These were reviewed as to their suitability given:-

• The changes to the Compact;
• Changes to some of the information collected; and
• The addition of new data sources.

2.2 The aim was to divide the indicators into 2 categories:-

• Core: in that they were central to Compact monitoring; and
• Secondary: ones that, whilst important, were seen as being less significant and could, if necessary, be omitted.

2.3 The detail of the analysis undertaken in given in Appendix One. This Chapter summarises the conclusions.

THE ORIGINAL INDICATORS

2.4 Of the 13 original indicators 6 were derived from the Executive's Direct Funding Database, which gives details of financial allocations to particular groups. Upon re-examination of these it was felt that 2 were no longer valid, largely as it would only be possible to generate the information following further analysis of the database. The other 4 were still felt to be relevant. However, one of these (dealing with the percentage of applications determined within 3 months) has been modified as a result of the addition of a specific question in the database (asking if grant recipients were notified of the success of their bids at least 3 months before funding was due to start). This is a more specific measure that the original proposed indicator.

2.5 Of the 4 indicators now to be derived from this source, one was felt to be Core, the other 3 Secondary.

2.6 In the 2003 Study, the Indirect Funding Database was felt to be of limited use for monitoring purposes as too much expenditure was being allocated under general headings. This is still the case (see Paragraphs A1.10 to A1.12 in Appendix One). Accordingly it is felt that no indicators should be derived from this database.

2.7 At the time the 2003 Report was written the Executive's Consultation Registration and Evaluation System was not operational. Now the Consultation Registration and Evaluation forms have been finalised and have been operational since December 2003. Analysis resulted in 2 of the original 7 indicators being dropped, as the forms do not provide a breakdown of responses by target audience. Of the other 5:-

• 3 are still valid, albeit in a slightly modified form as a result of changes to wording on the form; and
• it is proposed that 2 others be dropped as there are now alternative sources from which similar information can be derived.

2.8 The Consultation forms include additional questions from which new indicators can be derived. These are:-

• The range of consultation response origins (graded from full to narrow);
• The percentage of consultations for which Regulatory Impact Assessments were completed, as the proposed change would impact upon the sector;
• The percentage of consultations about which complaints were received from consultees; and
• The number of responses received from the sector as a percentage of consultation papers distributed to the sector.

However, some of these indicators will only be of use if it is possible to disaggregate the data, in order to specifically identify responses from the sector. Following discussion with the Executive it seems to be the case that all but one of the indicators proposed can be generated from the System. The one that currently cannot be produced (the number of responses received from the sector as a percentage of consultation papers distributed to the sector (Indicator 10 in Table 2.1)) could possibly be produced if it was felt sufficiently important to modify the system in order to generate it. Given this, it has been left in at the moment.

2.9 Of the 7 indicators derived from the Consultation and Registration Evaluation System, 3 have been classified as Core.

NEW DATA SOURCES

2.10 A potential source of monitoring information in the future might be the Compliance System. This was proposed by the Review Group in its 2004 Report (Scottish Executive, 2004, Paras. 4.6.6 and 4.6.7) and was intended both to record and resolve grievances resulting from non-compliance with Compact principles. However, at the time of writing this was still in the early stages and had not been approved. However, if introduced, it would allow for the recording, monitoring and potential resolution of issues related to non observance of the principles of the Compact. It would essentially provide some element of pressure, albeit one that was gently applied, on organisations to comply with the values of the Compact.

2.11 The Compact Review Group's Implementation Strategy contains 39 actions, of which 10 are "Specific", that is they are commitments to undertake certain activities and as such lend themselves to monitoring. Of these 8 have been, or are being, implemented covering such things as launching the revised Compact. The remaining 2 are still outstanding. They are, however, about "one-off" processes and as such do not lend themselves to on-going monitoring.

2.12 Advisory Group members were asked, as part of the indicator consultation exercise (Paragraph 1.11), if they knew of any other data sources that could be used for monitoring purposes. None were identified.

CONCLUSIONS

2.13 The outcome of the review of the original indicators and the data sources was the identification of 11 non-survey based indicators. Of these, 4 were Core indicators, the remainder Secondary. They are shown in Table 2.1. With the exception of Indicator 10 (see Paragraph 2.8) all of the information needed can be obtained from the relevant databases.

2.14 Of the 11 Activity Areas that the 2003 report identified as being central to Compact monitoring and evaluation these non-survey sources cover 7. The remainder, therefore, need to be covered through a monitoring survey. This is dealt with in the next Chapter.

Table 2.1 Non-Survey Based Indicators

Note:-
1. Before this is finalised there needs to be further discussion to see if the Consultation System can produce disaggregated responses. It is also the case that the sector will have ways of accessing consultations beyond the Executive's circulation list. Accordingly the indicator may need to be modified so that it measures the volume of consultation responses over time, regardless of whether respondees have received a formal invitation from the Executive.