Scottish Government

MONITORING AND EVALUATION OF THE SCOTTISH COMPACT Baseline Results 2004

APPENDIX ONE: DERIVING THE NON-SURVEY BASED INDICATORS

INTRODUCTION

A1.1 The 2003 study identified 13 indicators that were to be derived from the Direct Funding Database and the Consultation Registration and Evaluation System. This Chapter reviews these and looks at other suitable indicators that could be used to supplement them in the light of changes since 2003.

INDICATORS - DIRECT FUNDING DATABASE

A1.2 Of the 13 2003 non-survey based indicators, 6 were derived from the Executive's Direct Funding Database and were to be collected annually. These indicators, and the Activity Area they relate to, are shown in Table A1.1.

Table A1.1 Monitoring Indicators to be Derived from the Direct Funding Database

A1.3 One of the aims of this research was to identify indicators that were measurable, defined as being able to be obtained relatively easily. Applying this criterion to the indicators it was felt that:-

• Indicator 1 would be too difficult to generate, given that there would be a need to undertake further analysis on the Database and there would be difficulties in identifying a specific client group for all funded initiatives. For example, a number of Councils for Voluntary Service were funded that would be serving very diverse markets; and
• Indicator 5 (applications having no problems) would be impossible to collate information on from the Database. The percentage of applications approved (Indicator 3) would, however, seems to be a satisfactory surrogate. Accordingly Indicator 5 seems to be a duplicate.

Accordingly it was felt that these indicators (1 and 5) should be dropped.

A1.4 Of the other 3 (2 use the same data (Indicators 2 and 4)) 2 need to be treated with caution. These are 3 and 6. Applications approved and the percentage of the Executive's budget allocated to the sector, were likely to be a reflection of many factors in addition to Mutual Understanding and Trust. Accordingly, were they to remain as Indicators, they would need to be interpreted with care.

A1.5 Since the 2003 study was undertaken the Direct Funding Database has added some additional features that can be used for monitoring purposes. The key one is a question addressed to the various Executive funding Branches and Departments. This is:-

"Was the grant recipient notified of the success of their bids (sic) for funding at least three months before the new funding period commenced".

A1.6 This equates with the advice given to Executive Departments in the Funding Good Practice Guide where it states:-

" Departments should:

Wherever possible, notify the outcome of funding applications well in advance of the start of the funding year, preferably giving 3 months notice." (Scottish Executive, 2000)

Essentially this information is identical to Indicator 2 in Table A1.1.

A1.7 This additional question was only included for the 2003-04 grants round. As one might expect, the responses are still rather patchy. For example:-

• 91 (12%) of responses were Yes;
• 384 (52%) were No; and
• No entry had been given against 263 (36%) of grant applications.

A1.8 Information is also now provided about the breakdown of bids by Branch/Department. This shows:-

• The total number of bids;
• The number that were successful;
• The number that were unsuccessful; and
• The overall success rate and the success rate for individual Branches/Departments. For 2003-04 this varied from 5% to 100%, averaging 50%.

A1.9 In the light of the above the following indicators were derived, either fully or partially, from the Direct Funding Database:-

• The percentage of grant applicants ² notified of the success of their bids at least 3 months before the new funding period starts. This relates to the Funding Activity Area, in this case part of the Compact implementation process. However the Executive needs to ensure that all Branches and Departments provide this information if this indicator is to be of any use. If this problem is overcome than this indicator should be easily collated. It also has the merit of being unambiguous. Accordingly it was felt that it should be a Core indicator;
• There should then be 3 Secondary indicators:-
• The percentage of funding applications approved: information that again should be easily obtainable. This can be seen as a measure of Mutual Understanding;
• The total number of bids: information that can be seen as being indicative of confidence in government by the sector; and
• The percentage of direct Executive expenditure that goes to the sector: which can be seen as the Executive Trusting the sector. This would, however, need to make use of other data sources in addition to the Direct Funding Database.

INDICATORS - INDIRECT FUNDING DATABASE

A1.10 The 2003 GEN study rejected the Indirect Funding Database as a source of information, largely as too much expenditure was grouped under a number of undifferentiated headings. This still seems to be the case. For example, of the £112 million allocated under this heading, £47.6 million (42.5%) is allocated to various unspecified "multi-agency partnerships pursuing social inclusion strategies". In addition a small number of organisations account for a substantial proportion of spend. For example:-

• £7.4 million (6.7%) is allocated to Scottish Opera; and
• A further £15.6 million (14%) is allocated to 9 organisations, mainly involved in the performing arts (for example Scottish Ballet and Scottish National Orchestra).

A1.11 The remaining 37% of indirect grants are allocated to 688 individual organisations, some of which receive very small sums, for example less than £100.

A1.12 Given these flaws, this Database was not used to derive any monitoring indicators.

INDICATORS - CONSULTATION AND REGISTRATION EVALUATION SYSTEM

A1.13 Table A1.2 shows the Indicators that the 2003 Report proposed be derived from the Consultation Registration and Evaluation System, along with the Activity Areas they related to. At the time these indicators were proposed the System was not operational, having only being launched in December 2003. Accordingly the Indicators need to be revised in the light of the finalised System.

Table A1.2 Monitoring Indicators to be Derived from the Consultation Registration and Evaluation System

A1.14 Comparing the final versions of the Consultation Registration and Evaluation forms with the original Indicators the following conclusions can be drawn:-

• Indicator 1 (Table A1.2) can still be used as the finalised form asks if 12 weeks were given for written consultations. If the response is "No" then the number of weeks given has to be indicated;
• Indicator 2 can likewise still be derived as information will now be collected on the number of weeks after the consultation period ended that copies of reports were sent to consultees;
• Indicator 3 can be derived as the finalised form has a text question that asks for details of what happened as a result of consultation;
• Indicator 4 can be derived, in that the form enables liaison with the sector to be determined in so far as the sector is identified as one of the potential target groups. However, given that the sector is unlikely to be the sole target group this Indicator might only be of use if responses from the sector can be identified separately;
• Indicators 5 and 6 cannot be derived directly from the form as a breakdown of responses is not asked for by target audience;
• Finally Indicator 7 could be derived, but only, as with Indicator 4, if the sector is the only consultee or if sector responses can be disaggregated. Although the form asks for the number of consultation papers distributed, and the number of responses received, this is not broken down by target group.

A1.15 Accordingly it was felt that, in the light of the finalised Consultation forms, 2 of the original 7 Indicators should be dropped and, for a further 2, there needs to be further consultation to see if the responses need to be disaggregated to identify separately those responses from the sector. However, the finalised forms ask for additional information, which can be used to modify some of the original indicators.

A1.16 The additional information now asked for on the forms covers:-

• If there is any likely impact upon the voluntary sector, then respondents are asked if they have completed a Regulatory Impact Assessment;
• For written consultations, the extent of responses from the target audience are asked for (ranging from a full range to narrow);
• Whether copies of reports produced as part of the consultation process were sent to consultees; and
• Complaints about the consultations are also to be logged as is the response by the relevant Branch or Department.

A1.17 However, it may not be possible to use the results of the forms without a degree of manipulation. For example (as highlighted in Paragraph A1.14) it seems unlikely that the voluntary sector will always (or perhaps ever) be the sole audience for consultations. Accordingly there might be a need to isolate the sub-set of consultations to which the voluntary sector is a target audience. It is suggested that the appropriate consultations be those that have, as target audiences, the following groups (the groups being ones defined on the form):-

• Equality Groups;
• Voluntary Sector Umbrella Groups; and
• Voluntary Sector Community Groups.

A1.18 Accordingly the following Indicators should be derived from the Consultation Registration and Evaluation System for those consultations to which the sector (as defined in Paragraph A1.17) is one of the target audiences:-

• The percentage of consultations that allowed at least 12 weeks for responses. This should be a Core Indicator;
• The percentage of consultations that contain options that impact upon the sector for which a Regulatory Impact Assessment has been completed. Again this should be a Core indicator;
• When reports were published as a result of the consultations, the percentage of consultations in which copies of these reports were sent to consultees. Again we feel that this should be a Core Indicator; and
• The impact of the consultation, derived from the text response which asks for a 150 word summary of what was done as a result of consultation. Although this will be more time consuming to collate it relates to the outcome of the Compact and is therefore worth including. However, given potential challenges in relating cause and effect we do not feel that this should be a core indicator but a Secondary one.

A1.19 We then propose that a number of other Secondary Indicators be used. However, there needs to be discussion as to whether responses can be disaggregated to allow the sector to be separately identified before these are used. These indicators are:-

• The number of responses received as a percentage of consultation papers distributed. If the sector is only part of the target audience then this indicator would seem to be of limited use unless disaggregation of responses is possible. It may also be the case, as the Footnote to Table 2.1 indicates, that organisations will respond to consultations when they were not on the original list of consultees. Accordingly the indicator may need to be modified so that, over time, it tracks all responses received from the sector not just from those who were invited to respond;
• The range of response origins (full, reasonable, or narrow); and
• Percentage of consultations about which complaints were received.

A1.20 Currently there is no baseline data available from the Consultation and Registration and Evaluation system for any of these indicators. This now needs to be generated.

INDICATORS - COMPLIANCE SYSTEM

A1.21 Unfortunately it is not possible, at this stage, to know what potential use the Compliance System will be for monitoring. Decisions on what information to be collected, and how this will be processed, are still in their early stages having not yet been approved by Ministers. Accordingly this will need to be revisited once the System is established.

INDICATORS - IMPLEMENTATION STRATEGY

A1.22 The Compact Implementation Strategy (see Appendix Two) contains 39 actions. Of these 10 (26%) were classed as "Specific", that is they are commitments to undertake identifiable actions. These potentially lend themselves to monitoring.

A1.23 However, some of these have already been implemented. For example:-

• The revised Compact was launched in 2003 (Leadership);
• The revised Compact has been endorsed at Cabinet level in the Executive (Strategic Aims); and
• Progress, through this study, is being made in undertaking a monitoring survey and devising indicators (Monitoring and Evaluation).

A1.24 In other areas the actions are being implemented. For example, under Mainstreaming:-

• Voluntary Sector Liaison Officers (VSLOs) were appointed in each Executive Department in June 2004 and publicised through the Voluntary Issues Unit's (VIU) website and Scottish Executive's Grants Guide; and
• The VIU's Steering Group approved draft VSLO job descriptions in June and an introductory meeting was held in October 2004. The Guidance has now been circulated to the VSLO's for comments and will be finalised by the end of 2004.

A1.25 Under the Strategy's Raising Awareness and Best Practice headings there were 2 specific actions:-

• VIU was to have information about its work with the sector on the Executive's website. It will now ensure that the site is updated monthly. New pages on Futurebuilders Scotland and the VSLOs have recently been added, whilst the Strategic Funding Review page has been updated. There are plans to update the Volunteering Sections and other pages in due course; and
• VIU was to have a rolling programme of seminars to make staff aware of the Compact. These are to start soon.

A1.26 There is one specific action for Communications. This is piloting the compliance system in an Executive Department in the winter of 2003. Apparently this has not, as yet, been done.

A1.27 Finally there was to be a regular meeting of the Executive/Voluntary Sector Forum to review implementation progress. This meeting was held in November 2004 when a paper regarding progress with Compact implementation was tabled.

A1.28 In summary, of the Strategy's 10 actions:-

• 8 have been, or are being, implemented; and
• 2 are still outstanding, although one of these (staff seminars) is planned

A1.29 All of the outstanding ones are essentially about process: undertaking actions to help Compact implementation. As such they do not lend themselves to ongoing monitoring or development as indicators to assess outcomes. However, we would suggest that progress in implementing them be regularly reviewed over the next year in order to ensure that they are undertaken.

CONCLUSIONS

A1.30 With the exception of the Compliance System we have not identified any other data sources that could be readily used to monitor Compact implementation. Consultation with the Advisory Group also failed to identify any additional sources.

A1.31 Using the Direct Funding Database and the Consultation and Registration Evaluation System we have identified:-

• 11 Indicators, of which 4 are derived from the Direct Funding Database, the remaining 7 from the Consultation and Registration System;
• Of these we feel that 4 should be Core; and
• Of the Secondary ones we feel that, for those derived from the Consultation System, there needs to be further discussion to see if it is possible to disaggregate responses, thereby identifying the sub-set that is sector specific. If this is not possible then the utility of having these Indicators needs to be assessed.

A1.32 Table A1.3 summarises the 11 indicators and categorises them according to the Activity Area they fall under, their source and type.

Table A1.3 Non-Survey Based Indicators

Note:-
1. Before this is finalised there needs to be further discussion to see if the Consultation System can produce disaggregated responses. It is also the case that the sector will have ways of accessing consultations beyond the Executive's circulation list. Accordingly the indicator may need to be modified so that it measures the volume of consultation responses over time, regardless of whether respondees have received a formal invitation from the Executive.