The Scottish Government

« Previous | Contents | Next »

Listen

CHAPTER 12: MONITORING AND EVALUATION

12.1 Monitoring and evaluation systems

Introduction

Monitoring and evaluation will enable regular assessment of progress in achieving efficient and effective delivery of outcomes through the 2007-13 SRDP. Arrangements for monitoring and evaluation will follow the provisions of Articles 77 and 78 of Council Regulation ( EC) No 1698/2005 and will utilise the European Commission's Common Monitoring and Evaluation Framework ( CMEF). Progress will be assessed against the EU-wide common set of indicators in the CMEF, including those relating to the baseline situation as well as to inputs, outputs, results and impacts of the Programme. In addition, use will be made of additional indicators that relate to specific objectives of the SRDP.

The Scottish Executive established a Monitoring and Evaluation group in September 2006 to oversee the development of the monitoring and evaluation process. It is an internal group, with representation from the Scottish Executive family members including SNH and FCS. Its work has focused on developing and agreeing indicators to be used and considering the information that needs to be captured in the monitoring system.

Types of Indicator in the CMEF

Table 12.1 summarises the types of indicator in the CMEF.

Table 12.1 Types of Indicator in CMEF and Descriptions

Table 12.1 shows that the CMEF is based around 5 types of indicators: baseline, input, output, result and impact. The baseline indicators were used to inform the analysis of strengths and weaknesses in chapter 3. The baseline indicators also underpin the strategic plan.

Indicators in the SRDP 2007-2013

For each Axis in the rural development programme, the CMEF prescribes a common set of indicators. Additional indicators are allowed and have been proposed for the SRDP where an indicator in the CMEF is not detailed or specific enough to reflect the measure, whether at output, result or impact indicators.

Tables 12.2 to 12.5 summarise the indicators to be used in the SRDP for Axis 1, 2, 3 and LEADER. Each table notes the common indicators as well as additional indicators or programme specific indicators.

For Axis 1, additional result indicators include monitoring additional sales for producers as a result of support (for measure code 123) and additional impact indicators include employment creation (for measure code 123).

For Axis 2, additional result indicators include monitoring the use of pesticides and fertilisers (for measure 214) and improved and enhanced public access (for measure code 227). Examples of additional impact indicators are improvements in landscape (measure 323) and improvements in rural capacity (measure 41). An example of an additional result indicator is improved and enhanced public access (measure 313). The Scottish Executive has a national target to bring the special features of designated nature conservation sites into favourable condition. The following indicator is therefore added for measure code 214 "Bringing the special features of designated nature conservation sites including all the Natura Network into favourable conditions". Forestry Commission Scotland has a commitment on the contribution of forestry to mitigating climate change and therefore an additional impact indicator on carbon savings from forestry is included for measure codes 221 and 223.

For Axis 3, additional result indicators include satisfaction with services (measure code 321) and an additional impact indicator is safeguarding the sensitive aspects of landscape characters (measure code 341).

For LEADER, an additional impact indicator of increasing rural community capacity is included. LEADER will also contribute to the other axes and will therefore monitor progress against other relevant indicators.

In some cases, data is not available to report against the common indicators. All lead baseline indicators are presented. Work is in progress to develop all indicators. The CMEF notes that the indicator 'High nature value farmland' is under-developed across Europe. It can therefore not be reported against in the SRDP. An alternative indicator will therefore be used to capture high nature value farmland: hectares of land under farmland, woodland, urban, and other. Annual progress reports will report against this alternative indicator. Over the course of the programme, work will continue to develop the common indicator on high nature value farmland indicator and a technical group will be formed to explore its development and robustness. Owing to lack of data, an alternative indicator is also proposed for objective indicator 20 on water quality. Rather than gross nutrient balances, the indicator % of groundwater sites over 50mg/L is used as it is considered to be a good measure of diffuse pollution in particular.

Table 12.2: Output, Result and Impact Indicators for Axis 1

Table12.3a: Output Indicators for Axis 2

Table 12.3b Result and Impact Indicators for Axis 2

Table 12.4: Output, Result and Impact Indicators for Axis 3

Table 12.5: Output, Result and Impact Indicators for LEADER

Annex 3 to this document contains detailed information of the indicators to be used for each measure code. These accord with Guidance Note H of the CMEF. An example is shown in table 12.6 below. The table notes the type of indicator, gives a description and presents the indicative target.

Table 12.6: Example of Presentation of Indicators by Measure Code (for Measure 111)

# Impact indicators will be estimated based on output and result indicators

Targets

The indicator tables by measure code in Annex 3 contain information on indicative targets. The CMEF states that all indicators should be quantified and appropriate targets set, where the indicator concerned lends itself to quantification. Where quantified data are not available, qualitative data should be used.

The targets presented in Annex 3 are indicative estimates, based on past experience and expert judgement along with an assessment of whether targets are realistic in relation to funding. In some cases it has not been possible to set targets until further baseline data is collected. All result indicators in Axes 1, 2 and 3 do, however, have indicative targets associated with them. For many of the impact indicators, since they will be estimated based on output and result indicators, it has not been possible to set quantitative targets. Many of the impact indicators, such as labour productivity, require a change to be estimated. Setting a target on these requires knowledge of the baseline value (for example productivity, in the beneficiary population). Where this is the case, at this stage, qualitative statements have been used. Targets are, however, set for the 'employment creation' impact indicator.

After the programme has been adopted, and once further baseline information is available, indicative quantitative targets for impact indicators will be set, for reporting in 2010.

Similarly, for LEADER, the action groups are currently submitting business plans which include details of targets. Once these business plans have been approved it will be possible to confirm output and result indicators for LEADER.

Since targets are indicative, all targets will be updated during the course of the programme as new information becomes available.

Reporting frequency

Output and result indicators will be reported annually. Impact indicators will be reported at the mid-term and ex-post evaluation stages. Independent evaluators will estimation and quantify the impact.

The annual progress report will include:

• a table including the financial implementation of the programme giving, for each measure, a statement of the expenditure paid to beneficiaries during the calendar year
• the monitoring tables which will include quantitative information based on the common output and result indicators, and
• a summary of the on-going evaluation activities.

Monitoring system

In order to facilitate the monitoring and evaluation process, it is essential to give careful consideration to the information that needs to be captured in the monitoring systems. Experience from the mid-term evaluation of the SRDP 2000-2006 suggested that the monitoring system had not been designed in the most efficient manner for evaluation purposes and that this had led to difficulties in being able to monitor and evaluate the Programme. In the 2007-2013 Programme this problem has been addressed in two ways.

First, the information to be captured from the system has been identified and passed to the IT teams to be incorporated into the monitoring system. The list has been derived based on the agreed indicators as well as the monitoring reports that will be produced. Decisions have been taken on whether the best way to capture the data is through the application form, claim form or some alternative. This process has also taken into account the fact that the different delivery mechanisms for the 2007-13 SRDP ( LFASS, Rural Development Contracts, LEADER) have different data collection processes associated with them.

Second, common identifiers have been added to application and claim forms so that data sources can be linked to routine sources such as the agricultural census. This will facilitate monitoring and evaluation. Every application will be identified using a Business Reference Number and the applicant post code. This systematic approach will ensure effective links between all applications and claims forms. For all applicants, age and gender information will be recorded to allow reporting of indicators by these characteristics.

In general, input and output indicators will be measured through information collected at the point of delivery, generally through administrative records though there may be occasions when the information is better obtained by carrying out surveys. Result indicators may be measured either through administrative records or through evaluation methods such as sample surveys. Impact indicators, on the other hand, will be determined at the evaluation stage, using input, output and result information but also other tools and wider sources of data to build up a picture of the net impact of the programme on its wider strategic objectives.

Evaluation

Mid-term and ex post evaluations of the SRDP will be conducted as set out in the Regulation (1698/2005). Evaluation, including the provision of data for impact indicators, requires the collection of additional data to assess the following issues;

Additionality: would the activity have gone ahead without funding?

Attribution: can any change be attributed to the programme? In order to determine attribution, sampling of 'non-participants' may be required as a comparator.

Displacement: has the activity displaced other projects/activity?

Additional evaluation questions to those set out in the CMEF have been added to the indicator tables in Annex 3.

Part of the funding for Technical Assistance will be used to fund additional data collection required for monitoring and evaluation as well as to commission mid term and ex post evaluations of the SRDP.

12.2 Monitoring Committee

The Monitoring Committee, in conjunction with the Managing Authority, will oversee the effective implementation of the 2007-13 SRDP. It will be chaired by a senior representative of the Scottish Executive. Its members will include:

• economic and social partners;
• competent regional, local and other public authorities; and,
• other appropriate bodies representing civil society and non-government organisations including those responsible for environmental protection and enhancement and for the promotion of fairness and equality between different groups in society.

The Monitoring Committee will include a representative from the Structural Funds and Fisheries programmes in order to ensure proper complementarity between the SRDP and these programmes. The European Commission may participate in the Committee in an advisory capacity.

The Monitoring Committee will also maintain systematic links with the regional network of bodies set up by the Scottish Executive to determine priorities for the SRDP and to assess applications for funding under the programme (see Chapter 11). In particular, the Committee will liaise closely with the National Project Assessment Committee.

« Previous | Contents | Next »