« Previous | Contents | Next »
Listen
SECTION 3: PROCESS EVALUATION
Introduction
3.1 This chapter focuses on an evaluation of the advisory service and concentrates primarily on process issues. It includes a review of the rationale and objectives of the programme, and the procedures for appraisal, monitoring and reporting. The level of integration with other support mechanisms, future development and areas of best practice are also reviewed.
3.2 This section has been informed primarily by:
- the consultations undertaken with SCHRI development officers
- contacts in EST and HIE
- the survey of beneficiaries to assess the use of the advisory service and levels of customer satisfaction.
Rationale
3.3 Any public sector intervention in the economy needs to demonstrate a clear rationale which justifies the scale and nature of assistance. A case needs to be made that markets are not operating efficiently (market failure), which necessitates action in order to tackle existing market failures and support market adjustment.
3.4 In the case of support for the development of the renewables sector, the principal market failure relates to the existence of negative externalities in the form of pollution and climate change which are linked with the burning of fossil fuels. Costs associated with these externalities are not internalised in the market price of oil, gas, and coal, which means these energy sources are favoured over non-polluting and carbon neutral alternatives. Recent developments such as the Climate Change Levy ( CCL) and Renewable Obligations Certificates ( ROCs) provide incentives to the market to encourage the development of the renewable energy market.
3.5 Regarding the specific case of small scale renewables, the market remains relatively underdeveloped in Scotland, with only a small number of firms currently operating in the sector. The development of a subsidy, in the form of the SCHRI grant, is felt by both HIE and EST to be an important factor in encouraging the development of the sector, by stimulating demand and encouraging further investment in small scale renewables.
Strategic Objectives
3.6 The focus of any final evaluation of government funded initiatives will be the extent to which the initiative has met its stated objectives, targets and milestones. It is therefore important that the programme is able to demonstrate whether or not this is the case. As discussed earlier in the report, the initiative has three main stated objectives:
- to support the development of community scale renewable projects
- to support the installation of household renewables
- to raise awareness of renewable technologies and their benefits to Scotland
3.7 At this interim evaluation stage, it is worthwhile reviewing the current stated objectives against SMART criteria, which provide a framework against which the effectiveness of the objectives can be assessed. The five criteria are as follows:
- Specific - is there a description of a precise or specific behaviour/outcome which is linked to a rate, number, percentage or frequency
- Measurable - is there a reliable system in place to measure progress towards the achievement of the objective
- Achievable - with a reasonable amount of effort and application, can the objective be achieved
- Relevant - can the people with whom the objective is set, make an impact on the situation. Do they have the necessary knowledge, authority and skill
- Time-based - is there a finish and a start date clearly stated and defined.
3.8 The key objectives of the programme have focused on raising awareness of renewable energy in communities, and ensuring buy-in at a local level, by getting as many projects off the ground as possible across a wide range of technologies. Thus operational objectives do appear to be rather 'soft' and not SMART although this has allowed the programme a significant level of flexibility in the early stages of development.
3.9 However, raising awareness and supporting the installation of renewables is only a means to an end and is not an end in itself. Therefore, it is important that as the programme develops consideration is given to stressing the important of targets on outcomes. This will be important in determining the success of the initiative when it is subject to final evaluation.
3.10 A definition of inputs, activities, outputs and outcomes is given below, based on "The 3Rs Guidance" 5 produced by the Office of the Deputy Prime Minister ( ODPM).
- Inputs are the financial, in kind contributions and time resources used to fund the initiative
- Activities are the direct products or services provided or funded by the initiative
- Outputs are the intermediate effects of a project's actions. They represent the mechanism by which inputs and activities yield their intended outcomes
- Outcomes are the influence the initiative has on the various domains within the dimensions of environment, social and economic - quality of life conditions.
3.11 In this context, outcomes could be developed in relation to the amount and/or proportion of energy produced by small scale renewable energy systems. For example, a SMARTer objective than "to support the installation of household renewables" would be "to increase the proportion of domestic energy produced by small scale renewable systems to X% by 200Y". This is a specific and time-based objective against which clear progress can be measured.
3.12 In terms of being measurable, concentrating on the proportion and quantity of energy produced would necessitate better analysis of the baseline position and more robust ongoing monitoring systems. This would potentially involve closer links between SCHRI, installers and the Scottish Renewables Forum in order to provide a systematic way of measuring small scale renewable energy production and employment across Scotland.
3.13 There are no specific milestones or output targets associated with the third objective "raising awareness of renewable technologies and their benefits to Scotland". Specific activities related to this objective include:
- conferences, workshops and seminars developed through SCHRI
- wider marketing of the initiative
- development of the website and case studies.
3.15 While it is widely agreed that the SCHRI is successfully raising awareness, consideration needs to be given to whether there are specific measurable outcomes that should be related to this objective, in order to better demonstrate progress in this regard.
3.16 It is also recognised that the initiative needs to demonstrate how it contributes to strategic outcomes for the Scottish Executive, principally carbon displacement and economic activity/employment. The SCHRI was established to stimulate a micro-renewables market consisting of a wide range of technologies and was not strictly aimed at those technologies producing the largest reductions of carbon. Therefore, carbon displacement remains a secondary objective of the programme, and the programme has not been designed to maximise carbon displacement. Nevertheless, it is recognised that the initiative will need to increasingly demonstrate how it contributes to this objective, although there is a reluctance to attribute targets in this regard, as this would focus support on a smaller number of technologies, primarily biomass.
3.17 Supporting employment creation does also not appear to have been central to the development of the programme, which interfaces primarily with the community rather than the private sector. The initiative has supported the development of the sector by stimulating demand, but employment gains in installers are likely to be mostly displaced from companies installing conventional systems. However, there are potentially future gains in companies which have a manufacturing capability in Scotland such as the micro-wind sector. SCHRI support will stimulate demand at a critical stage in the market development. The issue of how the initiative contributes to these wider strategic objectives is discussed in further detail in the value for money assessment.
Appraisal, Monitoring and Reporting
Household stream
3.18 The administrative procedure for household applications is relatively straightforward. There are clear criteria for eligibility of projects and if projects meet these criteria then they will qualify for the grant. Contact between householders and development officers is limited, although in exceptional circumstances, the officer can arrange to make a site visit and provide technical advice over the phone.
3.19 Generally, householders are directed towards the centralised telephone helpdesk, which is based in the Orkney Energy Efficiency Advice Centre ( EEAC), for advice and guidance on installing a renewable energy system. In terms of the ongoing monitoring and reporting of household grants, the application states that applicants may be asked to supply information in relation to system output, estimated fuel savings and any equipment failures.
3.20 The requirement for all householders to submit self-completion monitoring information is not seen by EST as the most effective means of collecting information on cost and fuels savings and carbon displacement. Indeed, it has been made clear from the survey of beneficiaries, that very few households are clear about the actual fuel and cost savings associated with installation of the system. Instead, a number of site visits are to be arranged with a sample of households in order to collect this information. This is felt to be a more robust means of collecting this information, which may be influenced by a range of factors such as increased energy use due to lower heating bills.
Community Stream
Appraisal
3.21 The role of the development officer is central to the application procedure for community organisations. Any application for funding is submitted via the development officer and they generally provide a significant level of support and guidance in preparing the funding application. This means that the local development officers have considerable autonomy in terms of what projects are supported.
3.22 This in-built flexibility in the programme is reported by HIE and EST as being one of the key strengths of the programme. It ensures that funding decisions are not seen as being made by distant committees and builds strong linkages between SCHRI and community organisations. A 'bottom-up' and flexible approach to delivery was seen as being central to ensuring that a significant number of projects were developed and delivered in the first three years of the initiative.
3.23 However, as the initiative has developed, the weaknesses in this approach are more apparent. A high level of interest has now been generated in the programme, and the level of available funding is now not sufficient to fund all grant applicants. The bottom-up approach means that tensions have developed between the development officer's role as both an advocate for projects and as a representative of the funding body. As funds are distributed on an ongoing basis rather than competitively in 1-2 rounds, the main reason given for refusal of funding is that there is no more funding available rather than other projects being judged more able to meet funding criteria.
3.24 This is a particular issue for the lowland area managed by EST, which currently has very limited funding available for community projects. This is due to an increase need to allocate funds to the household stream, and a commitment on funding community projects from the last financial year 2004/2005.
3.25 Feedback from installers suggests that this has led to uncertainty in the market as projects have been delayed or cancelled, which were under the impression that they were going to receive SCHRI funding. From this perspective, there is a feeling that there needs to be clearer guidelines on the criteria for funding, and if these criteria are met funding should be made available.
3.26 An alternative approach would be to introduce a competitive funding round, possibly on a six monthly basis, whereby projects bid for a set amount of grant funding, with application deadlines available at regular intervals. This is the procedure for other related schemes such as the DTI Bioenergy Capital Grants Scheme which supports heat and electricity generating biomass projects. For this programme, value for money, as expressed by the total grant requested per kW installed, and the percentage of grant as a proportion of total costs are the two principal assessment criteria.
3.27 A competitive approach to appraising projects is not favoured by HIE or EST. It was felt that this would discourage community organisations getting involved in the initiative, many of whom are dependent on volunteers to bring forward applications.
3.28 However, there may be scope in exploring the merits of introducing a competitive funding stream directed primarily at local authorities and housing associations. These organisations are better resourced to submit bids for competitive funding, and it would provide them with greater clarity on whether or not funding was available during the financial year. Including value for money criteria in a competitive process would also encourage a high level of leverage and a low level of deadweight among project applications.
Monitoring and Reporting
3.29 As with the household stream, it is a condition of the grant that projects may be required to provide information on the system's output, estimated fuel savings and equipment failures. Generally, the development officers will maintain contact with projects after installation, although there is no formalised procedure for this, and further contact with projects can be rather ad-hoc.
3.30 Funded projects are required to complete a case study report, which summarises the outcome of the project. A pro-forma has been developed for this purpose, and completed case studies are made available on the SCHRI website. Details of the power output and estimates of carbon savings are included in only a small number of cases.
3.31 Indeed, it is clear from feedback from community projects that the level of awareness of actual quantifiable outputs arising from the installation of the system is currently very low, and in some cases beneficiaries report that anticipated energy savings are likely to be significantly lower than the level estimated by the installer. There is no specific funding allocated to undertake formalised evaluation of the performance of particular systems funded through the initiative, and so funding this type of activity would be dependent on funds being taken from the main funding allocation. EST had planned to undertake several evaluations of major projects this year, but given the need to fund the increase in demand arising from the household stream, this work has currently been postponed.
3.32 The need to collect robust evaluation data on the performance of systems will provide a much clearer idea of the cost and benefits of switching to a renewable energy system. This issue will be of particular importance when reviewing the performance of more significant, multi-technology projects, such as the PURE energy project in Unst and the Sir E Scott School and Sports Centre in Tarbert, where the issue of intermittency is balanced through a range of technology options. Although delivery will remain the primary focus of the initiative, there may be a case for allocating a separate fund for more formalised evaluation of the outputs arising from specific projects.
3.33 EST undertakes an assessment of the carbon savings arising from funded projects through its evaluation unit. The information collected through a range of site visits will improve the quality of the data currently held in relation to the estimated energy output and resultant CO2 displacement arising from various technologies. No such similar activity is being carried out in the Highlands and Islands, although some further work may be undertaken by a research student to provide further clarity on the level of carbon displacement resulting from supported projects.
3.34 The progress reporting to the Scottish Executive relating to both the community and household schemes consists mainly of reporting on inputs and activities. Thus, level of spend and leverage as well as number of projects completed and under development by type of technology are the main indicators of progress.
3.35 Consideration needs to be given to reporting on outputs and outcomes in relation to funded projects, as this provides a much clearer picture of how the initiative is developing in a wider context. In this sense specific indicators could relate to:
- installed power capacity ( MW) (output)
- renewable energy output (kWh) (outcome)
- cost of capital (£/kW) by technology (outcome)
- level of carbon displacement (outcome)
- employment in micro-power sector (outcome)
3.36 This would require a more formalised level of monitoring of employment supported through the initiative, which may involve a closer relationship with accredited installers to ensure that the data held on employment within the micropower sector is tracked over time. This will be very important in evaluating the success of the initiative at the final evaluation stage.
Integration
3.37 The SCHRI is branded as a one-stop shop for advice and guidance in relation to the development of small scale renewable projects. Generally grant beneficiaries felt this had provided them with a clear point of contact, although there was some confusion regarding the range of funding sources available for renewable energy projects.
3.38 A number of schemes are funded by various sources which directly and indirectly support the development of small scale renewable projects in Scotland. A detailed discussion of funding sources is provided in appendix II but the main sources include:
- DTI Major Photovoltaic Demonstration Programme
- DTI Bioenergy Capital Grants Scheme
- Scottish Power Green Energy Trust
- Department for the Environment, Food and Rural Affairs ( DEFRA) Community Energy Fund
- EST Community Energy Efficiency Grants
- Scottish Executive Central Heating Programme
- European Regional Development Fund
3.39 Two of the key issues regarding integration with other funding streams are how SCHRI interfaces with energy efficiency and fuel poverty. There is potential for a much greater level of alignment between SCHRI and these wider energy objectives.
3.40 A small fund is managed by EST to support energy efficiency measures in community groups, while other funds are available through the EST for households. However, the survey findings highlighted a relatively low take up of energy audits on the part of grant beneficiaries. Consideration needs to be given to make energy efficiency a much higher priority within the programme, potentially including energy savings measures as a compulsory part of the funding mechanism. Integrating energy efficiency measures is a key part of the new Low Carbon Buildings Programme for the rest of the UK.
3.41 The Central Heating Programme is designed to eliminate fuel poverty by 2016, by providing a package of energy efficiency measures and central heating systems. There is potential for the programme to be better aligned with wider sustainable development goals within the Executive, with a higher priority given to the development of renewable energy alternatives such as solar water heating, biomass and ground source heat pumps. The strong linkages between renewables and fuel poverty are already recognised in the combined DEFRA/ DTI Fuel Poverty Strategy, which includes a £5 million budget for renewables in remote locations that are off-grid or without mains gas connections.
Future Development
3.42 It is apparent that as the level of available SCHRI funding has remained at the same level each year, the level of demand for project funding has increased steadily. Funding up to one hundred per cent was available for community projects in the first two years; however, it is less likely that any project will now receive greater than fifty per cent funding in order to ensure that money is spread out more effectively. Ensuring that community groups are required to secure match funding for projects, does ensure that projects generate a certain level of buy-in, which may be less apparent if funding was allocated wholly through SCHRI.
3.43 The scheme is currently not being actively promoted among local authorities and housing associations as there is a concern that large scale projects of this type would "swallow up" most of the annual budget. The initiative has focused even more on developing projects within community groups. Feedback from beneficiaries demonstrates that the greatest "value added" in terms of the role of the advisors has been in relation to community groups, who are most likely to lack skills in developing renewable energy projects, and require a greater level of advice and guidance. Local authorities (and housing associations to a lesser extent) are more likely to view the funds as a useful source of funding renewable energy projects, rather than a source of advice and guidance.
3.44 The development of community scale renewables currently relies heavily on grant assistance, which does constrain the size and scope of potential projects. There is potential to further explore how loan and equity funding could be used to encourage the development of renewable energy projects. For example, local authorities are already able to access funds through the "Invest to Save" programme which promotes energy efficiency measures, which will produce future budgetary savings. This is focused primarily on energy efficiency measures rather than renewable energy projects, designed primarily to fund projects with a payback of five years or less. However, there is also scope to encourage investment in renewables via this route.
3.45 In addition, housing associations also have significant property assets against which they can borrow. It would be possible for these organisations to offset these borrowings against the future savings that will arise from investments in renewable energy systems. Communities Scotland, as the lead body in this sector, has the potential to play a much higher profile role in integrating renewable energy in social housing developments.
3.46 The further involvement of equity finance in the micro-generation sector needs to be explored further. As SCHRI has developed a one-stop shop approach to the development of community renewables, it is also appropriate that development officers are able to provide advice on a wide range of potential funding sources including grants, equity and loan finance. The Highlands and Islands Community Energy Company ( HICEC), which delivers SCHRI in the Highlands and Islands is currently in the process of developing a toolkit for the financing of community level micro-generation, and will also provide loan and equity finance to support projects.
3.47 Ensuring that the SCHRI Advisory Service is aware of and interfaces with private sector equity funds which are targeting community scale renewables, will be important in supporting the longer term development of the sector, promoting market adjustment, and reducing the future requirement for public sector support of the sector.
« Previous | Contents | Next »