The Scottish Government

« Previous | Contents | Next »

Listen

Developing Local Outcome Agreements for the Better Neighbourhood Services Fund

Chapter 4
Conclusions

Introduction

4.1 This section deals with:

• the advantages and disadvantages of the LOA approach
• best practice
• flexibility of the LOA approach
• lessons learnt
• future policy implications

ADVANTAGES

4.2 Stakeholders identified a range of advantages associated with the LOA approach:

• Local ownership - priorities are set by partners and the community to reflect local issues within a broad national framework.
• Focused - the LOA format makes partners think about the impact they hope to achieve rather than just what they are going to deliver; it also challenges them to consider what approach to delivery is the best to achieve the identified outcomes.
• Flexibility - the emphasis on outcomes rather than outputs allows partners flexibility in the programme as to what initiatives are supported; this is a positive feature, particularly from the standpoint of community involvement as the services and projects are not wholly pre-determined.
• Clarity - LOAs provide a clear statement of priorities and aims.
• Accountability - the community can see which partners sign up to an LOA and what they aim to provide and achieve. This allows the LOA to act as a reference document for the public (and other agencies).
• Partnership - LOAs encourage partner involvement and require trust among partners to achieve the identified outcomes.
• Evidence - the emphasis on outcomes means that LOAs have the potential to provide in-built monitoring and evaluation of effectiveness and thus to provide an evidence base to inform future policy development. If LOAs can be made to work well, they should be self evaluating.

DISADVANTAGES

4.3 At the same time, stakeholders identified a number of disadvantages or limitations of the LOA approach:

• Programme design challenges - designing a programme with an appropriate set of specific performance indicators, in consultation with local people, is now seen by Scottish Executive regeneration staff and local authorities alike as a challenging task.
• Consultations - for some Pathfinders the level of community consultation involved in the BNSF LOAs was excessive ⁵ while for others it was felt that not enough time had been allowed for genuine consultation.
• Time limited - despite the greater flexibility associated with the use of Revenue Support Grant (RSG), BNSF is still a time limited programme with all the difficulties associated with such initiatives, and in particular the difficulty of attracting and retaining staff for a temporary initiative.
• Project and Service Monitoring Issues - as the monitoring of individual services and projects appears to be less developed than other programmes, LOAs require trust between partners that they are working towards the joint outcomes and delivering what they said they would.
• Conflict - for a few Pathfinders the use of LOAs led to a deterioration of their relationship with the Executive, but this was by no means the rule. Other Councils felt that the Executive had been flexible and understanding of their needs.
• Bias in project selection - the emphasis on achieving outcomes may make partners focus on activity that is easily measurable without necessarily being the best option. Worthwhile activities associated with 'soft' outcomes may not be selected for funding.
• Partner commitment - to get partners to focus on outcomes was more time consuming and harder work than had been expected, and there may be a reluctance on the part of those concerned to engage again in such a complex process; alternatively the newness of LOAs may have contributed to the learning process for BNSF and future exercises might be more straightforward.

Best Practice

4.4 There were a number of examples of best practice LOAs, where Pathfinders have designed programmes with a series of specific actions and relevant outcome measures to address the needs of disadvantaged communities. These related for the most part to services to tackle disadvantage in particular deprived urban localities. The approach appeared initially to be more difficult to apply to rural areas, although the evidence to date suggests that an outcome oriented approach can work well in relation to rural services.

4.5 However, the name of the programme itself, and the guidelines and particularly the sample LOA, which were both written from a neighbourhood renewal rather than a rural deprivation perspective, do not convey the types of service which may be most appropriately developed in rural areas to address social exclusion issues.

4.6 'Best practice' LOAs were generally those with a clear geographical focus, a more tightly focused range of actions and clarity about the measures of success and the arrangements made to obtain data on these measures. In one of the best practice LOAs (East Ayrshire), the community is closely involved in assessing the programme's effectiveness. At least one of the Pathfinders appears to be demonstrating that the LOA approach, with its focus on meeting local needs and measurable improvements, can be made to work very effectively in the context of improving the quality of life in deprived neighbourhoods.

Flexibility of LOAs

4.7 A small number of Pathfinders had at the outset expressed concerns that LOAs might be too rigid in tying them to the content of the LOA for three years and not allowing variation to take account of later conditions. In our interviews however it transpired that the majority of Pathfinders felt that these concerns had so far proved to be unfounded with their LOAs seen as flexible enough to contend with changing circumstances. These Pathfinders felt that the LOA development process had been vindicated; they also felt the Executive's willingness to accept fine tuning had helped.

4.8 The Executive expects Pathfinders to retain the broad objectives and outcomes of the LOA across the lifetime of the programme. It has however been flexible in allowing Pathfinders to improve and/or clarify LOAs and associated outcome measures. Indeed, we (DTZ Pieda) have provided specific advice and support to Pathfinders to help them do this and identify relevant baseline data. The Executive has also accepted that some specific actions supported might change over time and thus that the outputs specified in the LOA might require adjustment.

4.9 A small number of Pathfinders still held the view that LOAs were too rigid. One Pathfinder felt that some services or projects that would have benefits for the neighbourhood but which did not contribute to the outcomes which had initially been chosen could not be supported by BNSF - this suggests that the setting of the outcomes is crucial. Despite the fact that the Executive has allowed Pathfinders to further define and clarify their LOAs, and specific outcome measures, there was also a suggestion from three or four Pathfinders that the Executive should allow greater flexibility for changes to the LOA. Three Pathfinders felt it was too early to tell whether the LOAs were too rigid.

Lessons Learnt

4.10 Design Issues: The main lesson that we would draw from experience so far is that the design of a programme of this type is demanding for all concerned - the Pathfinder itself, its community planning partners, communities and the Executive. Social inclusion covers a very wide range of topic areas and activities, and within the areas covered by the Social Justice Milestones, the range of potential outcomes which might be chosen is potentially very wide.

4.11 At a programme level it is hard to monitor and evaluate a programme which is as diverse as BNSF where 12 Pathfinders have over 300 separate outcomes. It is difficult to draw up guidelines which are specific enough to be relevant and helpful to all the different types of programme which may be supported. Given this, the task of summarising what has been achieved with the funding will be a challenging one. This is certainly true at programme level, and in some of the more complex individual Pathfinders, it may also be true at Pathfinder level.

4.12 Although the BNSF programme offered considerable freedoms to local authorities, they were also constrained by the need to secure agreement from community planning partners and to develop the Pathfinder in consultation with the relevant communities. While the arguments for both of these are strong, the involvement of other parties takes time, can complicate programme design, and make it harder to achieve a clear focus.

4.13 All concerned believe that the complexities of programme design were not fully appreciated at the outset, and that more training about how to approach the task of designing an outcome based programme would have been desirable.

4.14 Timeframe: The timescale for the Programme was felt by some Pathfinders to be unacceptably tight, and to have created difficulties in drawing up LOAs. The tight timescale also meant that the details of the LOA were being hastily worked out for presentation to the Executive, at times when consultation with interested parties was still ongoing. It was acknowledged by both sides that both Scottish Executive and Local Government are often caught between the desire to push ahead with implementation of policies and the timescale required to genuinely engage communities and partners.

4.15 Some Pathfinders felt that the timescale allowed for consultation with communities and partners did not permit the inclusive approach to LOAs desired by the Executive. In many cases, it was a case of going to other stakeholders with a plan rather than involving them at the earlier stages where they could have a direct impact on the design of the programme. It was felt by Pathfinders that ideally a lead in period of a year would be desirable although it was appreciated that this was probably not always practical. One authority commented that there is evidence from other programmes that these lessons about adequate lead in times are not being learned.

4.16 Guidance: The development of the Programme has been a learning experience for both the Executive and for Pathfinders. The initial Guidelines for Implementation did not give authorities a sufficiently clear indication of what the Executive expected, and while the sample local outcome agreement was clear and helpful, its timing was such that it did not impact fully on the design of LOAs.

4.17 It is our impression that Pathfinders had had much less exposure than Central Government to evaluation and performance measurement systems such as those advocated in the Treasury Green Book, the Framework for the Evaluation of Regeneration Projects and Programmes (often known as EGRUP) and the Logical Framework used by certain government departments. Pathfinders would have benefited from some specific guidance on what is meant by outcomes and outputs and on the choice of indicators.

4.18 Some of the more recently issued guidance from UK Government such as the revised Green Book ⁶, the new guidance on Assessing the Impact of Spatial Interventions ⁷ which replaces the EGRUP Framework, and 'Choosing the Right Fabric: A Framework for Performance Information' ⁸ provides much greater clarity on these issues, and would in our view have been particularly helpful had they been available at the time.

4.19 Rural Areas: The main focus of the programme has been on improving the quality of life of disadvantaged communities by improving neighbourhood services. This approach is not easily transferred to rural areas where deprivation is generally more dispersed. An outcome based approach to improving services is equally appropriate for rural areas, but the guidance would need to reflect the rather different circumstances in a rural area - the Guidelines which were issued in this case were written with a focus on improving services in deprived neighbourhoods. A separate sample LOA would have been desirable.

4.20 It could be argued that rather than try to 'fit' rural authorities into the guidelines for a programme designed to secure improvements to defined deprived localities in urban areas, it would be preferable to create a separate and parallel programme for rural authorities. This could focus specifically on addressing the specific disadvantages suffered by low income households in rural and isolated locations.

4.21 However although the Programme's title - the Better Neighbourhood Services Fund - was not particularly appropriate for populations in dispersed rural areas, the LOA approach itself has proved relevant in rural areas. Provided that guidance is designed to take into account the different needs of urban and highly rural areas, there is no reason why the LOA approach should not be adopted to tackle social exclusion and deprivation in rural areas also.

4.22 It is also important to recognise that smaller and specifically rural authorities may be disadvantaged by the demands placed on authorities to submit an LOA, because they seldom have staff who deal with external funding issues or special projects.

Future Policy Implications

4.23 Encouragingly for the Executive's Regeneration Division, all Pathfinders expressed support for the LOA approach to BNSF. Interestingly, the wider uptake in local authorities of the LOA concept - i.e. a more outcome oriented focus - was mentioned by more than half of the Pathfinder interviewees.

4.24 Both programme design and programme monitoring and evaluation for 'bespoke' LOAs are potentially time consuming and expensive. Many authorities have given considerable thought to the target group on which they wish to impact, which may not be a group for which ready made performance indicators or statistics exist. In several cases, LOAs require specially designed survey work or the development of a customised database to track impact.

4.25 This level of investment in design and monitoring is only appropriate for relatively large scale programmes. For authorities in receipt of more modest funding, selection of outcomes from a range of 'off the peg' outcomes may be more appropriate.

4.26 The range, scope and number of outcome measures developed through BNSF suggests that some reduction and standardisation in the number of indicators and target outcomes in use would be highly desirable. A smaller number of more standard outcome targets relating to issues such as child poverty, getting people back into work and creating better neighbourhoods would in our opinion be desirable.

4.27 We would suggest that some narrowing of the potential scope of programmes would be helpful, and that authorities would find it helpful to be offered a suggested 'menu' of outcomes which they might wish to address. This would not in any way constrain their ability to develop individual and customised services and projects to meet local needs, but might be helpful at programme design stage. The Guidance on the integration of Community Planning Partnerships and Social Inclusion Partnerships ⁹ makes reference to this issue and is likely to be followed up by a menu of outcome indicators which Community Planning partners may wish to use in the development of Regeneration Outcome Agreements.

4.28 We note also the need to avoid proliferation of outcome agreements, where different funding streams have outcome agreements which overlap. Again, the Guidance on SIP integration enables Community Planning Partnerships to develop a single Regeneration Outcome Agreement for SIP Funds, BNSF and regeneration funds associated with the transfer of council housing to community ownership.

4.29 There are a number of sources of indicators - for example the Audit Commission's library of quality of life indicators. As part of their work on a possible programme level monitoring and evaluation framework, DTZ briefly reviewed the range of existing indicators which might be used, and we concluded that the key issues were:

• attribution - i.e. is this indicator the best possible measure of the impact of the actions the Community Planning Partnership is proposing, or are there more appropriate measures to capture this impact?
• whether routinely collected data sources ¹⁰ may be used to measure impact
• how widely the indicator is used by others - where survey data is used, there are considerable advantages in using 'standard' questions such as those used in the Scottish Household Survey or the Scottish Crime Survey, so that national and in the case of SHS, local authority level, comparators are available.
• the cost/difficulty of obtaining information about impact - is it proportionate to the scale of the actions proposed?

4.30 However, the selection of these outcomes will be critical and will require considerable thought and discussion. The selection and agreement of appropriate outcomes for Regeneration Outcome Agreements may be just as taxing as the selection of appropriate indicators for BNSF has proved to be.

4.31 A final comment relates to the profile of less individualised outcome agreements. Failure to attain outcome targets common to a number of authorities is likely to be much more public and visible than a failure to reach a variety of self selected and relatively specific one-off targets. In policy terms, however this option still allows more freedom to CPPs to select targets which are relevant to their programmes rather than being required to measure progress against a centrally determined set of indicators which would be identical for all pathfinders.

Overall Conclusions

4.32 Authorities have found the evidence based approach implicit in Local Outcome Agreements to be an attractive, coherent and logical framework within which to develop new policies to combat disadvantage and subsequently to assess their effectiveness. While they have at times struggled with the demands of programme design and the selection of appropriate indicators, they have endorsed the overall value of the approach. Indeed, several authorities said that they have now adopted a similar approach in other areas of their work.

4.33 Comments included:

'LOAs are the way forward';
'Good concept, worth replicating';
'We are supporters of LOAs';
'Generally in favour';
'Focus on programme objectives is a positive feature';
'Must evidence change to justify the activity and the expenditure of tax payers' money'; and
'Despite the pain, we value the LOA!'

4.34 However Community Planning Partnerships would benefit from more specific guidance about the design of outcome agreements and in particular a more standardised and detailed approach to baseline preparation and subsequent monitoring of outcomes. Giving more specific guidance need not be incompatible with allowing scope for more innovative, one-off outcome agreements, where Community Planning Partnerships feel that this is desirable.

4.35 A menu of appropriate indicators, together with information on appropriate data sources and on data collection would do much to assist Community Planning Partnerships to make effective use of this type of agreement. The lessons learnt from BNSF are that the selection of indicators is critical, and that considerable thought must be given to issues such as attribution/plausibility and the cost, design and timing of any survey work needed to assess the overall effectiveness of the programme.

« Previous | Contents | Next »