The Scottish Government

« Previous | Contents | Next »

Listen

Early Approaches to Monitoring and Evaluation of the Better Neighbourhood Services Fund

CHAPTER THREE: CONCLUSIONS

3.1 At a broad level, Pathfinders saw monitoring and evaluation as a necessary and valuable activity - it is appreciated that it is important to understand and to record how the funds have been spent, what has been achieved with them and what the wider impact of the programme has been. It is not something which is done simply to meet Scottish Executive requirements.

3.2 Pathfinder co-ordinators/managers showed a genuine commitment to monitoring and evaluation as a way of monitoring the effectiveness of their activities - and in some cases to using continuous monitoring and evaluation formatively, to allow programmes to be adjusted and developed where necessary. The extent to which Councils had systems set up which would allow them to do this varied however. There are a number of Pathfinders whose monitoring and evaluation activity is likely to focus largely on a final evaluation. It is perhaps understandable that this approach has been taken, given the relatively short life of the programme, and the fact that certain Pathfinders started out with their baseline and monitoring arrangements incomplete. However, evidence that some Pathfinders have adjusted delivery as a result of monitoring information is particularly encouraging for the Executive, as BNSF was designed to be a programme that incorporated formative evaluation.

3.3 Pathfinders had some reservations however about the particular monitoring requirements which had been imposed on them in this programme, and left to their own devices, they would have done things differently - the main aspect that would be done differently would be to gather less information. The problems as they saw it were:

• the lack of clarity at the outset about what was to be in the baseline and what was to be monitored subsequently, which stemmed from the very broad nature of the April 2001 Guidelines - clearer guidance from the outset would have avoided some of these problems;
• the fact that the Scottish Executive's requirements were perhaps more rigorous than they themselves would have opted for;
• and that the Scottish Executive's requirements have evolved over time.

3.4 Earlier and clearer guidance from the Executive that required Pathfinders to be more explicit about their monitoring and evaluation requirements would have been useful. The sample LOA was very helpful, but for many came too late. The initial guidance would have been enhanced if it had contained the information later published in the sample LOA - details of the LOA content with examples of outcomes. In particular the requirement for outcome specific baseline information should have been clearer.

3.5 In addition, there was some evidence of a lack of clarity about what was meant by an 'outcome' and Pathfinders would have benefited from clearer guidance on this issue at an early stage. However, for many, monitoring terminology and usage is an arcane and rather boring topic, and how this topic is presented is important to retaining interest. While it is important that understanding is achieved about outcomes, it is also important not to introduce too much complexity. Future guidance should try to achieve the difficult middle ground of making it clear what may or not be appropriate as an outcome, without getting bogged down in the finer points of monitoring and evaluation terminology.

3.6 Soft outcomes are recognised by all involved - the Pathfinders, Scottish Executive and ourselves - to be another difficult area. Pathfinders have been attempting to capture soft outcomes by a variety of means particularly surveys, focus groups and interviews. Best practice has been achieved where surveys have been successfully used to gather changes in attitudes and perceptions of issues such as quality of life and neighbourhood safety - paradoxically soft outcomes have been gathered most effectively when the data is tangible. Where greater difficulty has been encountered has been capturing the outcome of community involvement and the involvement of community planning partnerships. Pathfinders should seek to show where the community or community planning partners add value to the process as this is more informative than merely monitoring the level of their involvement.

3.7 The issue of attribution is increasingly seen as important. It has been suggested by one Pathfinder that composite reporting of the impacts of all the regeneration initiatives operating in an area would be less labour intensive, and would also help evaluators to have a better overview of the totality of regeneration activity in an area or in relation to a particular client group. Communities Scotland's guidance on the integration of the SIPs into community planning partnerships moves in this direction. One potential drawback of the joint system would be disaggregating whether one programme worked but another did not.

3.8 Some authorities found it useful to use consultants to set up a monitoring and evaluation framework, a baseline and generally to advise on specifying surveys, etc. The evidence suggested that this worked well, but also that it was not the only way of achieving good practice in this area as those Pathfinders who addressed these issues without the use of consultants were at least as successful as those who relied on external assistance.

3.9 Despite the difficulties, a number of Pathfinders have managed to achieve best practice monitoring and evaluation arrangements, which we have highlighted in this report. The volume and general quality of the monitoring information being produced is testament to the arrangements which have already influenced delivery in some areas and will in all cases be valuable for the forthcoming evaluations.

« Previous | Contents | Next »