The Scottish Government

« Previous | Contents | Next »

Listen

Early Approaches to Monitoring and Evaluation of the Better Neighbourhood Services Fund

CHAPTER TWO: MONITORING AND EVALUATION IN BNSF - KEY ISSUES

2.1 This section of the report addresses the key monitoring and evaluation issues. A number of examples of best practice are also presented in text boxes throughout the section as supporting evidence of the issues discussed in the main text.

SCOTTISH EXECUTIVE GUIDANCE

2.2 Within the BNSF programme, LOAs must set out BNSF's expected impact on the target area/groups over the term of the programme, an M&E requirement which necessitates establishing a starting position, tracking changes over time and reporting on the end position for a number of locally selected indicators.

2.3 The initial Scottish Executive guidance published in April 2001 emphasised the need to establish arrangements to monitor and evaluate progress but in our opinion was not clear enough about the need for outcome-specific baseline information or the need to track changes in outcomes over time. In retrospect, Executive officials accept such a finding. However the subsequent sample LOA - published in October 2001 - included a section on monitoring and evaluation, with examples of methods (e.g. six monthly focus groups, annual household surveys and local crime surveys) which might be used to monitor impact. This sample LOA also provided examples of outcome-specific baseline data and made it clear that a baseline position would be needed for each target outcome.

2.4 Despite this, we concluded in our review of LOAs and subsequent monitoring and evaluation arrangements that the Executive's requirement for outcome specific baseline data was not fully reflected in the LOAs, with some LOAs providing baseline data of a very broad contextual type. This may have been because the need to establish a baseline position for each outcome was only specifically stressed by the Executive in the later-issued sample LOA.

VIEWS ON THE PURPOSE OF MONITORING AND EVALUATION

2.5 The views expressed in this subsection were gathered during the telephone interviews conducted for the purposes of this topic report with the sample of five Pathfinders. However, additional views expressed by Pathfinders during previous discussions with DTZ Pieda have also been included where relevant.

2.6 Monitoring was seen as meeting the following needs of the Pathfinders and the Scottish Executive:

• internally, to assess progress towards outcomes and review programme content to identify problems
• reporting progress to the Scottish Executive
• assessing the financial situation internally and for the Executive

Internal purposes

2.7 In the consultations undertaken specifically for this report, Pathfinders identified the main purpose of monitoring and evaluation as the means of assessing progress towards the achievement of their outcomes. The Pathfinder view was that the information was required to gauge how effective the interventions had been and whether they are having an impact on the target groups, as planned.

2.8 Furthermore, Pathfinders reported that monitoring information contributes to the early identification of problems so that remedial action can be taken where necessary to ensure that outcomes are achieved to the greatest extent possible. Monitoring is therefore seen as having an internal management purpose for Pathfinders. It was also suggested by one Pathfinder that it is monitoring activity which will demonstrate whether the focus and impact on outcomes - which is a key objective of BNSF - has been achieved.

Reporting progress to the Scottish Executive

2.9 Pathfinders also acknowledged that monitoring fulfilled the important role of updating progress to the Scottish Executive. However, all the Pathfinders consulted for the purposes of this report were of the opinion that the level of monitoring undertaken for BNSF has been more comprehensive than would have been required to meet their own internal needs. Pathfinders therefore felt that the level of monitoring was determined by the Scottish Executive's information requirements. Their views stemmed from the need to demonstrate to the Executive via the Annual Reports that satisfactory progress was being made and at the end of the programme to demonstrate that outcomes have been met. The amount of detail requested by the Executive was felt to be more in-depth than Pathfinders would have chosen to undertake themselves. For the Executive, the imperatives are to monitor expenditure and the outputs and outcomes being achieved with the funding, and to provide evidence of which approaches work well and which less well.

2.10 In approximately half of all Pathfinders the level of monitoring had created workload problems for staff responsible for gathering the information and data. This was particularly the case in the first six months of the programme when staff were being appointed, monitoring systems were being established and the requirements were being fully explained to partners and BNSF funded projects.

2.11 The Scottish Executive's requirements were seen by Pathfinders as relatively demanding. However, Pathfinders acknowledged that the Executive's monitoring requirements were important in allowing Ministers to identify what the programme had achieved from the 90 million investment. Accountability to the Executive was therefore a major driver of BNSF monitoring but there was no evidence from our consultations that accountability to partners or to the communities was seen as a major driver.

FINANCIAL INFORMATION

2.12 Pathfinders identified the collation of financial information as another key purpose of monitoring. This involves Pathfinders recording expenditure, keeping costs under control and ensuring that value for money is delivered. The financial information is utilised by both the Pathfinders and the Scottish Executive.

RESOURCES

2.13 The importance placed on monitoring and evaluation by the Scottish Executive was illustrated by the release of significant additional money by the Executive in early February 2002 specifically for this purpose. A total of 1.5m was made available by the Executive pro-rated across the twelve Pathfinders according to their initial allocation.

2.14 The majority of Pathfinders suggested that the resources they had allocated to monitoring and evaluation had been influenced by the type and volume of information required by the Scottish Executive. Expenditure on monitoring and evaluation included the appointment of monitoring staff, expenditure on community consultation exercises such as surveys and the use of consultants to gather baseline data and devise monitoring and evaluation frameworks. Less demanding requirements from the Executive might have resulted in a lower level of resources devoted to monitoring as fewer staff would have been required, community consultation might have been less frequent and the use of consultants not seen as necessary.

2.15 Approximately half the Pathfinders had used external resources. They have been used in three ways. A number of Pathfinders such as North Lanarkshire and East Ayrshire commissioned survey research companies to carry out survey work to gather community views and attitudinal data. Others such as West Dunbartonshire and Western Isles commissioned consultants to gather baseline service and socio-economic information and data. Glasgow and Dundee appointed consultants to assist with monitoring and evaluation arrangements.

2.16 Consultants were used in two types of situations. In some smaller Pathfinders such as Western Isles consultants were used when existing staff resources were limited and consultants were seen as the most practical and cost-effective way of meeting the monitoring requirements. In other Pathfinders, consultants were called upon to bring specialist knowledge of monitoring frameworks (e.g. North Ayrshire), surveys (e.g. East Ayrshire) or evaluation (e.g. Dundee).

2.17 However not all Pathfinders saw the use of consultants as appropriate - one Pathfinder had previously expressed the view that the use of consultants was not generally a part of the local authority's usual way of working, while another rural Pathfinder felt they would struggle to attract consultants to work in their area because of its isolation.

2.18 Fairly substantial resources have been devoted to monitoring and evaluation. For example, West Dunbartonshire has, in addition to a full-time monitoring and evaluation officer, a full-time co-ordinator, finance officer and admin officer, all of whom have some involvement in monitoring and evaluation; consultants were also commissioned to prepare their baseline study. Without detailed financial information for the Pathfinders it is not possible to say what level of resources is allocated to monitoring.

2.19 From our consultations for this report and previous discussions with Pathfinders the minimum monitoring and evaluation arrangements are one full-time person or three or four people part-time, where these people also have responsibilities for other programmes. There is no distinction between the quality of monitoring activity between these full-time or part-time arrangements.

2.20 Meeting the monitoring requirements clearly has a financial cost which is met from BNSF resources. One authority has used consultants for all monitoring and evaluation purposes, which they suggest has involved a cost of 10,000-11,000 a year, and they make the point that this is money which might otherwise have been spent on projects. However, it may be argued that this would appear to be a very cost-effective approach, relative to the cost of employing a full-time monitoring officer.

DEFINITIONS

2.21 The emphasis placed on the achievement of outcomes in the BNSF programme represents a change from previous programmes. The term 'outcome' was not in common usage when the programme was introduced, and it was not well understood. (Previous programme monitoring had tended to focus on high level monitoring indicators or on the reporting of project activity or project outputs). However, as the programme and the LOAs in particular also required outputs to be recorded, the distinction between the two terms has been a critical issue throughout the programme. Add to the equation other common terms in use such as 'inputs', 'activities', 'indicators', 'milestones' and 'performance measures' and it is clear why terminology has been a recurring issue for BNSF.

2.22 From our previous consultations with Pathfinders we are aware that clearer guidance from the Executive at the outset on these terms would have been welcome. We understand that some Pathfinders such as West Dunbartonshire and North Ayrshire issued their own definitions to stakeholders to provide clarity - but we have not seen details of the definitions issued. Outcome and output definitions were also provided by DTZ Pieda in October 2002 to all Pathfinders drawing on standard definitions taken from Treasury guidance and other official sources such as Guidance for SIPs on Evaluation. It is not known how widely these definitions were used by Pathfinders.

2.23 Consultations undertaken specifically for this report suggest that Pathfinders have progressed to a situation where those most closely involved in BNSF have a significantly better understanding than they had at the outset. However, neither Executive officials nor any of the Pathfinders we spoke to felt that there was a common understanding of all the terms. Although there was an appreciation of what inputs and activities were, confusion persisted surrounding the distinction between outputs and outcomes. There is a distinction that can be drawn in the level of understanding between those who are directly involved in the programme on a day-to-day basis such as co-ordinators and monitoring officers, who tend to have a good understanding, and those who are delivering services which are partly or wholly funded by BNSF.

2.24 In one Pathfinder the different level of understanding has created problems in agreeing outputs and outcomes that service deliverers are prepared to commit to, e.g. at a local level e.g. drugs/health workers were not really prepared to define or commit to outcomes. There may also be different levels of understanding among different types of partners - e.g. LA, LEC, health board, community.

2.25 Pathfinders stated that the lack of a universal understanding was not surprising as the issue can be complex - one Pathfinder said that there are 'grey areas' and we would tend to agree that usage is often confusing. Those on the periphery - for example supported projects - should not be expected to have a perfect grasp of these issues, given that even those who are more fully involved may use terms inconsistently. The distinction between outputs and outcomes can be difficult at times. Furthermore, Pathfinders suggested that some stakeholders should not be too exposed to these issues, which may seem quite semantic/technical, especially not if it proved detrimental to their interest in the programme.

2.26 It is important however that all involved understand that the programme is seeking to achieve broader impacts, even if, as with some health outcomes, these cannot be measured in the short term, and shorter term or intermediate output measures may also be needed. We felt that understanding of what is meant by the term 'outcome' has increased among all those involved, particularly those most closely involved. The consultations established that stakeholders are clear that the emphasis is on outcomes, which represent the ultimate impact the programme may have on an area.

2.27 Following support from DTZ and in consultation with the Executive, Pathfinders have been given the flexibility to refine and amend outputs and outcomes as their understanding has been enhanced and as the links between outputs and outcomes have become more apparent.

CAPTURING SOFT OUTCOMES

2.28 Two aspects of capturing soft outcomes and process issues are significant. Firstly, Pathfinders and the Scottish Executive agree on the need to capture soft outcomes and process issues which are viewed as crucial elements of the BNSF programme. The nature of the BNSF programme means that capturing soft outcomes and process issues is essential for a number of reasons. Firstly, some of the projects focus on in depth employability work with individuals and groups whose situation may be improved by the intervention, but perhaps not to the point where a hard outcome such as getting a job will be achieved. Secondly, a key objective for the programme is securing the involvement of community planning partners and the communities in the development of the LOA and the monitoring of outcomes in terms of real and substantial improvements to local services.

2.29 All the Pathfinders consulted for the purposes of this report stated that capturing information for soft outcomes is not straightforward; this view had previously been expressed to DTZ Pieda by the majority of Pathfinders. Capturing improvements in confidence - whether in project beneficiaries or stakeholders such as community representatives - is more difficult than capturing an improvement in a quantitative indicator such as unemployment. A small number of Pathfinders - particularly those concentrating work on young people - have successfully used accepted techniques such as the Richter scale, which attempt to measure the 'distance travelled' toward an objective such as re-entry into the labour market. Three or four other Pathfinders have used focus groups to gather softer information while we understand the majority of Pathfinders intend to gather such information in the future.

2.30 The Pathfinders consulted for this report also stated that capturing improvement in partnership working is difficult. Information has been gathered that shows that partnership working has taken place but deeper analysis of its effectiveness is more difficult. In our view, the focus should be less on the mere fact of partnership working and more on what it has achieved - but this is quite demanding.

2.31 Pathfinders have employed a range of methods to capture such information including interviews with stakeholders, small surveys of project staff, monitoring pro-formas, focus groups, workshops, case studies and narratives. In one case, video evidence of work with young people has been recorded. Pathfinders have also impressed upon stakeholders and projects the need to report to them any soft outcomes or process issues. Although all Pathfinders have reported difficulties in monitoring soft outcomes there was a consensus that the techniques employed had been successful in gathering information.

2.32 An additional issue raised by two Pathfinders was the difficulty of reporting some of these issues to the Executive within the first year's Annual Report format, which they felt allowed little space for in-depth reporting of such information because of page limits. However, many of the Pathfinders exceeded the page limit and the second Annual Report format was changed to take account of the perceived limitations, allowing additional space for unintended impacts and case studies. All Pathfinders provided this information in their second Annual Reports.

2.33 A further problem identified by one Pathfinder is when process issues related to internal staff or communications problems. The Pathfinder stated that they did not feel comfortable airing such problems publicly either in the Annual Report or in any other forum. Instead the Pathfinder felt that internal communication problems were better addressed internally.

2.34 One Pathfinder felt - based on their experience to date - that the Executive was more interested in tangible hard outcomes and less interested in soft outcomes/qualitative improvements. For example, the Pathfinder stated that an improvement in the confidence of their beneficiaries was a significant achievement which they felt was undervalued by the Executive. However, the widespread monitoring of soft outcomes undertaken across all the Pathfinders shows that Pathfinders were at least trying to report soft outcomes to the Executive. A different authority felt that the Executive's visits were a good opportunity to demonstrate soft outcomes.

ATTRIBUTION

2.35 The Executive's guidance stated that BNSF resources need not be used in isolation 'indeed partners may find it useful to make faster improvements to services already identified by the use of mainstream resources made available to target deprivation… in those cases it will be necessary to demonstrate precisely how the improvement results from the use of BNSF'. However, no specific guidance was given on how this should be achieved and the sample LOA made no mention of attribution. As a result the LOAs were developed without explicit plans being in place to specifically identify the benefits attributable to BNSF. As Pathfinders began to implement their LOA, some began to consider that the indicators which had been selected for monitoring are also subject to other influences and sought ways to take account of this. The issue was also raised by DTZ Pieda at the October 2002 networking event.

2.36 'Deadweight' or the counterfactual - what would have happened in any event - has to be considered, as does the impact of other programmes which may impact on the indicators selected. Generally, BNSF has to be reviewed in the wider context of all the activities and interventions that are taking place in the relevant area (or for the particular target group), and this is complex.

2.37 Pathfinders - and more particularly evaluators - need to develop an awareness of both issues. Some Pathfinders such as West Dunbartonshire have shown a particular awareness of this issue from an early stage. If both issues are to be considered, monitoring officers and evaluators will need to have an awareness of high level trends, and of the other programmes which are being operated in their area and which are likely to impact on the outcomes. Advice from DTZ Pieda and the Executive has emphasised the need to highlight other positive and negative influencing factors/programmes but has not suggested that Pathfinders undertake detailed work to ascertain specific attribution. In particular, the Executive's guidance for the second Annual Reports was intended to aid future interpretation of attribution which will be addressed during the final evaluations. Comprehensive information on other spending was presented in Pathfinders second Annual Reports which will greatly aid the evaluations.

2.38 Two problems with attribution have been identified by Pathfinders. The first has been recording of other funding sources expenditure. Where the other relevant programmes are being administered by a similar group of staff e.g. SIP programmes, difficulties are less than where there are a whole range of other interventions being developed by other departments and agencies, such as the Executive's funding for health improvement. The second problem has been Pathfinders' uncertainty of how to attribute outcomes to BNSF. The solution - to apportion outcomes on a pro-rata basis - is something that Pathfinders will not need to address until the final evaluation, now that the information is being gathered by the Annual Reports.

2.39 North Lanarkshire Council are attempting to develop a performance framework which shows impact from all funding sources as part of the Corporate Plan, the Community Plan and the Statement of Readiness. North Lanarkshire would like to use this framework in which the impacts of all the various regeneration and related (e.g. health) funds applying to a particular regeneration area would be monitored and recorded in a single exercise, rather than carrying out separate exercises for all the different funding bodies.

SUPPORT

2.40 The support received by Pathfinders was needed. Firstly, almost all Pathfinders felt that there was a need for much clearer guidance from the outset about what the Executive expected in the LOAs and therefore through monitoring and evaluation. Until the model LOA was issued, there was very little clarity about what was expected in terms of the provision of a baseline or of the types of indicators which might be selected. This is a key message for the Executive. Pathfinders felt that there had been a need for much greater clarity and training about the nature of the programme and of LOAs at the outset.

2.41 Secondly, a number of authorities indicated that they had found it helpful to use consultants to prepare both a monitoring framework and to pull together their baseline (see paragraphs 2.14-2.18).

Consultants

2.42 Consultants were either used where existing internal resources were already stretched or where external expertise was deemed necessary. Pathfinders were generally satisfied with the support provided by consultants as their use allowed the Pathfinder to meet deadlines or to gather information - survey output, baseline data and monitoring frameworks - that would otherwise not have been possible in that Pathfinder. Pathfinders that relied on their own internal resources were equally as successful in meeting deadlines or providing information. No information is available as to the best model as it was evident that both models were equally as successful and it was mainly a matter of selecting the model most appropriate to each authority.

Scottish Executive

2.43 The Scottish Executive Regeneration and Local Community Planning team manage and administer the BNSF programme. The team consists of two full-time officers, although their time is not exclusively devoted to BNSF. In addition support is provided by the Executive's researchers and economists.

2.44 In addition consultancy support from DTZ Pieda/Tribal HCH was made available to Pathfinders. This support consisted of a bank of 12 days consultancy per year that Pathfinders could access free of charge to deal with monitoring and evaluation matters. This element of the support has been accessed by seven Pathfinders for a variety of uses including revisions to outcome definitions, advice on monitoring soft outcomes and advice on completing the second Annual Report. The consultants also met each Pathfinder in December 2002 to review their monitoring and evaluation arrangements and their findings were reported to the Executive and the Pathfinders. Recommendations were made and Pathfinders have been asked in the second Annual Report to outline the changes they have made following the recommendations.

2.45 Generally Pathfinders stated that the support from the Scottish Executive had been satisfactory, given that it has been such a steep learning curve for all concerned.

2.46 The majority of Pathfinders had found it difficult to select SMART indicators and would have welcomed help in selecting appropriate indicators. Pathfinders would have welcomed more explanation of the type of information, with examples, which would have been acceptable to the Executive. A minority would also have welcomed consultancy support in choosing appropriate indicators. The provision of a suite of possible indicators would have been useful in our view and would have been welcomed by Pathfinders if it had been available at the outset. An attempt to develop a suite of indicators retrospectively in November 2002 ended following discussions with the Pathfinders, who did not want to see it introduced after they had selected their outcomes.

2.47 One Pathfinder consulted for the purposes of this report took the view that the Executive should have provided more training as well as improved guidance to local authorities on the design of LOAs. This would have helped with M&E requirements.

2.48 Pathfinders have taken a negative view of any changes in the information sought by the Executive and since October 2002 the majority have stated to us that they would like the Executive to be consistent. For example, changes to the information required for the second Annual Report was seen to be more onerous than the first year, in particular the match funding and leverage information which Pathfinders had not previously been asked for. It should however be noted that any changes to reporting requirements are discussed with Pathfinders in advance.

2.49 One Pathfinder felt that the BNSF networking events/seminars should be more participatory, with different authorities speaking about their experiences and less emphasis on 'top down' presentations. Another Pathfinder would have welcomed a greater number of networking events and indicated that the Pathfinder-led Rural Group had played a key role for them.

EVALUATION

2.50 The nature of the programme and of Local Outcome Agreements in particular is such that, had the Pathfinders set up systems to record key outcomes periodically in the way envisaged in the sample LOA, an important part of the evaluator's task - assessing impact - would be built in to the Pathfinder programme. At the end of the programme, its key achievements would then be apparent, without the need for a final evaluation to establish them. The focus of the final evaluation would then be more on 'what works', lessons learnt, and the outstanding needs of the area or the target group. There would also be a need to consider attribution, additionality and displacement in more detail.

2.51 However, not all of the Pathfinders are set up in the way envisaged in the sample LOA, (see Topic Report on LOAs) for reasons which have been well documented. In consequence, several of the Pathfinders see the final evaluation as playing an important role in ensuring that the information is collected which will allow impact to be assessed. A number of Pathfinders have suggested that they see a final evaluation as the best way of summing up the achievements of the programme.

2.52 For Glasgow, there are some special issues. The sheer size of the programme (27m) and its diversity makes it difficult to sum up achievements succinctly in the annual report format, and we understand that the final evaluation will be used as an opportunity to commission survey work which will allow the outcomes of the programme to be assessed.

2.53 A potential difficulty with the evaluations is the risk that key staff may move on before the evaluation starts. This is a common problem with time limited programmes but it can be lessened if individuals involved in BNSF are consulted after leaving their BNSF posts.

2.54 Although the Executive expects all evaluations to be complete by 31 March 2005, there are issues around the timing of individual evaluations as the Pathfinders are all at different stages of implementation. An overview report will collate the findings of the individual evaluations providing the overall achievements and lessons of the BNSF programme.

2.55 The consultant's initial brief included the provision of a programme level monitoring and evaluation framework. It was subsequently decided that this was not feasible for a number of reasons. A programme of this type which allowed participating authorities considerable freedom in the design of their Pathfinder programmes presents some difficulties for the design of a programme-level evaluation framework. Pathfinders have focused on different themes and on different target groups, and on a variety of different geographies. The diversity of the approaches supported and the fact that the programme was already well underway made it difficult to devise a programme-level framework which would have been meaningful and which would not have added to the Pathfinders monitoring and evaluation obligations. We concluded in our earlier monitoring and evaluation overview report that the final evaluation of the programme is likely to focus to a considerable extent on the effectiveness of the individual Pathfinders, judged against measures which are appropriate to their individual objectives.

« Previous | Contents | Next »