The Scottish Government

« Previous | Contents | Next »

Listen

CHAPTER EIGHT SELF MONITORING

Introduction

8.1 As has been indicated in Chapter 2 (Paragraph 2.15), the monitoring frameworks had the potential to be used as outcome agreements in that all schemes were to select indicators that related to their aims. These were then to have baselines and targets set for them and were to be monitored at regular intervals. Had this been done, then impact assessment would have been relatively easy. However, few schemes did this in a systematic way.

8.2 The purpose of this Chapter is to outline some of the issues that have emerged with monitoring and to look at the lessons that might usefully be taken account of by the Executive when setting up other initiatives. Tables 8.1 and 8.2 give an overview of the key monitoring indicators for the 30 schemes that submitted their 24 month reports in time to be included in this analysis.

Monitoring Indicators

8.3 The guidance issued by the Executive stated that at least 4 indicators were to be selected to assess progress, 2 statistical and 2 survey based (Scottish Executive, 2003b). The indicators could then be measured at regular intervals and progress towards attaining the schemes' objectives assessed.

8.4 The number of indicators selected varied considerably as Table 8.1 shows. For example:-

• Some schemes, such as East Ayrshire, had selected 2 survey and 2 statistical indicators as specified in the guidance;
• Some schemes had not followed the guidance, and had only selected either survey or statistical indicators. This was the case in the Aberdeenshire scheme, which selected 3 statistical indicators only, whilst the Highland scheme only selected survey based indicators; and
• At the other extreme some had selected large numbers. Thus Renfrewshire had selected 25 indicators, 23 of which were statistical.

8.5 It seems unlikely that monitoring progress by collecting large numbers of indicators is necessary. It is unlikely that all will change independently of one another, so that changes in one may be paralleled by changes in others. Given this, it seems likely that there will be a substantial degree of redundancy. Not only is this a waste of resources, given the time and effort that has to go into collecting this information, but it also may give the wrong impression as to the wardens' ability to micro-manage change.

8.6 These data collection problems were compounded in that, in some areas where the wardens operate in a number of discrete localities, data was collected separately for each one. For example, Aberdeen, Dundee and South Lanarkshire all collected data for distinct sub-areas.

TABLE 8.1 Summary of Monitoring and Evaluation Indicators (1)

TABLE 8.2 Summary of Monitoring and Evaluation Indicators (2)

The Appropriateness of the Selected Indicators to the Scheme's Aims

8.7 There are also issues about the appropriateness of the indicators selected, given the schemes' aims. Our assessment, based on the 24 month monitoring submissions, is that of the 30 schemes for which we had data, 12 (40%) had at least one indicator that we judged not to be appropriate given the schemes' stated aims (Table 8.1). This was an issue identified in earlier work undertaken as part of the evaluation (Hayton and Percy, 2005a and 2005b).

8.8 The diversity of schemes, and the freedom they were given in selecting indicators, also means that there is no single indicator that is common across all schemes. For example, of the 9 case studies, the single indicator that was selected by most schemes was vandalism. Yet even this had only been selected by 5 of the 9. This means that it is impossible to find a single measure that can be used to assess overall progress. It could be argued that this is as it should be, given the diversity of the schemes. Yet, at a strategic level, all schemes are trying to overcome antisocial behaviour problems, which, by their nature, are interrelated and reinforce one another. As such it could be argued that a single indicator should have been imposed by the Executive that all schemes had to monitor.

The Appropriateness of the Selected Indicators as the Schemes Develop

8.9 Whilst incorporating indicators at the start of the process seems commendable it needs to be remembered that most of the schemes were new (Paragraph 1.19 shows that only 8 schemes were based on earlier initiatives). As such there was limited experience as to how the wardens would operate and even less as to what impact they could have. As such there was always a possibility that the selected indicators would be found to be of limited use for measuring the impact of the wardens' activities.

8.10 When the relevance of the indicators was discussed with the scheme managers, as part of the final round of case study interviews, a number now felt that the indicators originally selected were no longer relevant. For example:-

• In Aberdeen the indicators related to rental income and void properties were felt to be irrelevant as experience had shown that the wardens were only one of many factors that impacted upon estate management;
• In Dumfries, although there were no plans to change the indicators, there was a degree of unhappiness with them, in part as they had been selected by someone else (due to staff changes);
• In Dundee the indicators originally selected in relation to car crime ²³ were not felt to be relevant as most of these crimes took place outwith the hours that the wardens were patrolling. Accordingly discussions were ongoing about replacing these with indicators related to the illegal use of motor cycles and removal of abandoned vehicles within 21 days. In addition, the scheme manager was reviewing the use of fire statistics as a means of monitoring the warden service. However, no changes were to be made until the scheme's future funding was secured;
• In East Renfrewshire the warden scheme was originally based in the housing department and some indicators related to such things as the numbers of void properties and rental income. The move of the wardens to the Community Safety Team, along with the demolition of large numbers of properties, meant that the manager now felt that these indicators were no longer relevant. However, they were still being used as it was not felt to be worthwhile making changes half way through the scheme's life; and
• Orkney felt that selecting indicators relevant to the unique characteristics of the island was difficult. It was also felt that occurrences of most crimes and antisocial behaviours were so few that any changes could appear to be very dramatic in relative terms. One implication might be that greater thought needs to be given to selecting indicators for initiatives in remote areas.

8.11 There was also a view, amongst some managers, that some indicators had been selected as they were measuring issues of considerable importance to the community, even though the wardens might have limited impact on the underlying causes. The best example of this was needle collections in Dumfries. The wardens might not be able to have much impact upon drug misuse but they could deal with one of the symptoms that caused considerable community unhappiness.

8.12 What, therefore, emerged was that some schemes were being evaluated against indicators that the schemes' managers now felt were no longer relevant. To some extent this may be inevitable as all initiatives evolve over time as experience is gained and new problems are encountered. However, some managers also felt that, with hindsight, more time should have been given to select indicators prior to the schemes becoming operational.

Setting Baselines

8.13 Once indicators had been selected then, if they were to be used, baselines needed to be set: that is the value of the indicator at the start of the wardens' intervention had to be measured. Again, even by the time the 24 month reports were submitted, progress in setting baselines was far from complete. For example, GEN's assessment was that, for the statistical variables, complete baselines had not been set by 13 (43%) of the 30 schemes for which we had information (Table 8.1).

8.14 Even where baselines had been set there were concerns about their validity. Some baselines that had been set covered different time periods from the year 1 and 2 data (or vice versa). For example, the Aberdeenshire scheme set a baseline for one indicator using data covering the 10 am to 10 pm period. However, the year 2 data covered 24 hours. Other schemes, such as Aberdeen, had set baselines using police geographies that subsequently changed.

Targets

8.15 Implementation progress was then to be assessed by setting targets for the ends of Years 1 and 2. The extent to which these were attained could then be used to monitor progress and make changes to activities and processes as necessary. As recently as the 24 month report, half of the 30 schemes had not set targets (Table 8.1).

8.16 Even when targets had been set there were often issues with them. For example:-

• Some were statistically meaningless, for example when targets were set to reduce incidences by 1% or when targets were set to reduce an indicator by 10% when there had only been 4 actual incidents. The most extreme example of this was the scheme that wanted to reduce 0 recorded instances of littering by 10%;
• Some schemes had set targets but had not set any baselines;
• Some had not set numerical targets but talked in general terms about "increases" or "decreases"; and
• At least one scheme had set a target for a service output, rather than an outcome.

Comparators

8.17 To be able to assess changes in the wardened areas there is a need to be able to benchmark the schemes against some wider area. Comparisons of relative changes in indicators will then allow assumptions to be made about the impact of the wardens.

8.18 Table 8.2 shows that only 4 schemes had selected comparators for all indicators, although more had selected comparators for some. The comparators selected fell into 3 main groups:-

• Those that were to use the wider council area to assess progress. This was helped by the fact that SHS data was available for the local authority areas every other year, thereby giving an authoritative comparator;
• The schemes that proposed to benchmark themselves against similar areas, as with Orkney that intended to compare itself to Shetland and the Western Isles; and
• Those schemes that would benchmark the wardened areas against the wider urban area in which the scheme operated. For example, Angus intended to compare its scheme, that operated in parts of Arbroath, with the whole of the town.

8.19 Despite some progress there was cause for concern given that, at least 24 months into the implementation of the warden schemes, most had not selected comparators and begun to gather appropriate data.

The Ability to Use the Data to Assess Progress

8.20 Table 8.2 shows which authorities submitted full year 1 and year 2 data sets thereby enabling progress to be assessed. It can be seen that:-

• 60% had submitted full year 1 data sets; and
• 53% had submitted full year 2 data.

Only 13 schemes (43%) had submitted full data sets for both years. Again the failure of more schemes to supply complete data is disappointing.

Survey Issues

8.21 The Executive's guidance asked authorities to select 2 survey based indicators. The analysis undertaken for the 6 month report found that, to collect these, the schemes were:-

• Commissioning consultants to undertake warden-specific surveys (as in Dundee and East Dunbartonshire);
• Doing surveys themselves (Midlothian and West Dunbartonshire); or
• Including questions relevant to the wardens in council-wide tenants' surveys (Edinburgh).

8.22 Details of the response rates were given for some of these surveys in the 6 month reports. In some instances, they were very low. For example, West Lothian had a 7% response rate whilst West Dunbartonshire's was 13.6%. Others had better responses. For example, Angus had a 32% rate, Orkney between 35% to 40% and Perth and Kinross 50%. Many of the other reports gave no details of the survey methodology or response rates. From the questionnaires seen, low response rates often seemed to reflect inadequate methodologies such as sending out questionnaires at holiday times, failing to offer incentives or giving lengthy return times.

8.23 As far as can be judged, few of the schemes made any attempts to see how representative of the demographic characteristics of the areas the responses were. Indeed, often this was impossible as respondent profile information was not requested on the survey forms.

8.24 The intention, in most authorities, was to repeat the surveys at a later date and compare the 2 sets of results. Any changes would then be ascribed to the wardens' interventions. However, we felt that there was a danger that, if no attempt was made to repeat the surveys using the same sample, then the 2 sets of results would not be comparable. Any comparisons over time, therefore, would be of limited use as those being sampled would be totally different. Only Stirling commented upon this in its 6 month return when it stated that it would repeat its community survey using the same respondent base.

8.25 It was also felt that the responses were unlikely to be representative, being largely drawn from those who were "professional complainers". This was hinted at in North Ayrshire's 6 month report which commented on the attitudes of elderly people within the wardened areas towards the young, whom they blamed for everything. Without a robust survey methodology we felt that there was a danger that such unrepresentative views would come to dominate the survey results.

8.26 As far as can be judged, very few of the schemes attempted to interpret survey results in the light of these deficiencies. Indeed many of the concerns identified in the 6 month report were repeated in the 12 month report (Hayton and Percy, 2005b). GEN, therefore, felt that there was a strong risk of widespread misattribution if reliance was to be placed on such surveys to evaluate the impact of the warden schemes.

8.27 Given these problems, our view, based on the analysis of the 6 month reports, was that many of the surveys that the schemes were undertaking were of dubious value and would be of limited use for assessing the impact of wardens' interventions. Given this, it was agreed that the evaluation would not draw on the survey data. Instead base and endline surveys would be undertaken as part of this evaluation in the case study and 2 control areas. These would then be used to look at changes in residents' perceptions. Full details of these surveys are given in Chapter 2 (Paragraphs 2.20 to 2.28).

Boundary Problems

8.28 The 6 month reports identified a range of problems related to the different geographies for which data was collected. The main one was a lack of correspondence between the areas patrolled by the wardens and police beats, which tended to be the lowest spatial units for which crime statistics were generated. This was identified as an issue by 9 councils when the 6 month reports were submitted.

8.29 However, generally most councils seemed to feel that boundary problems were relatively minor. Indeed these did not figure as an issue in the 12 month reports. As such they are probably best seen as teething problems.

Conclusions

8.30 Despite having built in systems for monitoring and evaluation of the wardens schemes at their inception, the results have been disappointing, with a widespread failure by many schemes to gather the data needed to populate the monitoring templates. A number of reasons can be put forward to explain this:-

• The schemes did not see monitoring and evaluation as being important alongside of the need to set up and establish the schemes. It had, therefore, always been given a low priority. This was an impression reinforced through the interviews with managers and by the difficulties experienced in getting schemes to submit their monitoring reports on time; and
• A seeming reluctance by the Executive to act when there was widespread evidence that monitoring was not been undertaken. Some of the Executive's actions, no doubt, sent the wrong messages to the schemes. For example, despite the schemes being asked to select 4 indicators, those who failed to do this were not challenged, likewise when the reports on the monitoring templates highlighted widespread data gaps little action was taken to remedy this.

8.31 In the light of this we would suggest that, should similar schemes be rolled out by the Executive, monitoring needs to be treated far more seriously. How this should be done is outlined in Chapter Seventeen.

« Previous | Contents | Next »