The Scottish Government

« Previous | Contents | Next »

Listen

CHAPTER TWO THE EVALUATION METHODOLOGY

Introduction

2.1 This chapter has 2 main aims, to:-

• Outline the objectives of the evaluation; and
• Look in some detail at the methodology used to attain these.

We start by looking at the Executive's objectives, as set out in the original research brief.

Research Objectives

2.2 This evaluation of the implementation and impact of community wardens was commissioned by the Scottish Executive in mid-2004 from GEN Consulting. It was intended that the work should be both formative and summative, with 4 main objectives, to:-

• Identify innovations and good practices that could influence and inform the development and implementation of warden schemes at national and local levels;
• Support the collection of monitoring information so that the impact of the schemes at the local and national levels could be determined;
• Examine, and explain, the processes underpinning the development and implementation of the schemes; and
• Provide an overall assessment of the warden programme, including value for money.

The details of each of these objectives were outlined further in the brief.

2.3 In terms of good practices the evaluation was to look at such factors as:-

• Recruitment and training;
• Scheme management;
• Relationships with partners; and
• Relationships with the community.

2.4 The brief identified a key aim as being to ensure that robust baselines and ongoing monitoring information was available so that the impact of the schemes could be assessed. This formative element had 2 main components:-

• Reviewing the existing arrangements for monitoring and evaluation and providing advice; and
• The production of summary monitoring reports at 6, 12 and 24 month intervals. These reports were to be compiled from templates submitted to the Executive by the 32 councils, that were to collect information under the guidance of the Executive's research staff.

2.5 There was also to be a summative element to the work, the information for which was primarily to be collected by undertaking a number of case studies. It was hoped that these would provide answers, amongst other things, to the following:-

• The impact of the community warden schemes on reducing crime, the fear of crime and antisocial behaviour and improving environmental quality;
• The extent to which the schemes in different authorities had achieved their specific aims and objectives;
• Views of local communities on the wardens' roles and the benefits of having the schemes; and
• The extent to which there was evidence of problems being displaced, as a result of the wardens' interventions, to other areas.

2.6 In terms of the processes underpinning the development and implementation of the schemes the following were tentatively identified:-

• Partnership working with other council services and external agencies;
• The impact of Neighbourhood Compacts upon the involvement of the local community;
• Interactions and relationships with the police;
• The development of relationships between the wardens and different groups in the local community, such as young people;
• The effectiveness of Environmental Audit Tool Kits as a means of monitoring the wardens' activities and ensuring community engagement;
• The factors that were felt to have facilitated and hindered the wardens from achieving their aims and objectives; and
• Views on the support offered by the Executive.

2.7 The outcomes of the work on impact and process were then to feed into recommendations, intended to improve the schemes' future effectiveness and impact.

2.8 Value for Money was recognised as being difficult to assess. Accordingly the brief proposed that a fairly basic assessment be undertaken.

Changes and Amendments to the Brief

2.9 In the course of the evaluation a number of changes were made to the brief. The main ones were that:-

• Glasgow was not to be included within the case studies as it was to be the subject of a separate formative evaluation of the Glasgow Taskforce, the results of which are expected to be published in the Summer of 2007;
• Value for Money considerations were not to be covered as they were to be included within a wider, separate evaluation of antisocial initiatives being undertaken for the Executive ⁴; and
• The effectiveness of Environmental Audit Tool Kits was not considered as, at an early stage in the Evaluation, the Executive found that no schemes intended to use them. As far as we are aware this has remained the case.

Given these changes the main focus of the evaluation has been upon identifying impact and looking at process issues. How this was undertaken will be covered in the next Section.

The Method

2.10 The brief identified 2 mandatory elements of the methodology:-

• Analysis of the 6, 12 and 24 month monitoring templates submitted by the councils; and
• Selection and then analysis over time of 6 to 8 case studies.

Additional elements were left to the discretion of the contractor.

2.11 Building on these mandatory elements, GEN proceeded to implement a number of complementary approaches intended to ensure that the objectives of the brief were attained. Each of these will now be outlined.

The Methodology - Impact Assessment - The Monitoring Reports

2.12 The initial guidance issued to the councils covered monitoring and evaluation, which was to be built in at the start of the implementation process (Scottish Executive, 2003b). To monitor progress each council was to select a number of monitoring indicators from a menu. The indicators fell into 2 categories:-

• Statistical ones of which 3 broad types were outlined in the Guidance:-
• • Impact on crime: based on the levels of recorded crime and focusing on crimes such as vandalism, littering, house breaking and harassment;
  • The impact on the recorded incidents of antisocial behaviour; and
  • The impact on empty properties and rental income generated; and
• Survey based ones, of which 6 broad types were suggested. Of these 4 were based on questions included in the Scottish Household Survey ( SHS), meaning that comparisons could be drawn with changes in the wider area.

2.13 All schemes were requested to select at least 4 indicators against which their progress was to be assessed. These were to be a mixture of statistical and survey based ones. In addition, schemes were allowed to select discretionary indicators that might more accurately reflect the focus of the activities of the wardens in their areas. A number were suggested in the guidance, covering such things as the costs of vandalism and numbers of neighbourhood disputes solved through mediation.

2.14 The selected mandatory indicators were then to be used to monitor implementation progress. Monitoring templates were to be submitted to the Executive according to the following timetable:-

• Six months from the scheme's start a template was to be submitted that looked at initial outputs such as the numbers of wardens;
• After 12 months, in addition to outputs, schemes were to report on their progress against the baseline for selected mandatory indicators; and
• At 24 months similar information was again requested.

The information at 6, 12 and 24 months was collected using a standard template. .

2.15 The basic form of the templates remained constant over the evaluation period, although additional questions were added when the analysis identified data deficiencies. The main impact data collected in the templates covered:-

• The aims of each scheme;
• The indicator, or indicators, selected to measure attainment of these aims;
• The source of the indicators;
• The baseline value of the indicators;
• Targets for each indicator;
• Attainment progress at the end of years 1 and 2; and
• Comparator information.

Essentially what was being set up was an outcome agreement (although these words were not used), against which progress could be measured.

2.16 Had the templates been populated as requested then monitoring and impact assessment would have been relatively easily done. In the event this proved not to be the case as is outlined in more detail in Chapter 8.

2.17 In addition to the quantitative data, the templates contained a variety of qualitative information. This covered such things as:-

• Details of the activities undertaken by the wardens;
• Self assessments of progress in setting up Neighbourhood Compacts and protocols with the police;
• Residents' views; and
• Publicity strategies.

2.18 As part of the templates analysis, all of this information was collated and reports submitted to the Executive. The 6 and 12 month reports also contained a number of recommendations intended to improve the schemes' impacts.

The Methodology - Impact Assessment - Independent Evaluations

2.19 In addition to the provision of this ongoing monitoring information, those councils that received financial allocations for wardens and other antisocial behaviour initiatives of more than £500,000 a year were expected to commission independent evaluations. The costs of these were to be met out of the Executive's total funding allocation ⁵. Some of these have been drawn on to inform this evaluation.

Methodology - Impact Assessment - Base and Endline Surveys

2.20 The original intention was that data from the survey indicators would be taken from the councils' monitoring templates and analysed. However, when the 6 month templates were analysed, questions were raised about the methodology that the councils were using and therefore about the validity of any data they were likely to produce. Following discussion with the Executive, it was agreed that GEN would undertake its own base and endline surveys in each of the case study areas and that the impact analysis would not make use of the councils' own surveys.

2.21 Accordingly a baseline attitudinal survey was undertaken within each of the 9 case study and 2 control areas in April/May 2005. This looked at the impact of the warden schemes on residents' perceptions and explored the schemes' additionality. To make the main body of the report more readable the majority of the Tables on which the analysis is based are presented in Appendix 2, which is cross referenced as appropriate. ⁶

2.22 Many of the warden schemes are covering a number of sub-areas. In most cases the wardens began to operate in these areas gradually. Accordingly, when selecting the areas in which the baseline surveys were to be undertaken, attempts were made to target sub-areas where the wardens were either not yet operating or (if this was not possible) had only recently become operational. In this way GEN was trying to ensure that the baseline results were, as far as possible, reflecting respondents' views prior to the wardens becoming operational.

2.23 The survey was then repeated, with some additional questions and modifications to others, in May/June 2006.

2.24 In each of the 11 areas some 100 face-to face interviews were undertaken in 2005 and 2006. The interviews were to be with named individuals using address lists generated from the electoral registers. The same addresses were targeted in 2006. If the occupiers had changed they were still interviewed, albeit that their responses were not analysed. From Table 2.1 it can be seen that, overall, 81% of those contacted in 2006 had been interviewed in 2005. The repeat interview rate was higher for the 2 control areas and also varied considerably across the 9 case study areas, from a low of 53% in Larkfield in Inverclyde to a high of 100% in Torry in Aberdeen City ⁷. These variations reflect the relative stability of the populations in the different areas.

TABLE 2.1 Survey Respondents 2005 and 2006

2.25 The subsequent analysis in Chapter 12 of this Report are based on those who were interviewed in 2005 and reinterviewed in 2006, that is the 710 respondents in the case study areas and the 179 in the control areas.

2.26 The 2 control areas (Oxgangs and Firrhill in Edinburgh and Rattray in Perth and Kinross) were selected with advice from the wardens' managers in these council areas. As far as possible, the intention was to have control areas that were similar in socio-economic terms to the areas in which the warden schemes were based and where no additional crime or antisocial behaviour initiatives were in place ⁸. Table 2.2 compares the case study and control areas on selected respondent profile characteristics. It can be seen that:-

• On most of the indicators the 2 types of area are reasonably similar, with the deviations being of the order of 10%;
• The control areas have fewer unemployed residents, 5% as against 6% in the case study areas; and
• The control areas have a higher percentage of retired residents (39% as against 29%).

The full profile data are contained in Appendix One.

TABLE 2.2 Comparisons of Survey Respondents' Profile Characteristics for Case Study and Control Areas

Note:-

1. The percentages are derived from Tables A1.1, A1.2, A1.3 and A1.5 in Appendix 1.

2.27 The attitudinal survey data for the control areas are presented in Appendix 2, alongside the data for the case study areas. What emerges when comparing the baseline data is that residents' perceptions in both types of area were broadly similar in 2005:-

• The control areas were generally similar to the case study areas on most (8) of the perception variables ⁹; and
• The 2 variables where there seemed to be differences were the perceived prevalence of Drug Misuse/Dealing (Table A2.14) and Rubbish/Litter (Table A2.17). With regard to Drug Misuse/Dealing the case study areas had a greater perceived incidence than had the control areas. For example, 34% of case study area residents perceived this was "Very" or "Fairly" common compared to 28% of those living in the control areas. For Rubbish/Litter the controls had a greater incidence than the case studies with 42% rating this as "Very" or "Fairly" common compared to 37% in the case study areas.

However, despite some differences it was felt that the general similarities justified the selection of the control areas.

2.28 Although similar in socio-economic terms the control areas were not subject to interventions by wardens. The theory is that, by comparing the changes in the control areas to those in the wardened case study areas, it will be possible to isolate the impact of the wardens' interventions. However, this needs to be done with caution as the increased policy and political interest in tackling antisocial behaviour means that wardens are now only one of a number of interventions and few areas have not seen some type of antisocial behaviour initiative established in recent years (see Paragraph 9.19).

Methodology - Impact Assessment - Base and Endline Surveys - The Scottish Household Survey

2.29 To enable comparisons to be made with the wider area within which the case studies are based, a number of questions in the surveys were worded so that comparisons could be drawn with similar questions included in the Scottish Household Survey ( SHS). However, although comparisons are possible for specific questions, this is not the case for the survey dates, as information for 2006 from the Household Survey will not become available until mid-2007. Accordingly we have used the 2003/04 Survey results to draw comparisons with the baseline data and the 2005 results (published in August 2006) for comparisons with the 2006 endline. This is not ideal but no other option seems possible.

2.30 The situation is further complicated by the fact that local authority level data, for the smaller authorities, is only collated every other year in the SHS, as it takes 2 years of data collection to achieve the required sample size for these smaller areas. Thus the 2003/04 SHS results are classified according to local authority boundaries, whereas those from the 2005 Survey are available, in some instances, only for wider geographical areas.

2.31 These problems need to be borne in mind when looking at the comparisons between perceptions in the case study areas and the wider local authority areas in Chapter 12.

The Methodology - Case Study Selection

2.32 The brief for the study specified that between 6 and 8 case studies should be undertaken over the course of the evaluation. Chapter 3 looks at the characteristics of the warden schemes based on analysis of the applications for funding submitted to the Executive. This analysis resulted in 9 classification criteria being identified. These were then used to group the schemes so that the case studies could be selected. The key criteria are outlined in Table 2.3. It was felt that they had a degree of permanency, reflecting fundamental factors about the schemes that were unlikely to change, such as their objectives, scale (financial and numbers of wardens) and management structures.

TABLE 2.3 Wardens Schemes Classification Criteria Derived from Scheme Analysis

Note:-

1. Most schemes had a generic focus, in that they were attempting to tackle all types of antisocial behaviour in their target areas. The few that were thematic intended to concentrate upon particular target groups. For example, the focus of the East Lothian scheme was originally to be young people.

2.33 The Executive, in its original study brief, outlined 2 additional case study selection criteria. These were:-

• That there should be a mix of schemes covering urban and rural areas; and
• At least one scheme should have received second round funding.

2.34 These 11 selection criteria were then used to select 9 case studies. The intention was to select at least one case study that met each of the criteria and the various sub-criteria. Following an initial selection of schemes, these were discussed with the Executive and some changes made, in particular adding more large urban schemes which were felt to be underrepresented. The final selection of schemes was:-

• Aberdeen;
• Dumfries and Galloway (the Dumfries scheme as the 2 schemes in the council area were separately managed (see Paragraph 1.18));
• Dundee;
• East Renfrewshire;
• Edinburgh;
• Inverclyde;
• Orkney;
• Perth and Kinross; and
• South Lanarkshire.

The fact that 9, rather than 6 to 8, case studies were selected reflects a feeling that the schemes seemed to be so different that any less than 9 would not capture their diversity.

2.35 Table 2.4 summarises the characteristics of the case study schemes in terms of the selection sub-criteria. Only 3 are not represented:-

• Schemes that solely have a focus upon crime;
• Schemes employing between 11 and 25 wardens (Medium); and
• Schemes having a thematic Focus.

The main reasons why schemes were not selected that met these criteria were varied. In one instance (Glasgow) other evaluation arrangements were in place whilst the Executive felt that some of the schemes that met the other criteria were not sufficiently advanced to justify selection.

2.36 Of the schemes selected, 3 were required, as a condition of funding, to undertake independent evaluations: Dundee, Edinburgh, Inverclyde and South Lanarkshire (see Paragraph 2.19).

The Methodology - Case Study Analysis

2.37 Having selected case studies then, over the duration of the evaluation, each was visited 4 times by the consultant team. The exception to this was Orkney which, because of its remoteness, was visited once, although there were additional telephone contacts. The case study fieldwork visits involved consultations with:-

• Community warden scheme managers at strategic and operational levels;
• Community wardens;
• Police, fire and rescue and council departments;
• Elected members; and
• Community representatives.

2.38 The case study analysis also involved a review of the information produced and recorded by the warden schemes. This included:-

• Literature produced by individual schemes for distribution to the public. This was generally information on the role of the wardens and how to use the service;
• Protocols (where they existed) with the police, fire and rescue service and other council departments;
• Neighbourhood compacts (where they existed);
• Community Warden handbooks and standard operating procedures; and
• Induction and training materials.

TABLE 2.4 Case Study Selection

Notes (see Table 2.3 for full definitions):-

1. Sub-criteria are: C = Crime; E = Environmental; C/E = Crime and Environmental;

2. Sub-criteria are: N = New; E = Based on an existing scheme;

3. Sub-criteria are: F = Few; S = Several; M = Many;

4. Sub-criteria are: F = Few; S = Several; M = Many

5. Sub-criteria are: F = Few; S = Several; M = Many

6. Sub- criteria are: S = Small; M = Medium; L = Large;

7. Sub- criteria are: S = Small; M = Medium; L = Large;

8. Sub-criteria are: P = Police; C = Council; MA = Multi-Agency;

9. Sub-criteria are: T = Thematic; G = Generic;

10. Sub-criteria are: RR = Remote Rural; M = Mixed; U = Urban; and

11. Sub-criteria are: 1 ^st = First Funding round; 2 ^nd = Second Funding round.

2.39 Each of the rounds of case study visits was to follow the implementation progress of the schemes in the 9 areas. In particular, the development of the schemes, community involvement, partnership working, relationships with other services and impacts were examined. The consultants also looked across the schemes to identify common issues and examples of good practice.

2.40 Appendix 3 gives details of each of the case study schemes and highlights the key findings from each, based on the research undertaken.

The Methodology - Case Study Analysis - Focus Groups

2.41 A series of focus groups were also undertaken in the case study areas. In most areas an initial approach was made to the scheme manager to identify suitable bodies that could be approached to help to organise a group. These bodies included youth clubs, schools, sheltered housing complexes and senior citizens clubs. In some instances the scheme managers themselves pulled groups together when there was limited local community infrastructure. The focus groups were targeted at those resident in the case study areas who were:-

• Young people, covering a range of ages; and
• Older people, with again a range of ages being covered.

The groups for young and older people were run separately, in mid to late 2006, towards the end of the evaluation.

2.42 The rationale for including young people and older people within the consultations was two-fold. Firstly, the early fieldwork found that young people were perceived to be the main perpetrators of antisocial behaviour by the community representatives in the case study areas. Secondly, there was a perception amongst community representatives that older people were the least tolerant of antisocial behaviour and therefore were more likely to use a service such as the wardens. Through exploring how community wardens were perceived in both the younger and older age groups these assumptions could be tested. These groups were also highlighted by the Executive in the evaluation brief.

2.43 Those present at the focus groups covered a range of ages, from primary school age to late teens and from the recently retired to the elderly living in sheltered housing. The extent to which participants had first hand experience of antisocial behaviour varied. Some of the young people were perpetrators and thus gave an interesting insight into the wardens' role and effectiveness. Others, especially the very young, had good relationships with the wardens. Some of the older people regularly experienced antisocial behaviour, for example intimidation from large groups of young people congregating near their homes. However, others did not have this first hand experience. The groups were therefore covering the spectrum of ages and experiences.

2.44 Each group was asked a number of standard questions, regarding their awareness and opinions of community wardens. These were designed to provoke discussion and debate amongst participants. The sessions were structured to ensure a degree of consistency in questioning across the areas, whilst allowing discussion to be tailored to the experiences and concerns of individual localities. It was, however, necessary to adapt, add or omit some questions depending on the characteristics of the focus group participants and their experiences.

The Methodology - Scheme Managers' Survey

2.45 In order to gain an insight into the views of the schemes' managers in areas other than the case studies, 3 email surveys of managers were undertaken over the duration of the evaluation. These were timed to coincide with the submission of the 6, 12 and 24 month templates to the Executive.

2.46 The first and second surveys were similar, covering such things as views on partnership working, good practices, relationships with the police, other local authority departments and the community, development problems and views on the support given by the Executive.

2.47 The final survey (which achieved a 66% response rate) was different in that, although some of the questions were the same, the majority were more reflective, for example asking managers if, in the light of experience, they thought that the original objectives of the scheme had been appropriate.

The Methodology - Community Wardens' Survey

2.48 Towards the end of the evaluation a postal survey of the wardens working in each of the case study areas was undertaken, with the co-operation of the scheme managers . The survey covered; the profile characteristics of the wardens, employment history, career aspirations, views on their role, relationships with various agencies and groups, the time spent on various tasks and perceptions of impact.

2.49 The survey was sent out to 195 wardens and 103 were returned, a response rate of 53%. Response rates varied across the schemes (Table 2.5). Perth and Kinross had a 100% response rate, South Lanarkshire 90% and Orkney 75%. At the other extreme, East Renfrewshire had a 13% response.

TABLE 2.5 Wardens Survey Responses from the Case Study Schemes

The Methodology - Wardens' Activity Analysis

2.50 The analysis of wardens' activities was largely taken from the 24 month templates. The templates provided 2 main sources of information:-

• Written descriptions of the activities that the wardens were involved in; and
• Nine also provided detailed breakdowns of the numbers of incidents that the wardens dealt with over varying periods ¹⁰, although the way these were classified tended to vary.

2.51 There were also variations in the timescales over which data was collected. The majority of schemes collected data for a 12 month period, although some, for example, North Ayrshire had data for 24 months. In the case of Inverclyde, the wardens' logs covered a 3 week period, by far the shortest timescale in the analysis.

2.52 In addition the wardens' surveys asked for perceptions of the activities on which the most and the least amounts of time were spent.

Other Activities

2.53 Over the course of the evaluation a number of other activities were undertaken, some planned and others ad hoc in response to requests from the Executive. The main ones were:-

• The establishment of a Help Desk in the first year of the evaluation. This was to provide advice and support to councils with their monitoring data. Although this was publicised by the Executive, and by GEN, uptake was limited, there being only 8 queries. Further details are given elsewhere (Hayton and Percy, 2005b, Appendix 3);
• A telephone survey of all schemes undertaken for the Executive in late 2004 (Hayton, Gray and Stirling, 2004). This looked at implementation progress and identified any problems that the schemes were experiencing;
• At various stages over the evaluation's life, telephone contacts were made with those schemes where the 6, 12 or 24 month templates' analysis had identified problems. These telephone calls were to offer and provide support as needed; and
• A short paper produced for the Executive in 2006, looking at the flexible deployment of wardens (Hayton, Percy and Doyle. 2006) ¹¹. This was based on research undertaken in those schemes that either deployed their wardens flexibly or were thinking of doing so.

2.54 We now turn, in Chapter Three, to look at the aims and objectives of the schemes.

« Previous | Contents | Next »