The Scottish Government

« Previous | Contents | Next »

Listen

3 THE PILOT PROJECT AND EVALUATION

3.1 The origins and establishment of the Pilot project

The overall aim for the national pilot project in Restorative Practices was to learn more about RPs in school settings and to look at whether there could be a distinctive Scottish approach, an approach that both complemented and offered something additional to Scottish practice.

The specific objectives of the pilot project and the evaluation together were:

• To identify the training and support which staff feel is required to enable them to implement the initiatives effectively
• To explore the different situations, contexts and areas of the curriculum where the new approaches are employed
• To analyse the ways in which different participants (teachers, classroom assistants, pupils, parents) respond to the innovative approaches and the conditions which appear to produce beneficial outcomes
• To identify the characteristics of schools, staff or others which contribute to positive or negative outcomes
• To identify the support required from Local Authorities to promote and support school-level implementation
• To develop a website for practitioners to exchange ideas and experiences
• To review the existing research and literature on Restorative Justice and Restorative Practice
• Produce a final report with implications for future developments

The Scottish Executive funded Restorative Practice pilots in the 3 local authority areas (Fife, Highland, and North Lanarkshire). Each area developed their pilots in different ways although underpinned by a broadly common philosophy.

The Executive also commissioned at the same time this evaluation of the pilots by a team from the Universities of Edinburgh and Glasgow. The evaluation was highly collaborative, and the levels of participation allowed the evaluation to be both formative and summative. Each school also identified its own aims and planned outcomes for the pilot project and the evaluation also examined how these aims and outcomes were delivered. The evaluation then, overall, offers some comparisons across schools and LAs but also focussed on assisting individual schools to develop and reflect on their own practice.

In each Local Authority, 6 schools were identified as pilot evaluation schools; these include 10 secondary schools, 7 primary schools and one special school overall. They are situated in urban, suburban and rural areas and in areas of severe economic poverty as well as areas of relative economic wealth; reflecting the diversity of Scottish communities. They also had varied histories in terms of existing approaches that could be described as restorative and had very varied expectations of the project.

3.2 Methods of Evaluation

The evaluation team began by systematically reviewing existing international research and literature on Restorative Justice and Restorative Practices in education. The team then negotiated a range of methods through which to evaluate the aims and outcomes specified for the pilot projects, as these developed in the different LAs and schools. The design of this evaluation forged new ground in a number of ways. It involved working with staff in the three LAs to clarify the nature and goals of the pilot initiatives and develop a methodology for the collaborative evaluation, in which participants, as well as researchers, played a critical part.

The evaluation team also worked throughout with a national Steering Group consisting of key managers from the LAs, schools and educational psychology services within each of the three Local Authorities. There are major advantages and disadvantages of working collaboratively and these are discussed further below.

3.3 Data Collection in Local Authorities and Schools

Data collection involved:

• Interviews with a range of LA and school staff
• Interviews, individual and group, with pupils
• School staff survey
• Pupil survey
• Observation of a range of meetings, activities and lessons
• Documentary analysis of school and LA policies
• Participation in a range of SEED, LA and school based meetings
• Analysis of national and school statistical data
• Focus group meetings with school and LA staff

Table 1 Staff, Parents and Pupils (Numbers of interviewees)

The Table above shows the numbers of individuals interviewed. Key staff members in each school were interviewed on a number of occasions over the period of the pilot. In summary, across the 18 schools, we interviewed 17 Headteachers; members of School Management Team/Principal Teachers (43); class/subject teachers (48); non-teaching support staff (30), other inter-agency staff (12), and educational psychologists (12). About 400 individual interviews took place with staff in schools. We also met directly with 138 primary pupils and 93 secondary pupils, either in groups or in individual interview. We met with 12 parents of primary pupils and 19 parents of secondary pupils. Each school was visited at least 5 times and findings at each stage of the research were fed back and discussed with key staff.

Staff and Pupil Surveys

Two surveys were carried out, one of school staff and one of pupils. These contributed to our understanding of each school's context and climate, as well as to the process of identification of key variables that might impact on the success of RP in different schools. The surveys were undertaken at Easter 2005 (staff) and September 2005 (pupils). They were not intended to be base-line data to be repeated later, as they represented a snapshot of the schools at very different stages of their journey.

Staff Survey

The survey of all staff (teaching and non-teaching) in the 18 schools was undertaken in 2005. A total of 1397 questionnaires were distributed to the schools. Of these 627 were returned, which represents an overall response rate of 45%. The response rates for the different types of schools differed with primary schools having an overall rate of 56% and secondary schools 42%. There are, of course, limits to any generalisations that can be made on the basis of this data due to the response rate and the fact that the survey was carried out early in the pilot. The findings, discussed in Chapter 6, do however offer a 'snapshot' of staff attitudes to, and understandings of, Restorative Practices.

Pupil Survey

Similarly, a pupil survey of all 18 schools was carried out in 2005, aimed at pupils in P5 and P7 in primary school and S2 and S4 in secondary school. The questionnaire complements the staff questionnaire and also allows for triangulation with the qualitative data gathered through observation, interviews and focus group meetings. The total number of questionnaires returned was 1163. Again though, there are limits to the generalisations that can be drawn, as the survey was undertaken at an early stage of the implementation of Restorative Practices in some schools. The findings, discussed in Chapter 6 offer an interesting picture of pupils' views.

Collaborative Evaluation Schools/Local Authorities

• Schools individually received regular 'visit' reports
• Interim LA Reports (May 2005)
• Staff Questionnaire report (January 2006)
• Individual school staff questionnaire summaries (January 2006)
• Pupil questionnaire report (May 2006)
• Individual school pupil questionnaire summaries (May 2006)
• Individual school reports (May 2006)
• Final LA Reports (June 2006)
• Final School Reports (June 2006)
• Final Focus Group - North Lanarkshire and Fife (August 2006)
• Full Evaluation Report (October 2006)

Advantages and Disadvantages of Collaborative Working

The evaluators were seen as helpful to the pilot projects in feeding back their observations and ongoing analysis, thus affecting directly and contributing to the projects. This continuous feedback allowed for honest reflection on the highs and lows of the intervention, and provided opportunities to change and adapt accordingly. The research team also clearly acted at times as 'critical friends', in some cases developing strong relationships with key staff in schools, sharing ideas and sometimes providing an empathetic 'ear' for the release of stressed staff. School staff mainly valued the school visits and some primary school Headteachers in particular, found them supportive. Some felt that it had reminded them to review how things were going:

It made me check (Headteacher).

You had to 'keep up-to-date' because you [evaluation team member ] were coming in' (Headteacher).

I actually enjoyed J***'s monitoring of me! (Support manager)

It was absolutely superb. You're so immersed in the doing of it, so to sit down with G****** and reflect! (Headteacher).

While some staff didn't enjoy the tape-recorder they appreciated its usefulness.

Did I really say that! (Teacher).

A number of staff from schools commented that they felt that the evaluation process had been a useful spur to the development process. There had been some initial concern that we may have needed a neutral person for appeal if there was conflict or strong disagreement between the research team and the schools. In the event this was not necessary, relationships remaining positive even though there were inevitable differences of view. Some people valued the different perspective of ' an outsider'.

Several indicated how useful they had found the post visit reports. One support teacher said that she had felt the process difficult at the beginning but that it had evolved over time. It took her a while to realise that it was a 2-way process, that she could influence what happened, and '… that giving feedback was OK - and that the evaluation team was prepared to compromise and correct'. However, being asked for more hard data was sometimes seen as more irritating and could be time-consuming by staff. It was often necessary to re-prompt for such data.

Final focus groups were held in North Lanarkshire and Fife with key staff from LAs and schools (distance and staff cover issues preventing this in Highland). In both there was a view that some continued level of external evaluation, emphasising the 'critical friend' dimension would be helpful.

3.4 Summary

The collaborative research model clearly involves major strengths in its facilitating of access to the sites of investigation and depth of discussion with participants. The relationships created through this process facilitated the collection of high quality authentic data but of course affected both the process of data gathering and our findings. We cannot claim that this has been a neutral and objective exercise - the research team and research participants were engaged reciprocally in the development of both the practice and the evaluation process. However we also gathered some more quantitative data, through the surveys and the statistics and these were used in an overall process of analysis.

The regular feedback of interim findings and discussion with schools meant that the research process and the development process were equally reciprocally influential, again with the advantages and disadvantages that this implies. The members of the research team however did not always share the same views about the value of particular practices or approaches and this led to a healthy debate within the team and with schools, the LAs and the national Steering Group.

There was also throughout an inevitable challenge of evaluating an initiative that was not operating in isolation from other initiatives with rather similar broad aims, particularly those emanating from Better Behaviour Better Learning. This was in some ways made easier, and others more problematic, by a number of the schools whose staff saw RPs as part of a broad approach to school development.

In summary the evaluation aimed to:

• Learn from schools about the different possible ways in which to implement Restorative Practices
• Establish how RP in schools might evolve most effectively within the Scottish educational, historical and cultural context
• Develop a collaborative approach, cognisant of the associated positive and negative implications for the findings

The data collection process was characterised by:

• Both qualitative and quantitative data gathering and analysis
• The gathering of a large bank of interview data
• Two major surveys of staff and pupils -those most directly involved by RP
• Identification of the difficulty of isolating the impact of one initiative in the complexities of daily school life
• Constructive, open and reflective discussion between the evaluation team and key school and LA staff.

In the following chapters, the findings from the evaluation of this pilot are described in detail. In Chapter 4 we report the experiences of the 3 Local Authorities as they implemented Restorative Practices. In Chapter 5, the findings from the implementation of RP in the 18 schools are described; the links with other initiatives, the different aspects of RP and how schools made use of each of these, early attempts to engage parents, and the views of all those involved. In Chapter 6 we look at some common indicators and discuss the overall achievements of the schools. This is followed in Chapter 7 by a discussion of key aspects of successful implementation under five broad themes; readiness, aims and change processes, training, leadership and issues raised by multiple innovation. Chapter 8 outlines the conclusions of the evaluation team.

« Previous | Contents | Next »