Scottish Government

Evaluation of Project 1 of the Assessment is for Learning Development Programme:
Support for Professional Practice in Formative Assessment
Final Report

3. Methodology

3.1 Design of the evaluation

The evaluation was undertaken taking account of principles fundamental to the philosophy of the project itself - the importance and value of self and peer assessment in learning and development. The extensive use of peer and self-evaluation in the planning, development and reporting stages of the project provided the basis for much of the overall evaluation. The evaluators collated data provided by:

• teachers' agreements;
• action plans;
• programme diaries;
• case study reports;
• relevant school self-evaluations.

This was supported by:

• the attendance of the evaluators at the meetings of each regional group;
• exploration of the experiences of the schools designated as comparators by the Education Authorities in relation to formative assessment;
• telephone interviews with all participating and comparator school head teachers;
• telephone interviews with Education Authority co-ordinators for the programme and any other involved EA managers;
• telephone interviews with project development officers and university support personnel;
• field visits to three schools within each regional group to provide in depth information about successful and less successful projects.

These activities served to validate the self-assessments.

Consistency of approach in the work with each regional group and in data collection was ensured by regular meetings of the evaluation team at the Institute of Education and the development of checklists of key aspects to be documented and guidelines for data collection. A Regional Research Co-ordinator from the evaluation team was assigned to each regional group to collect and collate data from that region.

3.2. Existing and emerging documentation

The evaluation relied to a great extent on the collation and analysis of data created as part of the self-evaluation element of the project. This was supplied by schools and through their submissions to project development officers.

3.3 Interviews

Telephone interviews were undertaken with head teachers in pilot and comparator schools, the programme co-ordinator from each Education Authority (EA) and any other involved EA managers, project development officers and university support personnel. Interviews were tape recorded. Notes were made of key points and the recordings referred to for confirmation of detail and the extraction of specific quotations as appropriate. A particular focus of the interviews, in addition to their role in the overall evaluation, was to identify instances where the project had not been successfully implemented and the perceived reasons for this.

3.3.1 Interviews with Head Teachers

The head teachers of comparator and pilot schools were contacted by e-mail or telephone and asked to outline their aims, aspirations and concerns relating to formative assessment and how they wished practices to change in their schools. This was used to provide an outline of the perceptions of formative assessment held within the educational community as a whole against which to consider the impact of the project nationally.

Project schools: Telephone interviews were undertaken with either the head teacher of the school or a member of the senior management team to explore:

• the aims of the project as developed within the school;
• how the project had been implemented;
• the nature of the action plans developed;
• any difficulties that had arisen in relation to implementation of the project;
• the perceived positive outcomes for teachers directly involved in the project, e.g. in relation to pedagogy, teacher understanding of assessment issues, attitudes, motivation;
• perceived positive outcomes for the whole school;
• perceived changes in the learning climate of the school;
• perceived positive outcomes in relation to pupil learning, motivation and behaviour;
• perceived positive outcomes in relation to parents' interest and involvement in their children's education;
• any unexpected benefits;
• any negative outcomes;
• the key features of the work perceived to have contributed to its success or failure;
• the effectiveness of the development and operation of the programme as a whole;
• whether, overall, the programme was perceived as worthwhile and, if so, why and in what ways.

The full interview schedule is included in the appendices.

Comparator schools: Head teachers or senior managers within comparator schools were interviewed by telephone to explore:

• their views of the role of formative assessment in the enhancement of learning and teaching in their school;
• their aspirations about how the project might enhance learning and teaching in their school;
• any development work being undertaken in their school relating to formative assessment and its effects;
• concerns they may have had about the implementation of the nationally developing assessment project.

The full interview schedule is included in the appendices.

3.3.2 Interviews with Education Authority Co-ordinators

The Education Authority co-ordinators and any other involved EA managers were interviewed by telephone and asked to outline their aims, aspirations and concerns relating to formative assessment in the schools in their Education Authority. They were also asked how the project was addressing these needs. Their views were also sought regarding the most effective ways of disseminating the good practice highlighted by the project to best ensure its implementation in schools.

The telephone interviews explored:

• the extent to which the project related to existing EA developments concerned with learning and teaching;
• the extent to which formative assessment was perceived as being central to promoting effective learning;
• the extent to which the implementation of the project was perceived as having been successful and why;
• how effective the project was in generating ideas and changing practice in relation to formative assessment;
• how good practice was to be disseminated within the EA.

The full interview schedule is included in the appendices.

3.3.3 Interviews with project development officers and university support personnel

Project development officers and university support personnel were interviewed by telephone to establish their aims, aspirations and concerns relating to the project, the extent to which they believed that these were being met, their observations regarding the implementation process, any difficulties experienced, and particular instances which they perceived as 'good' or 'poor' practice which had evolved during implementation. The telephone interviews explored:

• their perceptions of the overall success or failure of the project in the schools and regions within which they had been working;
• their perceptions of why the project had been successful or had failed in different schools or regions;
• the difficulties that they had experienced in supporting the implementation of the project;
• how effective the project was in generating ideas and changing practice in relation to formative assessment;
• the extent to which they had observed changes in teachers' thinking and practices;
• the extent to which they had observed changes in pupils' learning, motivation and behaviour;
• any changes they had observed in the nature or level of parental involvement with pupil learning;
• whether, overall, the project had been worthwhile and, if so, why and in what ways.

The full interview schedules are included in the appendices.

3.4 Attendance at meetings

The Regional Research Co-ordinators attended area meetings taking the opportunity to explore arising issues. Strengths and weaknesses of the approaches adopted in individual schools were noted and overall attitudes towards the project.

3.5 Field visits

Based on the self-evaluation documents and the data collected in the interviews, twelve schools were selected for field visits. The selection was made, in conjunction with the project team, to include primary, secondary and special schools, from a range of environments, representing the implementation of a range of different formative assessment strategies.

The field visits undertaken to the twelve selected schools by the Regional Research Co-ordinators included observation of two classes in each of these schools and interviews with the head teacher (or relevant member of the Senior Management Team), and groups of participating teachers and pupils from their classes. Interviews were tape recorded. Notes were made of key points and the recordings referred to for confirmation of detail and the extraction of specific quotations if appropriate.

A checklist was developed for ensuring consistency in the observations (see appendices for details).

A rating scale was developed for staff to evaluate the perceived effectiveness of the project in relation to a range of outcomes for pupils, teachers, the whole school and parents (see appendices for details).

3.5.1 Head Teacher / School Managers' interviews

Individual interviews were undertaken with either the head teacher of the school or a member of the senior management team to explore:

• the aims of the project as developed within the school;
• the nature of the action plans developed;
• how the project had been implemented;
• any difficulties that may have arisen in relation to implementation of the project and how these were overcome;
• the perceived positive outcomes for teachers directly involved in the project, e.g. in relation to pedagogy, understanding of issues of assessment, attitudes, motivation;
• perceived positive outcomes for the whole school;
• perceived impact of the project on the learning climate in the school;
• perceived positive outcomes in relation to pupil learning, motivation and behaviour;
• perceived outcomes in relation to parental involvement in their children's education;
• any unexpected benefits;
• any negative outcomes;
• the key features of the work that contributed to its success or failure;
• the effectiveness of the development and operation of the project as a whole;
• whether, overall, the project had been worthwhile and, if so, why and in what ways.

3.5.2 Teacher interviews

Teachers who were active participants in the project were interviewed to explore:

• their aims in being involved in the project;
• how they had implemented the project;
• the nature of the action plans developed;
• difficulties they experienced in implementing the project and how these were overcome;
• changes in their understanding of the nature and purpose of formative assessment;
• perceived positive outcomes relating to their own practice of taking part in the project;
• changes in their practice;
• any perceived negative outcomes in relation to their practice or professional life;
• perceived positive outcomes relating to pupils - learning, motivation, attainment and behaviour;
• perceived outcomes in relation to parental involvement in their children's education;
• any perceived negative outcomes;
• the key features of the work which contributed to its success or failure;
• the effectiveness of the development and operation of the project as a whole;
• whether taking part in the project was worthwhile and, if so, in what ways;
• whether the project had influenced the learning climate within the school.

3.5.3 Group interviews with pupils

Group interviews were undertaken with at least two groups of pupils in each school, to include a minimum of 12 pupils representing boys and girls of different levels of attainment, whose teachers had been directly involved in the project. The researchers sought informed consent from the pupils to be involved in the interviews through their school, in accordance with the recently published Code of Practice. The interviews explored:

• whether the pupils had noticed any changes in the activities in their classroom;
• the type of feedback they were given on their work and if and in what ways this helped them;
• their beliefs about the purpose of assessment;
• their experiences of formative assessment;
• their beliefs about the extent to which assessment supported their learning;
• the extent to which they had taken part in interactive discussions in their class;
• the extent to which they believed that their teachers listened to them and took account of their learning needs;
• whether and how they undertook self-assessment of their work;
• the extent to which peer assessment was encouraged in their class;
• the extent of their motivation to learn using feedback acquired through discussion or on written work;
• whether the changes made in relation to assessment had led to an improvement in their concentration and general behaviour;
• whether the changes had an impact on the extent of their parents' involvement in their school work.

The stimulus materials used to generate discussion were adapted to take account of the age of the pupils.

3.6 Observation of classes

In each school visited, two classes were observed where the principles underlying the project were believed to have been implemented successfully. Observations focused on the aims of the project as stated in that school. The observation focused only on the aspects of teaching within the classes which related to the project, i.e. the implementation of formative assessment. Observers took detailed notes and aimed to be as unobtrusive as possible. Points of interest were agreed with the teacher before the observation was undertaken. Examples of pupil work, if appropriate, were examined. This included examples revised to take account of feedback.

Researcher consistency in the observation process was ensured through the development of a checklist of aspects of teacher and pupil behaviour which were to be monitored. This was developed by the team prior to the initial observations on the basis of previous research relating to formative assessment. As the cycle of observations were undertaken any additional emerging aspects were monitored in subsequent observations. Depending on the particular focus on formative assessment adopted in the school consideration was given to:

• the extent to which the teacher had established pupils' learning needs and what was needed to support them in moving from their present level of attainment to the desired aim;
• the extent to which the quality of the work rather than its quantity or presentation was assessed;
• the extent to which pupils were encouraged by the teacher to adopt a mastery approach to learning, attempting to improve their own work rather than focusing on their position in relation to others;
• the extent to which work was graded;
• the extent to which the importance of effort was stressed;
• the extent to which pupil success was emphasised;
• the way in which pupils were supported in learning how to improve their performance;
• the way pupils were supported in developing their metacognitive skills;
• the extent to which pupils were given detailed feedback on their work (written or verbal);
• the extent to which pupils had knowledge of what constituted 'good' work;
• the nature and extent of opportunities for pupils to discuss what was being learned and clarify their understanding;
• the nature and quality of questioning and response in the class;
• the opportunities for thoughtful reflective dialogue between pupils and teacher.

3.7 Phase 2 of the evaluation

In Phase 2 of the evaluation further field visits were undertaken to eight schools including primary, junior high, secondary and one special school. The same interview schedules, classroom observation schedules and questionnaires were used as in the Phase 1 field visits.

3.8 Analysis of Data

The qualitative data was analysed to consider the extent to which the programme was perceived to have:

• changed practices in the classroom;
• improved pupil learning;
• changed pupils' understandings of the purposes of assessment;
• improved pupil motivation and behaviour;
• improved attainment;
• influenced change in teacher beliefs and attitudes;
• changed teachers' understanding of the assessment process;
• enhanced the school climate to promote a positive learning environment;
• changed parental interest and involvement in their child's education.

The strengths and weaknesses of particular initiatives in individual schools were noted and any obstacles to implementation. These were collated and emerging patterns noted.

The strengths and weaknesses of the process of the development of the project was assessed and how it might be improved for future initiatives within the Assessment Development Programme.

The data from Phase 2 enabled an assessment of the longer term impact of the project and the likelihood of its sustainability.