The Scottish Government

1 35 schools initially, with a further 10 commencing in February 2003, plus one school from the independent sector.
2 A development officer was appointed at a later stage but initial developments were supported by the HE representative
3 Over 500 teachers were involved in the projects

Projects 6 and 7 are national developments with national development teams involving teachers, higher education representatives, Scottish Qualifications Authority representatives and Scottish Executive statisticians. One National Development Officer acted as key AifL link for both projects 6 and 7. Some of the schools participating in AifL were active in the AAP; schools from Project 8 piloted the Online Assessment Bank for the new National Assessments.

2.4 Evaluation design

The evaluation of the programme was divided into two main phases. Phase 1 covered the period January to December 2003 and Phase 2 covered the period January to December 2004. Quantitative and qualitative data were gathered during both phases.

Phase 1: January - December 2003

The first phase of the evaluation focused on understanding the nature of the projects, the extent to which they had met the critical milestones set and how they were meeting their stated aims. It involved:

• gathering, collating and analysing the project documents available in hard copy from project development officers and from the Learning and Teaching Scotland website
• surveying the views and experiences of the key stakeholders (sector representatives and local authority managers). Questionnaires were sent to 32 local authority assessment co-ordinators, 9 project development officers (one development officer managed both projects 6 and 7) and 11 higher education representatives
• surveying some of the schools involved in the AifL Programme. Questionnaires were sent to 174 schools. This list of schools was derived from information received from the project development officers. A questionnaire was sent to each school for a member of the senior management team and, based on information received from the schools, 340 teacher questionnaires were distributed.
• identifying schools within case study authorities and undertaking initial interviews.

Documentary Analysis

Two stages of documentary analysis were undertaken during Phase 1. The first stage, in February and March 2003, involved scrutiny of publicly available documents in both hard copy and electronically from the Assessment is for Learning website. The purpose of this was to identify key stakeholders and determine the involvement of authorities and schools in the various AifL projects. The questionnaires for the key stakeholders were informed by issues which emerged from this analysis: for example, the role of the key stakeholders and the extent of their involvement in AifL; their network of contacts in implementing the projects; the main means of communication and collaboration between key parties and their effectiveness; and progress being made towards project and programme aims.

The second stage of documentary analysis from March to May 2003 involved reviewing a wide range of documents related to each project. These were either publicly available from the AifL website or supplied by the national development officers, and included school baseline audits, school action plans, school progress reports, reports of development officers' visits to schools and examples of 'products', if available. The purpose of this analysis was to identify emerging issues in relation to each project, to see if there were common issues, and to inform the questions to be included in the survey of headteachers and teachers.

Survey of key personnel

The Assessment is for Learning Programme is dependent on a variety of key stakeholder groups working together to enable teachers to implement and trial developments in the classroom. Important premises of the development programme are that it is informed by research, it is developing national priorities at a local level, and teachers are the key developers engaging in action research. In addition, policy makers, practitioners and researchers should work collaboratively to develop approaches which have a real impact on teaching and learning. Three key groups in this process are local authority assessment co-ordinators, project development officers and higher education representatives from the Faculties of Education.

The views of these groups were sought by questionnaire (see Appendix 1) in Phase 1 of the evaluation (May 2003), just over a year into the pilot projects. Survey responses were received from 29 out of the 32 local authority assessment co-ordinators, 9 development officers (one for each school based project [8] and one jointly for Project 6 and 7), and all 11 higher education representatives.

Survey of project schools

Views of those working directly with the projects within schools were sought through a series of questionnaires to teachers and headteachers (see Appendix 1). The questions sought information on involvement in individual projects as well as views on the overall programme.

A list of schools involved in Assessment is for Learning was drawn up from information supplied by the Scottish Executive Education Department, LT Scotland Project Development Officers and supplemented by information from the Local Authority Assessment Co-ordinators. In total, 174 schools were identified, allowing data to be gathered on all of the projects that that were school-based ( i.e. not Projects 6 and 7) from schools of varying sizes and in authorities across Scotland.

Directors of Education were asked to confirm the schools involved and permission was sought to contact them. Schools were then contacted with a request for information regarding the number of teachers involved in the school to enable the appropriate number of questionnaires to be sent. Responses were received from 92 headteachers from a possible 174 schools, and 189 teachers, representing 102 (59%) schools in total.

Identifying schools in case study authorities

Four authorities were selected to cover a range of factors such as rural/urban, remote/central and stage of implementation, and also to cover all projects. Initial interviews and further programme developments led to changes in the planned work. For further explanation see the section on case studies in Phase 2 below.

Phase 2: January 2004 - December 2004

The second phase of the evaluation was to focus on the use of the outcomes of the projects by practitioners, managers and policy-makers. This phase of the evaluation included:

• analysis of the degree of convergence of the systems of assessment
• collation, review and analysis of project materials received from schools, including their own evaluations
• interviews in the case study schools. In total, 26 schools were visited; in all cases a member of the senior management team and teachers involved in the project developments were interviewed. In addition, focus groups were held with parents in 5 schools and with pupils in 7 schools
• survey of a sample of the stakeholders. Questionnaires were sent to 32 local authority co-ordinators and 10 higher education representatives. The Development Officers were not included in this phase as there had been changes to the role and personnel since the first phase of the evaluation. Questionnaires were sent to 165 schools: this was based on a revision of the original database used in 2003. Some schools were removed as they had not actually taken part and several schools that joined the programme late were added
• survey of Associated Schools Group ( ASG) co-ordinators. The development of ASGs had taken place during the session 2003 to 2004 in order to take forward AifL developments within local authorities, focusing in particular on local clusters of schools. Although not part of the evaluation proposal, it was decided that it was important to gain insight into the progress of ASGs and the work they had been engaged in. A list of 111 ASGs was received from SEED. Information received from local authority assessment co-ordinators suggested that some had not been able to undertake much development during 2003-04 and that some had not collaborated as clusters. It was therefore decided to focus on ASGs where it appeared there had been collaboration and in which progress had been made during 2003-04. Questionnaires were sent to 81 ASG contacts.

Documentary analysis

The documentary analysis was intended to review project materials, including schools' own evaluations. As schools were producing case studies of their project activities, a sample of these was collected for this phase of the evaluation. Fifty-four case studies were received via the project development officers or directly from schools. These were reviewed, with about half being studied in greater detail to assist in the selection of schools for visits and in preparation for those visits. The amount of information and extent of self-evaluation varied between case studies. However, they provided a useful insight into the nature of the work being carried out in the schools for each project.

Survey of key personnel and school staff

In Phase 2 of the evaluation, survey responses were received from 30 local authority assessment co-ordinators and 8 HE representatives. Responses were received from 77 managers from a possible 165 schools, and 130 teachers, representing 87 (53%) schools in total. Additionally, 45 (56%) responses were received from the 81 ASG contacts to whom questionnaires had been sent. The questionnaire data were analysed using SPSSx. Open ended responses were analysed thematically (Appendix 2).

Case studies

In total, 26 schools were visited; in all cases a member of the senior management team and teachers involved in the project developments were interviewed. In addition, focus groups were held with parents in 5 schools and with pupils in 7 schools.

The original intention for the case studies was to focus on clusters of schools in 3 or 4 authorities to cover all projects involved in the programme. However, schools were not working in clusters on developments and it was proving difficult to gain sufficient access to all projects within only 4 authorities. Furthermore, evaluation of Projects 1, 2 and 3 by other evaluation teams meant that some schools were being approached by more than one evaluator. It was therefore agreed with SEED that QIE would focus on Projects 4, 5, 8, 9 and 10. Some of these projects were taking place in the authorities that had been identified initially. The advice of SEED and project development officers was taken with regard to schools that were displaying good practice and these were chosen for more in-depth study. The schools visited included one pre-5 centre, 13 primary schools, 9 secondary and 3 special needs schools.

For headteachers and teachers the interviews covered the following issues: a brief overview of the work that had been carried out under the auspices of AifL; the impact that involvement in AifL had on school policy; its impact on practice in the school; views on the support for development work; any aspect of the project which has not worked particularly well and why; sustainability of developments; extent to which developments have contributed to improvement in meeting the information needs of key stakeholders (see interview schedule in Appendix 3).

Pupil group interviews focused on the changes which had occurred as a result of their teacher being involved in the project. They were asked to explain what had been different, what they liked and disliked about it (if anything) and if they thought it helped with their school work. There were also some questions designed to explore their understanding of assessment.

The parent group interviews varied according to what their involvement had been, but broadly covered the issues of how they had found out about the project, what their involvement had been, what they had gained from involvement, ways in which it had helped them understand their children's learning, how it had helped their children, and what they wanted to know about their child's progress at school.

The data were analysed on a project basis, looking at the following themes for each project: how and why the participants had become involved in the project; key aspects of work developed as part of the project; next steps in development; impact on practice, impact on policy; introduction of other aspects of the AifL Programme other than their original projects; views on and understandings of the wider development programme; and the views of parents and pupils. In this report the data derived from the case studies are used to further illuminate findings from the surveys.

Interview with key stakeholders

Interviews were carried out with a number of key stakeholders representing local authorities, project development officers and higher education representatives. The purpose of these interviews was to gain further insight into issues which were emerging from the surveys and to obtain the views of the respective groups on the whole Assessment is for Learning Programme.

Co-ordinators from 4 authorities were interviewed, one via email communication (see Appendix 3 for interview schedule). Two HEI representatives were interviewed, one at the beginning because of extensive involvement in the programme, and one towards the end of the programme.

A senior member of the LT Scotland development team and the development officer for Projects 6 and 7 were interviewed; additionally, 2 development officers at SQA with responsibility for the National Assessment Bank and the Assessment of Achievement Programme were interviewed.

Changes to the evaluation design

As the 10 projects in the AifL Programme were developed and implemented during the two-year period, changes occurred as a result of the action research nature of the classroom-based projects and the programme growing and developing. These changes led to alterations not only in the implementation of the AifL Programme itself, but, consequently, in the design of the evaluation from that set out in the tender document.

As Projects 1, 2 and 3 were subject to separate external evaluations, it was agreed that they should not be asked to contribute significantly to the surveys and should be omitted altogether from the case study element of this evaluation. As noted above (p11), the nature of the planned case studies also changed during the evaluation process. The original proposal was to identify school clusters in 3 or 4 local authorities and to concentrate on developments within those clusters. However, the AifL pilot schools, which were the focus of the evaluation, were not working in clusters as anticipated, and coverage of all projects was difficult to establish. Therefore a spread of schools across more authorities focusing on Projects 4, 5, 8, 9 and 10 was targeted.

It was proposed that the second phase of the evaluation (2004) would focus on the use of the outcomes of the projects. Project specifications had indicated the production of guidelines, exemplification and dissemination of good practice as outcomes. So, for example, Projects 2 and 3 ( PLPs) were to produce 'the design of a single recording framework' and 'guidance on managing the record-keeping process'; Project 9 had to make use of these to produce a 'reporting framework and format which links to the records within the PLPs'. The original timescales suggested that these products would be available for wider use by the end of 2003. Schools were producing case studies and examples of locally designed PLPs, but it was not possible within the timescales to produce material for wider dissemination. As noted in more detail in Chapter 3 of this report, the reality was that schools needed more time to encompass developments and produce their case studies and, as the growth process was emphasised as being important, deadlines were treated flexibly; additionally, it was proving particularly difficult to agree a framework for PLPs. A consultation document on a PLP framework was issued in September 2004.

LT Scotland were commissioned to draw together the materials from the case studies and to prepare a toolkit ( www.LTScotland.org.uk/assess ) to enable wider understanding of the issues emerging from the programme and to enable schools to implement the ideas. This web-based toolkit was not launched until September 2004 and therefore it was not feasible to include its use within the evaluation project. Interviews and questionnaires in the second phase of the evaluation did focus on the extent to which participants were continuing developments of their own projects, their awareness of the outcomes of other projects and the extent to which they were taking on board wider AifL developments.

As a result of the separate evaluation of Projects 2 and 3 and the slower than anticipated development of PLPs, less emphasis has been given to objective (a) of the QIE evaluation: 'assess the extent to which the Personal Learning Plan developed within the programme is considered by pupils, parents, teachers and other education professionals to meet their perceived information needs'.

A further change to the evaluation design was the additional survey to Associated Schools Groups in the autumn of 2004. As explained above (p10) these were introduced during 2003 to 2004 to take forward developments in local authorities. They were not part of the original design of the programme and therefore were not included in the evaluation proposal. This illustrates the responsive nature of programme developments and hence the need for the evaluation team to be responsive to what should be the focus of the evaluation. The original design included further work with the case study authority schools at this stage, but with the agreement of SEED this was replaced by the ASG survey.

Thus, while the evaluation of the AifL Programme had a set of specific aims and a design to address these, it remained responsive to changes in the overall programme and plans were modified as appropriate. This reflects the philosophy behind the programme itself which gave considerable freedom to schools and teachers to develop practice within their own context at a pace and in a manner that reflected local needs. Regular meetings with SEED personnel reviewed progress and determined next steps in the evaluation process. An interim report was submitted in December 2003.

Sequence of evaluation activities

The evaluation was undertaken in two key phases to reflect the phases of the AifL Programme and school involvement. The sequence of activities undertaken by the evaluation team is outlined in Table 2.3 (January - December 2003) and Table 2.4 (January - December 2004). The final column in both figures indicates when reports were presented to SEED.