The Scottish Government

« Previous | Contents | Next »

Listen

Assessment of Achievement Programme: Report of the Sixth AAP Survey of Science (2003)

1. Introduction

1.1 Overview

The 2003 Science Survey was the sixth survey to be carried out in this curriculum area within the Assessment of Achievement Programme (AAP), and the second subject survey to incorporate the assessment of core skills attainment - the first being the Social Subjects Enquiry Skills Survey of 2002. The aims of the survey were to:

• assess the science attainment of pupils in P3, P5, P7 and S2;
• assess the core skills attainment of pupils in P3, P5, P7 and S2;
• compare attainment across the four stages and between boys and girls;
• report attainment in terms of 5-14 levels whenever possible;
• explore pupils' informed attitudes in science;
• provide a learning context against which to reflect on the attainment and attitude findings.

The 2003 survey differed from previous AAP surveys in science in three principal ways: pupils' Knowledge and understanding in science is reported with reference to 5-14 levels; science attainments have been assessed for pupils as young as P3; and an attempt has been made to explore the development of informed attitudes in science. Two important events led to the first two of these innovations. Firstly, in 2000 the National Guidelines for Environmental Studies were revised ¹, with the result that Knowledge and understanding in science is now, like other outcomes in this and other subjects, organised into strands, levels and attainment targets rather than key features and stage bands. Secondly, in 2002 Social Subjects joined English Language, Mathematics and Science in the AAP's subject coverage, demanding a shift from a 3-year to a 4-year subject cycle and inviting a parallel shift in focus from P4, P7 and S2 to P3, P5, P7 and S2.

The survey involved pupils in maintained and independent schools across Scotland (special schools were excluded), and testing took place in May and June 2003. Schools and pupils were randomly selected for survey participation in such a way that the tested pupils represented the national cohort at the stage concerned. In total, just under 9000 pupils in just under 600 schools took part in the survey: around 2000 pupils in around 155 primary schools at each primary stage, and just over 2000 S2 pupils in 130 secondary schools. Just over half the pupils attempted pencil and paper science tasks, while the remainder participated in core skills reading and writing assessment. The 'science' pupils in over half the survey schools also took part in practical science assessments, while the 'core skills' pupils in these same 'practical' schools also undertook ICT tasks or participated in group discussions. Full details of the sampling strategy and the resulting samples are given in Appendix B.

To provide a context for the attainment and attitude findings, samples of participating schools at P5, P7 and S2 were invited to complete questionnaires about the resources available to them for science and about their science provision, and their sample pupils were invited to complete questionnaires seeking their views about their science learning experience and exploring their attitudes to topical issues in science and technology.

1.2 The assessment framework and approaches to assessment

The assessment framework for the 2003 survey was in part established by a Science Reference Group, which comprised subject specialists, practising primary and secondary science teachers and other education professionals, and which met for the first time in August 2002.

1.2.1 Knowledge and understanding

The three Knowledge and understanding attainment outcomes in the 5-14 Environmental Studies framework are shown below, with their respective strands:

Understanding Earth and Space:

• Earth in space
• Materials from Earth
• Changing materials

Understanding Energy and Forces:

• Properties and uses of energy
• Conversion and transfer of energy
• Forces and their effects

Understanding Living Things and the Processes of Life:

• Variety and characteristic features
• The processes of life
• Interaction of living things with their environment

The survey was designed to assess and to report P3 attainment at Levels A and B, P5 attainment at Levels B and C, P7 attainment at Levels C, D and E, and S2 attainment at Levels D, E and F, for science as a whole.

Given the various constraints that national attainment surveys inevitably operate within, it would not be possible to assess and to report pupils' level-based attainment for each outcome separately. However, following the advice of the Reference Group, the three outcomes were equally covered within the entire set of tasks administered to pupils in the survey, so that some comment on comparative performance is possible.

Over time attainment comparisons are offered on the basis of tasks previously administered in the 1999 survey and re-used on this occasion. Given that P4 no longer features in the AAP survey programme, and given that Level F did not feature at S2 in the 1999 survey, it was accepted that temporal comparisons would be possible at P7 and S2 only, at Levels C (P7 only), D and E.

1.2.2 Skills in science - investigating

The three strands that comprise investigation skills in science are:

• Preparing for tasks
• Carrying out tasks
• Reviewing and reporting on tasks

In the 1999 science survey, pupils' skills attainment in science was explored using pencil and paper tasks ². For the 2003 survey, the Reference Group strongly advised that this strategy be abandoned in favour of skills exploration through pupil participation in purposeful practical investigations set in meaningful contexts. One consequence of this decision, given the financial and logistic challenges associated with implementing practical assessment on a national scale in schools, is that pupils' skills attainment at each stage is reported here on the basis of relatively few practical investigations. Another consequence is that there can be no comparison of skills attainment between 1999 and 2003.

1.2.3 Developing informed attitudes

Three strands comprise this outcome in the National Guidelines:

• A commitment to learning
• Respect and care for self and others
• Social and environmental responsibility

Pupils' attitudes to topics and activities in science have been explored in earlier surveys, and were explored again here. In this survey the pupil questionnaire enquiry incorporated an exploration into pupils' knowledge of and views about topical issues in science. In addition, a deliberate attempt was made to look specifically at the development of informed attitudes, an outcome of science education included in the National Guidelines but never before explored in any systematic way within a national survey. The Reference Group agreed that it was appropriate to begin addressing what up until now has been a gap in survey coverage, but at the same time advised against any attempt to assess the informed attitudes of individual pupils, even for the purpose of contributing anonymously to a population estimate. For this and other reasons the second vehicle for informed attitude exploration was focus group discussion.

1.2.4 Core skills

Pupil performance in the following core skills was also explored within the survey:

• Communication (reading and writing)
• Numeracy
• Using information technology
• Problem solving

The survey was explicitly designed to assess and report pupils' reading and writing attainments, along with their numeracy and ICT skills, while problem solving would be addressed naturally through performance in the science investigation tasks. Whenever possible, attainment in the core skills was assessed within a science context.

1.3 Task development

1.3.1 Knowledge and understanding

Pupils' Knowledge and understanding in science was assessed using relatively short pencil and paper tasks. The survey design called for a total of 360 such tasks, comprising 60 tasks at each of Levels A to F, with an equal representation of the three outcomes. AAP tasks that had been used in the 1999 survey furnished around one-fifth of the required tasks. Following the National Guidelines in operation in 1999, each task used in the survey that year had been classified into one of the three attainment outcomes and into one of three stage bands: P1-P3, P4-P6, P7-S2. In late 2002, in preparation for the 2003 survey, these tasks were reviewed. They were judged for continued framework relevance, revised where necessary to improve and standardise layout, and newly classified into 5-14 levels. The result was an expected shortfall in required numbers at every level, along with an outcome imbalance.

An appeal was made to the exemplification materials developed by a SEED-funded project team working in the then Department of Curriculum Studies in Northern College, now the Faculty of Curriculum Studies in the University of Aberdeen, to provide a further source of assessment tasks ³. The group that had produced the materials comprised university academics and practising teachers. Following the brief they were given, the task developers designed tasks for use by teachers in classrooms for formative purposes. Not all the tasks were therefore in a suitable form for direct use in a large-scale time-constrained survey such as this one, and none had been subjected to formal large-scale field trialling. Nevertheless, tasks from this source, often in modified form, served to fill the gaps.

1.3.2 Practical science investigations

Nine practical investigation tasks were newly developed for survey use, by a group of six teachers supervised by a staff member in the Faculty of Education in the University of Strathclyde. Each task was designed to allow pupils, working in groups, to plan an investigation, carry out the planned investigation, and report/review the investigation.

Individual tasks spanned two adjacent levels, in the sense that checklists were developed that in principle enabled assessors to classify pupils into one level or the other for each relevant attainment target, on the basis of their investigative behaviour. With the exception of Levels E/F, for which there was a single task, there were two tasks for every pair of adjacent levels. The investigations were trialled informally by the developers, with small numbers of pupils in local schools. Full-scale piloting occurred within the survey itself.

1.3.3 Reading and writing

New task development for reading and writing was organised by the Scottish Qualifications Authority (SQA). Fifteen reading tasks, three at each of Levels A to E, were developed by practising teachers for survey use. Each task comprised a text and a series of comprehension questions, resembling in form the reading tasks typically used in AAP English Language surveys and in National Assessments. Task topics were 'scientific', e.g. 'Seeds', 'Electricity' and 'Recycling', and topic selection ensured balance across the three Knowledge and understanding outcomes at each level. Any dependence on prior topic knowledge was minimised. The texts had been selected and the tasks developed to fit particular 5-14 levels. Level classifications were independently confirmed by teacher validators, and the tasks were put through national pre-tests before survey use.

At Levels C, D and E, the reading tasks were each complemented by an associated writing task, which focused on the same science-relevant topic.

1.3.4 Numeracy

The 15 numeracy tasks used in this survey were tasks that had previously been developed for use in the 2002 Social Subjects survey, or were adaptations of these. In that previous survey, numeracy tasks were embedded within larger level-based enquiry skills tasks, and related to the overall themes of the parent tasks ⁴, which might be 'Egyptians', 'River study', 'Advertising', or whatever. The tasks typically presented pupils with tables or charts, and required them to transfer some or all of the given information into different presentational forms, or to use it to answer simple questions, sometimes involving calculation.

Some of the numeracy tasks were re-usable in the science survey without modification. In others, context and content that were not 'scientific' enough could be modified without changing task format or skill demands. Thus, 'River study' became 'Water quality', with no changes to the task itself, 'Egyptians' became 'Bird table', with bird species replacing tomb jewellery in graph and questions, and 'Advertising' became 'Food complaints', otherwise unchanged.

1.3.5 ICT tasks

Six ICT tasks were commissioned from an independent consultant, and installed on CDs by Learning and Teaching Scotland (LTS). The interactive tasks featured exercises in word processing, spreadsheet and database use, internet searching and web page browsing (using simulated CD-held web pages). Three tasks were set in science-relevant contexts appropriate to pupils in P3 and P5, and focused on ICT skills at Levels B and C in the ICT National Guidelines ⁵, while the other three were set in contexts appropriate for pupils in P7 and S2, and demanded ICT skills at Levels D and E.

The development of these tasks benefited from previous experience, since similar tasks had been developed and used in the 2002 Social Subjects survey, and lessons were learned from the successes and failures of that survey. However, there was insufficient time available between task development and survey implementation for formal pre-testing, so that these tasks, like the science investigations, were essentially piloted within the survey itself.

1.3.6 Group discussions

Three topics were identified by SEED personnel as appropriate for 5-10 minute focus group discussions that might allow pupils to show evidence of informed attitudes in science. The topics had not only to be perceived by pupils as being 'scientific', but they would also have to hold interest for them and have some clear current relevance. The topics would also need to be capable of provoking discussion among pupils, allowing different points of view to be argued. This was no easy challenge, but the three topics finally selected were 'A school garden' (P3/P5), "Where have all the sparrows gone?" (P5/P7/S2), and 'Recycling' (P7/S2).

It was intended that trained field officers would introduce the discussion topics to small groups of pupils, and would animate and generally manage the ensuing discussions, rating pupils in real time for evidence of informed attitudes in science, using criteria-based checklists.

1.4 Task administration and marking

All the survey pupils were involved in some form of written assessment, and around half also participated in some form of practical assessment. The survey design anticipated that wherever possible 10 pupils in each school - the 'science' sample pupils - would be involved in the assessment of Knowledge and understanding in science, while a further eight pupils - the 'core skills' sample pupils - would be involved in the assessment of reading and, at Levels C, D and E, writing. In a subset of schools - the 'practical' schools - those pupils who had attempted pencil and paper science tasks would also take part in practical science investigations, while those pupils who had attempted reading and writing tasks would also attempt an ICT task or participate in a group discussion. Typically, therefore, 18 pupils in each school were involved in one way or another in the survey.

The pupils' own teachers organised and supervised written test sessions, but the practical assessments were the responsibility of itinerant field officers (see section 1.4.3). Once the written testing had taken place, the survey schools returned completed scripts to SEED for processing and marking. Field officers completed their assessments of pupils as they engaged in the practical activities.

1.4.1 Knowledge and understanding

Pupils at P3 and P5 were to be assessed at two levels only (A and B for P3; B and C for P5), whereas pupils at P7 and S2 were to be assessed at three levels (C, D and E at P7; D, E and F at S2). However, there was no possibility here of targeting tests to pupils, in such a way that every pupil would be given a test at or close to that pupil's current level. Rather than risk any possible distress should a pupil be faced with a single-level test at too high a level, or boredom and demotivation when faced with a single-level test at too low a level, it was decided to mix levels within each test booklet. For similar reasons it was also decided that every booklet would feature all three outcomes. In addition, each booklet would contain a single numeracy task.

The 360 Knowledge and understanding tasks, and the 15 numeracy tasks, were therefore distributed among a number of different test booklets at each stage in such a way that every pupil was faced with a collection of tasks from the two or three levels appropriate for their stage assessment, and spanning all three outcomes (see Appendix B for further detail). At P3 and P5, the test booklets comprised 12 science tasks each, plus a numeracy task, and were expected to take 30-40 minutes to complete. At P7 and S2, test booklets were longer, at 18 science tasks plus a numeracy task, and were expected to take 50-60 minutes to complete. Individual pupils were intended to attempt two different booklets, with a time interval between test sessions to be decided by the school.

Completed scripts were processed centrally in a series of 'response transcription' meetings organised by SEED. In these meetings, pupils' responses were transferred onto specially designed transcription sheets, for later keyboarding and machine marking (Chapter 2 offers further details).

1.4.2 Reading and writing

Reading tasks at Levels A and B were also presented to pupils in the form of test booklets, one task forming a single booklet. At Levels C, D and E, reading tasks were accompanied by their associated writing tasks. There were thus six 'reading only' booklets and nine 'reading and writing' booklets in total, with three booklets at each level. It was intended that each pupil would attempt two different reading booklets, the two booklets being at different levels and featuring topics from two different Knowledge and understanding outcomes (though not requiring any prior topic knowledge). This meant that individual P3 pupils would attempt booklets at Levels A and B, and individual P5 pupils would attempt booklets at Levels B and C. At P7, pupils faced booklets at Levels C and D, while pupils at S2 attempted tasks at Levels D and E.

Reading booklets were designed to have the same general time requirement as the science booklets, and it was expected that schools would organise both types of assessment to occur in the same test sessions. Completed reading booklets, like completed science booklets, were processed centrally, with response transcription forms completed by students for later keyboarding and machine marking. Random samples of pupils' writing were selected for marking by the teachers who had served as field officers in the survey (further details are given in Chapter 4).

1.4.3 Practical tasks and group discussions

Just over half the survey schools at P3, P5 and P7, and approximately two-thirds of those at S2, took part in the practical assessments. Typically, in each school eight pupils undertook science investigations, in groups of four. In addition, three or four pupils participated in a group discussion, and up to eight pupils individually attempted ICT tasks.

A total of 28 authorities provided the 148 practising primary and secondary science teachers who served as field officers. The field officers worked in pairs, half at P3/P5 and half at P7/S2, spending five days visiting schools located within reasonable travelling distance of their home areas. They spent one day in each of their five assigned schools, setting up and supervising the practical sessions, managing the group discussions, and rating pupils on-the-spot, using checklists to record observations and judgments (Chapter 3 provides full details). In preparation for this work, the field officers were given a day of task orientation in the period early May to early June 2003. Their involvement in the practical assessment was formally ended in a debriefing day held in the period late May to late June 2003, but half of these practising teachers contributed further by volunteering to be involved in marking pupils' writing.

1.5 Reporting pupil attainment

Pupils' Knowledge and understanding attainment, as well as their reading and writing attainment, is reported in terms of the percentage of pupils at each stage who were deemed to have attained specific 5-14 levels. For Knowledge and understanding and for reading, attainment decisions were based on the application of judgmental cut-off scores, whereas pupils' writing was classified into an appropriate level through application of 'best fit' criteria. For investigation skills in science and for ICT skills, cut-off scores have not been applied to test scores, but pupil performance is instead averaged over test items or behaviours classified at particular levels, and average facility values or rating distributions are given.

The resulting attainment findings are presented in Chapters 2, 3 and 4. Chapter 2 focuses on the assessment of Knowledge and understanding, Chapter 3 presents the results of the skills assessment in science, and Chapter 4 presents the results of the core skills assessment.

Chapter 5 presents findings from the pupil questionnaire enquiries, including the assessment of pupils' informed attitudes, while Chapter 6 presents findings from the school questionnaire. Finally, Chapter 7 summarises and reflects on the principal findings of the survey, and addresses some of the issues raised.

« Previous | Contents | Next »