The Scottish Government

« Previous | Contents | Next »

Listen

Assessment of Achievement Programme
Report of the First AAP Survey of Social Subjects Enquiry Skills (2002)

1. Introduction

1.1 Overview

The Assessment of Achievement Programme (AAP) has broken new ground with its 2002 survey of Social Subjects Enquiry Skills. For this is the first time that a national survey of pupil attainment has been carried out in Scotland in this area of the curriculum. It is also the first time that a core skills survey has been conducted in parallel with a subject survey.

The aims of the survey were:

• to assess the attainment of pupils in P3, P5, P7 and S2 in social subjects enquiry skills and in core skills;
• to compare attainment across the four stages and between boys and girls;
• to report attainment in terms of 5-14 levels whenever possible;
• to provide a learning context against which to reflect on the attainment findings.

The survey focused on pupils in mainstream schools across Scotland (special schools were excluded), and testing took place in May and June 2002. Schools and pupils were randomly selected for survey involvement in such a way that the tested pupils represented the national cohort at the stage concerned. In total, just under 10000 pupils in just under 600 schools took part in the survey: 2536 P3 pupils in 152 primary schools, 2668 P5 pupils in 158 primary schools, 2340 P7 pupils in 137 primary schools, and 2341 S2 pupils in 134 secondary schools. Full details of the sampling strategy and the resulting samples are given in Appendix B.

All the survey pupils attempted written tasks and a subsample of pupils also attempted practical tasks or took part in group discussions or one-to-one interviews. To provide a context for the attainment results, some of the participating schools at P7 and S2 - those not involved in the practical assessments - were invited to complete a questionnaire about the resources available to them for this curriculum area and about their social subjects provision, and their sample pupils were invited to complete questionnaires about their social subjects learning experience.

1.2 The assessment framework

1.2.1 Social Subjects Enquiry Skills

The principal and subsidiary strands in the 5-14 enquiry skills framework ¹ are shown below:

Preparing for tasks:

• Planning tasks in a systematic and logical way
• Identifying appropriate sources of information

Carrying out tasks:

• Selecting relevant information and/or equipment: observe, measure, find, select, record
• Processing information in a variety of ways
• Evaluating the usefulness and reliability of information

Reviewing and reporting tasks ²:

• Presenting findings in an appropriate and coherent way
• Presenting conclusions that are relevant to the given purpose or issue

Subject knowledge was not intended to be assessed in the survey, and every effort was made during the development of assessment tasks to minimise dependence on prior knowledge of specific facts. The tasks were, though, set in appropriate contexts, drawing ideas from the three 'knowledge and understanding' outcomes defined in the curriculum guidelines. These outcomes and their respective content strands are shown below:

People in the past:

• People, events and societies of significance in the past
• Change and continuity, cause and effect
• Time and historical sequence
• The nature of historical evidence

People and place:

• Using maps
• The physical environment
• The human environment
• Human-physical interactions

People in society:

• People and needs in society
• Rules, rights and responsibilities in society
• Conflict and decision making in society

1.2.2 Core Skills

Pupil performance in the core skills was also explored within the survey:

• Communication
• Numeracy
• Using information technology
• Problem solving
• Working with others

There is clearly considerable overlap between core skills and social subjects enquiry skills, and core skills demands were deliberately incorporated into social subject tasks whenever possible.

1.3 Task development

1.3.1 Design decisions

It was inevitable that within the context of a national survey the tasks used to assess enquiry skills would appear contrived to some extent. This is because such assessment is constrained by the need to minimise disruption to normal classroom activity whilst also maintaining control over the assessment process itself. It is clearly impossible in this context faithfully to reflect the richness of holistic social subject enquiries as engaged in by pupils over much longer periods of time during their normal schooling. It would never be possible to follow individual pupils through the entire process of an investigation, from initial problem identification, through planning, carrying out, evaluating and reporting - a chronological process reflected structurally in the enquiry skills framework given earlier.

Instead, the survey focused on parts of the investigative process, and attempted to explore and report pupils' performances on the parts rather than on the whole. It is arguable that performance on a particular aspect of a holistic investigation, such as identifying appropriate sources of information or presenting findings, might be more successful when a pupil is actively engaged in a real enquiry over a period of time, with teacher support, than it might be when the aspect is divorced from its normal context, performance is timed, and is without peer or teacher support. That said, the assessment tasks developed for use in the survey are authentic in terms of focus and demand, as well as being imaginative and motivating.

Before task development began in earnest, a Reference Group was established to help to identify appropriate modes of assessment and task types, and to work intensively as task developers. The Group comprised teacher trainers working in the field of social subjects, core skills specialists, practising primary and social subjects teachers, and staff from the then 5-14 Assessment Unit at the Scottish Qualifications Authority (SQA) (see Appendix A for membership details). All members had recent experience in the development of assessment tasks at various levels and across all three social subjects. The group included people with experience of setting at Standard grade for SQA, and others who had set and quality assured materials for the National Qualifications Assessment Bank.

Group members were brought together for a 3-day workshop in November 2001, to debate appropriate modes of assessment and types of task, and to launch the intensive phase of active task development. Also at the meeting were a small number of individuals who had long experience of task development in English Language and Mathematics, who would focus principally on core skills.

Preparing for tasks posed particular challenges. It was agreed that it would be difficult in such a survey context to assess planning skills with a high degree of validity. This is both because planning is an activity that requires time for thought and reflection, and because plans often evolve naturally as pupils, and indeed people in general, progress through the implementation phase of an enquiry. There was nevertheless agreement that some attempt should be made to assess planning skills, because of their importance in social subjects enquiry. There was strong support for attempting the assessment of planning skills through one-to-one interaction with trained assessors, and during the course of group problem solving activities.

It was the view of the Reference Group that most of the skills involved in Carrying out tasks could be assessed to some extent through pencil and paper testing. This is because these skills for the most part concern the location and processing of information from various sources, and many common sources of information lend themselves to paper-based presentation: for example, texts, charts and graphs, maps, diagrams and photographs. However, there was also a generally held view that this innovative survey should attempt to reflect as closely as possible the wide range of information sources to which children are regularly exposed during their everyday social subjects education in schools. The range includes a variety of non-print sources, such as video clips, audio clips, artefacts, websites and measuring equipment. For logistic reasons it would be difficult to feature these alternative kinds of information source in pencil and paper testing sessions. Therefore, in order to embrace this larger variety of information source in the survey, it was agreed that some assessment tasks would be developed for practical administration by field officers.

As far as Reviewing and reporting tasks is concerned, the general feeling of the Reference Group was that the skills of presenting findings in appropriate ways and of presenting conclusions 'relevant to the given purpose or issue' could be assessed to some extent in written tests, but that such assessment could usefully by complemented by assessment during practical group enquiry activities. In the event, and for a number of reasons, this strand was not assessed.

There was confidence among Reference Group members and the core skills specialists that the skills of Communication (reading and writing) and Numeracy could be invoked and assessed quite naturally within the pencil and paper enquiry skills tasks. There would also be opportunities for the assessment of oral communication skills during the practical testing - in tasks featuring audio and video clips as information sources - as well as in the one-to-one interactions with field officers and in group problem solving activities. The skills involved in Problem solving and Working with others could be assessed as pupils interacted in small groups, working on specially designed assessment tasks under the observation of a trained assessor, with checklists used to record pupil performance and apply assessment criteria.

The skills involved in Using information technology could, it was agreed, be assessed through tasks administered in a practical context, and in one-to-one sessions in which individual pupils would be observed by a field officer. The field officer, it was recognised, who would also need to be a technical trouble shooter prepared to deal with inevitable computer hitches. It was accepted that specially prepared CDROMs would probably feature, and that again checklists would be used to record pupil performance, with assessment criteria instantly applied.

With regard to the level-based reporting of attainment, it was agreed that for Carrying out tasks the assessment of pupils' enquiry skills would be at Levels A and B at P3, Levels B and C at P5, Levels C and D at P7, and Levels D and E at S2. In other words, assessment would focus, at P7 and S2 at least, on the target level for the stage and the level below target level. The nature of the activities and the constraints of survey testing precluded this degree of formality in reporting Planning for tasks. Pupil performance in this case, and in core skills, would inevitably, therefore, be less formally reported.

1.3.2 Pencil & paper tasks

Task development began during the November Reference Group workshop and continued in earnest during the weeks that followed. Priority was given to development of the written enquiry skills tasks intended to assess pupils' skills in Carrying out tasks.

The Group had agreed that each task was to be at a single level, and should be designed to require up to 45 minutes of testing time at P3 and P5, and up to 60 minutes at P7 and S2. Every task was to take the same general form: three subtasks requiring pupils to respond to a small number of short-response items by retrieving information from given sources, with each subtask featuring a different type of information source. One subtask would feature a text (the text-based reading subtask), the second a table or graph (the 'numeracy' subtask), and the third a drawing, map, street plan, chronology chart, or whatever (the 'other forms' subtask, requiring pupils to use information presented in other source types). Where computation items were included in the numeracy subtasks, these were to be at or below the level of the parent task

From Level B up, supplementary questions would require pupils to pull relevant information together from all three sources, perhaps to re-present this in some way or to support or refute a given inference. The three subtasks were to be presented in a single test booklet, which would also contain a writing task in which pupils were to summarise in some way what they had learned after working with all three information sources.

The specified content strands in the knowledge and understanding outcomes for Social Subjects were to be used to provide appropriate contexts for the skills assessment. This strategy was adopted purely to ensure a balanced representation of the social subjects through the vehicle of task contextualisation, and there was no attempt to assess factual knowledge per se - indeed, every attempt was made to minimise the influence of any such knowledge on the skills assessment.

1.3.3 Practical tasks and interviews

Development of the practical tasks and interviews took place during March and April 2002, once the pencil and paper enquiry skills tasks had been drafted. In principle, the practical tasks were to relate to one or other of the three skills strands - Preparing for tasks, Carrying out tasks and Reviewing and Reporting on tasks - and/or to core skills. Once again, tasks were devised to cover the three social subjects in terms of context.

In the event, five main types of practical activity featured in the survey:

• Group discussions
• One-to-one interviews
• ICT-based tasks
• Video-based tasks
• Reference skills tasks

Group discussions

Group discussions were designed to assess how well a small group of pupils could work together in a problem solving situation, and related to the core skills Working with others and Problem Solving.

One-to-one interviews

Individual interviews were designed to explore pupils' skills in Preparing for tasks. A photograph served as the stimulus for discussion, and formed the basis for a series of questions posed by a field officer.

ICT-based tasks

A number of tasks were developed to assess pupils' ICT skills in a social subjects context. Tasks were CDROM-based, and activities included modifying graphics and editing text, completing a spreadsheet, and searching the web (simulated).

Video-based tasks

Focusing on the strand Carrying out tasks, each of these tasks was based on a short 3-4 minute video clip, and had an associated worksheet. Some of the questions targeted information obtainable only from the video image, while others could only be answered from the soundtrack.

Reference skills tasks

These tasks, focusing on Preparing for tasks and Carrying out tasks, were designed to assess how well pupils can find information from reference books. Pupils were provided with front and back covers, contents lists, indexes and glossaries from two different books, as well as selected extracts from the content, and answer questions about these.

1.4 Task administration and marking

1.4.1 Pencil and paper tasks

A total of 45 different pencil and paper enquiry skills tasks were developed for use in the survey, three per social subjects outcome at each of five levels. Each task was presented in the form of a test booklet, with the relevant information sources contained in a separate booklet.

All the survey pupils were involved in the written assessment - typically 18 pupils in each school (some primary schools with low stage rolls supplied fewer sample pupils than this). Booklets were distributed to schools in such a way that every school received all the relevant tasks for their pupil stage. At most two pupils in each school, one boy and one girl, attempted any particular task combination, and for any individual pupil this task combination comprised two different tasks from two different social subjects outcomes and at two different levels - Levels A and B at P3, Levels B and C at P5, Levels C and D at P7, and Levels D and E at S2 (see Appendix C for booklet administration details). Thus, a P7 pupil might first try a Level C task featuring People and place and then a Level D task featuring People in the past (the lower-level task was always the first to be administered).

The pupils' own teachers organised and supervised the test sessions. Once the testing had taken place the survey schools returned the completed scripts to SEED for marking (see Chapter 2 for details).

1.4.2 Practical tasks and interviews

The practical tasks and interviews were administered in a subset of the survey schools at each stage: about 40% of the survey schools at P3 and P5, and around 30% at P7 and S2. In each 'practical' school, it was intended that all the pupils taking written tests would also participate in an interview or group discussion or would attempt practical tasks. Typically, in each school three or four pupils participated in a group discussion, four more were individually interviewed, three or four attempted an ICT task, three attempted a reference task and six attempted a video-based task.

Practising primary and secondary social subjects teachers worked as field officers in the survey. They underwent one day of training in June 2002, in four groups of about 25 at two locations in central Scotland (see Chapter 3 for further detail). They then worked in pairs, spending one day in each of their assigned schools, setting up and supervising practical sessions, and managing individual interviews and group discussions. They simultaneously rated pupils, using checklists to record observations and judgments, eliminating any need for post-survey marking.

1.5 Reporting pupil attainment

Pupils' performance when Carrying out tasks is reported in terms of the percentage of pupils at each stage who were deemed to have attained specific 5-14 levels. Since each of the 45 pencil and paper tasks used to assess this strand was designed to be at a particular level, it was possible to use the same reporting strategy for this survey as was used in the previous 2001 survey of English language ³. Three different criteria were adopted to decide whether or not a pupil attempting a particular task had attained the level of the task: pupils achieving 50% or more of the total marks on the task but fewer than 65% were classified as having demonstrated basic skills at the level; pupils achieving at least 65% of the available marks but less than 80% were classified as being secure at the level; pupils scoring 80% or more of the marks were classified as having considerable strengths at the level concerned ⁴.

Obviously there are measurement errors associated with these classification decisions. An individual pupil might fail to achieve 50% of the marks on a Level B task, and be deemed not yet to have demonstrated basic skills at this level, whereas had that pupil been given a different Level B task, or the same task on a different occasion, more than 50% of the marks might have been scored. Averaging attainment decisions over very many pupils and several Level B tasks serves to minimise the overall effect of decision error on the final attainment estimates for the whole pupil sample.

Writing attainment is also reported in terms of pupils' level attainment, but for Preparing for tasks and for the other core skills, performance is generally simply averaged over tasks, with no attempt to report level attainment in any formal way.

The attainment results are presented in Chapters 2, 3 and 4. Chapter 2 focuses on the enquiry skills written assessment, Chapter 3 presents the results of the enquiry skills practical assessment, and Chapter 4 presents the results of the core skills assessment. Finally, Chapter 5 reviews the questionnaire responses of the P7 and S2 pupils with regard to their social subjects school experience, and Chapter 6 presents salient findings that resulted from the school questionnaire enquiries. Chapter 7 summarises the survey, and addresses some of the issues arising.

« Previous | Contents | Next »