The Scottish Government

Content relevant to the various strands within each Knowledge and understanding outcome was used to provide appropriate contexts for the tasks, as noted in Chapter 1. This strategy was adopted purely to ensure a balanced representation of the social subjects through the vehicle of task contextualisation. Every attempt was made to minimise the influence of prior factual knowledge on the skills assessment. For People in the past the content strands are 'People, events and societies of significance in the past', 'Change and continuity, cause and effect', 'Time and historical sequence' and 'The nature of historical evidence'. For People and place they are 'Using maps', 'The physical environment', 'The human environment' and 'Human-physical interactions'. Finally, for People in society we have 'People and needs in society', 'Rules, rights and responsibilities in society' and 'Conflict and decision making in society'. The resulting variety of topics featured in the set of 45 tasks is illustrated in the task titles (see Table 2.1).

Tasks were designed to require up to 45 minutes of testing time at P3 and P5, and up to 60 minutes at P7 and S2. Each task took the same general form: three subtasks, each requiring pupils to respond to a small number of questions by retrieving information from one or other of three different information sources. The three information sources in each task included a text, a table or chart, and one other form, which might be, among a variety of possibilities, a map, a drawing, a street plan, a chronology chart, a photograph or a poster. At Levels B to E, a fourth subtask required pupils to pull together the information from all three sources - perhaps to evaluate relevance or to support or refute a given inference. Table 2.2 provides three example tasks, one each at Levels A, C and E and from the three different outcomes.

The test questions

The number of test questions that featured in each task was 22 at Level A, 24 at Levels B and C, 26 at Level D and 28-30 at Level E. At all levels the questions required a minimal written response that could readily be marked right or wrong, appropriate or inappropriate. Typically, a variety of different types of question were used in each task, selected from the list below.

Summary completion

Many of the assessment activities relating to texts involved the transfer of information in the form of a summary completion. Here, a summary of part or all of the text is written to reflect its content and purpose. Some of the principal meaning-bearing words in the summary are then deleted. The pupils, having read the passage, are required to fill the gaps with any word(s) that recreate the meaning of the original text. An understanding of both syntactic and semantic structures is required to complete the summary task.

Sentence completion

Here, pupils complete sentences either by providing appropriate words to fill word gaps, which could be in the body of the sentence or at the end of it, or by ending the sentence appropriately by selecting from three or four given phrase options.

Short answer questions

These would typically take the form "How many…?", "How far…?", "When…?, "What proportion…?", with pupils responding by extracting specific pieces of information from any of the different types of information source, viz. texts, tables, charts, drawings, maps, diagrams, etc.

Multiple-choice

A variety of multiple-choice techniques was used, including selecting one answer option from several given options, selecting two or more 'correct' statements from five or six, etc. Some of the sentence completion items described above could be classified into this question type.

Table and chart completion

Here, pupils are required to add additional information, counts for example, to a given table or chart (including tally sheets and pictographs at the lower levels).

Matching

Occasionally, pupils were asked to match attributes - for example, matching drawings of different types of vehicle to pie charts showing travel times over a given distance, or matching house styles to relevant climates.

True, False, Can't tell

Pupils judge each of a series of statements as true or false, or indicate, with "Can't say", that there is insufficient information in the text to permit such an evaluation. The "Can't say" option removes guesswork to some extent, as well as requiring a more accurate reading of the information given in the information source.

Open ended questions

These tended to appear in higher-level tasks, and generally took the form " Why do you think that….?" or "Give reasons for…".

The three tasks overviewed in Table 2.2 serve to illustrate how some of these question types were typically used.

2.1.3 Task development, validation and pre-testing

A team of task developers worked from mid-December 2001 to mid-February 2002 to create the tasks used in this survey. 'Shredding' meetings were organised by the task developers themselves during January, and draft tasks were submitted by mid-February to the then 5-14 Assessment Unit (FFAU) in the Scottish Qualifications Authority (SQA). SQA staff vetted and edited the tasks as they felt necessary, arranged for validation and organised small-scale piloting in schools.

A validation meeting was held in mid-March, during which the tasks were reviewed by practising primary and secondary social subjects teachers who had not been involved at any stage of task development. Given the novelty of level-based assessment in this area, combined with the complex nature of the lengthy tasks used here, level classification is not a straightforward activity. Nevertheless, the teacher groups reviewing the tasks at a particular level - the level to which the task developers had worked - were given the remit to confirm or question this level categorisation in all or individual cases, and to suggest possible improvements to task content. As a result, all tasks were confirmed on a group-agreement basis to be at the levels suggested by the task developers. Several suggestions were also made for modification to certain tasks.

Given the extremely curtailed timescale that was available for materials preparation for this innovative survey, it was not possible to follow normal procedures by trialling the assessment materials on a national scale. But the newly developed assessment materials were nevertheless pre-tested informally, by the task developers in their own or colleagues' schools. Their resulting experience of pupil reaction and response served as the basis for task refinement. The tasks were further pre-tested on a small scale by SQA staff before being finalised.

2.1.4 Task administration

Individual pupils were expected to attempt two tasks in the survey, at two different levels and relating to two different social subjects outcomes. The lower-level task was intended to be the first to be administered. Tasks were paired and randomly allocated to pupils in such a way that these intentions would be realised in the schools (see Appendix C). Thus, a P7 pupil might first have tried a Level C task featuring People and place, 'Scotland's Weather' perhaps, and then a Level D task featuring People in the past, such as 'Anne Frank'. The two tasks in each task pair were presented to pupils as a single test booklet, with the associated information sources under separate cover for easy reference.

At each stage, up to 18 pupils in each school took part in the written testing (in small primary schools fewer than 18 would be available). The schools organised their own assessment sessions within the timescale they were given, which was mid-May to mid-June. Since pupils were to attempt two different tasks, schools were advised to organise two separate assessment sessions, with a break between. They had the freedom to organise the two sessions to take place on the same day or on different days within the given period. The assessment sessions were supervised by the pupils' own class teachers, or by another teacher chosen by the head teacher of the school. The supervising teacher could explain what had to be done, but was not to provide answers or confirm that a pupil's answers were correct. The sessions were not necessarily timed, but it was expected that they would vary from about 30-40 minutes at P3 to 50-60 minutes at S2, allowing time for the writing task that was included in every booklet as well as for the enquiry skills task itself. Once a school's scripts were completed they were sent to SEED for marking (see below).

In the event, almost 10000 pupils participated in the survey, around 2500 at each stage, and around 95% of them attempted two written tasks as planned.

2.1.5 Script marking

The pupils' scripts were marked centrally by a team of undergraduates, during marking meetings held in June/July 2002. While all the test questions would eventually be marked right or wrong, or appropriate/inappropriate, the markers were not always required to award marks directly themselves. In some cases - for example, the summary completion paragraphs - the markers did make judgments about appropriate mark allocations on the basis of given marking schemes. In other cases - for example multiple-choice questions - they simply noted the response option selected by the pupil, and this was later machine marked.

Scripts were distributed among markers in such a way that any one marker marked only a small number of scripts from any one booklet. This served to reduce the possible effect of differences in marker judgment on the results, for the small minority of test questions that were not of objective format. Checks for consistency were also carried out. For each task at least one script per marker was re-marked by a different marker, and the results compared. Any discrepancies were then immediately investigated. Where discrepancies revealed clear errors on the part of an individual marker, the marker in question was alerted to the problem. Where discrepancies suggested different interpretations of the marking scheme for the task concerned, clarification was immediately given to all the markers and, where necessary, the mark scheme was modified. Any clarifications or changes were carried forward when the same task was re-encountered at a higher stage.

2.1.6 Reporting enquiry skills attainment

The criteria used as the basis for reporting pupil attainment in the 2001 English language survey are re-adopted here. Pupils successfully answering 65% or more of the test questions in a task are classified as 'secure' at the level concerned, ie. as having attained the level. In addition, pupils successfully answering at least 50% of the questions but not as many as 65% are classified as having demonstrated 'basic' enquiry skills at the level concerned, while pupils successfully answering 80% or more of the questions in a task are classified as demonstrating considerable strengths at this level ⁵. The proportions of pupils in each classification group were first calculated for each task separately, and then averaged over the tasks at a level to produce the national estimates of attainment presented in this chapter.

2.2 Overview of pupils' attainments