The Scottish Government

« Previous | Contents | Next »

Listen

Assessment of Achievement Programme: Report of the Sixth AAP Survey of Science (2003)

4. Core skills assessment

4.1 Introduction

The core skills of reading, writing, numeracy, problem solving and using information technology were assessed in this survey. Problem solving was assessed as an integral part of science investigation skills, and the findings are presented in Chapter 3. Here we describe the nature and outcomes of the assessment of reading, writing, numeracy and ICT skills, extending the overview given in Chapter 1.

Fifteen reading tasks were newly developed for use in the survey, three at each of Levels A to E. The 'scientific' topics that featured in the tasks represented in equal numbers the three Knowledge and understanding outcomes. At Levels C, D and E each reading task was accompanied by an associated writing task, which focused on the same topic. There were thus nine writing tasks in total.

Six different ICT tasks were also newly developed for use, each task carefully set in a 'scientific' context. These were administered in schools by trained field officers (see Chapter 1 for general information about the practical assessment). In addition, 15 numeracy tasks were administered in the survey. These originated from the 2002 Social Subjects survey, where they each formed an integral component in larger 'enquiry skills' tasks. Some of the numeracy tasks were used in identical form in the science survey, save for an occasional title change. Others were re-set in scientific contexts. The numeracy tasks were embedded in the science test booklets described in Chapter 2, one numeracy task per booklet.

All the survey schools - 155 primary schools at each stage and 130 secondary schools - participated in the assessment of reading, writing and numeracy. Each school provided up to 10 randomly selected pupils for numeracy assessment (these pupils were those who attempted the science test booklets), and up to eight randomly selected pupils for reading and writing assessment. Just over half the primary schools and two-thirds of the secondary schools, that is 80-90 schools at each stage, also participated in the assessment of pupils' ICT skills (and science investigation skills).

4.2 Reading

4.2.1 The reading tasks

Each of the 15 reading tasks administered in the survey took the same general form as those typically used in AAP English surveys, viz. a source text, or series of related texts, accompanied by a set of comprehension questions. In this case, however, the texts focused on science-relevant topics or issues, as the titles in Table 4.1 suggest. At Levels C, D and E, each reading task was accompanied by a writing task, focusing on the same topic. Table 4.2 describes one task at each of Levels A, C and E, and the writing tasks associated with them.

Table 4.1 Topics featured in the reading and writing tasks*

* Writing tasks accompanied reading tasks at Levels C, D and E only

Table 4.2 Overview of three reading tasks and two writing tasks

The number of test items varied across tasks, from 17-18 per task at Level A rising to 30-35 per task at Level E, and featured a variety of different formats, including multiple choice, summary completion, sentence completion, and short response. Items sharing common formats were grouped into sections, as illustrated in the task examples given in Table 4.2.

Reading tasks were classified by 5-14 level on the basis of the perceived level of their source texts. No prior science knowledge was needed in order to respond successfully to the tasks, although inevitably prior knowledge will occasionally have helped.

Tasks were presented to pupils in the form of test booklets. A single reading task comprised a booklet at Levels A and B, while a reading task and its associated writing task comprised a booklet at Levels C, D and E. Individual pupils attempted two test booklets, one at each of two consecutive levels: P3 pupils attempted reading tasks at Levels A and B, P5 pupils attempted tasks at Levels B and C, P7 pupils attempted tasks at Levels C and D, and S2 pupils attempted tasks at Levels D and E.

Completed booklets were processed centrally. Pupils' responses to the reading tasks were transferred onto specially designed transcription sheets for later keyboarding and machine marking, as described in Chapter 1. In total, just under 5000 scripts were analysed, emanating from over 2500 pupils in over 400 schools (some primary schools were too small to supply the 10 pupils needed for science assessment plus the eight pupils needed for core skills assessment, hence the lower numbers of schools involved here).

4.2.2 Pupils' reading attainment

The attainment results for reading were produced by applying the usual 65% cut-off score to the pupils' total task scores, as described in Chapter 1, and then averaging the proportions of 'secure' attainers over the three tasks at each level. The resulting proportions of pupils classified as 'secure' at the level concerned (the level of the task) are given in Table 4.3.

As Table 4.3 shows, reading attainment was relatively good at P3, with over 80% of the pupils demonstrating secure attainment at Level A, and over half demonstrating secure attainment at Level B. P5 attainment is a little lower at the levels assessed at this stage: 70% showed secure attainment at Level B and just under 40% at Level C.

Table 4.3 Reading attainment by stage ^*

(% pupils correctly answering 65% or more marks per task ,averaged over the three tasks at each level)

* Margins of error for the estimated proportions vary between 3 and 4 percentage points.

P7 and S2 performance is rather similar at the two levels assessed at these stages: around 60% of the P7 pupils showed secure attainment at Level C and around 40% did so at Level D, with the same picture emerging for S2 at Levels D and E. Interestingly, the reading results for P7 and S2 in this science survey are in line with the results reported for the AAP English Language survey conducted in 2001 (P3 and P5 did not feature in that survey), in which reading was assessed using very similar tasks but set in more everyday contexts (see Table 4.4). At Level E the attainment results are identical, at Level D they are close, and even at Level C the sample difference in favour of the English survey does not reach statistical significance.

Table 4.4 Reading attainment at P7 and S2: 2001 English survey and 2003 Core Skills survey

(% pupils correctly answering 65% or more marks per task, averaged over the tasks at each level*)

* Different tasks were used in the two surveys

Interestingly also, there is little evidence in Table 4.5 of the gender gap usually found in favour of girls in reading. Indeed, this time it is the boys who have produced the slightly better performances than the girls. None of the sample differences, however, reaches statistical significance.

Table 4.5 The reading attainments of boys and girls

(% pupils achieving 65% or more marks per task, averaged over three tasks at a level)

4.3 Writing

4.3.1 The writing tasks and writing evaluation

Each of the nine writing tasks was included in a reading booklet as a final "section", following the sections of test questions that explored pupils' understanding of the stimulus text (see the two examples described in Table 4.2). At P5, each pupil was invited to produce one piece of writing, relating to their Level C reading task. At P7 and S2 each pupil should have produced two pieces of writing, relating, respectively, to the two reading tasks they attempted at Levels C and D (P7) or Levels D and E (S2). This scale of testing resulted in a very large number of individual pieces of writing. In order to make the scale of writing evaluation manageable, a sample of just over 2500 pieces of writing was selected for writing evaluation. The sample was drawn at random, and included examples of writing from all nine tasks at the stages at which each individual task was used. Around 100 different schools at P5 and around 150 different schools at P7 and S2 each provided between one and 12 completed booklets, across all tasks at the relevant stage.

Pupils' writing productions were marked in autumn 2003, by 75 of the field officers who had undertaken the practical assessments in the survey. Since problems had arisen during the marking of writing in the 2002 Social Subjects survey, when an attempt was made to apply an adapted form of the National Criteria for writing, it was decided to use an holistic evaluation scheme here, based on 'best fit' descriptions (see Table 4.6).

The writing evaluators were given some minimal orientation for the writing evaluation activity after their post-survey debriefing. During the meeting they reviewed a small sample of scripts as a group, applying the holistic evaluation scheme. In the days following the meeting, and with access to exemplar materials on the SQA 5-14 web site, they each independently evaluated 50 scripts, drawn from several different tasks at a stage.

Because the evaluation of writing is inevitably a subjective process, the issue of rater agreement is always a concern. It cannot be assumed that different evaluators judging the same script will come to the same decision about writing relevance or quality, whatever the nature of the evaluation scheme that is used. In order to explore this issue here, a rater agreement study was incorporated into the general writing evaluation exercise, in that roughly every other piece of writing was independently evaluated by four different evaluators. The results of this exploratory study are shown in Figure 4.1.

Figure 4.1 Distribution of agreement rates over all evaluator pairs and writing tasks

As expected, it was relatively rare for all four evaluators to award the same attainment level to the same pieces of writing. The highest rate of unanimous agreement for any task was 18% and the lowest was zero. The average inter-rater agreement among pairs of evaluators, was 40% overall (the highest agreement rate per task was 60% and the lowest 7%); this fell to 33% for S2 writing, which was judged by secondary science teachers. Among other problems, the evaluators mentioned difficulty in judging sparse writing, difficulty in deciding content relevance, and difficulty in judging writing quality independently of the nature and accuracy of the relevant content. Averaging over all tasks, the proportion of scripts that could have been given a level classification based on 'majority vote' was 79%.

Table 4.6 The 'best fit' descriptions used to judge pupils' writing

4.3.2 Pupils' writing attainment

Table 4.7 presents the results of the writing evaluation exercise, while Figure 4.2 illustrates the picture of pupil attainment. The summary data are based on one single evaluator judgment per script. In the case of those scripts that were independently judged by more than one evaluator in the rater agreement study - just under half the total - the judgments of one only of the evaluators concerned have been used in these analyses, this individual having been selected at random.

In the event, the evaluators found that there was no writing to evaluate in almost 20% of the booklets they were asked to review. Just under 10% more of the scripts contained an insufficient amount of writing to permit a level judgment. The pupils concerned had presumably either run out of time before finishing their test booklets - the writing followed the reading - or were simply not motivated to make the effort to produce much, if any, writing. The presentation of the writing tasks as final sections within the reading tasks could have been a contributory factor here. Pupils might have viewed the task as an extended response reading task or as a restricted response writing task, and either way might not have felt inspired to generate much writing, or any writing, particularly under survey conditions. The proportions of pupils who failed to produce any writing at all varied across tasks, ranging from one in ten for 'Metals', a task used at P5 and P7, to around one in four for 'Light', one of the three tasks used at S2. Over all tasks, a higher proportion of the boys than of the girls produced no text at all (20% versus 14%).

An additional 4% of the scripts offered writing whose science content was considered irrelevant (while it was intended to be the quality of the writing that was judged, and not the accuracy or relevance of the content, the teacher evaluators, particularly the secondary science teachers, found it difficult to separate the two). The end result is that the evaluators were able to make judgments about writing quality for just 70% of the scripts they reviewed (see Table 4.7 and Figure 4.2).

Table 4.7 Writing attainment: Level distributions by stage and gender*

(% pupils classified into each 5-14 level, based on rateable scripts)

* Nine writing tasks featured in the survey: three tasks were used at P5 and P7, three more were used at P7 and S2, and three further tasks were used at S2 only.

Table 4.7 and Figure 4.2 provide clear evidence of stage progression in writing. While more than 40% of the rateable writing produced by P5 pupils was rated at Level A, the corresponding proportions at P7 and S2 are markedly lower, at fewer than 30% and 20%, respectively. The proportions rated at Level C or higher at the three stages are 16%, 34% and 51%, respectively. However, the data present a relatively bleak picture of writing attainment at the target levels for P7 and S2 pupils (Levels D and E, respectively): fewer than 10% of the rated scripts at P7 and S2 were judged to show evidence of writing at the target level or higher.

Figure 4.2 The picture of writing attainment P5 to S2 ^*

Interestingly, among those pupils who produced enough relevant writing to be given a level rating, there is no evidence in Table 4.7 of any gender difference in writing quality.

The fact that a high proportion of the survey pupils attempted both reading and writing tasks means that it is possible to explore the relationship between the two skills. Not surprisingly, given the rather poor performance the pupils in general demonstrated in writing, in the majority of cases pupils' judged writing attainments were lower than their reading attainments, with a minority showing the reverse pattern. Just over a quarter of the pupils were classified into the same level for both reading and writing.

4.4 Numeracy

4.4.1 The numeracy tasks

Each of the 45 pencil and paper enquiry skills tasks specially developed for use in the 2002 Social Subjects survey comprised three subtasks, between them featuring three different types of information resource: one task was based on a text, another focused typically on a table or chart of some kind (the 'numeracy' subtask), while the third featured a map, drawing, diagram, photograph or whatever. Among the numeracy subtasks, several were identified that could be used unchanged or that could be used after minor modifications to wording to render the context more 'scientific'.

This represented an opportunity to explore the possible effect of subject context on numeracy performance whilst at the same time providing information about numeracy skills within the science survey. In response, it was decided to seize the opportunity to capitalize on previous experience and to re-use some of the numeracy tasks. Fifteen different tasks were eventually selected for re-administration in the science survey, three at each of Levels A to E (the level of any numeracy task is simply the level of the original parent social subjects enquiry skills task - the skill demands in the numeracy tasks were designed to be at or below the level of the parent task). Table 4.8 overviews the tasks, while Table 4.9 offers brief descriptions of one task at each of Levels, A, C and E.

Every task comprised a number of atomistic test items, the number per task varying from four to ten. For the most part, the items required pupils to identify highest or lowest values, to count values, and to transfer values from one presentational form into another (chart to summary completion paragraph, table to bar chart, etc). In a very small minority of items, pupils were required to add or subtract values, to compare values, or to describe in words the trend shown in a set of data, usually in graphic form. Every item was scored dichotomously.

Table 4.8 The numeracy tasks

Table 4.9 Overview of three numeracy tasks

A numeracy task was inserted into every science test booklet. At P3 the numeracy tasks were at Levels A and B, at P5 they were at Levels B and C, at P7 they were at Levels C and D, and at S2 they were at Levels D and E. Pupils' responses to the tasks were processed as described in Chapter 1, and machine marked.

4.4.2 Pupils' numeracy performance

Table 4.10 and Figure 4.3 present the results of the numeracy assessment, in the form of average item facilities over all items and all pupils at each level and each stage (note that these data have not been produced by applying cut-off scores to test scores, and they do not therefore represent percentages of pupils attaining particular levels in numeracy).

Table 4.10 Numeracy performance P3 to S2

(average item facility over all items at a stage and level)

Figure 4.3 Numeracy attainment P3 to S2

(average item facility over all items and all pupils at a stage and level)

In general, Table 4.10 confirms that pupils' numeracy skills are good, the average item facility being over 50% in every case, reaching particularly high levels at P5 and P7 for the items at Levels B and C, respectively. Even where items demanded computational skills, facility values were relatively high: for example, item facilities of over 60% at P3 for simple pictograph counts in the Level A task 'Bird Table' (described in Table 4.9), 45-50% at P3 for three addition/subtraction items in the Level A tasks 'Travel Then and Now' and 'Keep Fit'; 55-65% at P3 and 75-85% at P5 for two addition items in the Level B task 'Children in Need'; 60-80% at P5 and 65-90% at P7 for four addition/subtraction items in the Level C tasks 'Nature Trail' and 'Water'; 35-55% at S2 for three subtraction items in the Level E task 'Food Complaints'.

The only cross-stage anomaly related to a pie chart item in the Level D task 'Genetically Modified Foods'. The pie chart showed the results of a fictitious opinion survey among a sample of 1000 people. Pupils were asked first what was the most popular opinion on GM foods, and were then asked what opinion on GM foods a quarter of the people held. These were well answered. Then came the calculation item: 'How many more preferred further testing to banning them altogether?'. This item required pupils to identify the proportion of individuals who were in favour of further testing (answer: 20%) along with the proportion who were in favour of banning GM foods altogether (answer: 12%), and then subtracting the one proportion from the other (in the answer line they were clued by the % symbol simply to difference the proportions - to get 8% - and not to calculate the actual number of people this would translate to). Over a third of the P7 pupils (37%) correctly answered the question compared with 28% of the pupils at S2 (the corresponding figures were 63% and 53%, respectively, in the social subjects survey); 39% of the P7 pupils and 56% of the S2 pupils, in almost equal proportions, gave 12% or 20% as their answer, that is one or other of the two percentages that should have been subtracted.

The pattern of numeracy attainment over the different levels and stages is similar in both the science and the social subjects surveys. Interestingly, however, there is a consistent, if small, difference in performance across the board in favour of the social subjects survey (see Figure 4.4).

Figure 4.4 A comparison of numeracy attainment in the Science and Social Subjects surveys

(average item facility over all items and all pupils at a stage and level)

The performance differences at any particular stage and level are not large, but the pattern of consistency is noteworthy. It is tempting to conclude that the different contexts in which the tasks were set (science versus social subjects) have influenced motivation levels and hence attainment. Non-response rates were indeed generally higher in the science survey than in the social subjects survey, particularly at P7 and S2. Since the numeracy tasks were embedded in the science booklets the generally higher task non-response rates in science will have affected the numeracy tasks as well as the science tasks. This will almost certainly have contributed to the lower numeracy performances for the science context. This issue of subject context effect is worthy of further research.

There were no differences in the numeracy performances of boys and girls in this science survey or in the previous survey of social subjects.

4.5 Using information technology

4.5.1 The tasks and their administration

The six ICT tasks used in the survey were developed with reference to the 5-14 Guidelines for ICT ⁹, and focused to varying degrees on the strands 'Using the technology', 'Creating and presenting', 'Searching and researching' and 'Communicating and collaborating'. While the tasks themselves were not assigned to specific 5-14 levels, three of the tasks assessed attainment targets at Levels B and C, and another three assessed attainment targets at Levels D and E.

Every task was set in a scientific context, and, while none required any prior science knowledge or understanding the contexts, were selected to cover the three outcomes 'Understanding Energy and Forces', 'Understanding Earth and Space' and 'Understanding Living Things and the Processes of Life' (see Table 4.11). The three Level B/C tasks featured contexts considered by the task developer to be suitable for pupils in P3 and P5, and the three Level D/E tasks were considered suitable for use with pupils in P7 and S2. In practice, in order to offer a comprehensive basis for cross-stage attainment comparisons, the Level B/C tasks were used at P7 as well as at P3 and P5, and the Level D/E tasks were used at P5 as well as at P7 and S2.

Table 4.11 The 'scientific' contexts of the ICT tasks related to Knowledge and understanding outcomes

Each task comprised a series of activities based on a set of web pages and other resources held on CDROM, and was planned to take about 30 minutes of testing time. The activities included modifying graphics and editing text (using Word), entering data into a spreadsheet and manipulating that data (using Excel), sending an email message and carrying out and interpreting a simulated web search. Table 4.12 overviews one of the Level B/C tasks and one of the Level D/E tasks.

The tasks were administered in around half the survey schools by trained field officers (see Chapter 1 for details of field officer numbers and training). Pupils worked on laptop computers provided by the field officers, and they were given a few minutes to familiarise themselves with these before the assessments began, using the mouse to control the cursor.

In each participating school, up to eight pupils were involved in the ICT assessment, each pupil working independently on one or two different randomly assigned tasks. After the brief laptop familiarisation, the pupil under assessment was given the relevant task sheet and asked to follow the instructions. If the pupil was struggling, the field officer directed him/her to the next question or brought the session to a close. Individual activities were rated as 'completed successfully', 'completed with some support', 'no real attempt', 'technical problem' or 'ran out of time'. At the end of the task the pupil's task sheet was collected in and filed with the checklist for later analysis.

Table 4.12 Overview of two ICT tasks

Before the attainment results are presented, some of the problems that arose during this innovative exercise merit comment. Firstly, as already noted, pupils were constrained to work with the laptops provided by the field officers, despite the fact that such a laptop environment was probably unfamiliar to most of them. Secondly, the tasks were only available for use on PCs, so that those pupils who were not familiar with the PC environment would have been disadvantaged. Thirdly, the text handling and data manipulation tasks required use of Word and Excel, software which was probably unfamiliar to most of the younger pupils. Fourthly, there were technical difficulties involved in leaving a simulated web page to edit text in Word or manipulate data in Excel and then returning to the web page. Field officers were advised to assist pupils should they meet problems in this sense. Finally, the text on some of the simulated web pages was not clear, the page images having been reduced to make space for task instructions to be shown alongside, and this might have made it difficult for pupils to answer some of the questions.

4.5.2 Pupils' ICT performance

Table 4.13 presents the overall ICT performance results, while Figure 4.5 illustrates the picture. Averaging over the activities and tasks used at any one stage, Table 4.13 records the percentages of pupils who completed the activities successfully without help from the field officer, the percentages who completed the activities successfully with some prompting from the field officer, the percentages of pupils made no real attempt to complete the activities, and the percentages of pupils who hit technical problems, or ran out of time.

Table 4.13 Response profiles across stages for the ICT tasks*

(% pupils classified into each response type, averaged over activities within tasks and over tasks)

* There were 400-450 pupils per task at P3 and S2,and typically 200-250 at P5 and P7

** 'Other' includes running out of time, meeting a technical problem, or not being rated for whatever other reason

Figure 4.5 Response profiles across stages for the ICT tasks

(200-450 pupils per stage per task)

As expected, Table 4.13 provides evidence of a clear attainment progression from one stage to another. On average, almost half the P3 pupils managed to complete the various Level B/C activities unaided, rising to almost 70% at P5 and over 75% at P7. For the activities at Levels D/E, just over 40% of the P5 pupils managed these without help, rising to 50% at P7 and over 60% at S2. Around a third fewer pupils completed activities without aid as they moved from Levels B/C to Levels D/E (P5 and P7 only). The 'No real attempt' figures follow the same pattern. These range from 10% of pupils to 30%, depending on the stage and level of skills demand: at P5 and P7, where pupils faced skills demands at both level groupings, activity non-response rates doubled from Levels B/C to Levels D/E.

As far as the four activity groups are concerned, viz. 'Navigating a document', 'Managing data', 'Searching and researching' and 'Communicating and collaborating' (in practice 'email communication' here), Table 4.14 illustrates the general picture. Over 90% of the pupils at all stages and in all the tasks managed to use the mouse to open a web page and then to follow links to access other web pages (90-92% at P3, 90-98% at P5, 95-98% at P7, 97-98% at S2). Success in the other skill areas was lower and very variable from one activity to another within general activity types.

Table 4.14 Unaided success rates across types of ICT activity*

(% pupils averaged over activities within tasks and over tasks)

* There were 400-450 pupils per task at P3 and S2, and typically 200-250 at P5 and P7; unaided success rates are averaged over between two and 11 individual activities per category

The majority of pupils at all stages, but rising with age, were able to enter text or data correctly into a given spreadsheet table (50-70% at P3 rising to over 80% at other stages). Much lower proportions were able to create a table from scratch before entering information into it (tested only in the tasks given to pupils at P5, P7 and S2). Around 10% of the P3 pupils, rising to 30-50% at P5 and P7, were able to centre text or data in columns without help (around half the P3 pupils and around 30% of those at P5 and P7 made no attempt at this activity, which was not assessed in the tasks used at S2).

In two of the tasks used at P5, P7 and S2, pupils were asked to produce an appropriate graph to re-present their table of data. In both cases the results were the same: just under 5% of the P5 pupils managed to produce a graph without help, rising to around 10% at P7 and around 35% at S2 (60-70% of the P5 pupils made no attempt to produce a graph, compared with 45% at P7 and just under 20% at S2). In principle the most challenging activity of all, two of the Levels D/E tasks required pupils to perform some computation on their table of data, finding a difference in one case and calculating an average in the other. Fewer than 5% of the S2 pupils and none of the pupils at P5 or P7 attempted to perform the calculations using Excel's function facility. Instead, 65% of the P5 pupils and around 80% of the pupils at P7 and S2 subtracted the numbers in question mentally, while 6% of the P5 pupils, around 20% of the P7 pupils and around 40% of the S2 pupils mentally calculated the average they needed. High proportions of pupils made no attempt to produce the required numbers by either method.

Of the various web site searches and emailing activities included in the tasks, most were reasonably well done, with 25-50% of the P3 pupils, and 40-80% of others, succeeding without help, the proportions varying with activity, level and stage. Three activities proved more difficult than most in this group. In one email activity pupils were to enter a particular email address into the Cc. box in the message header, and were expected to suggest using the attach facility to send a letter to a friend: 60-80% of the P3 and P5 pupils made no attempt to do either, compared with around half the P7 pupils. Around 10% of the P3 pupils, 20% of the P5 pupils and 35% of the P7 pupils did manage to enter the copy address successfully without help, but only 1% of the P3 pupils, 7% of the P5 pupils and 23% of the P7 pupils suggested using the attach facility. A similar pattern emerged when pupils at P5, P7 and S2 were asked to suggest three appropriate keywords when embarking on a given (simulated) web search.

There was no indication in the performance data of any differences in the ICT skills of boys and girls.

4.6 Summary

In addition to problem solving, which featured naturally here, the core skills of reading, writing, numeracy and using ICT were assessed in this 2003 Science survey. Reading, writing and numeracy were assessed using pencil and paper tasks, whilst ICT skills were assessed practically, with pupils working on laptop computers supplied by itinerant trained field officers. All the tasks used were carefully set in 'scientific' contexts.

Reading attainment was good at all stages. Over 80% of the P3 pupils were 'secure' at Level A (gaining at least 65% of the marks on their Level A task) and over half were secure at Level B; around 70% of the P5 pupils were secure at Level B and around 40% were secure at Level C. Roughly 60% of the pupils at P7 and at S2 were secure at Levels C and D, respectively, and around 40% were secure at Levels D and E, respectively. The results at P7 and S2 are in line with those reported for reading attainment in the 2001 English Language survey. In contrast with expectations, there were no apparent gender differences in reading in this survey.

The assessment of writing (conducted at P5, P7 and S2 only) produced two disappointing results. Firstly, an inter-rater agreement study highlighted the continuing problems that are inevitably faced when trying to evaluate pupils' writing. Despite training in the use of an holistic evaluation scheme, and access to exemplification materials, the inter-rater agreement rate was just 40% on average across pairs of raters and over several different writing tasks (this means that, on average, two different raters would allocate the same level classification to the same piece of writing in just 40% of cases). Secondly, the pupil attainment picture that emerged revealed evidence of relatively poor writing skills at all stages. In particular, fewer than 10% of the P7 and S2 pupils produced writing that was rated as being at the target levels for these stages (Levels D and E, respectively), and fewer than 15% of the P5 pupils produced writing deemed to be at Level C or higher. For most of the pupils involved in the reading and writing assessment, writing attainment was lower than reading attainment; just 25% of the pupils assessed in reading and writing were classified at the same level in both. Finally, while proportionally more of the boys than of the girls failed to produce any writing in the survey (20% versus 14%), there were no gender differences in the quality of the writing that was produced and evaluated.

Pupils' numeracy skills, on the other hand, appear to be quite well established. On average, more than half the pupils at every stage successfully answered their numeracy items, even where these involved computation. The numeracy tasks used here were derived from those used in the previous Social Subjects survey, and were used in the science survey in identical form or with simple changes to wording to render the contexts more 'scientific'. A comparison across surveys revealed slightly better performances across all stages and tasks in the social subjects survey than in the science survey, the lower achievements in the science survey reflecting generally higher non-response rates for all tasks in the science test booklets in which the numeracy tasks were embedded. There were no gender differences in numeracy attainment in either survey.

Despite a variety of problems associated with the exercise, the practical ICT assessment produced evidence of well-established skills in some areas and of clear progression elsewhere from one stage to another. Over 90% of the pupils at all stages, even P3, were skilled at opening documents and navigating through web pages. Around a third of the P3 pupils were also able unaided to add information to existing spreadsheet tables, compared with almost 60-70% of the pupils at other stages - at all stages lower proportions of pupils could centre information within columns. At P5, P7 and S2, where these skills were tested, rather low proportions of pupils managed to produce graphs to illustrate their tables of data, and none, or almost none, knew how to use the formula facility in Excel to carry out simple calculations on their data. Around a third of the P3 pupils were at ease simulating web searches while a quarter were able unaided to prepare to send an email; at the other stages the proportions were higher, ranging from 50% to 80%, depending on the activity and the stage. As was the case for the other core skills, there were no gender differences evident in the ICT attainment data.

« Previous | Contents | Next »