The Scottish Government

« Previous | Contents | Next »

Listen

Assessment of Achievement Programme
Report of the First AAP Survey of Social Subjects Enquiry Skills (2002)

4. Core skills assessment

4.1 Introduction

Core skills were assessed in this survey in a variety of ways. Numeracy and Communication (reading and writing) were assessed quite naturally within the pencil and paper enquiry skills tasks, given that the information sources used in the tasks deliberately featured a variety of different source types - maps, diagrams, tables, charts, and so on, as well as texts (see Table 2.2 in Chapter 2 for examples) - and with each enquiry skills task came an associated writing task.

The other core skills - Using information technology, Working with others and Problem solving - were assessed through use of practical tasks administered by field officers in a subsample of the survey schools. For Working with others and Problem solving the task concerned took the form of a group discussion aimed at a group solution of a problem posed by the field officer. ICT tasks were CDROM-based, and featured word processing, use of spreadsheets and web searches. Field officers worked in pairs in each of their assigned schools. Both were trained to organise and manage the group discussions, and to observe and rate pupil behaviours using a checklist, while one field officer was trained to set up and administer the ICT tasks (see Chapter 3 for further information about field officer training).

Discussion groups comprised up to four pupils each, and took place in around 60 schools at each of P3 and P5 and just under 50 schools at P7 and at S2. Experimental ICT tasks were administered in just under 20 schools at P7 and at S2, and were attempted by up to four pupils in each school, working individually under field officer supervision.

4.2 Reading and Numeracy

4.2.1 The enquiry skills subtasks

In every enquiry skills task one of the three information sources was a continuous prose text. Topic sophistication increased with increasing level, as did text lengths: from around 100 words at Level A to around 300 words at Level E. The variety of question type, however, was similar at all levels (summary completion, sentence completion, true/false, multiple-choice, short-answer, open-ended - see Chapter 2 for details), although summary completion featured more strongly in the Level B tasks than others. Every task also contained a set of questions based on a table or chart, and another set requiring pupils to handle information presented in some other form, such as a map, a poster, a picture, and so on.

Table 4.1 illustrates the nature of the texts and the associated test questions for the three enquiry skills tasks described fully in Chapter 2 (Table 2.2), along with the nature of the other information sources that featured and the type of activity pupils were engaged in when using these. Details of task administration and marking are given in Chapter 2.

4.2.2 Pupil performance

Since the number of test questions relating to each type of information source was small, at most eight at Level A rising to at most 12 at Level E, attainment data of the type presented in Chapter 2, applying the 65% success criterion, cannot sensibly be produced here. In its place we present the results of an analysis of average item facilities, ie. the average over all text-based items of the percentage of pupils answering the item correctly. Further, we compare text-based reading performance with pupils' performances when reading and using information given in the other types of information source. Table 4.2 presents the summary data for text-based and other-source items.

The most interesting feature in the data in Table 4.2 is the fact that there are no marked differences in performance for the two broad types of information source. The small sample differences seen in the table are not consistent in direction, and the only one to reach statistical significance is that for Level A at P3 - this could simply be an artefact of the particular items that comprised the tasks used at this level.

Of the almost 1400 test items contained in the 45 enquiry skills tasks, 50 demanded the application of computation skills, ie. they required pupils to count, add, subtract, multiply or convert to fractions and ratios. There were 14 such items at Level A (eight counts, four additions and one subtraction), nine at Level B (five counts, two additions, one subtraction and one multiplication), eight at Level C (three additions, four subtractions and one multiplication), eight at Level D (two counts, five subtractions and one conversion to a fraction), and 11 at Level E (two counts, three additions, three subtractions, two conversions to fractions and one conversion to a ratio). While it is difficult to generalise pupil performance on the basis of this small set of quite varied items, it is nevertheless the case that pupil performance was good: on average, between 50% and 75% of the pupils successfully answered the items. Table 4.3 gives the detail.

4.3 Writing

4.3.1 The writing tasks

Each writing task was linked to the particular pencil and paper enquiry skills task that it followed, in the sense that it used as stimulus material the same information sources as the enquiry skills task, and thus built on the work pupils had already done with their three information sources.

Having completed their allocated enquiry skills task, pupils were asked to produce a piece of writing on the general theme of the information they had been working with for the previous 40 minutes or so. For the pupil, the purpose of the writing was most usually to summarise what had been learned after working with the three information sources, or to use evidence from these sources to support or to refute a given conclusion. The three enquiry skills tasks described in Chapter 2 (Table 2.2) and above (Table 4.1) will serve to illustrate the nature of the associated writing tasks (see Table 4.4).

The national framework of writing criteria was adapted for the purpose of rating the pupils' writing in this survey. The framework is essentially a criterion by level grid, in which attainment targets define appropriate level-based attainment against each of nine different writing criteria. When using the framework, writing evaluators normally identify the highest level of performance demonstrated by the particular piece of writing, judging against each criterion separately before assigning an overall level on the basis of the resulting rating profiles . In this survey, however, it was decided not to attempt to allocate an overall level to each piece of writing, but simply to rate the pupil's demonstrated attainment level against each criterion separately.

The teachers who worked as field officers in the survey, mainly primary teachers with some S2 social subjects teachers, were invited to evaluate the pupils' writing, and four training sessions were held during October 2002 in different parts of the country. A number of problems arose, however, which reduced the quality of the resulting data.

Firstly, there was frequently too little writing produced by individual pupils - often none at all - for its assessment to be meaningful or even possible: in fact, just over 10% of the pupils produced no writing at all (15% at P3), while 10% of the almost 12000 pieces of writing reviewed were judged to be irrelevant, and just under 10% more were judged to be of insufficient length to be useful. This problem of low volume production could perhaps have been anticipated, given that the writing tasks were positioned at the end of the pencil and paper booklets, with the expectation that pupils would produce their writing after having already spent 40 minutes or so answering questions about their three information sources. Clearly, some pupils might have had too little time at the end of the test session to produce much writing, others might have been too tired to demonstrate their writing abilities fully, and still others might simply have lacked the necessary motivation to make the effort at all, given that this was for them a low stakes assessment with no direct personal consequences.

Secondly, the S2 social subjects teachers who took part in the writing evaluation were unfamiliar with the 5-14 approach to judging writing (most of the evaluators were primary teachers). This lack of familiarity, combined with the often small amount of writing they were faced with on occasions, led to some problems. In addition, whatever their current teaching experience, evaluators frequently experienced difficulty in distinguishing between writing showing 'insufficient evidence' to permit a judgment, writing that was 'irrelevant or incomprehensible', and writing that was 'below Level A'. Some markers also made these kinds of judgment against individual criteria rather than for the piece of writing as a whole.

The problems met in this attempt to evaluate pupils' writing were reviewed during development of writing tasks for use in the 2003 science survey, and a decision was made to discard the criterion-based framework in favour of an holistic evaluation scheme based on 'best fit' descriptions (see Appendix F). Experimentation with the holistic scheme confirmed that this scheme was easier to use than the detailed criterion-based scheme, but also revealed a need for increased exemplification for the raters. The fairly positive experience in the science survey led to a decision to re-evaluate a subsample of the writing produced in the social subjects survey, using the holistic rating scheme. The field officers who had previously been trained to use the scheme in the science survey were invited to take part in the new evaluation of social subjects writing, which took place in May 2004.

A total of 75 teachers - primary teachers and S2 science teachers - were involved in the re-evaluation of the social subjects writing, each judging 45-50 different pieces of writing. The writing selected for re-evaluation was produced by pupils in P5, P7 or S2, and related to one or other of the enquiry skills tasks at Levels C, D and E. In total, 1266 pieces of writing were randomly selected for rating - this is just over 10% of all the pieces of writing produced by pupils at the three stages in the survey.

The evaluation of writing is inevitably a highly subjective process, even when apparently tight criterion-based schemes are applied. Recognising this, and in an attempt to ensure that good quality evaluation data would be produced, it was decided to have each piece of writing rated by each of three different raters working independently, and to allocate a final level to each piece of writing on the basis of majority judgments. Where raters did not all agree on the appropriate level for a piece of writing the level allocated was that agreed by two of the three. Where there was no agreement the piece of writing was excluded from further analysis.

4.3.2 Pupil performance

Of the 1266 pieces of writing that were evaluated in the May 2004 re-evaluation exercise, 2.5% were judged to be 'irrelevant' (1% at P5, 2% at P7, 4% at S2) and another 15% could not be allocated to a particular level on the basis of majority rater agreement (13% at P5 and P7, 18% at S2).

Table 4.5 and Figure 4.1 show the level distributions for the writing at the three stages, for the 80% of the original 1266 pieces of writing that were relevant and that were able to be majority classified.

Table 4.5 shows clear evidence of a positive progression in level attainment in writing between P5 and P7, the 44% of P5 pupils attaining at or below Level A having become just over 10% at P7, and the 27% of P5 pupils attaining Level C or higher having become 57% at P7. But the pictures of attainment at P7 and S2 show greater dispersion in attainment at S2 compared with P7 rather than a positive movement upwards. Roughly the same proportions of pupils at P7 and at S2 were classified as having produced writing at or below Level B (43% and 45%, respectively) or at Level C or above (57% and 56%, respectively). But the level distribution at S2 is clearly 'flatter' than at P7, with slightly higher proportions of S2 pupils than P7 pupils showing very low attainment or very high attainment.

As far as gender differences in writing attainment are concerned, we see from Table 4.6 and Figure 4.2 that at P5 the performances of the girls and boys were very similar, except at the lower end of the level distribution where the proportion of boys classified as having produced writing at a level below Level A was more than twice that of the girls - 18% compared with 7% (of 80 boys and 113 girls).

At P7, in contrast, the distributions were uneven at the higher end, with proportionally more of the girls than of the boys categorised as having produced writing at Level D or E (26% versus 11%, respectively, of 235 girls and 134 boys); the level distributions between boys and girls reached statistical significance at the 5% level.

At S2, the girls generally produced work at a higher level than the boys, the gender difference in level distributions reaching statistical significance at the 0.1% level: 14% of the girls compared with 30% of the boys were classified as having produced writing at Level A or below, while 13% of the girls compared with 3% of the boys were classified as having produced writing at the standard of Level E (267 girls and 208 boys in total).

4.4 Working with others and Problem solving

4.4.1 Group discussions: Themes and structure

Group discussions were designed to assess how well a group of four pupils could work together in a problem solving situation, and related to the core skills Working with others and Problem solving. There were two different contexts - one for P3/P5 ('Rescue') and another for P7/S2 ('Survival). Table 4.7 overviews the two tasks.

Two field officers were involved in each discussion activity. The pupils were to work for about 15-20 minutes at the primary stages and up to 30 minutes at S2. One field officer organised, introduced and managed the group discussion, while the other observed one (pre-selected) pupil and used a checklist to record judgments about various aspects of the pupil's behaviour (see Table 4.8).

The managing field officer intervened only when the discussion waned, or when the group had arrived at some pre-determined stage. Where little intervention in the group discussion was needed, the first field officer assessed a second pre-selected pupil in the same way as the first. After completing the group work, two further pre-identified pupils were interviewed individually by the field officers to assess Problem solving and Working with others. Each pupil was asked a few questions about the way the group had gone about their task (see Table 4.9).

4.4.2 Pupil performance

Problem solving

The field officers recorded their evaluations of the eight different aspects of pupils' problem solving activity (see Table 4.8) by noting that in their opinion there was 'clear evidence' of the particular behaviour, 'some evidence' of the behaviour or 'no evidence' of the behaviour.

Interestingly, there were no general differences in the ratings given to the P5, P7 and S2 pupils, even though the P5 pupils were working on a different problem-solving task from their older peers. But there were statistically significant differences between the P3 pupils and all others. Table 4.10 provides the overall results, while Figure 4.3 illustrates the picture.

At all four stages, the proportions of pupils showing 'clear evidence', 'some evidence' or 'no evidence' were fairly stable across seven of the eight behaviours. The behaviour showing the lowest ratings at all stages was 'reviewing/reflecting on progress'. Here the proportion of pupils considered to have shown 'clear evidence' of this activity dropped from the typical 50-60% at P5, P7 and S2 to just under 40%, and from the typical 30-40% at P3 to just over 20%.

Working with others

Two of the behaviours that were rated during the groups' problem solving activity (see Table 4.11), and which concern Working with others, can be considered as desirable and related - 'acts on/builds on others' ideas' and 'reacts to others' suggestions with reasons' - whereas the other two would be considered undesirable - 'makes irrelevant comments' and 'disrupts the progress of the task'.

As far as the positive behaviours are concerned, rating profiles were similar for both, within each stage and across stages, again with the exception of P3, where pupils were significantly less well rated than others (see Table 4.11 and Figure 4.4).

Fortunately, undesirable behaviour was rarely in evidence at any stage, according to the observing field officers. On average, 90% of the pupils showed no evidence at all of contributing irrelevant comments - although dropping to 83% at P3 - and a higher 95% showed no evidence of disruptive behaviour.

Pupils' evaluations of their group's work

Here, the field officers rated the pupils' responses to each of the six questions (see Table 4.9), this time noting when pupils answered 'fully with minimal prompting' or answered 'with support', when they gave a minimal response even with support and when they gave an irrelevant response or no response at all. Again the rating profiles are similar at P5, P7 and S2 (see Table 4.12 and Figure 4.5), and are fairly stable across the six questions (65-75% 'full response' on average for each question).

Once again, the ratings given to the P3 pupils tended to be significantly less positive than those given to their older peers. Around 50% of the P3 pupils were judged as having given 'full responses' to the first three questions, with lower proportions for the last three questions: falling progressively from 38% ( Do you think the group worked well together?) through 33% ( Did the others in the group listen to what you said?) to 30% ( Is there anything you wish you had done differently?).

4.5 Using information technology

4.5.1 The tasks

ICT skills have become increasingly important for enquiry in social subjects and a small number of tasks were developed to assess such skills in this survey. Perhaps inevitably, given that this was the first time ICT assessment had been attempted within the AAP, there were problems during task administration within the survey itself. Consequently, whereas the initial plan had been to assess pupils at all four stages in all relevant schools, pupils were in practice assessed only at P7 and S2, and in a relatively small number of schools. This particular exercise should therefore be considered as a pilot rather than a full survey of pupils' ICT skills. The nature and results of the exercise are simply reported here for interest.

The ICT tasks were developed with reference to the 5-14 Guidelines for ICT, and related in particular to the strands Using the technology, Creating and presenting text, Collecting and analysing, Searching and researching, and Communicating and collaborating. Each task comprised a series of activities based on a set of web pages held on CDROM, and was planned to take about 30 minutes of testing time. The activities included modifying graphics and editing text, completing a spreadsheet, carrying out and interpreting a simulated web search, and preparing to send an email. There was one task for each of the three social subjects outcomes, and all the activities within any particular task related to a single common theme (see Table 4.13 for details), selected from among those that featured in the enquiry skills written tasks described in Chapter 2. Table 4.14 overviews one of the P7/S2 tasks.

Pupils were required to work with laptops carried to the schools by the field officers, and before starting the assigned task each pupil was given a few minutes to become familiar with the laptop, using the mouse to control the cursor. The pupil was then given the relevant task sheet and asked to follow the instructions. If the pupil was struggling, the field officer directed him/her to the next question or brought the session to a close. The pupil's progress and any related prompts were recorded on an assessment checklist, and at the end of the task the pupil's task sheet was collected in and filed with the checklist for later analysis.

4.5.2 Pupil performance

In total, 69 P7 pupils in a total of 18 primary schools attempted an ICT task, as did 60 S2 pupils in 17 secondary schools. Table 4.15 presents the overall results for these pupils on the three tasks used at these stages, while Figure 4.6 illustrates the picture.

Averaging over the activities within each task for each pupil, Table 4.15 records the percentages of pupils who completed their given task successfully without help from the field officer, the percentages who completed the task successfully with some prompting from the field officer, the percentages of pupils who failed to complete the task even with field officer prompts, and the percentages of pupils who made no attempt to complete the task. As might be expected, in each of the tasks proportionally more of the S2 pupils compared with the P7 pupils successfully completed the task unaided, although the differences vary markedly from one task to another.

As Table 4.15 and Figure 4.6 show, not only did the performance gaps between the P7 and the S2 pupils vary from one task to the other, so did the general difficulty of the tasks. 'World War II' was apparently quite difficult for many pupils at both stages, while 'Advertising' appeared to have been the most straightforward of the three tasks. The individual activities within the tasks are clearly relevant here.

The data suggest that the pupils at both stages found chart creation within spreadsheets the most challenging individual activity: creating a bar chart in 'Advertising' and pie charts in the other two tasks. Relatively high proportions of pupils made no attempt to create the charts: 20-25% at S2, 20-35% at P7.

As far as any possible differences in ability to word process, use spreadsheets and use databases are concerned, no useful comment can be offered. The relative difficulty of the three sets of questions within each task resulted in an inconsistent picture of difference across tasks. Moreover, very small numbers of pupils in even smaller numbers of schools took part in this exercise, contributing to the inability to generalise performance findings. It is clearly not possible either to offer gender comparisons.

What we can say, however, is that the pupils here did relatively well on the ICT tasks, given that they were using laptop computers that they were not used to working with in the normal course of events.

4.6 Summary

Core skills were assessed in this survey in a variety of ways. Numeracy and Communication (reading and writing) were assessed via the pencil and paper 'enquiry skills' tasks. These tasks deliberately featured a variety of source types with relevance for assessing Numeracy including, diagrams, charts, and tables. In addition, every task included one subtask based on a continuous prose text, to facilitate the assessment of text-based reading. Finally, associated with each enquiry skill task was an extended writing task, included to permit the assessment of pupils' writing skills within a given context. The other core skills - Using information technology, Working with others and Problem solving - were assessed through use of practical tasks administered by itinerant field officers in a subsample of the survey schools.

Analysis of test items based on texts and those based on other types of information source (diagrams, charts, maps, etc) showed no evidence of any difference in pupil performance. On the few items requiring the application of computation skills, pupils performed quite well, with 50-75% of them on average successfully answering the items concerned.

As far as pupils' writing skills are concerned, the picture for P7 and S2 was fairly similar, with roughly 45% of the pupils attaining at or below Level B, and 55% attaining at or above Level C. This compares with 45% of the P5 pupils attaining at or below Level A. A gender gap in favour of girls was apparent at every stage, increasing through the stages: in particular, at S2 almost 20% of the boys were classified as performing below Level A compared with under 10% of the girls.

Group discussions were designed to assess how well a group of four pupils could work together in a problem solving situation, and related to the core skills Working with others and Problem Solving. A different problem task was used at P3 and P5 compared with P7 and S2. However, the findings are clear. While there are no evident differences in the discussion behaviour of pupils at P5, P7 and S2, nor indeed in their ability to evaluate their own and their group's work, there is significant improvement in these respects between P3 and P5.

ICT tasks were used only with P7 and S2 pupils, and while rather few pupils were involved at each stage - around 60 from under 20 schools - the findings show very clearly that proportionally more of the S2 pupils compared with the P7 pupils were able to complete their given task unaided. Pupils at both stages found chart creation within spreadsheets a more challenging activity than editing text, adding information to spreadsheet tables or using web searches.

« Previous | Contents | Next »