The Scottish Government

« Previous | Contents | Next »

Listen

Assessment of Achievement Programme
Report of the First AAP Survey of Social Subjects Enquiry Skills (2002)

3. Enquiry skills: practical assessment

3.1 Introduction

A number of different practical tasks were developed for administration in this survey, along with discussions and interviews. Some of these focused on the assessment of social subjects enquiry skills - these are reported in this chapter - while others focused on core skills (reported in Chapter 4).

Practical assessment is always more expensive and logistically more difficult to implement than is written assessment, especially in a large-scale application like a national attainment survey. For this reason, relatively few tasks were developed for administration in this first social subjects survey, and these were administered in a subsample only of the main sample of survey schools: 59 schools at P3, 60 at P5, 42 at P7 and 44 at S2 (ie. just under 40% of the schools at P3 and P5 and just over 30% at P7 and S2). Also, while the schools involved in the practical assessment were drawn from across the country, they were not selected entirely at random: two important criteria for involvement were (i) that the school should have sufficient pupils at the stage concerned to justify a day visit by two field officers, and (ii) that it should be within easy travelling distance of the field officers' home base.

All 18 survey pupils in each 'practical' school took part in some form of practical assessment, as well as having attempted one or two of the written tasks described in Chapter 2. Of relevance here is that typically four of the pupils participated in one-to-one interviews with a field officer, designed to explore the skills involved in Preparing for tasks, while three pupils individually attempted a reference task and six pupils undertook a video-based task, both focusing on the strand Carrying out tasks.

The tasks were administered in the schools by trained field officers. As noted in Chapter 1, the field officers were practising primary and social subjects teachers, released by their schools and authorities for six days each - one day for task orientation, five days for assessment and one day for debriefing. They were given the task orientation, in late June 2002, in groups of around 25 at two locations in central Scotland. Among the resources used during the training was a specially produced video of children attempting some of the practical tasks. Field officers were in addition able to observe a group tackling one of the Problem solving tasks described in Chapter 4, and the video also included individual interviews with several children.

Field officers were paired for school visits, and each pair visited five different schools in their own geographical regions within their allocated 2-week period. Both field officers were involved in running the practical circuits, one taking particular responsibility for the video-based task. One of the field officers conducted the pupil interviews. In addition to setting up the tasks and managing the assessment sessions, the field officers completed observation and rating checklists on the spot.

In this chapter the enquiry skills practical tasks are fully described, as is pupil performance on them. Given the particularly novel nature of the tasks, the relatively few tasks used of each type, and the administration of the practical tasks in fewer schools than is the case for the written tasks, the practical assessment should be considered exploratory, and its findings, interesting as they are, should be considered indicative rather than robustly generalisable.

3.2 The one-to-one interviews

3.2.1 The interviews

Individual interviews were designed to explore pupils' skills in Preparing for tasks. Each interview-based task used a photograph as the stimulus for discussion, and as the basis for a series of questions posed by the field officer.

Table 3.1 records the nature of the photographs that were used at the various stages. There were four photographs (or sets of photographs) in total. Two of these were used at all four stages: 'Child in close/housing' ( People in the past) and 'Emergency services' ( People in society). For People and place two different photographs were used - one for P3/P5 ('Weather') and another for P7/S2 ('River market') - to guarantee appropriate and familiar contexts for the pupils' age and stage. Any one pupil worked with only one photograph (or set of photographs), and the order of use of the photographs with pupils was pre-determined. The stimulus photograph(s) formed the basis for a series of questions posed by the field officer. The 'Weather' interview, overviewed in Table 3.2, illustrates the general format; a copy of the notes prepared for the field officers is included here in Appendix E.

Interviews were intended to last around 10-15 minutes at P3, and 15-20 minutes at other stages. The field officer began each interview by explaining to the pupil what the task involved, before moving on to invite a dialogue about the given photograph(s). During the dialogue the field officer used a list of pre-determined questions (see Table 3.3) to explore how the pupil might plan an enquiry in a logical and systematic way, what sources of information the pupil might use in such an enquiry, and what advice they would give other children that they might find useful in searching for, recording and using information when working on an investigation.

The field officer could re-phrase questions in cases where poor linguistic skills were creating problems for the pupil, and questioning continued until the pupil concerned appeared to have exhausted possibilities. The field officer recorded pupil responses on a checklist, and noted prompts where these proved necessary. For each question posed, the pupil was judged as having answered fully with minimal prompting, as having given an appropriate answer with some support, as having given a minimal but nevertheless correct response, or as having given an incorrect or irrelevant response or no response at all.

3.2.2 Pupil performance

At all stages, as would be expected, the proportions of pupils giving full responses with minimal support from the field officer decreased as the questions became more challenging. Figure 3.1 illustrates the general response pattern for the P3 pupils, averaged over the three tasks used at this stage.

Thus, while, on average, half the P3 pupils responded fully to question 1 - "Tell me what you can see in the photograph", a lower 34% responded fully to question 2a - "Is there anything in the photograph that you would like to know more about?", falling further to 27% for the follow-up question 2b - "Can you think of a question to ask to help you find out more?". The corresponding proportion for question 5 "If you were asked to use this photograph as part of a talk to your class, what kind of books could you use to find out more?" was 18%. The lowest proportion giving a minimally prompted full response was 10%, for the last question (question 9), which asked what advice the pupil would give other pupils about searching, recording and using information.

The same general response profile shown in Figure 3.1 for the P3 pupils emerged also at the other stages. The only difference is that the proportions of pupils giving full responses increased and those giving incorrect/irrelevant responses or no response at all decreased from one stage to the other.

The structure of the interviews was the same at every stage, with the same nine questions posed to the pupils (see Table 3.3). We can therefore look more formally across the stages for evidence of progression. Table 3.4 presents the data, while Figure 3.2 shows the resulting picture of skills improvement through the four stages.

It is interesting to note in Table 3.4 and Figure 3.2 that the proportion of pupils classified by the field officers as having given a full answer but with prompting was fairly stable across the stages, at 25-30%. It is in the proportions giving full and confident answers, with minimal if any prompting, and in the proportions giving inadequate or incorrect responses, including "Don't know", that stage progression is most clearly seen. So that while on average (over questions and tasks) just under a quarter of P3 pupils gave full responses, the proportion increased steadily to reach over half of the pupils at S2. Stage progression would appear to be most marked between P3 and P5.

At P7 and S2 there was a tendency for proportionally more of the boys than of the girls to give confident answers to many of the questions, but the differences were not large enough to reach statistical significance.

3.3 The video-based tasks

3.3.1 The tasks

Six video-based tasks were developed, one for each stage (P3/5 or P7/S2) for each of the three social subjects outcomes. The strand focus was Carrying out tasks.

Each task was based on a short 3-4 minute video clip, and had an associated worksheet. Six different video clips were selected for use in the survey, two per outcome. Three clips were used at P3 and P5 and a different three at P7 and S2 (see Table 3.3). All six clips were held on a single video tape for the convenience of the field officers. At each stage the clips were used in strict rotation.

Pupils watched their video in groups of six, but worked individually on the associated questions. The field officer started pupils on the task by first distributing the relevant worksheet and going over the instructions with them. Then the video clip was played twice, with time between each showing for reflection. After the second showing pupils answered a series of 10-15 associated questions using the worksheet. Table 3.4 provides an overview of one of the P7/S2 video tasks.

The task developers felt it was important to differentiate between viewing and listening as ways of collecting video-based information, and for this reason some of the questions related to information only obtainable from the video image, while others could only be answered from the soundtrack (there were typically five 'image-based' questions and 10 'audio-based' questions in each task). The pupil was required in some questions to tick or circle the answer, choosing from three or four alternatives. In other questions, specific information had to be written in short answer form. At the end of the P7/S2 worksheets, some 'Level D/E' questions required more open-ended answers, such as drawing a conclusion and giving reasons to justify it. Worksheets were evaluated by the field officers at the end of the day, and assessment criteria applied.

3.3.2 Pupil performance

Even though the video tasks were not level specific, it is interesting to apply the usual cut-off scores (50%, 65% and 80%) to the video test results to classify pupils in the same way as for the enquiry skills written tasks described in Chapter 2, and to average the classification proportions over the three tasks used at each stage. Table 3.5 presents the results, and Figure 3.3 illustrates the picture of progression

Since the tasks used at P7 and S2 were different from those used at P3 and P5, it is not useful to compare the performances of the P5 pupils with that of the P7 pupils. Within each stage grouping, however, we see that, as expected, the pupils in the higher stage performed better than those in the lower stage. Interestingly, if we look at those pupils who successfully answered at least 80% of the questions in their video task (showing 'strengths' in Table 3.5 and Figure 3.3), we see a difference between the two stages of around 20 percentage points in each case, ie. between P3 and P5 and between P7 and S2.

As to any differences in response to image-based questions as opposed to sound-based questions, the number of questions whose responses depend on image alone is very small, at 5 per task, and the relative difficulty of image-based versus sound-based questions varies from task to task. This makes it difficult to offer comment on this issue, other than to say that there is no evidence in the response data of important consistent differences in the abilities of pupils to handle the two modes.

Pupil numbers were relatively small for each task, and groups of pupils were assessed on the same task within any one school. This means that any gender differences would need to be very strong to reach statistical significance in these circumstances. In the event, there is no evidence in the data of any gender differences in the video-based task performance.

3.4 The reference tasks

3.4.1 The tasks

The strand focus here was both Preparing for tasks and Carrying out tasks. Three tasks were designed to assess how well pupils could find information from reference books, one for P3, one for P5 and one for P7/S2. The books used were selected as appropriate for the pupils' age/stage. As Table 3.6 shows, the two books used at P7 and S2 were both about the Victorians. At P5 one of the books was about cars and the other was about transport. The book about cars was used also at P3, along with a book about hedgehogs. Initially, it had been intended to use the books themselves in the test sessions, but because of cost and time pressure high quality colour photocopies of the relevant pages were substituted.

In each task, pupils were provided with the contents lists, indexes and glossaries from the two books, as well as, at P7 and S2, the inside cover text. They were to use their reference materials to answer three sets of questions. The first set of questions required them to arrive at an appropriate choice between the two books; questions were typically of the form "Which book might tell you about…?". The second and third sets of questions required them to access information in the contents list, index and/or glossary of first one book and then the other. Questions took a variety of forms, including, for example, "Who was the author of this book?", "On which page would you find…?", "Which page tells you about…?", "Which topic would you look up to find out about…?", "Use the glossary to find out the meaning of the words…?", etc. Table 3.7 provides an overview of the P7/S2 task.

After putting the pupil at ease and explaining the task, the field officer provided the pupil with the materials, and was then to work through the three sets of questions, using a checklist to record responses and prompts.

3.4.2 Pupil performance

There was essentially no difference between the performances of the P7 and S2 pupils on their reference task: 70% average score at P7 compared with 73% at S2. Table 3.8 presents the results for the individual questions and Figure 3.4 illustrates the pattern.

Interestingly, Table 3.8 shows that rather low proportions of the P7 and S2 pupils successfully identified which, if any, of their two books would be the appropriate one to find out about the history of education (for which 'The Victorians at School' might seem the obvious choice): 31% at P7 and 42% at S2. More positively, almost all the pupils at both stages could name the author of a given book, without prompting, and high proportions (95%+) could also give page references from the contents list where the requirement was to locate particular kinds of factual information. Once again, lower proportions could give references to pages where they might find pictures (39% at P7 and 51% at S2).

While quite high proportions of pupils could identify a topic in the contents list under which they might find extra information about libraries (70%), much lower proportions could do the same when the subject of interest was women teachers in Victorian times (27% and 21% of P7 and S2 pupils, respectively). Quite low proportions correctly answered the single question that required use of an index (42% and 56% respectively).

The task at P7/S2 differed from those used at P3 and P5 in one important sense, and this is that here pupils were also required to locate and retrieve additional information from the book jacket (eg. from 'Books to read' and 'Places to visit'). Success rates varied between 70% and 90% in most cases, rising to 100% at P7 and 98% at S2 where a single fact was to be retrieved.

The survey findings suggest that P5 pupils also have well developed referencing skills in general. On their reference task the P5 pupils in the survey achieved an average score of 85% on their one task. Detailed results at the individual question level are shown in Table 3.9.

High proportions of the P5 pupils correctly identified which of their two books would be the most appropriate to meet given needs (90%+), even when the correct answer was 'neither' (82%). Almost all the pupils could name the author (98%, with just 3% of them requiring prompting) and could give page references for specific kinds of information. Paralleling the finding at P7/S2, page referencing was less well done when pupils were to identify where they might find pictures of particular objects in the book concerned (47%). When asked to use a glossary or index, rather than a contents list, relatively high proportions of pupils needed a prompt.

As Table 3.10 shows, reference skills also seem to be fairly well established among P3 pupils: the average task score at this stage was 81%. The majority of the pupils were able correctly to identify which of the two books "has facts and fun" (although it must be said that the words 'facts' and 'fun' appeared on the front cover), which might tell them about an animal family, and which might tell them "about things with wheels". Many fewer (a drop from 90%+ to 56%) correctly indicated that neither book might tell them about trains (question 3).

A high proportion of pupils (95%) correctly identified the author of the book about hedgehogs, although 18% did so only after prompting - presumably these pupils were not aware of the meaning of 'author'. High proportions of the pupils were able to use the contents lists of the two books to locate the correct page reference for information about specific facts or topics (85% to 95% for most of the questions). Lower proportions demonstrated ability to use a glossary and index (79% and 61%, respectively), many doing so only with the benefit of prompting (42% and 28%, respectively). Moreover, non-response rates were also relatively high for these last two questions, at 13% and 18%, respectively.

The data suggest that the pupils at all stages were well practised in inferring the general content of books from their covers, and in using contents lists to locate specific kinds of information, but they were less good at using indexes and glossaries.

The only evidence of any gender difference in performance emerged for the reference task at S2, where the girls achieved an average score of 78% compared with the boys' 68%.

3.5 Summary

In addition to the written testing described in Chapter 2, pupils' enquiry skills were assessed more practically in this survey. There were one-to-one interviews with field officers, to explore the skills involved in Preparing for tasks, as well as reference tasks and video-based tasks, focusing on the strand Carrying out tasks. The practical assessment was undertaken in a smaller number of schools than the full written survey: about 40% of the schools at P3 and P5 and about one-third of the schools at P7 and S2. All the pupils involved in the practical assessment would previously have taken part in the written assessment.

For the one to one interviews, a photograph was used as an initial stimulus for the discussion. Not surprisingly, at all stages the proportions of pupils giving full responses to questions, with minimal support from the field officer, decreased as the questions became more challenging. Equally unsurprisingly, the proportions of pupils giving correct responses increased with increasing age, from an average over questions of around a quarter of the pupils at P3 to over half of the pupils at S2. Stage progression would appear to be most marked between P3 and P5.

Six video-based tasks featured in the survey, three at P3/P5 and three at P7/S2. Each task comprised a series of information recall questions, using either the image or sound as source. Within each stage grouping - P3/P5 or P7/S2 - pupils in the higher stage performed better than those in the lower stage, by similar margins (there was a difference between stages of around 20 percentage points in the proportions of pupils who successfully answered at least 80% of the test questions).

Three reference skills tasks were designed to assess how well pupils could find information from reference books using contents, index and glossary. Although different pairs of books were used at P3 and at P5, the survey findings suggest that pupils already have very well developed referencing skills at these early stages. The same reference task was administered at P7 and S2, and it is interesting to note that the improvement in reference skills from P7 to S2 was only marginal. At all stages pupils were less competent when using indexes and glossaries than they were when using contents lists.

There was no evidence of any general and consistent gender differences in the survey data for any type of task.

« Previous | Contents | Next »