The Scottish Government

« Previous | Contents | Next »

Listen

Assessment of Achievement Programme: Report of the Sixth AAP Survey of Science (2003)

3. Skills in Science - Investigating

3.1 The assessment process

3.1.1 The assessment tasks

Practical assessment is always more expensive and logistically more difficult to implement than is written assessment, especially in a large-scale time-constrained application like a national attainment survey. For this reason, relatively few investigating tasks - nine in total - were specially developed by teachers under the supervision of an academic staff member in the University of Strathclyde's Department of Education.

Each task was designed to allow pupils, working in groups, to plan, carry out and review/report a meaningful investigation set in a realistic context. Rating checklists were developed that in principle enabled field officers to assess a pupil's investigative behaviour, and to classify the pupil into one of two adjacent 5-14 levels with respect to each relevant attainment target. There were two tasks for every pair of adjacent levels A/B, B/C, C/D and D/E, and one further task which spanned attainment targets at Levels D, E and F. Perhaps inevitably, given the time normally needed to set up and to follow investigations in the biological sciences, the nine investigation tasks were virtually all set within the physical sciences, as the task names and investigation questions listed in Table 3.1 illustrate. Two of the tasks are overviewed in more detail in Table 3.2.

Table 3.1 The investigating tasks

Every task was designed to take 45 minutes to one hour of pupil time. The first 15 minutes were devoted to task orientation. Here the field officer would start by demonstrating some phenomenon - a ball rolling, a crystal dissolving, an object floating - and would engage the pupils in discussion about it, asking what they had observed, why they thought it had happened, and so on. In this way, the pupils would be guided towards the question that they were to investigate.

After being shown the resources available to them, the pupils were to be given worksheets and invited to make notes in the 'planning' section of these before starting work. The field officer would observe the pupils at work, occasionally asking questions and recording observations on associated checklists.

Table 3.2 Overview of one Level A/B task and one Level D/E task

This practical activity was intended to last about 20 minutes, at the end of which pupils were individually asked what their conclusions were, whether they were happy with the investigation, if not how they would improve it should they be able to do it again, and what other questions they might like to investigate in the future.

3.1.2 The rating checklists

As pupils worked together on their investigation, they were to be individually rated for evidence of the attainment of relevant attainment targets, using a task-specific checklist. For ease of use, the rating checklists were designed to hold records for up to four different pupils, i.e. for all the pupils in the investigating group.

Each rating checklist had three sections, corresponding with the three strands comprising this attainment outcome - 'Preparing for tasks', 'Carrying out tasks' and 'Reviewing and reporting on tasks'. Within each section were listed the associated attainment targets at the levels relevant to the particular task. Table 3.3 overviews the checklist items for the Level A/B task, 'Changes', described in Table 3.2. The 'Changes' checklist is typical of all others, in the sense that one of the two levels has more attainment targets listed than the other, reflecting exactly the strand-target framework in the National Guidelines.

Table 3.3 The structure of the rating checklist for the Level A/B task 'Changes'

Through observing the pupils at work, talking to them as they worked, and reviewing any notes or other materials they had prepared whilst carrying out their assigned investigations, the field officers were to assess each of the pupils in the investigating group, deciding which attainment targets had been demonstrated, and at what levels. They were also to record whether the pupil concerned had managed to show the evidence unaided or with some prompting. In addition, they were to note occasions when they felt that they could not make an assessment, either because the pupil did not respond to the task or because technical problems arose.

The investigating tasks were trialled informally by the task developers, but full-scale trialling occurred within the survey itself. The consequence of this is that the checklists had not been fully tested for applicability in the field before their use in the survey, and neither had any rater agreement trials been possible. Albeit unavoidable in the circumstances, this situation was unfortunate, given the potential value of this inherently interesting exploration of practical investigation skills. These limitations should be borne in mind when the performance data presented later in this chapter are reviewed, as should the effect of the structure of the task itself on level attainment.

3.1.3 Task administration

For cost and logistic reasons, the tasks were administered in a subsample only of the main sample of survey schools: 87 schools at P3, 85 at P5, 94 at P7 and 90 at S2 (these figures represent just over half the survey schools at each primary stage and around two-thirds of those at S2). Although the 'practical' schools were drawn from across the country, they were not selected entirely at random: two important criteria for involvement were (i) that the school should have sufficient pupils at the stage concerned to justify a day visit by two field officers, and (ii) that it should be within easy travelling distance of the field officers' home bases.

Of the nine tasks, sets of three were administered at each stage (see Table 3.4). Two of the three were used uniquely at a particular stage, but one was used at two consecutive stages in order to allow stage comparisons across three rather than two levels.

Table 3.4 Task distribution by stage

The tasks were administered in the schools by 148 field officers, who were practising primary teachers and secondary science teachers released by their schools and authorities for seven days each - one day for task orientation, five days for assessment and one day for debriefing. They were given their task orientation in June 2003, in groups of 18-25 at various locations in Scotland. Field officers were paired for school visits, and each pair visited five different schools in their own geographical regions within their allocated 2-week period, administering the same two investigating tasks throughout. In addition to setting up the tasks and managing the assessment sessions, the field officers completed their rating checklists on the spot.

In the selected schools, the pupils who attempted the investigating tasks were those who had already participated in the assessment of Knowledge and understanding in science. Pupils typically undertook their assigned investigations in groups of four, and two such groups were assessed in most of the schools involved.

3.1.4 Reporting skills attainment

In the ideal case, pupils might have been rated at the same level across the various attainment targets included in each checklist, preferably without assistance from the field officer. Had that happened as a general case then pupils could have been classified into appropriate 5-14 levels for their investigation skills. But, as most teachers could have predicted, this did not happen. Individual pupil profiles typically demonstrated a mixed pattern of level judgments, some merited without prompting and some awarded only after help had been given. In addition, there were often gaps in the rating profiles, where the field officers could not make a judgment, because the pupil concerned had not produced any, or enough, evidence of the skill in question, or because there had been a rare technical problem of some kind, or for some other reason. Given this, the only clear option for data analysis was to look across pupils at a stage, to see what proportions were judged as having demonstrated the various attainment targets at the different levels within the three strands. The attainment data to be presented in this chapter are therefore quite different in nature from those presented in Chapter 2 for Knowledge and understanding.

Finally, a word of caution: given the particularly novel nature of the tasks, the relatively few tasks used at each stage, the administration of these tasks in fewer schools than was the case for the written tasks, the fact that the schools involved were not randomly selected, and the fact that no formal piloting had taken place ahead of time, the practical assessment should be considered very exploratory, and its findings, interesting as they are, should be considered indicative rather than robustly generalisable.

3.2 Overview of pupils' attainments

3.2.1 The attainment picture across the stages

Table 3.5 presents the global findings from this exploration of pupils' investigation skills, while Figure 3.1 illustrates the general attainment picture.

The 'prompt' column in Table 3.5 indicates the proportions of pupils, on average, who needed assistance from the field officer before being able to demonstrate rateable behaviours that could be judged to be at one level or another (these pupils do not feature in the 'level' columns). 'No rating' indicates the proportions of pupils, on average, who could not be rated at all because they did not demonstrate the behaviour concerned, with or without field officer help (in a small proportion of cases field officers indicated that there had been a technical problem). The data indicate that on average a quarter to a third of the pupils were able to demonstrate achievement of the attainment target at one level or the other after having been assisted by the field officer, while 10-25% of pupils could not be rated on individual attainment targets for whatever reason. The proportions varied from one task to another, but the variation was not great.

As far as the attainment figures in Table 3.5 are concerned, it should be noted that these are not intended to represent the percentages of pupils 'secure' at the levels concerned, or showing 'basic skills' or 'considerable strengths' - the investigation data produced here do not lend themselves to this kind of analysis and interpretation. Rather, at each 5-14 level the table shows the proportions of pupils judged to have demonstrated attainment targets at that level without help from the field officer, averaged over all attainment targets at the level concerned (typically 8-10 in each task), and also averaged across all three tasks used at the stage in question.

Table 3.5 The general pattern of investigation skills attainment

(% pupils demonstrating attainment targets, averaged over targets and tasks)

* This indicates that a judgment was made, but only after field officer prompting helped the pupil.

** 'No rating' means that a pupil did not demonstrate evidence of the attainment target, with or without help.

Thus, at P3 we see a figure of 53% under Level B. This indicates that on average, over the nine Level B attainment targets that featured in each of the three tasks used at this stage, 53% of the pupils were judged as having demonstrated the behaviour concerned, without assistance from the field officer. The tasks concerned were 'Magnets', 'Changes' and 'Floating and sinking', and the nine attainment targets over which ratings are averaged are those shown in Table 3.3 for 'Changes'. The 53% at P3 compares with 34% of the pupils at P5, after averaging over the same nine Level B attainment targets rated in each of their three tasks ('Keeping the beat', 'Melting' and 'Magnets').

Figure 3.1 The attainment picture across stages

(% pupils attaining level-based targets, averaged over 8-10 attainment targets and three tasks per stage)

Despite possible questions about rater reliability, the attainment data show clear evidence of stage progression in investigation skills. In particular, we see that, on average, around one-third of the S2 pupils were rated as having demonstrated the attainment target skills at Level E, with just under half having shown evidence of skills attainment at Level D. Around one-third of the P7 pupils showed target skills attainment at Level D, 40% of the P5 pupils did the same for Level C, and over half the P3 pupils were deemed to have shown appropriate evidence of target skills attainment at Level B.

Table 3.6 presents corresponding data for the nine individual tasks, and confirms the general picture, despite some variation from one task to another.

Table 3.6 The pattern of skills attainment task by task

(% pupils demonstrating attainment targets, averaged over attainment targets)

There was no consistent pattern of difference at any stage across the three strands: 'Preparing for tasks', 'Carrying out tasks, 'Reviewing and reporting tasks'.

3.2.2 Gender comparisons

There was no evidence of any gender difference in investigation skills at any stage, even at the level of individual attainment targets.

3.3 Summary

Just over half the survey schools at the primary stages and around two-thirds of the survey schools at S2 participated in the practical assessment of pupils' investigation skills in science, that is 80-90 schools at each stage. In most schools two groups of four pupils were involved, the two groups working on different assigned investigations.

Nine investigation tasks were newly developed for use in the survey. Each task had an accompanying rating checklist containing attainment targets at two adjacent 5-14 levels, which field officers completed in real time as they observed and questioned the investigating pupils. Sets of three tasks were administered at each stage, two of the three being used at one particular stage and the third being used at two adjacent stages. The tasks were administered in the schools by 148 field officers, who were practising primary teachers and secondary science teachers released by their authorities for survey participation.

Across the stages, and averaging over attainment targets and tasks, a quarter to a third of the pupils had to be given assistance by the field officers before they could show enough evidence of relevant skills for them to be judged at one level or the other for each of the attainment targets included in the checklists; 10-25% of the pupils could not be judged at a level at all, because they did not show evidence of the behaviour concerned with or without prompting from the field officer.

The attainment data provided by those pupils who could be judged show clear evidence of stage progression in investigation skills. Again averaging over attainment targets at a level within and across tasks, around one-third of the S2 pupils were rated as having demonstrated the attainment target skills at Level E, with just under half having shown evidence of skills attainment at Level D. Around one-third of the P7 pupils showed target skills attainment at Level D, and 40% of the P5 pupils did the same for Level C. Over half the P3 pupils were deemed to have shown appropriate evidence of target skills attainment at Level B.

There was no evidence in the attainment data of any gender difference in investigation skills.

« Previous | Contents | Next »