« Previous | Contents | Next »
Listen
Section 4 Writing and Talking
Assessing Writing attainment - extended writing
4.1 No assessment of extended writing took place within the survey itself. Instead, schools were asked to provide an estimate of a level in writing for all sampled pupils, and to submit a piece of extended writing produced during normal class lessons for a random third of the sampled pupils in each school, a total of 3,000 pupils at each stage. Schools were asked to provide functional, personal or imaginative writing for each sampled pupil involved, the genre having been randomly assigned to pupils beforehand. The topic, theme or context of the writing within these three broad categories was entirely a matter for the schools to decide.
4.2 The piece of writing selected was to reflect the level at which the pupil was currently working. Teachers were given advice on selecting appropriate material and how much teacher support was permitted. Schools were advised that each piece of writing should be assessed by the class teacher and one other teacher or a promoted member of staff from the school, using the 5-14 national writing criteria. Schools were told that a proportion of randomly selected scripts would be centrally moderated by a team of teachers recruited from education authorities.
4.3 Scripts with level judgements attached were received for around 2,000 pupils at each stage, which is just over two-thirds of the pupils originally selected for writing submission. There were several reasons given by schools for not submitting writing, or submitting pieces in a genre other than specified. In some cases, there was no suitable piece available that had been produced recently enough to show the level that the pupil had reached. Schools may wish to consider whether they give pupils sufficient appropriate opportunities to write in a variety of genres.
Assessing Writing attainment - Writer's Craft
4.4 In addition, a sample of over a thousand pupils (250-300 at each stage), from a relatively small number of sampled schools, undertook Writer's Craft tasks as part of the survey's practical assessments with field officers. Up to four pupils at a stage attempted the same Writer's Craft task in any one school.
4.5 Writer's Craft tasks assess both reading and writing skills, thus bringing together different strands relating to writer's craft and genre from the reading and writing attainment outcomes in the 5-14 English language guidelines. Field officers read aloud a 'starter' text and asked pupils to identify some of the features particular to the kind of writing. Through further questioning appropriate to the level, pupils were helped to identify features of the writer's style and to suggest what might happen next. They were given a short time to think alone and/or to share thoughts in pairs or groups and then invited to write the next part of the story in the same way as the writer, paying close attention to how the writer had told the story so far.
4.6 The Writer's Craft scripts were assessed by the field officers, using 'best-fit' criteria, the same criteria used by teachers to assess pupils' understanding of the writer's craft through National Assessments.
Moderation of Writing
4.7 Moderation of writing took place over five days, at the end of August 2005. All education authorities were invited to nominate up to two teachers to participate in the moderation event, with the intention of having at least one representative from each authority and two from the larger authorities. In the event, 42 teachers took part, 30 primary teachers and 12 secondary teachers, from the 24 cooperating local authorities. The aims of the event were to increase the validity and reliability of the survey attainment data, to evaluate the national writing criteria and to offer professional development for participating teachers.
4.8 Representative samples of extended writing scripts were randomly selected for moderation and organised into batches. The scripts in each batch were marked by two teacher moderators working independently of each other. Therefore, each moderated script, between 600 and 850 at each stage, had three independent judgements as to the level: two moderator judgements and the original school's judgement. In over 90% of cases, a 'majority' decision emerged: at least two of the three judges agreed about the level to be recorded. For around one in three of the pieces of writing moderated there was unanimous agreement about the level the pupil had demonstrated. There was no evidence of any difference in attainment related to genre.
4.9 The extent of agreement between teachers and moderators would suggest that teachers have a reasonably good understanding of the standards expected at the various 5-14 levels in writing, particularly at P5 and P7. Many, though not all, local authorities can be confident that the levels schools are reporting to them are reasonably accurate: the proportions of scripts - all stages combined - for which one or both moderators agreed the same level as the original class teacher(s) varied across authorities from around 65% to 80%. This encouraging position could reflect the considerable effort that has gone into improving teachers' confidence and expertise in the teaching and assessment of writing over the past several years.
4.10 Writer's Craft scripts were also subject to moderation, though in smaller numbers, since priority was given to the extended writing scripts. As the Writer's Craft scripts had already been assessed by field officers, any script handled during the moderation would have at least two judgements assigned to it. Of the Writer's Craft tasks completed, moderators assigned levels to around half of P3, P5 and P7 scripts, and a third of S2 scripts. In around 40-45% of cases at P7/S2 and 55% of cases at P3/P5, moderators and field officers gave the same level judgement to scripts.
Extended Writing attainment
4.11 The results below are for writing attainment as reported by schools for all their sampled pupils. (These are not the levels for pieces of writing actually submitted).
4.12 At P3 and P5, schools reported almost all their sampled pupils as attaining one of two levels. - Around nine out of ten of sampled pupils were reported as attaining Level A or B, or Level B or C, respectively.
- About a third of P3 pupils were reported as attaining Level B and a half of P5 pupils as attaining Level C.
- Few P3 and P5 pupils (less than one in ten) were reported as attaining Level C and D, respectively, or as not yet having attained Level A or B respectively.
|
4.13 At P7 and S2, schools also reported almost all their pupils as attaining one of two predominant levels. - At P7, schools judged nearly two-thirds of pupils as having attained Level D or above, with one in seven attaining Level E or above. They reported nine out of ten of P7 pupils as having attained at least Level C.
- At S2, the corresponding proportions were just over half at Level E or above and just over one tenth at Level F, with over four fifths attaining at least Level D.
- For lower attaining pupils, schools reported one in ten of P7 pupils and more than one in seven of S2 pupils at Levels B and C respectively, that is not yet attaining the levels below those expected for their stage.
|
4.14 Patterns of attainment reported by schools across the four sampled stages are very consistent. Rather more P5 pupils are reported as attaining Level C than Level B at P3, suggesting steady gains in skills between the two stages. Proportions of pupils at S2 attaining Levels D and E or above are around 10% lower than the proportions attaining Levels C and D or above in P7. Very similar proportions of P7 and S2 pupils are reported as attaining Levels E and F, the levels beyond those expected.
Writer's Craft attainment
| 4.15 Field officers rated just under four fifths of the responses to the writer's craft tasks from sampled P3 pupils as Level A or above, and just over four fifths of tasks from P5 pupils at Level B or above. Just over a fifth of P3 pupils' tasks were judged to be at Level B or above, and nearly half of P5 tasks to be at Level C or above, beyond the expected level. |
| 4.16 For P7 and S2, just over three-quarters of P7 tasks were rated as Level C or above, and just under two-thirds of S2 tasks were rated as Level D or above. Field officers judged around a quarter of the P7 tasks to be at Level D or above and around a third of S2 tasks to be at Level E or above, the expected levels for these stages. |
4.17 Field officers' judgements of levels for the relatively small sample of pupils' Writer's Craft tasks were consistently less positive than teachers' judgements about writing overall, particularly at P7 and S2, where differences were between 20 and 40 percentage points.
Gender Differences in Writing
4.18 For extended writing, gender differences in level profiles are statistically significant at all four stages, with girls' writing generally more highly rated than boys' writing.
Changes in Writing attainment over time
4.19 Figure 5 shows the estimated attainment results for writing in 2001 and 2005 at P7 and S2 (no data are offered for P3 or P5, since these stages did not feature in the 2001 survey).
Figure 5
Writing attainment rates at P7 and S2 in 2005 and 2001
(% pupils at the given level)
4.20 While there are differences shown in Figure 5 between 2001 and 2005, it cannot necessarily be assumed that population attainment has indeed changed in the interim. This is because in 2001 results are based on the judgements of single teacher raters (though not the pupils' own class teachers) judging submitted scripts, whereas in 2005 they are based on the majority agreements of three independent evaluators (the class teacher and the two external moderators). The 2005 data are in consequence more reliable than the 2001 data. Indeed, the message that should be taken from Figures 6a and 6b is that writing attainment rates have emerged as closely similar in the two surveys, except at P7 Level E, despite these differences in procedure.
4.21 Figures 6a and 6b compare the writing attainment trend information from the 5-14 National Survey of Attainment (1998-2004) and the AAP/ SSA (2001 and 2005, respectively).
Figure 6a
Trends in Writing attainment at P7 at Levels C, D, E
(% pupils 'having well-established skills' at the indicated levels)
Figure 6b
Trends in Writing attainment at S2
(% pupils 'having well-established skills' at the indicated levels)
Note: National Survey results are 1998/99-2003/04 National Survey of 5-14 Attainment Levels, 2004/05 are sampled teacher judgements as provided in the SSA. AAP/ SSA results are from the AAP in 2001 and the SSA in 2005.
4.22 At P7, the AAP/ SSA results show a relatively stable picture across the period at Levels C and D and a decrease at Level E (it must be remembered, however, that the data from the 2001 and 2005 surveys are not directly comparable, since the 2005 data are moderated and those of 2001 are not). The data deriving from teachers' judgements show steadily increasing trends over the first five years at all levels, levelling off or even beginning to decrease slightly in the last two years. Teachers' judgements and survey results are in close agreement in both survey years and at all levels, with the exception of Level E in 2001, where the survey showed a higher attainment proportion.
4.23 The AAP/trend in SSA results at S2 show little if any change over the period at Level E, but an increase in attainment proportions at Level D (but again, it should be remembered that the data from the 2001 and 2005 surveys are not necessarily comparable, given that the 2005 data were moderated and those of 2001 were not). The national 5-14 survey data show steady increases at all levels, reaching a plateau and even slightly decreasing most recently. Survey results and teachers' judgements most closely coincide at Level D, but less so at Levels E and F.
Assessing and reporting Talking
4.24 The practical assessments involved relatively small numbers of pupils in a sub-sample of the survey schools, 250-300 at each stage. While no inter-rater agreement trials were conducted before the survey, at their initial briefing sessions field officers watched a video of pupils undertaking the talking tasks, applied the relevant criteria, and discussed their judgements about levels with one another.
4.25 To assess pupils' talking skills, the field officers talked individually with randomly selected pupils in each sample school, engaging the pupil in a dialogue and eventually allocating a 5-14 level to the pupil's performance. Each pupil was given a choice of dialogue topic: their interests (hobbies, clubs, etc), TV/Films, school, or a piece of their own writing. The 'best fit' criteria used to assess pupil talk were derived from attainment targets and programmes of study for the outcome Talking in the 5-14 English language guidelines.
4.26 The highest proportion of pupils, at just under 40% at every stage, chose to talk about their interests, followed by school and TV/Films, each chosen by around a quarter of the pupils, with their own writing the least popular subject, at just over 10% of pupils. Boys tended to choose to talk about their interests and, at P7/S2, TV/Films. Girls tended in greater numbers than boys to choose to talk about school. There were no differences in the attainment results related to topic choices.
Talking attainment
4.27 The results for Talking provide a general overall picture of performance in individual talk across the various 5-14 levels set out in national 5-14 guidelines, although the sample of schools was small and not fully representative (see section 1)
4.28 Looking at patterns of attainment across the stages, the proportions at each stage estimated as attaining the successive levels in general decreases with age. Whereas around two-thirds of P3 were reported as attaining Level B in Talking, the proportions at P5 and P7 at Levels C and D are around a half, and by S2 at Level E have reduced to a third of the sampled pupils. |
| 4.29 The patterns for slightly lower attainment for the four stages, at Levels A, B, C and D, are very similar. For the lowest attaining pupils, the numbers estimated as not attaining the level below that expected increase steadily from stage to stage, with twice as many pupils in this category in S2 as in P5. |
4.30 It is interesting to note that P3 pupils were rated more highly for talking than for writing at Level B, as might be expected. P5 pupils were rated similarly for talking and writing at Level C. In contrast, P7 and S2 pupils were rated more highly for writing than for reading, by 10% in P7 and by 20% in S2, perhaps reflecting the priority given to written work at these later stages.
4.31 Field officers reported that in general, primary school pupils were more willing to engage in open dialogue than secondary pupils. They felt that this could be attributed to the natural reticence of many 13-14 year-old pupils, particularly when talking with strangers.
Gender differences in attainment in Talking
4.32 At all stages the gender differences in level profiles are statistically significant. Girls' talking skills were generally more highly rated by the field officers than were the boys'.
Assessing and reporting Working with Others and Problem-solving
4.33 As for all practical tasks, the sample size for Working with Others and Problem-solving was relatively small in comparison to samples for the written tasks used elsewhere in the SSA. Just over 800 pupils participated from just under 250 schools, around 200 pupils from 50-60 schools at each stage.
4.34 The purpose of the tasks was to assess how pupils worked together to solve a problem within a language-based theme. There were four different Problem-solving tasks, two involving sequencing segmented text, one creating a display and one designing an advertisement. The two sequencing tasks used texts in a variety of genres, with some texts used with pupils at adjacent stages. For the display task, pupils at P7 and S2 worked with a greater range and complexity of information than P3 and P5. The same advertising task was used at all stages.
4.35 All four tasks were used at all four stages. Pupils worked in groups of up to four. The field officers explained to the pupils what they were to do and what it was that the field officers would be looking for. They talked pupils through instructions for the task and then only provided support if absolutely necessary.
4.36 The assessment focused on describing how individual pupils worked together with others to solve a problem, suggesting new ideas, building on others' ideas and showing motivation to complete the task. The two sets of skills were assessed together. Each pupil was rated for Working with Others while the group was rated for Problem-solving. The group was rated on a 3-point scale, from showing no evidence of the behaviour or skills, to showing some evidence, with support from the field officer, to showing strong evidence, with little support.
4.37 Field officers completed an assessment grid based on observation of pupils as they engaged in the Problem-solving activity, using the same grid for all the tasks and at all stages. For Working with Others, the assessment focused on Getting the task done, by contributing New ideas, Building on others' ideas, and showing Motivation to participate and complete the task. During the discussion, field-officers observed pupils' skills in participating and adopting roles within the group. For Problem-solving, they considered pupils' approaches to Understanding & exploring the problem with others in the group, Resolving the problem by reflecting critically on progress and adapting suitable strategies, and eventually Completing the task. After pupils completed the activity the field officer invited the group to reflect on what they had just done, using agreed prompt questions.
Attainment in Working with Others and Problem-solving
4.38 Field officers rated individual pupils' skills in Working with Others. - Across all the stages, the general picture is that eight out of ten of pupils contributed new ideas, and nine out of ten listened to others' ideas and built on them, were motivated to complete the task, and contributed to the discussion without dominating it or being disruptive.
- There were differences between P3/S2 and P5/P7. Roughly twice the proportion of pupils in P3 and S2 compared with P5 and P7 made little or no contribution to the discussion and/or showed no evidence of listening to the ideas of others and building on them.
- Rather more pupils at P3 and S2 compared with those at P5 and P7 adopted a disruptive role or showed signs of being disengaged.
|
Problem-solving attainment
4.39 Field officers rated groups for Problem-solving. - Over 90% of groups across all stages showed some evidence or strong evidence of completing the task successfully.
- For Understanding and exploring the problem and Resolving the problem there was again a difference in ratings for P3/S2 and P5/P7. Only just over 10% of groups at P5 and P7 showed no evidence of these skills, whereas at P3 and S2 the equivalent proportions were nearly 40%, and nearly 30%.
|
4.40 These results suggest that pupils' skills in participating and working in groups develop well through the primary years. As might be expected, pupils and groups at P3 needed more support than those at later stages to participate in group activity and complete a problem-solving task successfully. Profiles for pupils at P5 and P7 were very similar, with pupils showing greater confidence, participating in their groups and completing the tasks very successfully.
4.41 As for individual talk, the S2 picture is less positive, with pupils not participating or committing themselves to completing tasks as well as the primary pupils. This finding is supported by comments from the field officers at their de-briefing. Evidence from the teachers' and pupils' questionnaires suggests that S2 have regular opportunities to work in pairs and groups in English lessons, but it may be that in the survey context, where they might be working with pupils they did not know well, they were less willing to commit themselves to the task in hand.
« Previous | Contents | Next »