The Scottish Government

« Previous | Contents | Next »

Listen

Section C: Assessment materials and procedures

C.1 Reading

The reading tasks

Seventy-two reading tasks were administered in the survey, comprising 12 at each of Levels A to F. Each task took the same general form; a source text or texts followed by a series of questions (test items). Over half (38) of the tasks were newly developed, one-third (24) were pre-existing tasks that had been used in previous AAP surveys of English Language and which had been available in the 5-14 National Assessment Bank for use as National Assessments, and 10 tasks were drawn from those used in the 2004 AAP survey of Mathematics and core skills. Among them, the tasks covered the three different genres identified in the 5-14 English Language guidelines: informative, personal and narrative (the 10 tasks from the 2004 AAP survey were all informative, with texts having a mathematical context). A variety of different themes were featured in the tasks, as the three examples overviewed in Table C1 will serve to illustrate.

Table C1
Overview of three reading tasks

All newly developed tasks had a set number of items at each level, conforming to the situation in National Assessments: Level A tasks had 21 items, Levels B and C tasks had 24 items, Level D tasks had 27 items and Levels E and F tasks had 30 items.

New tasks were developed with particular 5-14 levels in mind, unlike the tasks originally developed for use in the AAP, which were each assigned a 5-14 level retrospectively for the purpose of this survey. A large number of potential source texts, more than were ultimately needed, were each first independently reviewed by at least five teachers. Approximately 100 texts were divided by SQA task developers into two piles on first reading, as being appropriate for lower/middle primary pupils and upper primary/lower secondary pupils. Five teacher validators with experience at the lower/middle stages were then sent the texts in the lower/middle batch, and five teacher validators with upper or secondary experience were sent the texts in the upper/secondary batch. Using their professional judgment, and working independently, the teachers were asked to assign a 5-14 level to each text, as well as rating it for likely pupil interest level (high, medium or low). Only those texts for which at least four of the five validators assigned the same level and rated the task as having medium to high interest value were selected as source texts for task development.

Test items were then produced for each validated source text. In accordance with previous AAP English Language surveys, in their totality the items were designed to tap a variety of different component reading skills

• understanding the main ideas and details of the text;
• distinguishing between different feelings or points of view;
• attributing feelings or atmosphere to characters or situations;
• identifying and understanding aspects of the writer's craft, including figurative language;
• making generalisations and inferences;
• understanding the sequence or structure of events or ideas in the text;
• reconstructing and summarising the overall meaning of the text;
• distinguishing between different types of text and different genres.

Since different source texts lend themselves to differing degrees to the development of different types of test item, item developers were not given a standard test specification to work to. They were simply asked to produce a required number of test items, embracing a small variety of item format, and to avoid an over-emphasis on open-ended items, particularly at the lower levels.

At all levels, the test items ultimately used required only a minimal written response (particularly at Levels A and B), and could be easily marked as right/wrong. Within newly developed tasks, the number of test items per task varied from 21 at Level A through 24 at Level C to 30 at Level E (in readiness for future use as National Assessments). A variety of different formats featured, including multiple choice, aided and unaided summary completion (see Figures C1a and C1b), sentence completion, sequencing (see Figure C2), short response, and, at higher levels, open-ended (see Figure C3). Items sharing common formats were grouped into sections, as illustrated in the task overviews given in Table C1, and in Figures C1a, C1b, C2 and C3). To ensure that pupils tackling Level A tasks were given the amount of support suggested by the 5-14 English Language guidelines, all tasks at this level included an example: "The first one has been done for you" (see Figure C1a) .

Pre-testing and formal task validation

Once fully developed, the resulting tasks were pre-tested on a national sample of pupils at appropriate stages, in cooperating schools that were not at the time intended also to participate in the main survey (although some schools were in fact later asked to participate in the survey as well, if they were located in reporting authorities).

Each new task was completed by at least 100 pupils in two rounds of pre-testing. The pupils' performance results were analysed to ensure that the overall range of difficulty of the items was consistent with the level of the associated text. Items, sections or whole tasks that were found to be problematic in any way were either amended or discarded. Pre-testing was also used to refine answer keys to be used for automatic marking (and later for manual marking by teachers when the tasks eventually became available for use in National Assessments). Devising a completely appropriate answer key for summary completion items in the absence of pupil performance data raises particular problems, as it is difficult to predict the whole range of potentially acceptable answers. Pre-testing and follow-up analysis in advance of the survey were used to construct a provisional set of possible right and wrong responses, that would be confirmed and fine-tuned as necessary at the time of response transcription following the survey.

Figure C1a
Aided summary completion: Flip (Level A), Section D ^*

* This is not a reproduction of Section D as the pupils saw it: both layout and typeface (originally 16pt) are changed here for ease of report presentation.

Teachers participating in pre-testing were given the opportunity to comment on the text, items and mark schemes. Pupils were also given the opportunity to comment on the texts: a Level F task was shelved following pre-testing because pupils gave the text a low interest rating ( topic: Growing herbs).

The tasks which remained after pre-testing were then validated for 5-14 level. Ten practising teachers again participated in the independent validation of tasks, a 'task' being defined as the combination of source text and test items - the whole package. Each new task was independently judged by five different teachers, with lower primary or upper primary/lower secondary teaching experience, as appropriate, and considered validated at a level when at least three of the teachers made the same level judgment.

Figure C1b
Summary completion: Attila the Hen (Level C), Section C ^*

* This is not a reproduction of Section C as the pupils saw it: both layout and typeface (originally 13pt) are changed here for ease of report presentation.

Figure C2
Sequencing: Attila the Hen (Level C), Section B ^*

* This is not a reproduction of Section B as the pupils saw it: both layout and typeface (originally 13pt) are changed here for ease of report presentation.

Figure C3
Open-ended questions: Anglo-Saxons (Level F), extracts from Section C ^*

* This is not a reproduction of Section C as the pupils saw it: both layout and typeface (originally 12 pt) are changed here for ease of report presentation.

Task administration and attainment estimation

At each stage pupils assessed in reading attempted three different reading tasks at three consecutive levels: Levels A, B and C at P3, Levels B, C and D at P5, Levels C, D and E at P7, Levels D, E and F at S2. The reading tasks were presented to pupils as individual test booklets. There were 72 different test booklets in total, one for each of the 72 tasks. At each stage 36 different booklets were administered, 12 booklets at each of the three relevant levels. Pupils were randomly pre-assigned triplets of booklets in such a way that all booklets would be attempted by similarly representative pupil samples, both nationally and within reporting authorities.

The tests were administered by the schools themselves. Supervising teachers could explain what had to be done, but were instructed not to provide answers or confirm that a pupil's answers were correct. The tests were not timed, and schools had flexibility in how they were administered. They were encouraged to give the pupils a break between tests, if possible allowing a day between assessment sessions.

Completed test booklets were processed centrally, with pupils' item responses recorded onto specially designed recording sheets, by circling the appropriate option among a presented list of possible responses. The response data were then keyed, and the responses automatically marked as right/wrong and allocated binary scores. Alpha values (i.e. values of Cronbach's alpha coefficient) for the 72 reading tasks, based on the survey data, are given in Table C2 below.

Table C2
Alpha values for the 72 reading tasks*

The proportions of pupils nationally and in each reporting authority showing 'very good' performance at the level of the task, as being 'well established' at the level, or as having made a 'good start' at the level were then estimated (using criterion cut-off scores of, respectively, 80%, 65% and 50%), with data weighting applied to correct for imbalances in school and authority representation in the pupil samples. In total, over 15,000 pupils were assessed in reading, in over 1,300 schools. The resulting attainment estimates are given in Section E.

C.2 Numeracy

There were two components to core skills numeracy within the survey; 'atomistic' numeracy items drawn from the 5-14 National Assessment Bank and 'mathematical literacy' tasks newly developed for use in the survey and comprising a source leaflet and a set of associated numeracy questions or items.

Numeracy skills were principally assessed in this survey through administration of a total of 530 different atomistic 'pencil and paper' test items: 80 items at Level A and 90 at each of Levels B to F. The items were drawn from the 5-14 National Assessment Bank, pre-classified by level, outcome and strand.

The items were drawn from two of the four outcomes defined in the 5-14 Mathematics Guidelines, namely Number, money and measurement and Information handling; Shape, position and movement and Problem solving were not included. Within the two outcomes the following strands were represented:

Number Money and Measurement

• Range and type of number
• Money
• Add and subtract
• Multiply and divide
• Fractions, percentages and ratio
• Measure and estimate
• Time

Information Handling

• Interpret information

The items were selected at random from the National Assessment Bank, to meet a given survey specification (i.e. numbers of items of given types). The specification demanded 30 'Interpreting information' items at each of the six levels, with 52 items from Number, money and measurement at Level A and 60 at each of Levels B to F. At Level A, 30 of the 53 'number' items were to focus on computation items (add and subtract only, following the 5-14 Guidelines) compared with 40 of the 60 number items at Levels B to F (add, subtract, multiply and divide). Half of the computation items at every level were to be 'context free' and half set in a written context: in 'context free' items the operation to be used is given, whereas in contextualised items it is not.

Figures C4a to C4c give examples of 'context free' and 'in context' number items, as well as examples of 'interpret information' items.

The mathematical literacy tasks

Eighteen 'mathematical literacy' tasks were newly developed for use in the survey. The purpose of these was to assess how well pupils could apply a combination of reading and numeracy skills in order to interpret and use information presented as a mixture of text, tables and other graphical formats.

Each task comprised a multi-coloured source leaflet and 12 related questions, six of which were 'number' items, drawn from the Number, money and measurement strands identified above, and six of which were 'interpret information' items. The source material featured contexts that could be assumed to be familiar even to the youngest pupils, and the information presented was of a kind that pupils at the different stages might reasonably be expected to access and use in their everyday lives:

• a selection of postcards offering goods for sale in a post office window.
• leaflets about the insects and other creatures at an insect 'zoo'.
• a catalogue describing a range of craft goods offered for sale.
• a postcard about undelivered mail is put through someone's letterbox.
• a leaflet explaining the cost of posting mail in the UK.
• a leaflet issued by a city council to encourage householders to recycle garden waste.
• a leaflet explaining how to fill a shoebox with gifts to help families in Romania.

Figure C4a
Examples of 'atomistic' numeracy items at Level C

Figure C4b
Examples of 'atomistic' numeracy items at Level E

The individual test items were level-related, higher levels being associated with increasing complexity of information and more demanding numeracy skills. Entire tasks were targeted at a single level, in the sense that all the items comprising the task were classified at the same level. Most of the contexts exploited in task development lent themselves to the development of more than one task, at one or other of the five levels; one context, 'Insect World', was developed into five tasks, one at each of the five levels (see Figure C5).

Task development and pre-testing

The development of 'mathematical literacy' tasks for the survey built on the model developed for the 2004 AAP Mathematics Survey to assess 'core skills' mathematics, when a small number of tasks were used to investigate the combination of reading and mathematical skills.

It was decided to use a tighter specification than that used in 2004 when developing new tasks for use in the 2005 survey. The two core skill areas of using number and using graphical information were equally represented in each assessment task, and test items were to be level-related, based on the relevant parts of the 5-14 Mathematics Guidelines.

Initially 12 tasks were commissioned, two at each of Levels A, B and C and three at each of Levels D and E. The sources were independently assigned an appropriate 5-14 level by three different reviewers, and then reviewed by the group and, where necessary, a consensus reached about level. However, it was clear that some sources could be used to generate items at more than one level. This was particularly true at Levels D and E, so much so that for two of the contexts the tasks at these two levels shared the same source material. Similarly, some source material could, with a little editing or augmentation, readily be adjusted to cover the level above or below that for which it was originally considered appropriate. In this way the original 12 sets of source material were extended to serve 17 different tasks, three at each of Levels A, B and C and four at each of Levels D and E.

Once developed, each task was pre-tested, and also reviewed once again by between 10 and 14 practising teachers. The reviewers were asked, among other things, to indicate the level which they considered the source material most suited, and also the level they would attach to the majority of the test items in the task. Teachers were encouraged to comment on test items which they considered inappropriate for the 'majority' level, or which had caused difficulties for any reason. In most instances the teachers' judgments confirmed the level of the tasks (both source material and test items), with only one or two teachers out of the 10 to 14 opting for a higher level for any particular task (no teacher opted for a lower level than the majority). The reviewers' comments, along with the pupil results from the pre-tests, helped inform the process of finalising the tasks prior to use in the survey.

Item/task administration and attainment estimation

Paralleling the survey aim in reading, 'atomistic' numeracy attainment was to be estimated at three consecutive levels at each stage: Levels A, B and C at P3, Levels B, C and D at P5, Levels C, D and E at P7, and Levels D, E and F at S2. Mathematical literacy attainment was to be reported at two levels per stage: Levels A and B at P3, Levels B and C at P5, Levels C and D at P7, Levels D and E at S2.

Figure C5
Overview of the mathematics literacy tasks 'Insect World' and example test items

At each stage, multiple matrix sampling was used for administration of the entire set of relevant items and tasks to pupils. Since no prior attainment information was available for the pupils, and since it was considered unacceptable to consider presenting a pupil with a single-level numeracy test that might be a level or two above where that pupil was currently working, or a level or two below, the items and tasks were assembled into mixed-level test booklets for administration, following a given test specification (length and composition).

There were 20 different 'atomistic' test booklets at each stage, each containing a set of items from the three relevant levels: in fact there were 10 booklets pairs, with one booklet in each pair presenting the given set of items in a randomised order with the other simply presenting the items in the reverse order. Items at a level within a booklet at one stage were carried across to appear in one booklet at another stage; these items comprised level-specific 'testlets'. Each booklet at P3 contained 15 Number, money and measurement items and nine Information handling items, thus 24 items in total; at P5, P7 and S2 the corresponding numbers were 18 items and nine items, respectively, viz. 27 items in total. Testlets comprised 16 items at P3 and 18 at other stages.

The mathematical literacy tasks were paired to create test booklets, each booklet containing tasks at two different levels.

Prior to the survey, test booklets were randomly distributed among the survey pupils in such a way that any one pupil was allocated two different 'atomistic numeracy' booklets and one mathematical literacy booklet. The 'atomistic numeracy' assessment was given priority in the survey, and for this reason the two numeracy booklets were to be attempted in the first two of the three assessment sessions that schools had been asked to organise for their pupils, with the mathematical literacy booklet taken in the last session.

As in reading, pupils' item responses were recorded onto specially designed response record sheets. 'Transcribers' simply circled the response option matching the pupil's response, within a presented list of letter codes or keyterms indicating the alternative response possibilities item by item. The responses were then automatically marked, and dichotomous scores allocated. Table C3 records the Cronbach alpha coefficient values for the various level-specific testlets, based on the resulting survey data.

Also as in reading, pupils were deemed to have shown 'very good' attainment at a level, in numeracy or in mathematical literacy, if they successfully answered 80% or more of the items they were presented with at that level. They were deemed 'well-established' at the level if they answered 65% of the items correctly but not as many as 80%. They were deemed to have made a 'good start' at the level if they successfully answered half the items at the level but not as many as 65% ⁹. The proportions of pupils in each reporting authority, and in the country as a whole, falling into each of these attainment bands were then estimated (see Section F), as were the corresponding margins of error.

C.3 Writing

Two different types of writing featured in the survey: class-based writing tasks and writer's craft tasks (this latter type of writing was undertaken within the practical component of the survey - see section C.4). Schools were asked to submit a piece of extended writing, completed prior to the survey as part of normal course work, for designated pupils . The genre of writing required from individual pupils was prescribed, through random allocation: functional, personal or imaginative. The sub-genre, topic, theme or context of this writing was entirely a matter for the schools to decide.

Table C3
Alpha values for the numeracy testlets*

Selecting and assessing class-based writing

Teachers were given advice on selecting appropriate material, and on how much teacher support was permitted. The piece of writing selected was to reflect the level at which the pupil was currently working. Schools were advised that each piece of writing should be assessed by the class teacher and one other teacher or a promoted member of staff from the school, using the 5-14 national writing criteria. Teachers were asked not to annotate the level with '+' or '-' or to record two levels e.g. D/E. The level awarded was to be recorded on the register provided but not recorded on the script itself. Where there was disagreement between the two markers, teachers were asked to discuss and come to a final decision.

Schools were informed that a proportion of randomly selected scripts would be centrally moderated by trained education authority representatives. In total, schools submitted 7600 pieces of pupil writing, of which around 3000 were randomly selected for moderation.

The moderation of submitted class-based writing

The moderation activity took place over five days, at the end of August 2005. All 32 local education authorities were invited to nominate up to two teachers experienced in judging writing to participate in the exercise, with the intention of having at least one representative from each authority and two from the larger authorities. In practice, there were 42 participants - 30 primary teachers and 12 secondary teachers - from 24 education authorities.

The set of randomly selected scripts was organised into batches of approximately twenty scripts each for issue batch by batch to the moderators throughout the week. Teachers with lower or middle primary experience were assigned P3 or P5 scripts, while those with upper primary or lower secondary experience were assigned P7 or S2 scripts. For logistic convenience, the 42 participants were grouped into pairs. Teachers with lower/middle primary experience were paired together, and primary teachers with upper stage experience were paired with secondary teachers. During the moderation, meeting participants did not know who they were paired with, since each was to work independently of the other when judging scripts.

In addition to judging scripts, the participants were addressed from time to time by guest speakers, and a number of activities relevant to writing also took place. In particular, the week was punctuated with plenary discussions focusing on particular pieces of writing (which were not themselves moderated), in order to facilitate a shared understanding of standards and an evaluation of the 5-14 national writing criteria.

The process began with teachers reading the piece of writing and then offering comments on the strengths and weaknesses. Participants were encouraged to challenge any comment with which they disagreed. The next step, using professional judgement, was to suggest a 5-14 level for the piece of work. The outcome typically straddled two levels, and sometimes more. An analytical approach was then adopted using the bullet point descriptions of attainment stated in the national criteria for extended writing. Each bullet point across the levels was discussed until a level was finally agreed. This was a time consuming process, but all teachers agreed that it was a necessary and invaluable experience. In addition to promoting an understanding of standards, the discussion permitted the production of material for creating exemplification of extended writing and evaluation of the writing criteria. Throughout the week, participants also had opportunities to share information about any innovative practice that might be going on in a particular class, school or authority.

Throughout the week, the teachers also identified factors which made assessing scripts using the extended criteria problematic and suggested that in future surveys the writing requested from schools should be continuous writing - i.e. not posters, leaflets or sets of instructions. Some participants suggested that a note of the task, purpose and audience should be attached to each piece of writing submitted. However, a significant number of others cited the workload implications of such a request.

In total, 3050 scripts were processed in the moderation meeting, each script therefore benefiting from three independent judgements about level, two moderator judgements and the original class teacher's judgement - a good basis for exploring the reliability with which pupil's writing can be judged for level. There were no genre differences evident in the data, and neither did the moderator agreement rates change from the beginning to the end of the moderation week.

Of the 3050 moderated scripts, 93% attracted the same level judgments from at least two of the three independent evaluators (class teacher and two 'moderators'), and so were able to be given an 'agreed' level through 'majority decision'. In one-third of cases, all three evaluators came to the same independent decision about level.

At P7 and S2 75% of the scripts with a majority decision after moderation retained the level originally allocated by the pupils' class teachers, with a higher 80% doing so at P3/P5. Table C4 records the results of the moderation in detail, level by level.

Table C4
The effect of moderation on class teachers' original level judgments*
(% scripts at each level after moderation: 2842 in total)

The majority (just under 80%) of the scripts originally rated as Level F (just 44 scripts in total) were 'downgraded' to Level E after moderation, while a fifth of the scripts originally rated as Level E by the class teachers became Level D and 15% of those originally judged as Level D became Level C. In contrast, a quarter to a third of the scripts originally rated as Level A or below were 'upgraded' to Level B. Level B showed the highest proportion of unchanged level judgments, at almost 90%, with no particular trend towards an increase or decrease in level for those 10% that were changed.

C.4 Practically assessed skills

Pupils' performances in a number of different skill areas were assessed within the practical component of the survey, with field officers organising and supervising assessment sessions and in some cases making attainment judgments. ICT skills, problem solving skills, 'working with others' and 'talk' all featured here. Listening skills were also included in the practical component - partly to relieve pressure on the written survey and partly because of the need for audiovisual equipment to be used. There was also some assessment of writing ('writer's craft') and 'knowledge about language'.

All the field officers were involved in all the different types of assessment, working in pairs in their assigned schools. They were given one day of pre-survey task orientation before visiting their five schools, spending one day in each to conduct the assessments, and participated in one de-briefing day after the event. In the assessment of pupils' listening skills the field officers were simply required to organise and supervise assessment sessions - test booklets would be marked centrally before analysis. In ICT they were required to observe pupils in action, and to use a given protocol to record their observations, and sometimes their judgments, for later analysis. In the assessment of talk, the field officers had clearly to take an interactive role, engaging individual pupils in dialogue whilst subjectively rating their talking skills.

The types of task used in each assessment and the procedures adopted to produce attainment results are described below. The attainment results themselves are given in Section H.

Listening

Pupils' listening skills were assessed using six different tasks. One task featured a short video clip as source material, whilst five further tasks featured short audio clips. In keeping with the English Language focus of the survey, four of the six clips had a language focus.

Pupils' information recall was explored through a series of questions, or test items, contained in a test booklet. The video task and two of the audio tasks were used at all four stages, although with different test booklets at P3/P5, P5/P7 and P7/S2. The remaining three audio tasks were stage-specific: one was used at P3/P5, one at P5/P7 and one at P7/S2.

The test questions (multiple-choice items) were based on the attainment targets contained in the 5-14 Guidelines for Listening/Watching, with around 20 questions in each test booklet. Questions in the P3/P5 booklets were at Levels A, B and C, those in the P5/P7 booklets were at Levels B, C and D, while those in the P7/S2 booklets were at Levels D, E and F (see Figure C6 for examples). Several questions were used at more than one stage, and each audio-based task contained two or three 'Knowledge about language' questions (these did not contribute to the assessment of listening skills).

Figure C6
Examples of listening test items

The audio and video clips were validated for level and stage suitability by a group of currently practising teachers. The same group also validated each test item for level suitability. The tasks were pre-tested in schools prior to use in the survey

Pupils' listening skills were assessed within the practical component of the survey, with assessment sessions organised by the field officers. Typically, in each school visited three to four pupils attempted one task, watching the video clip or listening to the audio clip together before individually answering the questions in their test booklet. Booklets were processed centrally, with pupils' item responses being recorded onto specially designed record sheets, as for reading and numeracy, and later automatically marked as right or wrong.

Level attainment is reported in terms of average facilities on items at the same level across the different listening tests (see Section H.1).

Talking

Talk was assessed through one-to-one conversations between a field officer and a pupil, with the second field officer observing and eventually making a judgment about the pupil's level of attainment, using a 'best fit' scheme (see Table C5). Pupils were invited to choose one of four topics to talk about, topics which is was hoped would have interest value for pupils and about which they would be able to speak for several minutes in conversation: school, a hobby/pastime, their favourite TV programmes/films, or a piece of their own written work.

In order to ensure a degree of standardisation in the field officers' level judgments, a training DVD was produced for use in task orientation. During task trialling, forty-eight pupils (twelve at each stage) in various schools across Scotland were filmed in conversation with an assessor. From these 48 video clips, 30 were selected for moderation by a group of currently practising teachers, who assigned a 5-14 level and attached explanatory commentary to each clip using the 'best fit' descriptors. These clips and commentaries were then used as part of the training process for the field officers.

In each of the schools that participated in the practical assessments, typically three to four pupils were involved in the assessment of talk. The results of the exercise are given in Section H.2.

Writer's craft tasks

The writer's craft tasks required pupils to analyse, with teacher support, a short piece of text, and then to continue the text in the same style used by the original author. The tasks focused on the strands 'Reading to reflect on the writer's craft' and 'Awareness of genre', within the outcome Reading in the 5-14 National Guidelines for English Language. The stimuli were narrative texts and required a written response using the imaginative genre.

Identifying the 5-14 level of a writing task is not an exact science, and it is accepted that one task can provide sufficient challenge for pupils writing at two consecutive levels. Thus several tasks were classified as Level B/C tasks or Level D/E tasks. Level A and Level F bring their own particular challenges in terms of task validation, and for this reason writing tasks were specifically created for the relative degree of skill and experience of the target audience at these levels. Table C6 offers an example of context for one task at each level or level pair.

The development, administration and assessment of the writer's craft tasks used in the survey followed the model used for national assessment tasks. A large number of texts covering the range of 5-14 levels was gathered and then sent out for independent validation by five teachers. As well as using their professional judgement to assign a 5-14 level, validators also gave each text an interest level (high, medium or low). Only those texts that were assigned the same 5-14 level by at least four of the five raters, and which attracted medium to high interest ratings, were selected for developing into writer's craft tasks for use in the survey.

Table C5
The 'Best Fit' Scheme for judging level attainment for Talk

Source: Scheme derived from the attainment outcome Talk in the 5-14 English Language Guidelines

Table C6
Overview of four writer's craft tasks

The selected texts were trialled in schools. During the trialling, some of the cooperating class teachers were asked to provide information on how they administered the scripts, and to forward a selection of pupils' responses and task reactions. This information was used to compile protocols and exemplification for field officers to use when administering the tasks within the survey and assessing the resulting pupil scripts.

In total, 12 different tasks were developed for survey administration: three each at Level A, Level B/C, Level D/E and Level F. P3 pupils attempted Level A tasks and Level B/C tasks, P5 and P7 attempted Levels B/C and Levels D/E tasks, and S2 attempted Level D/E and Level F tasks. Three to four randomly selected pupils attempted a writer's craft task in each 'practical' school, all the pupils independently attempting the same task after some group stimulation by the field officer.

Field officers read aloud the starter text and asked pupils to identify some of the features particular to the kind of writing. Through questioning - "Who are the characters?", "What is happening?", "Where is the story taking place?" - pupils established involvement with the text. Through further questioning appropriate to the level, features of the writer's style used to tell the story were identified, i.e. choice of language, content, sentence structure, punctuation and tone. Pupils were asked for a few suggestions as to what might happen next. They were given a short time to think alone and/or to share thoughts in pairs or groups. They could make notes if they wanted to. Then pupils were asked to write the next part of the story in the same style as the writer, paying close attention to how the writer had told the story so far. Pupils were not asked to finish the story, nor even to bring it to any conclusion. This was to be a short piece of writing.

The pupils' writing was assessed by the field officers using a 'best fit' scheme - the same scheme that is used by teachers when evaluating pupils' work in the National Assessment Programme.

The 'best fit' scheme acknowledges that a pupil needs to be able to fulfil most of the description for a particular level to be awarded, but if there is one weakness in the writing - perhaps in spelling or punctuation - the level concerned may still be achieved. If there is more than one weakness, however, it cannot be said that the pupil is confidently working at that level. Field officers were given advice on assessing writer's craft tasks during their task orientation, and they were provided with exemplification and guidance documents. A ceiling was placed on the level which could be awarded a script depending on the task level, e.g. if the task had been validated at Level B/C then the highest level which could be awarded was Level C.

Pupils' attainments at each stage are reported in Section H.3 in terms of the percentages awarded the different 5-14 levels.

Knowledge about language

Pupils' 'Knowledge about language' was assessed using three tasks at each stage. Each task consisted of a short passage accompanied by a set of associated questions, or test items. The test questions were based on the 'Knowledge about language' strands contained in the 5-14 Guidelines for Reading and Writing. Tasks were presented to pupils in the form of test booklets.

The three tasks used at the different stages were linked, in that the source passages used at P3 became longer and more complex for P5, and so on. The number of items also increased from 12 in the P3 tasks to 15 in the P5, P7 and S2 tasks, with some items used across stages. Questions were at Levels B and C in the P3 booklets (there are no KAL attainment targets at Level A), at Levels B, C and D at P5, at Levels C, D and E at P7, and at Levels D, E and F at S2 (see Figure C7 for examples). Some 'Knowledge about language' items also appeared in listening test booklets (see above).

Figure C7
Examples of 'Knowledge about language' test items

The tasks were trialled in schools prior to survey use, and practising teachers validated each item as being appropriate for its given level.

In each school visited by the field officers three to four pupils typically attempted one booklet each, different pupils attempting different booklets. Findings are given in Section H.4.

ICT skills

The framework for the assessment of ICT was developed following consultation with an ICT Focus Group consisting of teachers from a variety of schools across Scotland. Six newly developed, two for use at P3/P5, two for use at P7/S2 and two for use at all four stages (see Table C7).

The tasks focused on the assessment of the strands 'Searching & Researching', 'Creating & Presenting' and 'Using the Technology', as defined in the 5-14 Curriculum Guidelines for this subject. The tasks were set in a language context, and each required pupils, working in a small group, to produce a multimedia presentation for a given audience on a choice of topics. The general aim was to assess pupils' ability to find information, to transform the information into a form suitable for their purpose, and to present the information in a way that took into account the audience for their presentation. Specially developed checklists in principle allowed an overall attainment level to be awarded to each pupil for ICT skills.

An accompanying questionnaire, one version for P3/P5 and one for P7/S2, was used to explore pupils' attitudes to ICT and to gather some contextual information (see Section D.3). A self-assessment element focused on knowledge and skills within the strand 'Using the Technology'.

Pupils worked in pairs on their assigned task. One field officer introduced the task, and familiarised the pupils with the laptop computer that they were to use and the resources made available to them (a selection of task-relevant text and graphics, held in a folder on the computer with print copies also supplied). The field officer then encouraged discussion about the task, which the pupils were allowed approximately 40 minutes to complete. The field officer observed the pupils as they worked and noted any observations on a recording sheet. After saving their work the field officer carried out a structured discussion with the pupils. This discussion was based on a protocol consisting of questions moving from Level A through to Level E (see Table C8).

The field officer then used information gathered from observations during the task and from the post-task discussion to award an ICT level to each pupil using 'best fit' descriptors based on the 5-14 Guidelines for ICT (see Table C9).

The resulting attainment findings are presented in Section H.5, while the questionnaire results are discussed in Section I.8.

Table C7
Overview of the ICT tasks

'Working with others' and 'Problem solving'

Pupils' behaviours and achievements when 'Working with others' and 'Problem solving' were observed and assessed by the field officers, using four different discussion-based problem solving tasks (see Table C10 for an overview). The purpose of the tasks was to assess how pupils worked together to solve a problem within a language-based theme.

Two of the tasks were sequencing tasks, involving sorting and recreating the sequence of different segmented texts. Two different sequencing combinations were used at each stage, using texts which covered a variety of genres: stories, instructions on how to play a game, a recipe, an information text about pancakes and instructions on how to juggle. The third task focused on creation of a display about the Loch Ness monster, while the fourth was an advertising task that required pupils to choose a name for a new snack bar and decide how to advertise it. All the tasks were trialled in schools prior to use in the survey, and judged to be appropriate for the stages to be assessed. Trialling did not extend to inter-rater studies.

Table C8
The ICT post-task questioning protocol ^*

^* Responses were recorded as 'Y' for 'Yes - answered/demonstrated appropriately', 'N' for 'No - answered/demonstrated incorrectly' or '?' for 'Unable to respond'.

Table C9
The 'best fit' rating scheme for ICT

Table C10
Overview of the problem solving tasks

All four tasks were used at the four stages, with some slight variations in source material and/or resources. In each school a group of up to four pupils in the main survey sample were randomly selected to take part in the problem solving discussion, with different tasks used in different schools on a rotation basis. The field officers explained to the pupils what they were to do and what it was that they, the field officers, would be looking for. They used task-specific instructions to take pupils through the task and then only provided support if absolutely necessary.

There was no attempt to assign levels to pupils' problem solving skills, either individually or as group, nor to assign levels for their skills when 'working with others'. The assessment focused rather on describing how pupils individually worked together to solve a problem. For example, did a pupil contribute lots of ideas or only a few, did the pupils listen to each other's ideas and respond in a positive fashion, did a particular pupil dominate or disrupt a discussion? The field officer completed a checklist based on observation of individual pupils as they engaged in the problem solving activity, using the same checklist for all the tasks and at all stages. The behaviours rated in the checklist for 'working with others' are shown in Table C11.

Table C11
Behaviours in the 'Working with others' checklist ^*

*Each pupil was rated for working with others while the group was rated for problem solving.

The pupils were given around 20 minutes to complete their problem solving activity, after which time the field officer invited the group to reflect on what they had just done, prompted with questions from task-type specific protocols (an example is given in Table C12).

Table C12
Questioning protocol: advertising task

As a group, the pupils were then assessed on their problem-solving skills, in terms of how they approached the task, how they decided to proceed with solving the problem, whether they reviewed/amended their strategy as they carried out the task, whether roles were assigned or adopted within the group, and whether the task was successfully completed (see Table C13).

Table C13
Aspects rated under 'group problem solving' ^*

* Groups were rated 0, 1 or 2 for, respectively, 'no evidence', 'some evidence' and 'strong evidence with minimal support'.

The results of this assessment by field officers are presented in Sections H.6 and H.7.

« Previous | Contents | Next »