The Scottish Government

« Previous | Contents | Next »

Listen

Annex II: Assessment materials and procedures

II.1 Administration and marking of Social Subjects, reading & numeracy

Test booklets were randomly distributed among the survey pupils in such a way that any one pupil was allocated two different numeracy booklets and one social subjects enquiry skills booklet, or three reading booklets. The two numeracy booklets were to be attempted in the first two of the three assessment sessions that schools had been asked to organise for their pupils, with the enquiry skills booklet taken in a third session. Pupils were assessed at the following levels.

Table 1
Allocation of 5-14 levels by stage and subject

The tasks were administered by the schools. The supervising teachers were asked to give out the assessment materials and to supervise the pupils while they were working. The teacher could explain what had to be done, but was not allowed to provide answers or confirm that a pupil's answers were correct. The tasks were not timed, but were designed to last around 40 minutes at P3 / P5 and around 60 minutes at P7 / S2. Schools had flexibility in how they were administered. They were encouraged to give the pupils a break between each task, and if possible set each task on a different day. Schools returned all the test material for marking once all the tasks had been administered.

Pupils' responses were recorded onto specially designed response record sheets. 'Transcribers' simply circled the response option matching the pupil's response, within a list of letter codes or numeric answers or keyterms indicating the alternative response possibilities item by item. The data from the coding sheets were professionally keyed and the responses were then automatically marked and dichotomous scores allocated.

II.2 The Social subjects enquiry skills written tasks

Nature of the assessment

Thirty enquiry skills tasks were administered in the 2006 survey, six at each of levels A, B, C, D and E; all were newly developed tasks. Within each level, two tasks were developed for each of the three 5-14 Social Subjects attainment outcomes ' People in the past', ' People and place', and ' People in society'. Some topics were developed at more than one level.

The tasks comprised a multi-coloured source leaflet and associated questions. The sources contained a mixture of text, pictures, graphical information (tables, graphs, charts, maps), web pages, contents lists, index and glossary, as appropriate to the level and stage being assessed - i.e. levels A and B at P3, levels B and C at P5, C and D at P7 and D and E at S2. The sources were representative of the sorts of information that pupils at the different stages might reasonably be expected to access and use as part of a social subjects enquiry.

The tasks were designed to assess the 'carrying out tasks' strand within the enquiry skills outcome in the 5-14 Social Subjects guidelines. Within each task the following sub-strands were assessed:

• Reference skills
• Finding/selecting/processing information in a reading context
• Finding/selecting/processing information in a numeracy context.

At Levels C, D and E a fourth sub-strand was assessed also:

• Evaluating.

All tasks developed for the survey had a set number of items at each level; level A tasks had twenty-two items, level B tasks had twenty-six items, and levels C, D and E tasks each had thirty items. At level A these items comprised six reference skills items and eight items each of finding/selecting/processing information in a reading and in a numeracy context. At level B, the items comprised six reference items and ten items each of finding information in a reading and in a numeracy context. Levels C, D and E mirrored level B with the addition of four evaluation items. Questions were level-related, were objectively scored and were each awarded one mark.

The assessment task source booklets also varied in length according to level. The level A sources were four pages long. At level B the sources were six pages long, while at levels C, D and E the sources comprised eight pages. At all levels the source booklets were designed in an attractive style, and contained colour photographs, graphics, maps, charts, and illustrations as well as text.

Each source booklet also became more complex according to level. Figure 1 below illustrates this.

Figure 1: Overview of enquiry skills tasks 'Island Holiday 1-4'

Figure 1.1: level C

Figure 1.2: level D

Source:http://www.islandferries.co.uk/lewis.htm

Item descriptions

Reference skills items

Pupils were asked to use a contents list to find information contained in the booklet. Below are examples of two reference skills items, the first at level A and the second at level E.

Figure 1.3: reference skill item at level A

Figure 1.4: reference skill item at level E

Items relating to finding/selecting/processing information in a reading context.

Below are two examples of this type of item, the first at level B and the second at level D.

Figure 1.5 : level B reading item

Figure 1.6: level D reading item

Finding/selecting/processing information in a numeracy context

Below are two examples of this type of item, the first at Level C and the second at Level E.

Figure 1.7: level C numeracy item

Figure 1.8: level E numeracy item

Evaluating skills

These skills were assessed at levels C, D and E. Below are two examples of this type of item, the first at level C and the second at level E.

Figure 1.9: level C evaluating item

Figure 1.10: level E evaluating item

Task development and pre-testing

Topics and associated material were identified by practising teachers who were trained as item writers. All sources were initially independently validated by between three and five teachers, to produce an appropriate set of source materials. They were further validated at pre-testing stage. As well as using their professional judgement to assign a 5-14 level, validators also gave each text an interest level (high, medium or low). Only sources with unanimous or near unanimous agreement of the 5-14 level and with medium to high interest ratings were selected for developing into enquiry skills tasks for use in the survey.

All enquiry skills materials were pre-tested. Each enquiry skills task was pre-tested by at least 110 pupils. Analysis of the results was used to ensure that the overall range of difficulty of the items was consistent with the level of the associated text. Items, sections or whole tasks that were found to be problematic in any way were either amended or discarded. Pre-testing was also used to refine coding sheets to be used for marking.

Teachers participating in pre-testing were given the opportunity to comment on the sources, items and mark schemes.

Task performance

Alpha values ( i.e. values of Cronbach's alpha coefficient) for all of the tasks, based on the survey data, are given in Table 2 below. Cronbach's alpha is a measure of a test's internal consistency; values of 0.8 or higher are generally considered to indicate acceptable levels of test reliability.

Table 2
Alpha values for the 30 social subjects tasks*

* Individual tasks were typically attempted by 250-300 pupils at the relevant stage

II.3 Reading

Nature of the assessment

Seventy-two reading tasks were administered in the 2006 survey, comprising twelve at each of levels A to F: thirty five were newly developed tasks, twelve tasks were previously used in the 2001 AAP survey of English Language and 2005 SSA survey of English Language, three tasks were previously used in the 2004 AAP survey of Mathematics and core skills and the 2005 SSA survey of English Language, and twenty two tasks were used for the first time in the 2005 SSA survey of English Language. Among them, the tasks covered the three different genres identified in the 5-14 English language guidelines: informative (reading for information), personal and narrative. The contexts for the newly developed tasks and the three tasks from the 2004 AAP Survey related to the focus of the particular survey. Consequently, with one exception at each of level A and level F, these tasks were all informative with texts having either a social subjects or mathematical context respectively. A variety of different themes were featured in the tasks, as the titles in Table 3 illustrates. With the exception of those previously administered in 2001, which were allocated a 5-14 level post-hoc, tasks were developed with particular 5-14 levels in mind.

Table 3
The 2006 survey reading tasks, with texts classified by genre and 5-14 level

Note
Newly developed tasks for 2006 survey ¹
Re-used tasks from 2001 and 2005 survey ²
Re-used tasks from 2004 and 2005 survey ³
Re-used tasks from 2005 survey ⁴

Each task took the same general form; a source text or texts followed by a series of questions (test items). All tasks developed for the 2004, 2005 and 2006 surveys had a set number of items at each level; level A tasks had 21 items, levels B and C tasks had 24 items, level D tasks had 27 items and levels E and F tasks had 30 items. Figure 2 describes one task at each of levels A, C and E.

Figure 2 Overview of three newly developed reading tasks

In accordance with previous surveys, reading tasks were designed to assess various reading skills as follows.

• Understanding the main ideas and details of the text.
• Distinguishing between different feelings or points of view.
• Attributing feelings or atmosphere to characters or situations.
• Identifying and understanding aspects of the writer's craft, including figurative language.
• Making generalisations and inferences.
• Understanding the sequence or structure of events or ideas in the text.
• Reconstructing and summarising the overall meaning of the text.
• Distinguishing between different types of text and different genres.

Although most items used at all levels required only a minimal written response (particularly at levels A and B), tasks newly developed for SSA 2006 provided more opportunities for pupils to give open responses, for example in addressing a range of aspects of the writer's craft. To ensure that pupils tackling level A tasks were given the amount of support suggested by the 5-14 English language guidelines, all tasks at this level included an example: "The first one has been done for you"

Task development and pre-testing

When developing the new tasks administered in this survey, a large number of texts were independently validated by at least five teachers, to produce an appropriate set of source texts. As well as using their professional judgement to assign a 5-14 level, validators also gave each text an interest level (high, medium or low). Only texts with at least four out of five agreement of the 5-14 level and with medium to high interest ratings were selected for developing into reading tasks for use in the survey.

Comprehensive pre-testing of new reading materials was carried out. Each of the new reading tasks was completed by at least 100 pupils. The results were analysed to ensure that the overall range of difficulty of the items was consistent with the level of the associated text. Items, sections or whole tasks that were found to be problematic in any way were either amended or discarded. Pre-testing was also used to refine coding sheets to be used for marking.

Teachers participating in pre-testing were given the opportunity to comment on the text, items and mark schemes.

Task performance

Alpha values ( i.e. values of Cronbach's alpha coefficient) for the seventy-two reading tasks, based on the survey data, are given in Table 4. Cronbach's alpha is a measure of a test's internal consistency; values of 0.8 or higher are generally considered to indicate acceptable levels of test reliability.

Table 4
Alpha values for the 72 reading tasks*

* Individual tasks were typically attempted by 250-300 pupils at the relevant stage

As explained earlier in this section, almost half of the tasks used to assess reading were newly developed for the 2006 survey. These new tasks were all in the Social Subjects context and were of the informative genre. Additional analysis was conducted on these tasks, in particular comparing how pupils performed in the new tasks compared with those used in the 2005 survey. The results of this analysis are shown in chart 1.

Chart 1
Comparison of pupil performance in repeated ('old') and 2006 only ('new') reading tasks
(% pupils with 65% or more of the items correct)
The statistics in this table derive from unweighted sample data

The analysis shows that, on average, pupil performance was consistently lower in the newly introduced tasks than in the tasks used in 2005 for all levels except F. It cannot necessarily be inferred that the difference is in any way explained by the fact that the new tasks were informational, given that some of the 'old' tasks were too, or even that they had a Social Subjects context. Further investigation is required.

II.4 Numeracy

Nature of the assessment

The core skill of numeracy was assessed in the survey through administration of a total of 532 different atomistic 'pencil and paper' test items: 80 at level A and 90 at each of levels B, D and F, and 91 at levels C and E. The numeracy items used in the survey were randomly drawn from the 5-14 National Assessment Bank, pre-classified by outcome, strand and target. The majority of the items pre-existed in the bank. Additional items were newly developed, to extend the content of the bank in accord with intended curriculum coverage. In total, just over half of the test items were different from those used for the assessment of numeracy in the 2005 survey.

The items were drawn from two of the four outcomes defined in the 5-14 Mathematics Guidelines, namely number, money and measurement and information handling. Shape, position and movement and problem solving were not included. Within the two outcomes the following strands were represented.

Number Money and Measurement

• Range and type of numbers
• Money
• Add and subtract
• Multiply and divide
• Fractions, percentages and ratio
• Measure and estimate
• Time

Information Handling

• Interpret Information

There were thirty items for information handling at each of the six levels A to E, sixty for number, money and measurement for levels B to F and fifty-two for level A.

The numeracy items were distributed among ten different test booklets at each stage, each containing items from three consecutive levels and spread over the various strands. P3 booklets contained items at levels A, B and C, at P5 the levels were B, C and D, at P7 the levels were C, D and E and at S2 the levels were D, E and F. Items were randomly ordered within booklets and booklets came in two versions, one version simply reversing the item order of the other.

After the booklets were generated, it was decided to make use of 'redundant' survey space to create two additional booklets at each level. These booklets were to be 'clones' of two existing booklets but all items would be converted to an alternative format: if the original item was in multiple choice format, then it was to be converted to an open (supply) format, and vice versa. These additional booklets would allow an investigation into whether question format has any effect on pupil performance. The two forms of item were then distributed between the two parallel booklets so that each booklet contained a mixture of multiple choice and supply response formats.

Each booklet at P3 contained fifteen number, money and measurement items and nine information handling items (24 in total); at P5, P7 and S2 the corresponding numbers were eighteen items and nine items respectively (27 in total). All items were binary scored. Two thirds of the number items at levels B to F (60% at level A only) related to assessing the four mathematical operations of add, subtract, multiply and divide. These were split between items set in a written context where the operation to be used was not immediately obvious, and context free items where the operation to be used was given. The remaining items from the number, money and measurement outcome covered money (levels A to C only), fractions (levels B to F), percentage and ratio (levels D to F) while range and type of number, measure and estimate and time could appear at any level. Items were chosen, at random, from the assessment bank for the specified strands/sub-strands to a pre-determined specification.

Figures 3.1 to 3.4 give examples of 'context free' and 'in context' items as well as examples of 'interpret information' items.

Figure 3.1 Examples of 'atomistic' numeracy items at Level B

Figure 3.2 Examples of 'atomistic' numeracy items at Level E

Figure 3.3 Example of 'atomistic' interpret information item at Level A

Figure 3.4 Example of 'atomistic' interpret information item at Level D

Task performance

Alpha values ( i.e. values of Cronbach's alpha coefficient) for the forty eight numeracy testlets, based on the survey data, are given in Table 5. Testlets, twelve per level, comprised sets of sixteen (P3) or eighteen (P5, P7, S2) items at a level distributed over two different test booklets at each stage. Cronbach's alpha is a measure of a test's internal consistency; values of 0.8 or higher are generally considered to indicate acceptable levels of test reliability.

Table 5
Alpha values for the numeracy testlets

As explained earlier in this section, almost half of the items used to assess numeracy in 2006 were different from those used in the 2005 survey, because items were randomly selected for inclusion from the national assessment bank, to meet a given booklet specification. Additional analysis was conducted on these items, in particular comparing how pupils performed on the new items compared with those previously used in the 2005 survey. The results of this analysis are shown in Chart 2.

Chart 2
Comparison of pupil performance in repeated ('old') and 2006 only ('new') numeracy items
(% pupils with 65% or more of the items correct)

The analysis shows that, on average, pupil performance was about the same in tasks used only in 2006 as in tasks used both in 2005 and 2006.

II.5 The writing tasks

In line with SSA 2005, schools were asked to submit a piece of extended writing, completed prior to the survey as part of normal course work, for designated pupils . The genre of writing required from individual pupils was prescribed, through random allocation: functional, personal or imaginative. The sub-genre, topic or theme was entirely a matter for the schools to decide, but schools were asked to select writing within a social subjects context where possible.

Selecting and assessing class-based writing

Teachers were given advice on selecting appropriate material and how much teacher support was permitted. The piece of writing selected was to reflect the level at which the pupil was currently working. Schools were advised that each piece of writing should be assessed by the class teacher and one other teacher or a promoted member of staff from the school using the 5-14 national writing criteria. Teachers were asked not to annotate the level with '+' or '-' or to record two levels e.g. D/E. The level awarded was to be recorded on the register provided but not recorded on the script itself. Where there was disagreement between the two markers, teachers were asked to discuss and come to a final decision.

Schools were informed that a proportion of randomly selected scripts would be centrally moderated by a group of teachers nominated by their education authorities.

The moderation of submitted class-based writing

The moderation event took place over five days, in early September 2006. All education authorities were invited to nominate up to two teachers to participate in the moderation event, with the intention of having at least one representative from each authority and two from the larger authorities. In fact, there were forty eight nominations - thirty nine primary teachers and nine secondary teachers - from all 32 education authorities.

In addition to judging scripts, the participants were addressed by guest speakers from Scottish Executive Education Department ( SEED), and a number of activities relevant to writing also took place.

In particular, the week was punctuated with plenary discussions focusing on particular pieces of writing (which were not themselves moderated), in order to facilitate a shared understanding of the standard and an evaluation of the 5-14 national writing criteria. Throughout the week, participants also had opportunities to share information about any innovative practice that might be going on in a particular class, school or authority. At the request of a number of primary teachers, a teacher from the secondary sector introduced a piece of writing judged to be at level F by two moderators, with an explanation of the qualities which made it level F.

A qualitative evaluation of the scripts took place towards the end of the week where moderators had the opportunity to share their perceptions of the general strengths and weaknesses of the writing they had moderated.

Moderators were also given the opportunity to discuss issues and concerns relating to assessment policy and practice in general.

The moderators were organised into pairs (with two groups having three members for part of the event) and assigned scripts from one stage only; P3, P5, P7 or S2. Teachers assigned P3 or P5 scripts had experience in the lower or middle stages of primary, and those assigned P7 or S2 scripts had experience in upper primary or secondary. Primary teachers with upper stage experience were paired with secondary teachers.

Over 9,000 scripts were submitted for moderation. Scripts were randomly selected for moderation and organised into batches of approximately twenty five each. Each batch was moderated by two teachers working independently of each other so that each writing script had three independent judgements as to the level, two moderator judgements and the original class teacher's judgement. The intention of the meeting was to moderate around 1,300 scripts at each stage, and this number was generally achieved.

Throughout the week exemplar scripts were discussed in plenary sessions, and levels agreed. The process began with teachers reading the piece of writing and then offering comments on the strengths and weaknesses. Participants were encouraged to challenge any comment with which they disagreed. The next step, using professional judgement, was to suggest a 5-14 level for the piece of work. The outcome typically straddled two levels, and sometimes more. An analytical approach was then adopted using the bullet point descriptions of attainment stated in the national criteria for extended writing. Each bullet point across the levels was discussed until a level was finally agreed. This was a time consuming process, but all teachers agreed that it was a necessary and invaluable experience. In addition to promoting the understanding of standards, the discussion permitted the production of material for creating exemplification of extended writing and evaluation of the writing criteria.

The moderators echoed the message from the 2005 moderation exercise that the experience of working with colleagues from different schools, authorities and sectors is invaluable.

During moderation each script was 'levelled' by two moderators working independently. Wherever possible, scripts were then assigned the level at which at least two of the three raters (two independent moderators plus the original class teacher) judged it to be. For over a third of the scripts a level was assigned by a unanimous judgement, with both moderators agreeing with the original level assigned by the teacher (the moderators were not aware of the level assigned by the class teacher). For a further third of the scripts a majority level was assigned to the script; here the level judgement of one of the moderators, but not both, agreed with the level judgement assigned by the class teacher. For a quarter of the scripts, a majority level was assigned to the script when the two moderators' judgements agreed with each other, but differed from the level assigned by the teacher. For the remaining scripts (approximately 7%), the levels assigned by the three individuals were all different, leaving the script without an agreed level. More detailed results are shown in Chart 3.

Chart 3
Percentage of pupils by teacher assigned level and moderation result

Comparisons of attainment levels between different assessment methods

Chart 4, explores in more detail the difference in levels between the level of the submitted script and the teachers' judgements. This analysis takes the results obtained from the submitted pieces of extended writing and then compares the teachers' judgements which were provided for just those pupils. Full results are available in Table 6.

As described earlier, for stages P3 and P5, the submitted script was generally of a higher level than the teachers' judgement for writing, while at P7 and S2 the level of the submitted script was slightly lower than the teachers' judgement.

Chart 4
Writing attainment based on submitted pieces of extended writing compared with teacher judgements
(% pupils at indicated levels)

* Approx. pupil sample sizes:1,700-2,400. Teacher judgements are only for those pupils for whom a script was submitted.

Table 6
Moderation of Submitted Scripts
(% of pupils at moderated level by original level assigned by teacher)

II.6 Practically assessed skills

In the 2006 survey there were three practical tasks: social subjects enquiry skills, working with others/problem solving and ICT. Each of the practical tasks was administered and pupils' performances assessed by field officers during school visits. Field officers were teachers nominated by their local authorities as having an expertise and interest in Social Subjects. Approximately 130 teachers received field officer training. Working in pairs, they visited between four and ten schools, assessing up to twelve pupils at a stage at each school.

II.6.a Social Subjects enquiry skills

Task description

Social Subjects enquiry skills were assessed by field officers taking part in one-to-one conversations with pupils during school visits. Schools were asked to ensure that pupils being assessed in this way took with them to the conversation some artefact connected with a recently conducted school-based Social Subject investigation. The artefact might be a model, a poster, a written report, or anything associated with the research; the artefact was not assessed, but acted as an aide memoire for the pupil and a stimulus for the conversation.

The conversation might range over the planning and the carrying out of the investigation, but field officers were asked to assess the pupils on their attainment of the third strand of the enquiry skills curriculum framework, reviewing and reporting. The assessment was based on a set of 'best fit' descriptors (see Table 7) which required field officers to make a judgement about a pupil's level of attainment by matching to the descriptor which most closely described the performance of the pupil.

The topic of the investigation (and therefore the conversation) was entirely dependent on the school's programme of study, but field officers were asked to record the topic, along with the pupils' attainment levels. The topics were later coded to assign each to one of the 5-14 Social Subjects attainment outcomes (People in the past, People and place, People in society).

Task development and pre-testing

In order to create exemplars, pupils were interviewed about recently conducted Social Subjects Enquiry Skills work. A CD containing thirty-six recordings of pupils taking part in one-to-one conversations was created. Twelve of these recordings were moderated by a group of teachers and were assigned levels using the 'best fit' descriptors. The moderators' comments were developed into commentaries for each recording. Field officers received the CD with the moderated levels and commentaries, in advance of field officer training, in order to allow them the opportunity to familiarise themselves with the assessment process and the levels. On each training day, further moderation took place with each group of field officers listening to and discussing more recordings and working towards a consensus.

II.6.b Working with others/Problem solving

Task description

The core skills of working with others and problem solving were assessed by field officers observing groups of pupils during school visits. Four packs were created, based on Social Subject contexts: Calanais, The Scottish Parliament, Clean Our Burn, and Fair Trade. Each pack was differentiated at two levels of complexity and contained brightly coloured stimulus materials and information, together with a task card.

Working in groups of four, pupils were asked, for example, to design a poster advertising a visit to Calanais. Pupils were given one hour to complete the task. Field officers observed and assessed the pupils using the Working with others grid (see table 7.1). At the end of the group activity the field officer put task-specific questions to the group as a prompt to completing the Problem solving grid (see table 7.2).

Task development and pre-testing

Each topic and pack was developed by teachers and pre-tested on a small-scale informal basis at various stages. An example of one the problem solving tasks is given in Figure 4.

Figure 4: Example of working with others/ problem solving task

Table 7.1 SSA 2006 Working with Others grid (individuals)

Table 7.2 SSA 2006 Problem Solving grid (the group)

Group: rate 0 = no evidence; 1= some evidence, given support; 2 = strong evidence with minimal support

II.6.c ICT

Task description

The tasks were developed to assess concepts, confidence, knowledge and skills involved in the use of ICT equipment and in the use of ICT as a core skill within a Social Subjects Enquiry Skills context. The tasks were assessed by a field officer through observation of, and in discussion with, the pupil. Pupils used their school computers so that they were familiar with the technology. An internet connection was not required. The final level awarded by field officers was based on observation of the pupil at work, their responses to the questions in the ICT protocol (Table 8) and using the best fit descriptors (Table 9).

The 5-14 ICT strands assessed were 'Searching & Researching' and, to a lesser extent, 'Using the Technology'. The tasks were designed to assess pupils' ability to find and transform information into a form suitable for the specified purpose. All the assessment tasks had a common theme (finding and transforming information) and required pupils to find, select and extract information to add to a 'Fact File'. Pupils selected the information from a range of 'captured' websites saved on a CD Rom in order to replicate as closely as possible genuine on-line searching. Each task included a mock up search engine page as the 'entry' page.

Six tasks were developed to allow pupils to demonstrate skills, knowledge and understanding in ICT capability across the 5-14 levels. The six topics for the tasks were:

1. Weather
2. Famous Victorians
3. France
4. Ancient Egypt
5. A Scottish Holiday
6. Make Poverty History

The tasks were used as follows:

P3 Tasks 1, 2, 3
P5 Tasks 2, 3, 4
P7 Tasks 3, 4, 5
S2 Tasks 4, 5, 6

As an example, Task 4, Ancient Egypt, is shown in Figure 5.

Figure 5: Example of ICT task used at P5, P7 and S2

Task development and pre-testing

The tasks were developed to reflect the way pupils use ICT to support learning in Social Subjects. The topics were selected as topical and relevant to pupils at all stages. The website material 'captured' using an off-line web browser was copied to CDs which field officers used as a resource in schools.

Tasks were trialled on a small-scale informal basis. The main purpose of this exercise was to trial the 'protocol' (question sheet - Table 8) and the best fit descriptors (Table 9) that would be used by the field officers. The results provided an indication that the level descriptors were useful in making consistent judgments about pupils ICT enquiry skills.

Table 8 The ICT protocol

Ask Question 1 and continue through questions, working across then down the page as far as possible.

The final level awarded is based on observation of the pupil at work, the replies to the questions and from use of the best fit criteria.

Table 9 Best Fit descriptors used to assess ICT

« Previous | Contents | Next »