The Scottish Government

« Previous | Contents | Next »

Listen

2. Mathematics

2.1 The assessment process

2.1.1 The assessment tasks

The principal intention of this survey was to assess the mathematics attainment of pupils in P3, P5, P7 and S2 at two or three consecutive levels: Levels A and B at P3; Levels B, C and D at P5; Levels C, D and E at P7; Levels D, E and F at S2. To this end, a total of 1120 atomistic 'pencil and paper' test items (single-item tasks) were administered in the survey, every item having been classified by outcome, strand and level prior to survey use. The majority of the items pre-existed in the 5-14 National Assessment Bank. Additional items were newly developed, as necessary, to fill gaps in the intended curriculum coverage of the survey.

The items took a form which everyone with any experience of mathematics assessment is familiar, presenting pupils with a short stem and inviting a single quick response (see Figure 2.1a to Figure 2.1f for examples).

Figure 2.1a
Example test item: Level A
Number, Money & Measurement: Subtract

Figure 2.1b
Example test item: Level B
Information Handling: Interpreting information

Figure 2.1c
Example test item: Level C
S hape, Position and Movement: Symmetry

Figure 2.1d
Example test item: Level D
Problem Solving and Enquiry: Problem solving

Figure 2.1e
Example test item: Level E
Shape, Position & Movement: Range of shapes

Figure 2.1f
Example test item: Level F
Number, Money & Measurement: Algebra

Table 2.1 shows the intended distribution of the survey items over the assessment framework. Where no items appear in particular cells in the grid, this is usually because the strands concerned do not feature at the levels concerned within the national guidelines for this subject, for example 'multiply and divide' at Level A.

In addition to the 1120 'pencil and paper' items, a total of 384 'mental' test items were also administered in the survey, comprising 64 items per level, Levels A to F. With the exception of three 'problem solving' items, all the mental items were from the outcome Number, Money & Measurement. An important point to note regarding the level classification of the mental items is that, following the definition of targets given in the 5-14 Guidelines ³, some items were classified at one level above the level that would normally be assigned to the item were it to be presented within a written test booklet; thus a Level D 'pencil and paper' item would be considered a Level E item in a mental test. Table 2.2 shows the distribution of these items over the assessment framework, while Table 2.3 provides examples of the type of item involved.

The written and mental atomistic items were complemented by longer newly developed multi-item tasks, 'maths literacy tasks'. Each of these tasks comprised 10 or more test questions (items), all based on the same set of text-based and graphical stimulus materials. Thumbnail sketches of three of the tasks are offered in Table 2.4. Practical tasks of different kinds also featured - these are described and reported in Chapter 3.

In principle there were three multi-item maths literacy tasks at each of Levels A to E. In practice there was disagreement about the appropriate level classifications of many of the tasks when four different teachers were invited independently to validate them. Any level-based results given later in this chapter should in consequence be considered indicative rather than robust.

2.1.2 Task administration

Even with an unlimited amount of testing time per pupil, no single pupil could ever be expected to undertake all the items and tasks concerned at the levels proposed for that pupil's stage. As in all large-scale attainment surveys there must be a limit to the testing time demanded of pupils and teachers. Here, as in previous AAP surveys, the time assumed was around 30-40 minutes per session at P3, 40-50 minutes at P5 and 50-60 minutes per session at P7/S2, with two separate assessment sessions available.

The Mathematics Reference Group was consulted about the most appropriate numbers of items that pupils at the different stages might reasonably be expected to attempt within these timescales. The advice was: around 21 items at P3, around 25 items at P5, and 30-35 items at P7/S2. It was also confirmed that no pupil should be expected to take a test at one level only, given that there was no prior information available about the attainment level that pupils might be working comfortably at when the survey was planned to take place. It was considered ill-advised to contemplate giving a P5 pupil a Level D test, for example, when that pupil might currently be working at Level C or even at Level B. For different reasons, it was considered equally inappropriate to give a P5 pupil a Level B test when that pupil might already be working at Level D. It was therefore decided to include items from more than one level in every written test booklet and in every mental test.

The 1120 'pencil and paper' items were consequently subdivided into a number of different test booklets, each designed to take around the given testing time, and each comprising the same mix of test items in terms of curriculum coverage and levels. There were 20 different test booklets in total at each stage, comprising 17 items at P3, 27 at P5, 28 at P7 and 30 at S2 (see Appendix B, section B.2, for the rationale behind these numbers). At P3 every booklet contained items from two levels (A and B), whereas at the other stages every booklet contained items from the three relevant levels. The order of presentation of items within a booklet was randomised, so that pupils did not face a string of items at the same level or from the same outcome, and booklets were produced in two versions, the second version simply reversing the order of item presentation. Where pupils at different stages were to be assessed at the same level (e.g. P3 and P5 at Level B) the same items were used as the basis for the assessment.

The 384 mental test items were also distributed among a number of different mental tests, 16 tests at each stage. At P3, tests were eight items long, with four items representing each of Levels A and B. At the other stages tests were 12 items long, with four items from each of three consecutive levels (Levels B, C and D at P5, Levels C, D and E at P7 and Levels D, E and F at S2).

The written and mental assessment sessions were organised by the schools themselves, within the period mid-May to mid-June, with teachers in the schools delivering the mental tests and supervising the written testing. Prior to the survey, every pupil in the sample was randomly allocated two different written booklets, to be attempted in two separate assessment sessions, with a break between to be decided by each school. Since mental testing requires oral delivery of items, it would clearly not be feasible to expect teachers to deliver numerous different mental tests to their pupils. Therefore, every school was randomly allocated two different mental tests, each to be delivered to the whole pupil group before the pupils embarked on their individualised written test booklets. Thus, all the pupils in any one school would take the same two mental tests, but different pupils would take different pairs of written tests. Completed scripts were returned to SQA for marking. Typically, each written test booklet, and hence every 'pencil and paper' test item, was attempted by 200-250 pupils at P3, P5 and P7 and 250-300 pupils at S2. The mental tests were each attempted by 250-300 pupils at the primary stages and 350-400 pupils at S2.

The multi-item maths literacy tasks were administered to a subsample of the pupils in a subsample of the schools by the itinerant field officers responsible for the practical assessments in mathematics. Each task was attempted by 150-200 pupils, the numbers varying by stage.

2.1.3 Marking

The mental tests, written test booklets and maths literacy tasks were processed by the SQA, with students employed to mark pupils' responses during a continuous marking period throughout July 2004. After instruction in marking procedures, the markers were organised into five teams of five, and at any one time, four teams were involved in marking and one team in checking. Booklets and tasks were marked stage by stage, starting with P3.

All the 'mental' items and the majority of the pencil and paper items were allocated one mark each. A minority of the pencil and paper items, particularly algebra items at Level F, such as that shown in Figure 2.1f, were allocated two marks, usually one for method and one for a correct answer. Non-responses were noted, as were those occasions when pupils selected more than one answer option in multiple-choice items.

2.1.4 Reporting attainment

Pupils' written mathematics attainment is reported in terms of the percentages of pupils deemed to be working at the 5-14 levels assessed at the stage concerned. Attainment decisions were based on the application of cut-off scores to the total marks achieved by pupils on the set of items at the same level distributed across their two booklets. The numbers of items at each level across two booklets were as follows: 16 at Level A, 18 at each of Levels B, C and D, 20 at Level E and 22 at Level F. Following practice in recent AAP surveys, pupils achieving 80% or more of the marks were considered as having demonstrated "considerable strengths" at the level concerned; pupils with 65% or more of the marks but less than 80% were considered as being "secure" at the level; pupils achieving 50% or more of the marks but less than 65% were considered as having "basic skills" at the level ⁴. For each pair of booklets at a stage the proportions of pupils achieving or exceeding the cut-off scores were calculated, and the results were then averaged over all booklet pairs to produce the attainment figures presented in the next section.

Given the small numbers of items/tasks attempted by any individual pupil relating to the separate outcomes (four to six items per level per pupil), mental mathematics (eight items per level per pupil) and mathematical literacy (one task per pupil), performance reporting is in these cases in terms of average mean scores rather than level attainments.

2.2 Overview of pupils' attainments

2.2.1 The attainment picture across the stages

For the written assessment of mathematics, Table 2.5 provides an overview of attainment at all four stages, in terms of the proportions of pupils meeting the 65% success criterion on the items they attempted at particular levels, averaged over all booklet pairs. Figure 2.2 illustrates the picture.

Table 2.5 and Figure 2.2 show an expected pattern: 'secure attainment' rates rise through the stages at any one level, and decrease within each stage as levels increase. At P3, almost 90% of the pupils tested were at least 'secure' at Level A and around one-third were also secure or better at Level B. At P5, almost three-quarters of the pupils were at least secure at Level B, 40% also at Level C and 10% at Level D. At P7 we see a similar picture, but removed by one level: almost three-quarters of the pupils were secure or better at Level C, over 45% at Level D, and almost 20% at Level E, the level above their expected level. Meanwhile, at S2, 60% of the pupils were secure at Level D, over 35% at Level E and over 5% at Level F.

* Figures show the percentages of pupils demonstrating attainment at the indicated level or higher: 2000-3000 pupils at each stage, weighted data; margins of error around 1.5 percentage points.

Figure 2.2
'Secure' Mathematics attainment P3 to S2*

* Each bar shows the percentage of pupils achieving 65% or more of the marks at the level concerned: 2000-3000 pupils at each stage

Within each level we can look further than the proportions of pupils who are at least secure at the level, extending the picture to those showing considerable strengths as well as to those achieving only basic levels of attainment or even lower than basic. Table 2.6 provides the relevant data.

Perhaps one of the most interesting features in Table 2.6 is the general similarity in the attainment profiles of the pupils at P7 and S2 at Levels D and E, respectively their expected levels, and of the pupils at P3 and P5 at Levels B and C, respectively. The pattern is illustrated in Figure 2.3.

* '< basic' means fewer than 50% of marks achieved, 'basic' is between 50% and 64%, 'secure' is 65% to 79%, and 'strengths' is 80%+; weighted data

Figure 2.3
Profiles of Mathematics attainment P3 to S2*

* '< basic' means fewer than 50% of marks achieved, 'basic' is between 50% and 64%, 'secure' is 65% to 79%, and 'strengths' is 80%+.

While there is some variation in the proportions of pupils showing considerable strengths at the respective levels, this variation is not great between P5 and S2: 15% at P5 for Level C, 17% at P7 for Level D, 18% at S2 for Level E. At the other extreme, we see similar variation in the proportions of P3, P5 and P7 pupils showing 'below basic' levels of skill and understanding, with around one-third falling into this category. At S2 this rises to just over 40% demonstrating 'below basic' levels of skill and understanding at their expected level.

Focusing now on the lowest attaining pupils, 20% of the S2 pupils failed to demonstrate basic skills at Level D, the level below their expected level. At P5 and P7, a lower 10% or so of the pupils showed similarly low attainment at Levels B and C, respectively, while at P3 fewer than 5% of the pupils failed to demonstrate basic skills at Level A.

In total, 40, 60 or 80 different items were administered in the survey to represent each of the four outcomes at each level, the number depending on outcome and level (see Table 2.1). These numbers allow us to explore pupil performance in the different outcomes, in terms at least of averaged percentage item scores if not in terms of percentages of pupils attaining a level (individual pupils attempted a handful only of items from any one outcome). Table 2.7 provides the results of this comparison.

* NMM - Number, Money & Measurement, IH - Information Handling, SPM - Shape, Position & Movement, PS - Problem Solving & Enquiry. The figures in this table are averaged percentage item scores, and not percentages of pupils attaining levels (weighted data).

The clearest feature evidenced in Table 2.7 is the lower performance typically associated with Problem Solving items, at all stages and levels. This finding might be expected given that items were principally classified as problem solving items if correct answers required pupils to carry out two or more operations rather than one, and without any cueing. The Level D item shown in Figure 2.1 is an example. In this item, pupils are told that a P7 class was making a school trip, with the children travelling by coach, minibus or car: 4/7 by coach, 2/7 by minibus and eight children by car. They are asked how many pupils in total went on the trip. To answer this question, the survey pupils concerned had first to add 4/7 and 2/7 to find the total fraction travelling by coach or minibus, subtract from 1 to find the fraction travelling by car (1/7), equate this fraction to 8 pupils, and then multiply 8 by 7 to reach the answer of 56 pupils. Items asking pupils to find 1/7 of 56 or to find the sum of 4/7 and 2/7 would be classified as Number, Money & Measurement. Clearly, the more steps are involved in arriving at a successful conclusion the more opportunities there are for error.

It is more difficult to interpret differences between the other three outcomes. The performance scores for these outcomes are generally rather similar, and apparent differences are not consistent from one level or stage to another. Even where performance differences appear to be replicated at two or more stages (for instance, Level B at P3 and P5, where Information Handling appears to be 'easier' than Number, Money & Measurement and Shape, Position & Movement), this could quite simply be an artefact of the particular items used to represent these outcomes at this level, or even a reflection of the way that the 5-14 assessment framework has defined this level for these outcomes.

Moving on to mental mathematics, where all but a handful of the 384 items were from the outcome Number, Money & Measurement, Table 2.8 presents the relevant performance data, and also provides a direct comparison with the results shown in Table 2.7 for the 'pencil and paper' items from this outcome (the items in the two modes were different items).

* The figures in this table are averaged percentage item scores, and not percentages of pupils attaining levels. Corresponding figures for the pencil and paper assessment of Number, Money & Measurement are shown in brackets.

The performance data in Table 2.8 reveal a very clear picture of difference, with item scores 10-15 percentage points higher on average for the mental as opposed to the pencil and paper items, at all stages and levels. It should be remembered that some items presented orally would be classified at one level higher than they would be had they been presented on paper (see section 2.1.1). Moreover, while every one of the 'mental' items reproduced in Table 2.3 could very readily be presented to pupils in pencil and paper form (classified at a level below), the same cannot be said in reverse for the 'pencil and paper' items shown in Figures 2.1a and 2.1f, where the memory demand in terms of the source information is clearly too great to make oral presentation a fair option (compare the Level A pencil and paper item in Figure 2.1a with the Level A mental item reproduced in Table 2.3, both of which involve calculating change). That said, the picture of attainment in mental mathematics is a positive one.

Finally, the performance of pupils on the few mathematics literacy tasks administered by the itinerant field officers is shown in Table 2.9. In principle, three tasks per level were administered, for Levels A to E, individual pupils attempting just one task in their 'practical' assessment sessions. With just two exceptions, the tasks comprised ten questions, or test items, each, based on information presented in various forms (see Table 2.4 for examples).

Despite some problems classifying the tasks by 5-14 level, the data in Table 2.9 reveal a strikingly similar pattern of performance from one stage to another at the various levels. At P7 and S2 the averages of the mean percentage task scores ⁵ for the three tasks at Levels D and E, respectively, were just over 45%, falling to just over 40% for Levels B and C, respectively, at P3 and P5. At one level below at each stage the average mean percentage task score was 60-70%. Average item performance for the atomistic 'pencil and paper' assessment was typically higher at every level and stage.

* The figures in this table are average percentage task scores, and not percentages of pupils attaining levels. Bracketed figures are average percentage item scores(average facilities) for the 'atomistic' pencil and paper assessment (all outcomes combined).

2.2.2 Gender comparisons

There were no statistically significant gender differences at any stage at any level, for any type of mathematics.

2.2.3 Change over time

The previous AAP mathematics survey took place four years earlier, in 2000. In 2001 the stages assessed in the AAP programme changed from P4, P7 and S2 to P3, P5, P7 and S2. It is therefore not possible to offer comment on change over time for P3 and P5. At P7 and S2, while it is possible to offer comment, this can only be on the basis of mean item facilities and not in terms of percentages of pupils attaining levels. This is because in preparation for the launch of the National Assessment Bank in the autumn of 2003, the reservoir of test items provided by the AAP through surveys prior to the 2004 survey was reviewed, and every item re-evaluated in terms of the 5-14 framework and, if framework-relevant, classified by level. Wherever possible, test items were drawn from the 2000 pool within the National Assessment Bank for use in the 2004 survey, but these items were scattered at random throughout the test booklets.

Table 2.10 presents the resulting performance picture for the two years, a picture illustrated in Figure 2.4.

Figure 2.4
Mathematics attainment at P7 and S2 in 2000 and 2004
(average facilities for 84-120 items/level that were administered in both surveys)

At P7, there is literally no difference between the average item scores at Levels C and D, and the small sample difference at Level E is not statistically significant. While the small sample differences shown at all levels at S2 are in favour of the pupils in the year 2000, none of these differences reaches statistical significance either. There is thus no evidence of any real change in pupil performance at these stages and levels between 2000 and 2004.

2.3 Summary

A number of different types of mathematics assessment featured in this survey. Pupils' mathematical knowledge and skills were assessed through the administration of 1120 'atomistic' test items presented within numerous 'pencil and paper' test booklets, through the administration of 384 'mental mathematics' items presented to pupils orally by their own teachers, through use of a small number of multi-item 'mathematical literacy' tasks, administered to pupils by the itinerant field officers, and through a number of practical mathematics tasks (reported in Chapter 3).

On the basis of their performances on the atomistic 'pencil and paper' items, pupils were deemed as 'secure' at a level if they achieved at least 65% of the marks for items attempted at that level, as demonstrating 'considerable strengths' if they achieved 80% or more of the marks, as having 'basic skills' if they achieved half the marks but less than 65%, and as being

'below basic' if they could not achieve half the marks. As might be expected, 'secure attainment' rates rose through the stages at any one level, and decreased within each stage as levels increased. At P3, almost 90% of the pupils tested were at least 'secure' at Level A and around one-third were also secure or better at Level B. At P5, almost three-quarters of the pupils were at least secure at Level B, 40% also at Level C and 10% at Level D. At P7, almost three-quarters of the pupils were secure or better at Level C, over 45% at Level D, and almost 20% at Level E. At S2, 60% of the pupils were secure at Level D, over 35% at Level E and over 5% at Level F.

At P5, P7 and S2, 15-20% of the pupils showed 'considerable strengths' at Levels C, D and E, respectively. At the other end of the performance scale, 30-40% of the pupils at P3, P5, P7 and S2 showed 'below basic' levels of skill and understanding at, respectively, Levels B, C, D and E.

One-fifth of the S2 pupils failed to demonstrate basic skills at Level D, the level below their expected level, while 10% of the pupils at P5 and P7 showed similarly low attainment at Levels B and C, respectively, and fewer than 5% of the P3 pupils at Level A.

On the basis of average item scores, among the four outcomes of mathematics the lowest performances were associated with Problem Solving, at all stages and levels. The performance scores for the other three outcomes were generally rather similar, and any differences were not consistent from one level or stage to another.

As far as mental mathematics is concerned, item scores were 10-15 percentage points higher on average for the mental as opposed to the pencil and paper test items, at all stages and levels, a difference at least in part reflecting the nature of items administered in the two modes along with the fact that some items were classified at a level above normal for oral presentation.

Average percentage mean scores on three 'mathematical literacy' tasks per level were just over 45% for P7 and S2 at Levels D and E, respectively, and just over 40% for P3 and P5 at Levels B and C, respectively. At one level below, the average percentage mean score was 60-70% at each stage. Average facilities for the atomistic 'pencil and paper' items was typically higher at every level and stage than for the multi-item maths literacy tasks.

There were no statistically significant gender differences at any stage at any level, for any type of mathematics.

Finally, on the basis of around 100 repeated items per stage and level (on average), there is no evidence of any real change in P7 performance at Levels C, D or E or in S2 performance at Levels D, E or F since the last mathematics survey in 2000.

« Previous | Contents | Next »