The Scottish Government

« Previous | Contents | Next »

Listen

Appendix A: The Survey Design Team and the Mathematics Reference Group

The Survey Design Team

The Mathematics Reference Group

Appendix B: Sampling, task distribution and attainment estimation

B.1 School and pupil sampling

The 2004 mathematics survey was designed to assess the mathematics and core skills attainment pupils at P3, P5, P7 and S2 in mainstream schools in Scotland - including educational authority, self-governing, grant-aided and independent schools. Special schools and Gaelic Medium schools were excluded from the sampling frame.

Representative pupil samples were selected for testing using two-stage proportionate stratified sampling, with an overall sampling fraction of 4-5% of the pupil population at each stage. Separate school samples were drawn for the four pupil stages. Before school sampling began, the population of schools was first stratified by education authority grouping, roll size and percentage free school meals entitlement. The 32 education authorities were classified into four groups for this purpose (maintained schools only), based on their general population densities (see Table B.1). In addition, maintained schools were grouped into two size bands (primary schools: under 280 pupils on roll, and 280 or more on roll; secondary schools: under 150 S2 pupils on roll, and 150 or more S2 pupils on roll) and two bands for free school meals entitlement (primary schools: <10%, and 10% or more; secondary schools: <15%, and 15% or more). School size and free school meals entitlement classifications were based on the most recent school census data available at the time, viz. census data at September 2002 and at January 2003, respectively. Independent schools formed a separate national stratum.

Schools were selected from within strata, without replacement, with probability proportional to stage size. At each stage, between 200 and 300 schools were selected and invited to participate in the survey. Of these, at each primary stage around 90% of the invited schools agreed to participate, with 80% of the secondary schools agreeing for S2 (see Table B.2).

The sample pupils were selected in a second stage of sampling, from within those schools that had agreed to participate. Wherever possible, i.e. in those schools with sufficient numbers of pupils available in the stage concerned, 22 pupils were randomly selected within each primary - 10 for involvement in the assessment of mathematics, six (at P3 and P5) or nine (at P7) for involvement in the assessment of reading and writing, and six (or three) reserve pupils, to act as substitutes for pupils absent on the assessment days. In very small schools mathematics assessment took priority over the assessment of reading and writing. In other words, where schools had fewer than 16 pupils available in the relevant stage, 10 would do mathematics assessments and the remainder would take reading tasks and produce writing. Where schools had 10 or fewer pupils available at the relevant stage all of these would take mathematics assessments and none would be subject to the assessment of reading and writing. There were thus some primary schools in the survey sample that would take part in mathematics assessment only. In secondary schools 32 S2 pupils were randomly selected, 20 for mathematics assessment, nine for reading/writing assessment, and three reserves. Again, where schools had fewer than 20 pupils in their samples then there would be no reading/writing assessment.

Where pupils with special educational needs were selected in school samples, these were included in the test sessions at the head teacher's discretion. Pupils who were not withdrawn for any reason but were absent on the day of testing were replaced with 'reserve' pupils.

In a subset of the schools the 'mathematics' pupils also took part in the assessment of practical skills, while the 'reading/writing' pupils took part in the assessment of ICT skills or other practical activities. Although the 'practical' schools were drawn from across the country, they were not selected entirely at random: two important criteria for involvement were (i) that the school should have sufficient pupils at the stage concerned to justify a day visit by two field officers, and (ii) that it should be within easy travelling distance of the field officers' home bases. In the event, just over 45% of the schools that participated in the survey were involved in the practical assessments (see Table B.3).

B.2 Task distribution

Mathematics
In order to assess pupils' mathematics knowledge and skills, and to report attainment in terms of the 5-14 levels, 1504 different level-specific test items were administered in this survey: 1120 in written mode and 384 delivered orally by class/subject teachers. The 'written' items were drawn from all four mathematics outcomes - Information Handling, Number, Money & Measurement, Shape, Position & Movement and Problem Solving & Enquiry; 84-100 items at each of Levels C, D and E had been used in the 2000 survey at P7 and S2, and formed the basis for an attainment comparison between 2000 and 2004. All the 'mental' items related to Number, Money & Measurement. Table B.4 shows how the items distributed across the 5-14 mathematics framework.

A multiple matrix sampling strategy was adopted for distribution of items to pupils, in that the items were distributed among several different and unique test booklets, each with the same general composition, and booklets were randomly allocated to pupils.

There were 20 different written test booklets in total at each stage, comprising 17 items at P3 (8 Level A items and 9 Level B items), 27 at P5 (9 items at each of Levels B, C and D), 28 at P7 (9 items at each of Levels C and D and 10 items at Level E), and 30 at S2 (9 items at Level D, 10 at Level E and 11 at Level F). These booklet lengths are all within those proposed by the Mathematics Reference Group as appropriate at the different stages, assuming testing times of 30-40 minutes at P3, 40-50 minutes at P5, and 50-60 minutes at P7/S2.

The order of presentation of items within a booklet was randomised, so that pupils did not face a string of items at the same level or from the same outcome, and booklets were produced in two versions, the second version simply reversing the order of item presentation. Booklets at the different stages overlapped, in the sense that the items at a particular level in one stage booklet were carried across to represent that level in a booklet at the next stage.

The 384 mental test items were distributed among 16 different mental tests at each stage. At P3, tests were eight items long, with four items representing each of Levels A and B. At the other stages tests were 12 items long, with four items from each of three consecutive levels (Levels B, C and D at P5, Levels C, D and E at P7 and Levels D, E and F at S2). An answer page was included at the front of every written test booklet for pupils to use when responding to their mental test.

Test booklets were randomly assigned to survey pupils, two different booklets per pupil, before being despatched to participating schools. In any one school different pupils were allocated different written test booklets (see Table B.5 for the allocation scheme). In this way, all the booklets were distributed across the maximum number of survey schools, thus minimising any possible school effect on the attainment results: within a primary school just one pupil would attempt any particular booklet, whilst in secondary schools two pupils at most would do so.

* Primary schools were sent up to10 of the booklet combinations in either Block I or Block II, while secondary schools were sent all booklet combinations in both blocks.

Since mental testing requires oral delivery of items, it would clearly not be feasible to expect teachers to deliver numerous different mental tests to their pupils. Every school was, therefore, randomly allocated two different mental tests, each to be delivered to the whole pupil group before the pupils embarked on their individualised written test booklets.

The written and mental assessment sessions were organised by the schools themselves, within the period mid-May to mid-June, with teachers in the schools delivering the mental tests and supervising the written testing. Completed scripts were returned to SQA for marking.

The 15 multi-item maths literacy tasks, three per level for Levels A to E, were administered by the itinerant field officers responsible for the practical assessments in mathematics, to a subsample of the pupils in the subsample of 'practical' schools at each stage. At P3 pupils attempted tasks at Levels A and B, at P5 at Levels B and C, at P7 at Levels C and D, and at S2 Levels D and E.

Arrangements for the mathematics practical tasks are described below.

Reading and writing
There were 15 reading tasks in total, three at each of Levels A to E. Each task comprised a text and associated test questions, and was expected to take the same time to complete as a mathematics booklet at the stage concerned. At all levels writing tasks were devised that were loosely linked to the reading tasks: each writing task focused on the same source material as one of the reading tasks, but the source material was not required reading for the purpose of producing an appropriate piece of writing. P3 pupils were assessed at Levels A and B, P5 pupils at Levels B and C, and P7 and S2 pupils at Levels C, D and E.

Again, a multiple matrix sampling scheme was employed to allocate tasks to pupils. At each stage a number of different task pairings were created (six at P3/P5 and nine at P7/S2), each pair comprising tasks from two different levels. Task pairs were then randomly allocated to the pupils in each school that had agreed to participate in the survey and that had pupils available for reading assessment. In this way every task would have been attempted by similar numbers of pupils across the survey, in similarly representative subsamples, and no more than two pupils would attempt the same task in any particular school.

Practical mathematics skills and ICT skills
Four practical mathematics tasks were administered in the survey, along with six ICT tasks. The pupils who undertook mathematics practical tasks were drawn from those included in the written mathematics assessment in the 'practical' schools, whereas those pupils who attempted ICT tasks were drawn from among those involved in the assessment of reading and writing in these same schools. Within these groups, tasks were allocated to pupils at random.

In the majority of the schools that participated in assessment in this area, four pupils were assessed for their mathematics practical skills and a further four for ICT skills.

B.3 Achieved pupil sample sizes

Almost 15500 pupils were assessed in the survey, around two-thirds of them in mathematics and one-third in reading and writing. A subsample of around 15% of the 'mathematics' pupils attempted practical mathematics tasks and/or mathematical literacy tasks, while a subsample of around a quarter of the reading/writing pupils also attempted ICT tasks. Table B.6 provides details of the final analysis samples.

* These were scripts randomly sampled for evaluation from the larger numbers submitted.

In total, over 9900 pupils were involved in the written and mental mathematics assessment, drawn from the 835 participating schools. Just under 9% of the pupils were 'reserve' pupils, substituted on the day for pupils who had left the school since the pupil sample was drawn, or who were simply absent on the day of testing. Just over 4000 pupils attempted a mathematics literacy task; 12% of these pupils were substituted reserve pupils at P3 and P5, with fewer than 1% in this category at P7 and S2.

Typically, each mathematics test booklet, and hence every 'pencil and paper' test item, was attempted by 200-250 pupils at P3, P5 and P7 and 250-300 pupils at S2. The mental tests were each attempted by 250-300 pupils at the primary stages and 350-400 pupils at S2. Each mathematics literacy task was attempted by 150-200 pupils, the numbers varying by stage.

In total, reading assessment data were analysed for 5505 pupils and writing assessment data were analysed for a total of 3110 pupils (the writing produced by these pupils having been randomly selected for evaluation from all of that submitted). Again just under 9% of the pupils were substitute pupils. Individual reading tasks were attempted by over 350 pupils at P3/P5, over 400 at P7 and over 250 at S2.

Performance data were analysed for a total of almost 1500 pupils for mathematics practical skills and over 1300 for ICT skills; substitution of 'reserve' pupils was as high at P3 and P5 as in the written assessments, but there were virtually no substitutions at P7 and S2. The number of pupils who undertook any particular task varied from stage to stage: 150-200 pupils per stage for maths literacy, and 70-100 for maths practical tasks and ICT.

B.4 Attainment estimation

In mathematics and reading total scores were first computed for pupils, for each of their level-based 'tests'. In mathematics these 'tests' comprised those items at the same level across the two test booklets that a pupil attempted: 16 items at Level A, 18 items at each of Levels B to D, 20 items at Level E and 22 items at Level F. In reading, the level-based tests were the tasks themselves: comprising 21 items at Level A, 24 items at each of Levels B and C, 27 items at Level D and 30 items at Level E.

Cut-off scores were then applied, and pupils classified into one or other of three attainment groups on the basis of these: 'basic skills', 'secure attainment' or 'considerable strengths' ¹⁸. The proportions of pupils classified into the three groups at relevant levels were calculated separately for every booklet pair in mathematics, and for every reading task, with the attainment data weighted appropriately to adjust for imbalances in sample representation caused by the non-participation of some schools. The resulting proportions were then simply averaged over pairs of mathematics booklets (20 pairs per stage) or reading tasks (three per level) to produce the population attainment estimates reported in Chapters 2 and 4, respectively.

Margins of error for the attainment estimates arising from a single booklet pair in mathematics would be a maximum of around six to seven percentage points, reducing to a maximum of around one and a half percentage points for the final averaged population estimates at a level. Margins of error for the attainment estimates deriving from a single reading task would be a maximum of five to six percentage points, reducing to a maximum of one and a half to three and a half percentage points for the final population estimates at a level. It should be noted that these figures cannot take account of any measurement error that will have arisen from the possible incorrect classification of individual pupils, for some of whom the decisions made might have been different had the pupils concerned been assessed on a different day or on the same day with a different reading task or pair of mathematics booklets (test reliabilities - alpha values - are typically in the range 0.7-0.9 for each mathematics 'test', and 0.8-0.9 for each reading task). Neither do they take account of the measurement error that will have arisen from the fact that the tasks and items used in this survey are merely representative of all the similar tasks and items that might have been developed and used in their place.

In the case of writing, practising teachers evaluated pupils' scripts and allocated level judgments. As always with extended writing, judgements of quality were subjective to some extent, as the inter-rater agreement study described in Chapter 4 confirms: the average 'majority' agreement rate when applying a 'best fit' evaluation scheme (i.e. at least two of the three independent raters agreeing the same level) was 82%. With this in mind, the resulting writing attainment data have been presented in Chapter 4 as sample statistics only.

Given the nature of the practical assessment tasks - which were novel in nature and which did not lend themselves to pupil classification by level - no attempt has been made to produce weighted estimates of practical skills attainment on this occasion. School and pupil questionnaire findings are also presented in this report as sample statistics rather than formal population estimates.

Appendix C: Field officer guidance for practical mathematics assessment

The practical tasks are designed to be progressive, with each pupil given the opportunity to work through a series of tasks as far as they can manage.

There are four different practical tasks (task 1.1 to 1.4). Each of the four pupils in a school will do a different task.

One field officer should take the pupil through the task and the other field officer should observe the interaction and record the pupil's responses.

Conducting the assessment

• introduce the practical activity and explain to the pupil what you are going to do. You may wish to carry out an introductory activity to put the pupil at ease. This is not assessed. (Such an activity is likely to be more appropriate with younger pupils.)
• for older pupils you may wish to explain that some of the activities you will be asking them to do may seem very easy/simple; say that you want to find out how older pupils do things and to compare that to how younger pupils do the same thing. Try to make them aware that what you are asking them to do is not trivial.
• work through the sequence of tasks from the start until you feel that the pupil has done as much as he/she can do
• you may offer support to encourage a pupil to complete an activity
• through your questioning, try to explore the pupil's mathematical understanding, asking for clarification if necessary
• you may terminate the activity at any time if a pupil is obviously struggling or in any way distressed

Recording the results

• use the recording grid specific to the task
• make sure you complete the details at the top of each sheet
• for each sub-task you should record the pupil response (Y, N, ? = correct/incorrect/don't know) and also the degree of support given (0/1/2 = minimum/some/considerable)
• under 'mathematical language' make a note of the language used to express mathematical ideas - circle those given and add any others used
• under 'explanation' note the pupil's responses to the questions such as 'how did you work that out?' etc. Try to record the actual words/expressions used by the pupil. The aim is to gain a better understanding of how children think mathematically and how they express mathematical ideas.
• if a pupil has used any mathematical equipment, make a note of what was used and how competent they were (rated 2/1/0 from most to least).

Appendix D: The pupil questionnaires

There were four different questionnaires used at each stage - the same four at every stage. One set of questions was common to all four versions, while other enquiries, such as lesson activity frequencies and importance of mathematics for jobs were distributed across the four versions. For illustration purposes version 2 is shown here.

Assessment of Achievement programme

2004 Mathematics Survey

Appendix E: Selected pupil questionnaire results

Appendix F: The teacher questionnaires

Two questionnaires are reproduced here: the questionnaire for primary head teachers and the questionnaire for P5 class teachers. The questionnaire for secondary principal teachers was identical to the questionnaire for primary head teachers, save for a focus on S2 rather than P7. The class/subject teachers' questionnaires at P7 and S2 were also identical to the questionnaire included here for P5, save for a re-focus on the appropriate stage, references to 'department' in the S2 version in place of 'school' in the primary versions, and additional space in the S2 version for a third 'subject taught' and for information about class sizes and contact hours for up to four classes rather than one.

ASSESSMENT OF ACHIEVEMENT PROGRAMME 2004: MATHEMATICS

QUESTIONNAIRE FOR PRIMARY HEAD TEACHERS

Your school agreed to participate in this year's national survey of Mathematics. As part of the survey some of your P7 pupils recently took two written tests. We are grateful to you for collaborating with us in this way.

Now that the pupil testing is over, we very much hope that you will be prepared to help us further, this time by completing the attached questionnaire. The questionnaire has been designed to gather information about the organisation, emphasis and coverage of Mathematics in Scottish schools at the P7 stage. This information will prove a rich and informative background against which the results of the pupil testing can be reviewed. Taken together, the questionnaire information and the test results will provide a full picture of teaching and learning in Scotland in this general area of the curriculum, which will be of as much interest and value to schools as to others involved in the educational process.

Your personal responses to the questionnaire will be known only to those members of the survey team responsible for data analysis, and they will be treated as confidential. Neither you nor your school will be individually identified in the survey report or elsewhere when findings are presented and discussed. Your contribution to the general picture, however, will be invaluable. We urge you to give us the modest amount of time you will need to complete the questionnaire, and we thank you very sincerely for this further assistance with this innovative survey.

ABOUT MATHEMATICS IN YOUR SCHOOL

PLEASE GIVE YOUR VIEWS ON THE QUALITY OF THE FOLLOWING:

HOW WOULD YOU EVALUATE THE FOLLOWING?

IF YOU WOULD LIKE TO MAKE ANY FURTHER COMMENTS, PLEASE DO SO IN THE SPACE BELOW :

ASSESSMENT OF ACHIEVEMENT PROGRAMME 2004: MATHEMATICS

QUESTIONNAIRE FOR TEACHERS IN P5: MATHEMATICS

Your school agreed to participate in this year's national survey of Mathematics. As part of the survey some of your P5 pupils recently took two written tests. We are grateful to you for collaborating with us in this way.

Now that the pupil testing is over, we very much hope that you will be prepared to help us further, this time by completing the attached questionnaire. The questionnaire has been designed to gather information about the organisation, emphasis and coverage of Mathematics in Scottish schools at the P5 stage. This information will prove a rich and informative background against which the results of the pupil testing can be reviewed. Taken together, the questionnaire information and the test results will provide a full picture of teaching and learning in Scotland in this general area of the curriculum, which will be of as much interest and value to schools as to others involved in the educational process.

Your personal responses to the questionnaire will be known only to those members of the survey team responsible for data analysis, and they will be treated as confidential. Neither you nor your school will be individually identified in the survey report or elsewhere when findings are presented and discussed. Your contribution to the general picture, however, will be invaluable. We urge you to give us the modest amount of time you will need to complete the questionnaire, and we thank you very sincerely for this further assistance with this innovative survey.

ABOUT YOU AND YOUR TEACHING IN P5

IN MATHEMATICS TOPICS, HOW OFTEN DO YOUR PUPILS SPEND TIME……

In teaching Mathematics topics to your P5 class, how much use do you make of…..

Appendix G: Teachers' reports on lesson activities

* See Appendix E for the exact wording of the activity descriptions

« Previous | Contents | Next »