The Scottish Government

Listen

ISBN 0 7559 2789 3
This document is also available in pdf format (216k)

The 2004 AAP Mathematics and Core Skills Survey

Executive Summary

1. Survey overview

1.1 The 2004 Mathematics Survey was the seventh survey to be carried out in this curriculum area within the Assessment of Achievement Programme ( AAP), and the third subject survey to incorporate the assessment of core skills.

1.2 The stages assessed were P3, P5, P7 and S2 and testing took place in May and June 2004, when around 15500 pupils in just under 850 mainstream schools across Scotland took part, that is 3000-4000 pupils at each stage.

1.3 All the survey schools participated in the written assessments of mathematics and the majority also took part in the assessment of reading and writing. In addition, a sub sample of the schools participated in the assessment of pupils' practical skills. Around two-thirds of the survey pupils attempted pencil and paper mathematics items as well as mental maths tests, while the remainder participated in core skills reading and writing assessment.

1.4 The 'mathematics' pupils in just under 350 of the survey schools (around 40% of the schools) also took part in practical mathematics assessments, while the 'core skills' pupils undertook ICT tasks, or worked in pairs on problem solving tasks or participated in experimental group discussions.

1.5 To provide a context for the attainment and attitude findings, half the participating schools were invited to complete a questionnaire about the resources available to them for mathematics and about their mathematics provision, and those survey pupils undertaking mathematics assessments were invited to complete questionnaires about their mathematics learning experience. For the first time in an AAP Mathematics survey, the pupil questionnaires included an exploratory assessment of pupils' perceptions of the importance of mathematics in different professional occupations.

1.6 The 5-14 National Assessment Bank provided most of the mathematics test items used in the survey, and some of the reading and writing tasks. Where necessary, new items and tasks were commissioned by the SQA for survey use; all the practical tasks were newly developed. School and pupil questionnaires were developed by SEED.

1.7 The SQA was responsible for printing the test booklets and distributing these to schools. The survey schools organised and supervised their own written test sessions within the period mid-May to mid-June. Completed test booklets were received and processed by the SQA, and pupil response data keyed by Datapro Data Preparation Ltd and analysed by Assessment Europe.

1.8 Practical assessments were the responsibility of itinerant field officers, practising primary teachers or secondary mathematics teachers released by their education authorities for survey involvement.

2. Reporting Mathematics attainment

2.1 Almost 10000 pupils in almost 850 schools participated in the written/mental assessment of mathematics, that is around 2000-3000 pupils at each stage. In total, over a thousand pencil and paper test items and almost 400 mental items were administered in the survey. Individual pupils attempted mixed-level booklets of items at two or three different consecutive 5-14 levels: Levels A and B at P3, Levels B, C and D at P5, Levels C, D and E at P7, Levels D, E and F at S2.

2.2 The attainment criteria first introduced into the AAP English survey of 2001 were adopted again here: 'considerable strengths' at a level is indicated by 80% or more of items correct at that level, 'secure' attainment by at least 65% of items correct but fewer than 80%, and 'basic skills' by at least 50% of the items correct but fewer than 65%.

3. Mathematics attainment results

3.1 At P3, over 95% of the pupils assessed in the survey were estimated to have shown at least basic skills at Level A, just under 90% were 'secure' or better at this level and almost 70% showed considerable strengths. At Level B the corresponding proportions were over 60%, around one-third and just under 10%.

3.2 At P5, 90% of the pupils were estimated to have demonstrated at least basic skills at Level B, over 70% were at least 'secure' and just over 40% showed considerable strengths. At Level C the corresponding proportions were 65%, 40% and 15%, respectively, and at Level D around 25%, just under 10% and 2%.

3.3 At P7, just under 90% of the pupils were estimated to have demonstrated at least basic skills at Level C, almost 75% were 'secure' and over 40% showed considerable strengths. At Level D the corresponding proportions were almost 70%, over 45% and over 15%, respectively, and at Level E almost 35%, almost 20% and just over 5%.

3.4 At S2, 80% of the pupils were estimated to have basic skills at Level D, 60% were secure and 30% showed considerable strengths. At Level E the corresponding proportions were just under 60%, almost 40% and just under 20%, respectively, while at Level F the figures were over 15%, over 5% and 2%.

3.5 One of the most interesting features that emerged in the survey is the steady progression in the attainment profiles of the pupils from P3 to P7 at Levels B to D. The proportions of pupils secure at the relevant level, and showing considerable strengths, gradually increases. Although the proportions attaining the expected levels for the stage are not high, nevertheless schools do appear to be building on pupils' achievements during each 2-year period. The pattern is illustrated in Figure 1 below.

3.6 At S2, however, the pattern is different. The proportion of pupils showing considerable strengths remains stable, but the proportion showing secure attainment has noticeably reduced. The proportion showing basic skills remains stable, but there is an increase in the numbers scoring fewer than half of the available marks in the tests at Level E.

Figure 1
Profiles of mathematics attainment P3 to S2*

* '< basic' means fewer than 50% of marks achieved, 'basic' is between 50% and 64%, 'secure' is 65% to 79%, and 'strengths' is 80%+.

3.7 This would suggest that in the first two years in secondary school able mathematicians are well catered for. Some pupils, however, are not enabled to build on their secure attainment at Level D as they work towards the next level; and some of those with only basic skills at Level D find themselves in considerable difficulty as they encounter the greater challenges of Level E work. The pace of learning, and particularly approaches to consolidation of previous learning, rather than 'pushing on' to the next level, may both need to be reviewed, particularly for S1-S2.

3.8 At all stages and levels Problem Solving was the mathematics outcome showing the lowest average attainments. This finding is at least partially attributable to the fact that items were generally classified as problem solving items if pupils were required to carry out two or more operations rather than one to reach a correct answer, and without any cueing. Clearly, the more steps are involved in arriving at a successful conclusion the more opportunities there are for error. Problem solving is an area that continues to need attention, particularly at S1-S2.

3.9 Focusing on the lowest attaining pupils, one-fifth of the S2 pupils failed to demonstrate basic skills at Level D, the level below their formal target level. At P5 and P7 around one in ten pupils showed similarly low attainment at Levels B and C, respectively.

3.10 The attainment picture for numeracy is essentially as described above for mathematics as a whole.

3.11 Average item scores were generally 10-15 percentage points higher for the mental testing at each level (the small numbers of mental items attempted at each level by individual pupils preclude the reporting of percentages of pupils attaining levels). However, pupils' relative success in mental maths does not seem to have impacted on overall mathematics attainment in the way anticipated in the 2000 mathematics survey report.

3.12 There was no evidence in the survey data of any difference in the mathematics attainment of boys and girls. In this respect, Scotland differs from a number of other countries, where boys' mathematics attainment is consistently reported as higher.

3.13 On the basis of pencil and paper test items used in both the 2000 survey and this 2004 survey at P7 and S2 (100 items per level on average), there is no evidence of any change in pupils' mathematics attainment in the interim.

3.14 Relatively small numbers of pupils (150-200) attempted multi-item themed pencil and paper 'mathematics literacy' tasks within the practical assessment sessions. Three tasks at each of two levels were administered at each stage, and the findings reveal a steady attainment progression through the stages: average percentage scores were in the range 60-70% for the lower level tasks at each stage dropping to 40-45% for the higher level tasks. It is possible that tasks of this kind, involving language as well as maths, might provide support for problem-solving and that more experience of tackling them could help to improve overall attainment in mathematics.

4. Practical mathematics

4.1 A small number of resource-based practical tasks were administered at all four stages in the survey. The tasks focused on one or other of 'fractions, percentages and ratio', 'money', 'time and measure', 'shape, angle and direction', with each task demanding activities at increasing 5-14 levels. Between 70 and 100 pupils at any stage attempted the individual tasks. The pupils were guided through the tasks by itinerant field officers, the assessment ending when pupils seemed to have reached their personal levels of capability.

4.2 At P5, P7 and S2, 90% or more of the pupils successfully carried out the various activities involving money. Performance was lower at P3, particularly when pupils were required to calculate and deliver change rather than simply to offer coins up to a given price. Discount problems were handled well by half to three-quarters of the S2 pupils.

4.3 Time (setting analogue and digital displays) also proved relatively unproblematic for the P7 and S2 pupils, 80% or more of whom successfully completed the activities at all levels unaided. At P3, 50-70% of the pupils managed the Level B activities unaided, whereas 75-85% of the P5 pupils successfully tackled the Level C activities. A roughly similar picture emerged for the task focusing on measure.

4.4 The fractions/percentages task revealed a steady attainment progression through the four stages, with particularly marked attainment gaps between P5 and P7 at Level D (percentages) and between P7 and S2 at Level E (ratio).

4.5 A variety of activities involving naming and drawing shapes and angles again revealed clear evidence of stage progression. Turtle programming was successfully achieved by 20-30% of the P3 pupils, 45-60% of the P5 pupils and 70% or more of the pupils at P7 and S2. Similar proportions of pupils at the different stages were successfully able to use a compass to identify direction.

5. Core skills attainment results

Communication - Reading

5.1 Reading attainment was assessed using multi-item tasks based on texts set in mathematics contexts ( e.g. featuring famous mathematicians). No prior mathematical knowledge was needed in order to show evidence of reading comprehension.

5.2 At P3 and P5, tasks at two consecutive stages were administered: A and B at P3, B and C at P5. At P7 and S2, tasks at three different levels were used: C, D and E. At every stage each pupil attempted two different tasks, one at each of two different levels.

5.3 Over 65% of the P3 pupils were estimated to be 'secure' or better at Level A ( i.e. gaining at least 65% of the marks on their Level A task) and over half were also secure at Level B; over 80% of the P5 pupils were estimated to be secure at Level B and over half were also secure at Level C.

5.4 Between P7 and S2 there is similarly clear evidence of continued stage progression, particularly at Levels D and E. At P7, 80% of the pupils were estimated to be secure at Level C, 60% at Level D and just over 15% at Level E. Corresponding proportions for S2 are 85%, just under 75% and just under 30%.

5.5 There were significant gender differences in attainment at P3, P7 and S2, the girls generally outperforming the boys, and with strong evidence of topic effects. This gender difference in reading appears to be nearly universal, according to international studies.

Communication - Writing

5.6 Pupils attempting reading tasks were also invited to produce a piece of writing, with the reading task content as stimulus material. In total, over 3000 pieces of writing were evaluated using a 'best fit' scheme, the scripts having been drawn at random from a larger set. Each script was evaluated by three independent teacher raters, and around 80% of the scripts were classified in the same way by at least two raters.

5.7 There was an expected stage progression evident in the resulting attainment decisions: the proportions of scripts judged to be at Level B or higher rose from almost 40% at P3 through around 75% (56%, 57%) at P5 and just over 85% (89%, 74%) at P7 to over 90% (77%, 80%) at S2. Around a quarter (29%, 23%) of the S2 pupils were judged to be at Level D and just under 10% (9%, 7%) at Level E. (Improvement at P5 at Level B; similar (dismal) picture across 3 surveys for S2).

5.8 There were significant gender differences in writing quality at all stages, with the girls tending to produce the better writing. Again, this is an expected finding that has emerged in other international studies.

Using ICT

5.9 Over 1300 pupils in over 340 schools participated in the assessment of ICT skills: 250-400 pupils per stage. Pupils undertook their tasks individually, using a laptop computer supplied by an observing field officer. The findings show clear evidence of age-related progression, with the majority of P7 and S2 pupils competent in most aspects.

5.10 High proportions of the pupils at all stages (typically 90%+) claimed familiarity with use of keyboard and mouse. Familiarity was also high for P7 and S2 with respect to the operating system ( PC/Windows) used for the assessments, but markedly lower at P5 and P3. High proportions of P7 and S2 pupils were also familiar with Word, proportions dropping to a third for P3 pupils; familiarity with Excel was much lower, from over half the S2 pupils to almost none of those at P3.

5.11 The majority of pupils at P7 and S2 were competent when using the technology (opening a folder, scrolling a file, saving text under a new name, etc), the proportions being lower for P5 and lower still for P3, with variation depending on the specific task.

5.12 Text handling and graphic manipulation ('creating and presenting') were also well done in general by most pupils at P7 and S2. Justifying text, creating two or more columns of text, bulleting and wrapping text were the most challenging tasks.

5.13 Table handling ('collecting and analysing') proved more difficult than text handling and graphic manipulation in general, but again there was very clear evidence of skills development with age.

6. Pupils' learning circumstances and views about mathematics

6.1 Very high proportions of the survey schools at P5, P7 and S2 (90% or more) returned completed pupil questionnaires (pupil questionnaires were not administered at P3).

6.2 High proportions of the responding pupils at P5, P7 and S2 had a quiet place to study at home, had home access to a computer and the internet, and had a calculator and dictionary available to them at home. The most common family pastimes were watching TV and videos, and listening to music. Cultural outings of various types were uncommon, while reading and talking about books with family members decreased markedly in popularity with age; these activities were more frequently mentioned by the girls than the boys.

6.3 The majority of pupils belonged to out-of-school clubs or groups of various kinds, with the girls more likely to be club/group members than boys in the primary stages and the reverse at S2.

6.4 In the mathematics classroom the most frequent individual activities were writing in a jotter or file, working quietly alone and reading texts/reference books.

6.5 Using computers in class, working in the school grounds and visiting places outside school were infrequent activities in mathematics lessons, particularly at S2. For ICT, this suggests that high-quality provision is under-utilised in schools, although home access to computers is good. Use of spreadsheets in particular is limited. There may be considerable scope for fuller use of ICT in mathematics teaching.

6.6 On their own reports, the S2 pupils spent on average one hour more on mathematics homework than did their younger peers.

6.7 Job aspirations were very varied at all stages, and heavily gender typed. The majority of pupils considered mathematics to be important for later learning in other subjects and for future occupations, and they wanted to do well in the subject.

6.8 The majority of the pupils at all stages found mathematics an easy subject, were happy with the pace of work, were rarely left behind, and rarely found it difficult to catch up if they missed a lesson.

6.9 The majority of pupils at all stages thought that they were expected to work hard in their mathematics lessons, that they were given help when they needed it, that everyone had a chance to say what they thought, and that they found it easy to concentrate and work hard in class.

6.10 Pupils were less positive about whether they looked forward to lessons and enjoyed reading books about the subject, enthusiasm for mathematics learning decreasing with increasing age.

6.11 Teacher assessment more frequently took the form of short tests at S2 than in the primary sector.

6.12 There were no differences between boys and girls in terms of their reports on their mathematics learning, nor in perceptions of subject importance for later learning or jobs.

7. Teachers' reports on mathematics in the schools

7.1 Questionnaire enquiries were incorporated into the survey at all stages to gather information from teachers about the provision and resourcing of mathematics in the schools, and to invite their views about the quality of different aspects of mathematics experience in classrooms.

7.2 A school manager's questionnaire and a class/subject teacher's questionnaire were sent to a randomly selected half of the survey schools. The school managers' questionnaire was completed by 245 primary head teachers and 58 secondary mathematics principal teachers, and the class/subject teachers' questionnaire was completed by 239 primary teachers and 52 S2 mathematics teachers. These figures translate to response rates of around 60-70% on the part of the schools concerned.

7.3 In the primary schools just over half the current mathematics programmes had been introduced in 2000 or later, compared with two-thirds of the S2 programmes in the secondary schools. Two in five of the primary programmes were currently under revision, along with the majority of the programmes at S2. The most popular bases for mathematics programmes in both sectors were the national 5-14 guidelines, and commercial textbooks and resource packs. The school's (or department's) own materials featured significantly more often at S2 than in the primary school, with the reverse holding for local authority 5-14 guidelines. Other possible resources were rather little used in either sector.

7.4 Almost all the schools in both sectors reported to parents in terms of 5-14 levels, with high proportions also using teachers' own comments. Compared with the situation in the primary sector, pupil progress at S2 was significantly more often reported in terms of test/exam results or marks/grades for effort. In contrast, comment banks were significantly more often used by the primary schools for reporting to parents than by the secondary schools.

7.5 When invited to rate the quality of various resources, differences in the rating patterns of primary head teachers and S2 principal teachers frequently reached statistical significance, with primary head teachers being the more positive.

7.6 Among the primary head teachers the most positive quality ratings were given to computer and internet access for teachers and for pupils, and to pupil attendance at, and behaviour in, subject classes. Parental support for their children's learning was only moderately well rated in both groups, as was the availability of learning support or enrichment for individual pupils in mathematics.

7.7 Most school manager respondents in both sectors gave high ratings to teacher expectations of pupil achievement in mathematics. But for pupil motivation to learn and teacher morale, the primary head teachers gave significantly more positive ratings than the S2 principal teachers. Just over a quarter of the class/subject teachers in both sectors responded 'very well' when asked how well motivated their pupils were to learn mathematics.

7.8 Class sizes varied very widely in both sectors, with an average of around 24 in the primary schools and 27 at S2. Class contact time was six hours per week on average in the primary schools and four hours at S2. Lesson preparation time was claimed by the primary teachers to be four hours a week on average, whereas among the S2 teachers the weekly average was just two hours.

7.9 Over half the S2 mathematics teachers indicated that they met with colleagues once a week or more frequently to discuss mathematics education issues, compared with around one-third of the primary class teachers.

7.10 According to the majority of the teachers in both sectors, pupils were taught as a class for at least some of the time in most lessons, while group teaching was significantly less prevalent at S2 than in the primary sector. Pupils also spent at least some of the time in most maths lessons writing in their jotters and files, and worked in pairs for at least some of the time in most lessons in the majority of the S2 classes.

7.11 The evidence from the teachers' responses is that S2 pupils more often used tools and instruments in investigations than did their younger peers, and they also used maps, drawings and diagrams more frequently. As the pupils themselves reported, the use of computers in class was relatively rare in both sectors, but particularly at S2, and mathematics lessons were rarely carried out in the school grounds or further afield.

8. Issues for consideration

8.1 The findings from the survey highlight a number of issues relevant to future developments in learning and teaching mathematics in primary and lower secondary school. There are clearly positive messages to be taken from this survey but it also raises issues for staff development and review of provision within the 5-14 stages.

8.2 Any review of the curriculum should re-consider levels and expectations, based on experiences over multiple AAP surveys and other evidence that might guide such a review, so that realistic expectations are made of pupils at each of the key stages.

8.3 The pace of pupils' learning needs to be reviewed and consolidation of study is required to ensure that 'secure' knowledge and understanding or 'considerable strengths' can be demonstrated across the different attainment outcomes at each stage. The reference group acknowledged the need to strive for 'secure' knowledge and 'considerable strengths', favouring depth of treatment in the earlier stages over any pushing on in a premature fashion. This calls for teachers at all stages to interpret the 5-14 guidelines in the fullest way to challenge the pupils in their care. There is scope for more inventive and extensive interpretation of the guidelines when it comes to reviewing 'examples' within any strand e.g. Time, Fractions & Percentages. The aim should be for 'secure' knowledge and understanding or 'considerable strengths' to be demonstrated within strands and attainment outcomes.

8.4 Pupils' enthusiasm for mathematics learning decreases with increasing age, yet the majority of pupils across all stages considered mathematics to be important and they wanted to do well. Pupils' appreciation of the importance of mathematics and their enthusiasm for learning needs to be capitalised upon to ensure they remain motivated as they move through the stages. It is worth considering what scope there is for broadening pupils' experiences within the level of study, embracing more problem solving, mathematical literacy tasks and generally consolidating knowledge, skills and understanding to gain secure status and considerable strengths in preference to any premature acceleration, that may be a factor in the declining standards witnessed in the S2 profiles.

8.5 Problem solving is an area that continues to need attention, particularly at the S2 stage. Pupils have not reported a lot of interesting topics or investigations in their study of mathematics. Mathematics literacy tasks provide contexts that potentially support problem solving and in turn pupils' general mathematical attainment. There are significant gender differences in reading attainment and in the quality of writing across most stages, with girls tending to outperform boys. Core skills of reading and writing are the only aspects that have shown a gender difference in favour of girls.

8.6 The picture of attainment in mental mathematics is a positive one, although this has not as yet significantly affected other aspects of mathematics as suggested in the report on the 2000 AAP mathematics survey.

8.7 The survey shows no significant gender differences in attainment in mathematics, at any stage. There are significant gender differences in reading attainment and in the quality of writing across most stages, with girls tending to outperform boys. In international studies, in many countries boys outperform girls in mathematics. However, girls outperform boys in reading and writing almost all countries.

8.8 As all teachers have a responsibility to promote literacy, perhaps the model explored in the core skills part of this survey could be pursued more widely in mathematics classrooms. Books 'about mathematics' are not widely accessed by pupils, or strongly identified as a positive motivator, but they could provide a link with literacy, something that is currently missing. Such activity might close the gender gap through more regular exposure to mathematical texts with the opportunity to write about them within a supported framework.

8.9 High quality ICT provision is under-utilised. Class teachers and pupils report limited use of ICT in mathematics classes yet home access is good, suggesting scope for fuller use within subject teaching. Use of spreadsheets seems to be particularly limited. Given the high quality of ICT reported as being in schools, it is important to fully utilise these resources with pupils in the classroom. Pupils appear to be confident in their use of ICT, increasingly gaining access at home, so it clearly makes sense to capitalise on this ever-expanding learning resource in a more formal capacity.

8.10 Staff development opportunities will need to be supported. An aspect that appears to be worthy of support is an unpacking of what 'consolidation' might entail to ensure staff are comfortable with their interpretation of the guidelines, going beyond the stated targets but working within the level of study and resisting the temptation to accelerate towards the higher levels. Another staff development opportunity would be within the ICT domain, providing support for staff to extend their personal knowledge (subject and pedagogical) and confidence with their use of ICT to maximise the classroom use of the resources that are now reported to be in schools.