The Scottish Government

« Previous | Contents | Next »

Listen

Programme for International Student Assessment (PISA) 2003: Initial Report on Scotland's Performance in Mathematics, Science and Reading

Chapter 2: Student Proficiency in Mathematical Literacy

How Mathematical Literacy is Defined

The following paragraphs briefly outline the definition and assessment of mathematics literacy used for PISA. A full account is to be found in The PISA 2003 Assessment Framework, 2003 (OECD) and in Chapter 2 of the international report on the PISA 2003 survey results referred to earlier.

Centrally, the PISA study is predicated upon an understanding of mathematics as now being of significance for all adults in society rather than just for a minority of specialists as has been assumed, either overtly or implicitly, in previous studies. The concern is consequently not with how well students confront problems designed specifically to assess the concepts and techniques taught at school but whether they can stretch out beyond these potentially restrictive and artificial problems to apply their learning to situations similar to those they will meet as adults in their working and personal lives. Whereas the primary objective of many national examination systems is to ascertain whether or not the student has acquired the necessary mathematical foundation to see him, or her, into higher education in some specialism, the primary objective in PISA is to establish the extent to which those now leaving school will be able to meet the mathematical demands of living at the beginning of the 21 ^st Century.

The tasks simulate, as closely as possible within the limitations of an assessment context, situations that students could well encounter in their present and future lives, with many items drawn from real-life examples provided by the participating countries. An expert group of mathematical educators was responsible for the selection, and formulation, of items. They applied themselves to ensuring that the final selection of items effectively tapped into students' ability to activate the mathematical knowledge and skills needed to solve such problems.

As problems in everyday life rarely present themselves with the mathematical route to their solution apparent, so in PISA, students have to decode the tasks set and translate them into a suitable mathematical form before they can start to solve them. Process and situation are key factors in the PISA concept of mathematical literacy. Simple technical, or even conceptual, competence in mathematics is not enough. Students must also demonstrate the ability to unravel the core of each task and, when a unique solution exists, find a suitable mathematical model for solving it or, when one does not exist, the model they judge provides the best answer to the task set.

How Mathematical Literacy was Assessed in PISA 2003

The assessment of mathematics was set in a framework defined by three factors ⁶: the mathematical content of the tasks, the processes required for interpreting each task in a mathematical form, and the various situations in which such tasks might be encountered.

Four content zones were covered:

space and shape
change and relationships
quantity, and
uncertainty.

Tasks were categorised at three levels of process:

the reproduction cluster
the connection cluster, and
the reflection cluster.

Tasks were set in four situational contexts:

personal
educational and occupational
public, and
scientific.

A brief description of these follows.

Four Content Areas

Space and shape, as the name implies, draws upon the discipline of geometry. It requires students to recognise similarities and differences in the shapes of objects when presented in different representations and in different dimensions, as well as the concepts of relative position and movement.

Change and relationships relates most closely to algebra. It involves, besides an understanding of the functional dependency between variables, an awareness of inequalities, equivalence, divisibility, etc, as well as a recognition that relationships can be expressed in various mathematical forms, and that changing between representations may be the key to solving a problem.

Quantity covers those aspects of mathematics bearing upon number. Students have to demonstrate competence in the many facets of this, from an understanding of relative size, and the use of numbers as representations of objective properties of bodies, through to the higher level of quantitative reasoning, which requires the understanding of the meaning of arithmetical operations.

Uncertainty lies within the area of probability and statistics. PISA regards the understanding of statistical ideas and the ability to follow statistical arguments as of increasing importance, if citizens are to participate effectively in democratically organised societies.

Process

The process of translating a problem to an appropriate mathematical formulation, is rarely achieved in one single step. A complete formulation often requires several levels of conceptual refinement. Similarly, the actual mathematics needed to solve it may require several different levels of proficiency. Some steps may be straightforward arithmetic, others may require careful algebraic manipulation. Finally, the complete answer to a problem may require several partial solutions, each at different conceptual levels. Therefore, real-life problems will rarely fall neatly into one single level of any framework devised to categorise process. Consequently, the main value of such a framework lies in confirming that the desired conceptual range is adequately covered by the collection of items used. It has less value for the reporting of results. The framework adopted by PISA assembled cognitive activities into three clusters, loosely hierarchical in structure. These are:

The reproduction cluster covers those competencies necessary to solve familiar, routine problems. Students essentially utilise practised knowledge, standard methods, and straightforward calculations.

The connections cluster comprises competencies that students have to deploy to solve those problems which, although set in a familiar situation, do not present an immediately recognisable solution. Such problems typically involve a greater degree of interpretation for their solution than those in the previous cluster.

The reflection cluster covers the highest levels of competencies required by PISA. Tasks in this cluster require some insight, reflection, and creativity. These tasks typically involve more mathematical elements than others and require students to explain and justify their reasoning and methods.

Situation

Students were presented with situations that they could conceivably meet in their own lives and which mathematical methods would help them analyse and resolve. Such situations fell into four broad categories: personal, educational/occupational, public, and scientific. A short description of each follows.

Personal situations relate directly to the student's own daily life. They have at their core the ways in which the individual perceives and is affected by an immediate, personal context. The student has to utilise their mathematics to appreciate, or interpret, some specific aspect of each situation.

Educational/occupational situations include settings that could arise in a student's school or work life. The core of these situations is how particular school and work settings present students with problems requiring a mathematical solution.

Public situations require students to observe aspects of their broader surroundings. They are generally situations located in the community and their core consists of the relationships that exist between the operative factors. A mathematical evaluation of the aspects of such relationships that have consequences for public life is wanted.

Scientific situations comprise those more abstract contexts typically involving a technological process, a theoretical one, or an explicitly mathematical problem. PISA includes within this situation abstract problems frequently confronted in the mathematics classroom, without attempt at contextualisation.

How the Mathematical Literacy Results are Reported

Results are presented in two principal ways.

The first uses a scale of scores obtained by modelling the patterns of item responses from each student. As each student sat just one booklet from the thirteen test booklets used in the assessment, statistical modelling of the responses is necessary to place all students on a common score scale. This scale was set to have a mean across OECD countries of 500 and a range such that two-thirds of students would score between 400 and 600 ⁷. Scales were derived for each of the four content areas, and for mathematics as a whole.

The second method of reporting results uses six 'proficiency levels' - descriptions of the kind of mathematical competency demonstrated by students. Summary descriptions of the proficiency levels are provided in Appendix B. Full descriptions can be found in the OECD publications arising from the study, which have been mentioned already. Descriptions of the highest and lowest levels are given below for convenience. Each test item used in PISA 2003 was matched to one of the six proficiency levels, and students were then placed at a specific proficiency level depending on how they had answered the set of items allocated to this level. More specifically, a student was placed at a particular proficiency level if he or she could be expected to answer correctly at least 50% of a hypothetical range of items spread evenly across the difficulty range for that level.

At Level 6, the highest level:

" .. students can conceptualise, generalise, and utilise information based on their investigations and modelling of complex problem situations. They can link different information sources and representations and flexibly translate among them. Students at this level are capable of advanced mathematical thinking and reasoning. These students can apply this insight and understanding along with a mastery of symbolic formal mathematical operations and relationships to develop new approaches and strategies for attacking novel situations. Students at this level can formulate and precisely communicate their actions and reflections regarding their findings, interpretations, arguments, and the appropriateness of these to the original situations."

At Level 1:

"...students can answer questions involving familiar contexts where all relevant information is present and the questions are clearly defined. They are able to identify information and to carry out routine procedures according to direct instructions in explicit situations. They can perform actions that are obvious and follow immediately from the given stimuli."

Summary of Mathematics Results for the OECD and for Scotland

Proficiency Levels

Results for the combined mathematics scale are given in Table 2.1. Figure 2.a presents the same results in graphical form and orders countries on the basis of the percentage of their students that reach Level 2 or above. Results for the individual broad content areas will be presented later.

Just below one-third (31%) of students across the OECD area as a whole performed at the top three levels of mathematics (Level 4 or above), and 3.5% at the highest level, Level 6. In broad terms, although OECD country results do vary widely, Belgium, Korea, and Japan, have the greatest proportions of their students achieving both the top three levels jointly, and Level 6, the topmost, as well. Mexico has the lowest proportion of its students at these levels.

With 41.2% of Scottish students attaining Level 4 or better Scotland is well placed on the broader criterion covering Levels 4 to 6, but placed similarly to OECD students as a whole on the narrower criterion, with 3.9% of students achieving Level 6.

At the lower end of the scale, more students in Scotland, 89%, were operating at or above Level 2, compared with 74% in OECD countries as a whole. While Scotland thus appears well placed in respect of the poorer performing students, Level 2 merely requires that students can interpret and recognise situations in contexts that require no more than direct inference and that they can extract information from a single source and make use of a single representational mode. That 11% of our 15 year old students fail to reach Level 2, may give cause for concern.

Table 2.1. Percentage of students at each level of proficiency on the mathematics literacy scale

Figure 2.a. Percentage of students at each level of proficiency on the combined mathematics scale

Mean Scores in Mathematics Literacy

Table 2.2 gives the mean scores for the 29 OECD countries and Scotland, along with mean scores for male students and female students separately.

Table 2.2. Student performance on the mathematics scale, all students and by gender

Positive differences indicate that males perform better than females
Negative differences indicate that females perform better than males

As previously noted, the mean score for the mathematics scale was set at 500 for the OECD overall. Scotland's mean score of 524 is therefore significantly above the OECD average. Eight OECD countries have mean scores higher than Scotland. These are: Australia, Belgium, Canada, Finland, Korea, Japan, The Netherlands, and Switzerland. However, the statistical uncertainty associated with extrapolating these test scores to the average performance to be expected from all 15 year old students means that only for Finland, Korea, and The Netherlands can it definitely be said that their national attainment is better than Scotland's. For the other five this may be the case, but it is not definitely so. Nine OECD countries have mean scores that do not differ significantly from the Scottish one and 17 have mean scores that are significantly lower. Table 2.3 summarises the position.

Table 2.3: OECD countries whose mean scores differ significantly from the Scottish mean, or do not differ significantly from this.

Changes in Mean Scores in Mathematics Between PISA 2000 and PISA 2003

As mathematics was the primary domain in PISA 2003 more mathematics items were used for this survey than in PISA 2000: 85 items in 2003, compared with 31 in PISA 2000. Twenty (20) of the PISA 2000 items (those not publicly released in the reports on that study) were re-used in 2003 to provide a common core for the two studies.

Several factors conspire against drawing strong conclusions about changes in countries' mean scores between the two surveys. The three principal factors are: firstly, the relatively small number of items linking the two studies; secondly, the fact that the items carried forward were not distributed evenly across the four content areas of mathematics, but came largely from just two, namely Space and Shape, and Change and Relationships; and thirdly, that context changes between the booklets used in 2000 and those in 2003 introduce appreciable statistical uncertainty into the matching of the scales for each survey. For these reasons, comparisons between the two surveys are necessarily general in nature and confined to the two content areas mentioned.

Space and Shape

Performance could be compared across 25 OECD countries and Scotland. Figure 2.b shows the comparison in graphical form. Small black squares denote the PISA 2000 mean scores and small white squares, the PISA 2003 ones. Shaded areas show countries with significantly different mean scores between the two surveys.

The average score across these 25 countries as a whole did not change significantly. On a common scale, the mean score in PISA 2000 was 494, while that in PISA 2003 was 496 points. Four OECD countries show a significant improvement in mean score and two a significant drop. The four showing these improved scores are: Belgium, The Czech Republic, Italy, and Poland. The two showing a drop in scores are: Iceland and Mexico.

Scotland shows a slight, though non-significant drop in performance, with a mean score in PISA 2000 of 511 points and in PISA 2003, of 507 points. This drop was essentially uniform at all levels of student attainment. Comparing scores in 2000 with those in 2003 for students at a number of levels of attainment, the 5 ^th, 10 ^th, 25 ^th 75 ^th, 90 ^th and 95 ^th percentile levels, ie the 5% of students at the lowest level through to the 5% at the highest level, scores change by 5 points at most. A difference of 5 points is well within the plausible limits of statistical uncertainty.

Figure 2.b. Differences in scores between PISA 2000 and PISA 2003 on the mathematics space & shape scale

Source : OECD PISA 2003

Change and Relationships

Performance in this content area could be compared also across 25 OECD countries and Scotland. Figure 2.c shows the comparison in graphical form. For these the average score did, in contrast to Space and Shape, show a significant change. On a common scale, the mean score in PISA 2000 was 488, while that in PISA 2003 was 499 points. Ten OECD countries show a significant improvement in mean score and none a significant drop. The ten showing improved scores are: Belgium, Canada, Czech Republic, Finland, Germany, Hungary, Korea, Poland, Portugal, and Spain.

The Scottish mean score decreased, but by just 1 point, between the two surveys. As with Space and Shape, there was no significant difference in performance observable at any level of attainment. The greatest difference at any level of attainment in Change and Relationships was just 3 points.

Although not visible in Figure 2.c, all but four OECD countries showed an increase in attainment in this content area between 2000 and 2003. This would imply that it is an area that many countries are focusing on developing, in which case Scotland's relative stand-still may be worrying. On the other hand, Scotland's score of 529 is already one of the highest and it may be that other countries are merely now catching up. Of the five countries with higher mean score than Scotland in PISA 2003 ⁸: two (Korea and Japan) already had higher scores than us in 2000; three (Belgium, Canada and Finland) made gains of between 14 and 22 points to enable them to leap-frog over Scotland.

Figure 2.c. Differences in scores between PISA 2000 and PISA 2003 on the mathematics change & relationships scale

Source : OECD PISA 2003

« Previous | Contents | Next »