The Scottish Government

« Previous | Contents | Next »

Listen

Programme for International Student Assessment (PISA) 2003: Initial Report on Scotland's Performance in Mathematics, Science and Reading

Chapter 3: Student Proficiency in Reading Literacy

How Reading Literacy is Defined

Reading literacy as defined in PISA focuses on the ability of students to use written information in situations that they encounter, or may encounter, in their lives. Reading literacy is understanding, using and reflecting on written texts, in order to achieve one's goals, to develop one's knowledge and potential and to participate in society. This notion goes beyond the traditional notion of decoding information and literal interpretation of what is written, towards more applied tasks.

How Reading Literacy was Assessed in PISA 2003

The assessment of reading for PISA 2003, as in PISA 2000, was framed along three dimensions: the type of reading task, the form and structure of the reading material, and the use for which the text was intended. Three broad types of task were used. Some required the retrieval of information, others the interpretation of the texts, and others called for reflection and evaluation of the texts. The text forms themselves were classified as 'narrative', 'expository', 'descriptive', 'discursive', 'instructive', and 'documentary'. Both continuous and non-continuous prose forms as well as charts, tables, diagrams, etc, were used. A wide variety of intended uses were covered under such categories as 'personal use', 'occupational use', and 'reading for education'. Further discussion of how the assessment was constructed is to be found in Chapter 6 of the International Report and in more detail in the PISA 2003 Assessment Framework .

Reading literacy was the main topic in PISA 2000, but a minor one in PISA 2003. Twenty eight of the 132 items used in PISA 2000, constituted the test of reading in PISA 2003. These 28 items were carefully chosen to give as balanced a picture of attainment in reading literacy as could be achieved within the limitations imposed by the study design, while at the same time providing a sound cross-section of the items used in PISA 2000. Each of the three aspects of reading: 'retrieval', 'interpretation', and 'reflection' were covered; as was the full difficulty range of the PISA 2000 items.

How the Reading Literacy Results are Reported

The more limited scope of the PISA 2003 assessment restricts the reporting of the 2003 results to the single combined scale of attainment, and not the three distinct sub-scales also used in PISA 2000 ¹⁰: 'retrieving information', 'interpreting texts', and 'reflecting and evaluating texts'. Results are, however, still reported by overall proficiency levels as well as by a scale score.

Appendix C gives a summary of the descriptions of performance expected at each proficiency level, and the full definitions can be found in the international reports on PISA 2000 ¹¹. Descriptions of the lowest and highest proficiency levels are provided below for illustration. Each test item used in PISA 2003 was matched to one of the six proficiency levels and students were then placed at a specific proficiency level depending on how they had answered the set of items allocated to this level. More specifically, a student was placed at a particular proficiency level if he or she could be expected to answer correctly at least 50% of a hypothetical range of items spread evenly across the difficulty range for that level.

At Level 5, the highest level:

"Students proficient at Level 5 on the reading literacy scale are capable of completing sophisticated reading tasks, such as managing information that is difficult to find in unfamiliar texts; showing detailed understanding of such texts and inferring which information in the text is relevant to the task; and being able to evaluate critically and build hypotheses, draw on specialised knowledge, and accommodate concepts that may be contrary to expectations."

At Level 1:

"Students at Level 1 can recognise an author's main theme or purpose in a text about a familiar topic, if this is prominent; they can make connexions between information in a text and common everyday knowledge; and they can locate one or more pieces of explicitly stated information in a text."

The PISA 2003 results are scaled on the same scale as used in PISA 2000. This scale was set to a mean of 500 for the 27 OECD countries that participated in that study and a range such that two-thirds of students scored between 400 and 600. As the Slovak Republic and Turkey joined the OECD in 2003 and The Netherlands met all the technical standards in 2003, but not in 2000, and conversely for the UK, 29 OECD countries are now included in the PISA 2003 results. For these 29 countries, the overall OECD mean for reading literacy is 494, while the range remains unchanged.

Summary of Reading Results for the OECD and Scotland.

Proficiency Levels

Table 3.1. Percentage of students at each level of proficiency on the reading literacy scale

Figure 3.a. Percentage of students proficient at each level of reading literacy

Source : OECD PISA 2003

Proficiency level results for reading literacy in the 29 OECD countries and Scotland are given in Table 3.1 and shown graphically in Figure 3.a.

In the whole OECD area, 8% of students are proficient at Level 5, the highest of the levels. More than 16% of students in New Zealand and more than 12% of the students in Australia, Belgium, Canada, Finland, and Korea are at this level. Nine percent (9%) of Scottish students reach this same level, slightly more than in the OECD as a whole. Less than 1% of students in Mexico reach this level.

If the results for the top three levels, Levels 3, 4 and 5, are combined, Australia, Canada, Finland, and Korea have over 70% of their students attaining these levels. Scotland has 68% of students attaining these levels. Overall, 55% of students in the OECD attain one of these three highest levels.

Taking a wider range, 78% of students in the OECD as a whole performed at Level 2 or above, but there are wide differences between countries. In Finland and Korea, about 94% of students reached these levels, but these are exceptions. In all other OECD countries, between 48 and 91% of students did so. Scotland is at the upper end of this range, with 89% of its students attaining Level 2 or above.

In the combined OECD area, 14% of students perform at Level 1, and 8% below Level 1, but there are wide differences between countries. In Finland and Korea, only around 5% of students perform at Level 1, and about 1% below it, but, again, these countries are exceptions. In all other OECD countries, between 10 and 52% of students perform at or below Level 1. One-third of the OECD countries have between 2 and 6% of students performing below Level 1. In Scotland, 8% of students performed at Level 1 and 3% below this.

Mean Scores in Reading Literacy

The first column of Table 3.2 gives the mean scores for the 29 OECD countries and Scotland, along with mean scores for male students and female students separately.

Table 3.2. Student performance on the reading scale, all students and by gender

Positive differences indicate that males perform better than females
Negative differences indicate that females perform better than males

Only the three top performing OECD countries on the PISA 2003 reading literacy scale (Finland, and Korea and Canada) had mean reading scores that were significantly higher than that of Scotland. Scotland's mean score of 516 is (statistically) significantly above that for the OECD as a whole, and the table below lists the OECD countries whose mean scores are significantly greater or less than that for Scotland.

Table 3.3. OECD countries whose mean scores differ significantly from the Scottish mean.

Changes in Mean Scores in Reading between PISA 2000 and PISA 2003

Figure 3.b. Differences in scores between PISA 2000 and PISA 2003 on the reading scale

Source : OECD PISA 2003

Comparative data is available for 26 of the 27 countries that participated in PISA 2000, the UK being the missing country, and for Scotland. Figure 3.b shows the two mean scores for these 27 countries sequenced from left to right in ascending order of the differences between the two scores. Countries on the left did better in PISA 2000, those on the right did better in PISA 2003.

None of the two mean scores for the 17 OECD countries lying between France and Korea differ significantly. Essentially attainment has not changed in these 17 countries. In two OECD countries, Luxembourg and Poland, mean performance rose significantly by 38 and 17 points respectively. In the other seven OECD countries, mean attainment fell significantly, by between 11 and 24 points.

Attainment in Scotland dropped by 11 points, a drop that is statistically significant, but at the 10% level of significance only. It is standard practice to measure statistical significance at the 5% level and at this level this is not a significant drop. However, as 11 points is one of the larger falls observed, it may be prudent not to ignore it, despite its borderline statistical position. While Scotland maintained its position just above the middle of the scale, average performance did shift closer to the OECD mean.

Scores in Reading for Students at Various Percentile Levels of Attainment

Improving attainment among the bottom 20% of students is an important policy priority for the Scottish Executive Education Department. The following table gives the scores attained by Scottish students at the lower levels of attainment ¹² in PISA 2000 and PISA 2003. It shows the point below which the lowest performing 5%, 10% and 25% of students scored (i.e. the scores that mark off each of these percentiles).

Table 3.4: Scores attained by students at the lower end of the ability scale in reading literacy in PISA 2000 and PISA 2003

Scores for the two lowest groups have improved by approximately 9 scale points, but scores for the third group have not changed appreciably. An improvement of 9 points is not statistically significant.

In the OECD area as a whole, the performance of students at these three levels has fallen, not risen (by approximately 15 scale points at each of the percentiles).

Overall, as just noted, the Scottish mean score in reading literacy fell by 11 points between PISA 2000 and PISA 2003. This overall decline is largely attributable to lower performance by pupils at the upper end of the ability range, those at the 75 ^th, 90 ^th and 95 ^th %-ile, whose scores dropped by between 19 and 34 points. These differences are statistically significant. In the OECD as a whole, scores for these three groups also fell, but for each percentile by approximately 7 points only.

It is therefore reassuring that, although there has been a drop in Scotland's mean score, this drop has not been experienced across all ranges of ability. Given the emphasis on 'closing the gap' and promoting equity of attainment, it is particularly reassuring to note that, despite the trend across the OECD as a whole, the performance of our lowest 25% of students has improved (though not significantly).

Gender Differences in Reading

Figure 3.c. Gender differences in mean score in reading literacy (score difference in favour of males)

In every OECD country the mean reading score for female students is significantly higher than that for male students. The data are given numerically in Table 3.2 and shown graphically in Figure 3.c. This graph plots the difference between mean score for male students and that for females. A bar to the right of the centre line means male students scored higher than female, while one to the left means female students did better than males.

On average, across the OECD, the difference in performance between female and male students is 34 points, equivalent to half a proficiency level. There is, though, considerable variation between countries: from 58 points in Iceland, to 21 in Korea, The Netherlands and Mexico. In Scotland female students outperformed male students with a difference in mean score of 24 points. In PISA 2000, Scottish female students had a 30 point advantage over male students.

Variation in Reading Scores Between Low and High Achievers.

Figure 3.d. Difference in reading literacy scores between students at the 25th and 75th percentile levels of attainment

Figure 3.d shows the score gap between students at the 75 ^th percentile level, ie those at the top end of the attainment range, and those at the 25 ^th percentile level, at the bottom of the attainment range. Each pair of joined points corresponds to one of the 29 OECD countries or to Scotland. Countries are sequenced from left to right in increasing gap size. The smallest gap (105 points) is found in Finland, and the largest (153 points) in Germany.

The graph compares the degree of equity in reading attainment between countries. That is, how great the achievement gap is between those who do well and those who do not. Is this gap narrow or wide? A country may have high overall reading achievement, as indicated its mean score, but may provide a very unequal education to its students, or it may not. The gaps shown in this graph reveal where each country is placed in this respect.

Scotland is well placed in fourth position with a gap of 116 points, just slightly wider than Denmark's.

In PISA 2000, Scotland's score gap (between the 25 ^th and 75 ^th percentiles) in reading was 136 points. This means that between 2000 and 2003, our score gap narrowed by 15%. No other OECD country narrowed its score gap by as much as this.

« Previous | Contents | Next »