The Scottish Government

« Previous | Contents | Next »

Listen

Programme for International Student Assessment (PISA) 2003: Initial Report on Scotland's Performance in Mathematics, Science and Reading

Chapter 4: Student Proficiency in Scientific Literacy

How Scientific Literacy is Defined

As in the case of mathematics, the central concept defining science in PISA is its utility in the lives of all citizens in the early 21 ^st Century, and not the focused requirements of students who are going forward to specialise in a scientific discipline. Understanding scientific data and reasoning are seen as increasingly relevant for effective participation in our present world. Consequently, the PISA items focus on assessing students' ability to: recognise scientific questions; identify what is involved in scientific investigations; relate scientific data to claims and conclusions; and communicate these aspects of science.

How Scientific Literacy was Assessed in PISA 2003

The science assessment comprised 35 items. These were the items used to assess scientific literacy in PISA 2000. No new items were added. The items were substantially as used in previously, although some were slightly modified in the light of experience.

A diverse range of items were employed. They varied in difficulty and covered a range of topics. The hardest items required complex conceptual skills, the less difficult required sound scientific thinking, and the easiest required straightforward recall and use of simple scientific knowledge.

In designing the assessment framework, the specialist science team took into account: scientific knowledge and concepts; scientific processes; and the science-based situations in which these need to be deployed. The framework covered three specific applications of science that raise issues for today's and tomorrow's citizens:

Science in life and health
Science in Earth and environment
Science in technology.

As there were relatively few scientific literacy test items, the framework domains were used only to ensure a due range of coverage in the item pool, and not for reporting purposes, and for the same reason, no proficiency levels were defined. Only scores are reported.

How the Scientific Literacy Results are Reported

The PISA 2003 results are scaled on the identical scale as used in PISA 2000. This scale was set to a mean of 500 for the 27 OECD countries that participated in that study and a range such that two-thirds of students scored between 400 and 600. As the Slovak Republic and Turkey joined the OECD in 2003 and The Netherlands met all the technical standards in 2003, but not in 2000, and conversely in the case of the UK, 29 countries are now included in the PISA 2003 results. For these 29 countries, the overall OECD mean for science literacy is 496 while the range remains unchanged.

Summary of Science Results for the OECD and Scotland

Mean Scores in Scientific Literacy

Table 4.1. Student performance on the science scale, all students and by gender

Positive differences indicate that males perform better than females
Negative differences indicate that females perform better than males

The first column of Table 4.1 gives the mean science scores for each of the 29 OECD countries and Scotland.

The three top performing countries on the scientific literacy scale are Finland, Japan, and Korea. Scotland's score of 514 is significantly above the OECD average.

Extrapolating PISA results to the performance to be expected from all 15 year old students in each country introduces an element of statistical uncertainty into mean scores. Consequently, of the eight OECD countries with higher mean scores than Scotland, only the top three countries just mentioned can be said with certainty to have students who do better, on average, than those in Scotland. The higher mean tests scores in the other five countries may or may not imply better performance by their 15 year-olds. One cannot be certain. Similarly, of the 21 OECD countries with lower mean tests scores, only for 14 countries, those listed below, can it be said with certainty that their 15 year-old students do less well, on average, than Scottish 15 year-old students.

Table 4.2. OECD countries whose mean scores differ significantly from the Scottish mean

Changes in Mean Scores in Science Between PISA 2000 and PISA 2003

As noted, essentially the same item set of 35 items was used in PISA 2003 as in PISA 2000, and the scoring scale was the same.

With relatively few items to cover the whole domain of science literacy, it would be unwise to place too great an emphasis on changes in countries' mean scores between the two surveys. However, the main trends observed are as follows.

Comparative data is available for 26 of the 27 OECD countries that participated in PISA 2000, the UK being the one missing country, and for Scotland. Figure 4.a shows the two mean scores for these 27 countries in ascending order, from left to right, of the difference between the two scores. The PISA 2000 scores are shown by small black squares; the 2003 ones by small open white square. Countries on the left did better in PISA 2000; those on the right did better in PISA 2003.

None of the 11 OECD countries lying between Ireland and Portugal, show a significant change in performance between 2000 and 2003. In the 10 OECD countries to the right of Portugal on the graph, mean performance rose by between 8 points (in Italy) and 40 points (in Luxembourg). In the other five OECD countries, mean attainment fell significantly, by between 10 points (in Canada ¹³) and 30 points (in Austria).

Performance fell in Scotland between PISA 2000 and PISA 2003, but the 8 point drop, from 522 to 514 points, is not statistically significant. Scotland maintained its position just above the middle of the attainment scale.

Scores in Science for Students at Various Percentile Levels of Attainment

The drop in science literacy performance by students in Scotland between PISA 2000 and PISA 2003 was more or less uniform across the whole ability range. Performance by students at the very lowest levels of ability, those at the 5 ^th %-ile and 10 ^th %-ile levels fell by approximately 5 points, while those at the 25 ^th 75 ^th 90 ^th and 95 ^th %-ile levels fell by between 7 and 9 points.

In the OECD as a whole, students at the 5 ^th, 10 ^th, and 25th %-ile levels of the ability range dropped in score by between 12 and 16 points. Performance by those at the 75 ^th level changed very little and performance by those at the 90 ^th and 95 ^th levels improved by 5 and 8 points respectively.

Figure 4.a. Differences in scores between PISA 2000 and PISA 2003 on the science scale

Source : OECD PISA 2003

Table 4.1 also gives the mean scores for male and female students, and the differences between these two are shown graphically in Figure 4.b. This graph plots the difference between mean score for male students and that for females. A bar to the right of the centre line means male students scored higher than female, while one to the left means female students did better than males.

In five OECD countries the mean scores for male and female students are equal to within 1 point. In 21 OECD countries the mean score for male students is higher than that for female students by between 1 and a maximum of 18 points (but the difference is statistically significant in only 11 of these countries - Korea through to Portugal). In three countries female students score higher than male students by between 2 and a maximum of 10 points (the difference is significant in Finland and Iceland). Korea shows the largest difference in favour of males, while Iceland, shows the largest difference in favour of females. Thus, while in general male students did better than female in the OECD at large, this is not universally the case.

Gender Differences in Science Literacy

Figure 4.b. Gender differences in mean score in science literacy (score difference in favour of males)

In Scotland there is a difference of 8 points in favour of male students, but this difference is not statistically significant. In PISA 2000 ¹⁴, the gender difference in Scotland was just 1 point in favour of males. Quite possibly, therefore, Scotland's relatively high position in Figure 4.b may simply reflect the statistical uncertainty inherent in the assessment and not be a cause for concern.

Variation in Science Scores Between Low and High Achievers

Figure 4.3. Difference in science literacy scores between students at the 25th and 75th percentile levels of attainment

Figure 4.c shows the score gap between students at the 75 ^th percentile level, i.e. those towards the top end of the attainment range, and those at the 25 ^th level, i.e. those towards the bottom of the attainment range. Each pair of joined points corresponds to one of the 29 OECD countries or to Scotland. Countries are sequenced from left to right in increasing gap size. The smallest gap, of 115 points is found in Mexico, and the largest, of 157 points, in Germany.

This graph compares the degree of equity in science attainment between countries. That is, how great the achievement gap is between those who do well and those who do not. Is this gap narrow or wide? A country may have high overall science achievement, as indicated by its mean score, but may provide a very unequal education to its students; or it may not.

Scotland stands in the centre of the range, with a gap of 140 points. Scotland's rating on this equity scale is neither particularly good nor particularly bad, compared with the OECD countries. However, both in respect of quality, as measured by the overall mean score, and equity, as measured by the differences above, Scotland does fall well below what Finland shows can be achieved.

In PISA 2000, Scotland's score gap between the 25 ^th and 75 ^th percentiles for science was 142. This means that between 2000 and 2003 we narrowed the gap by only 2 score points. However, only three OECD countries managed to narrow their gap in science between these two sweeps of PISA: Belgium (by 1 score point); Denmark (by 3 score points) and Hungary (by 8 score points). In all other OECD countries the gap increased.

« Previous | Contents | Next »