27 November 2009

	Home | About | News | Publications | Consultations | Search | Links | Contacts | Site Map | FAQs | Help

< Previous | Contents | Next >

Programme for International Student Assessment: Scottish Report

Introduction

The Programme for International Student Assessment (PISA) was set up by the OECD to assess the performance of 15 year olds in reading literacy, mathematical literacy and scientific literacy in its member countries. The results are intended to contribute school outcome measures for the OECD educational indicators programme. PISA is a three phase study with the first phase in 2000, the second in 2003 and the third in 2006. In 2000 the main domain assessed was reading literacy and mathematical literacy and scientific literacy were subsidiary domains. This means there were more reading literacy test items than mathematical literacy or scientific literacy ones. In 2003 mathematical literacy will be the main domain and in 2006 scientific literacy will be the main domain. The domains covered by PISA are defined in terms of:

• The content or structure of knowledge that students need to acquire.
• The processes that need to be performed.
• The contexts in which knowledge and skills are applied.

PISA looked at young people's ability to use their knowledge and skills in order to meet real-life challenges rather than how well they had mastered a specific school curriculum. For instance, PISA defines reading literacy as the ability to understand, use and reflect on written texts to participate effectively in life.

PISA provides a broad assessment of comparative learning outcomes towards the end of compulsory schooling. This can guide policy decisions and provide insights into the factors that contribute to the development of knowledge and skills, and the extent to which these factors are common to different countries. In the 2003 and 2006 studies a proportion of the test items from the previous study will be retained to provide a measure of performance change over time.

Design of PISA

In 2000 over a quarter of a million students took part in PISA from the 32 participating countries (one OECD country's results are not included in this report and four non-OECD countries participated). A further 13 countries will undertake PISA in 2002. PISA used pencil and paper assessments, lasting two hours for each student, including multiple choice questions and questions requiring students to construct their answers. Each student's assessment tasks were drawn from a total of seven hours of assessment items with different students taking different combinations of these items. Students also completed a questionnaire that took about 30 minutes and head teachers completed a questionnaire about their schools. In Scotland the testing sessions were supervised by the schools whereas in most other countries, including England and Northern Ireland, external administrators were used.

In Scotland our intended school sample was 120 and we received completed tests and questionnaires from 99 schools, an 82% response rate, which was within the PISA sampling limit. The sample of schools was drawn as a stratified, random sample representative of all secondary schools in Scotland. In total over 2500 students in the sample schools completed tests and questionnaires, the sample drawn randomly from students with a date of birth during a specified period.

Performance of Students

Student Proficiency in Reading Literacy

Reading literacy was measured on three scales - on retrieving information, on interpreting and on reflecting and evaluating. The scores on each scale represent degrees of proficiency in particular aspects of reading literacy. Each of the three reading literacy scales is divided into six levels of knowledge and skills. Level 5 corresponds to a score of more than 625, level 4 to scores in the range 553 to 625, level 3 to scores from 481 to 552, level 2 to scores from 408 to 480, level 1 to scores from 335 to 407. Students at a particular level not only demonstrate the knowledge and skills associated with that level but also the proficiencies required at lower levels. Students scoring below 335 points are not able to show the most basic skills that PISA sought to measure. Such performance should not be interpreted to mean those students have no literacy skills at all, but they could be said to have serious deficiencies in their ability to use reading literacy as a tool for the acquisition of knowledge and skills in other areas.

Students performing at the highest PISA proficiency level are likely to enhance their country's pool of talent. In the combined OECD area just under 10% of students were proficient at this level, level 5. More than 15% were proficient at this level in Australia, Canada, Finland, New Zealand, and the UK but only 5% or fewer were at this level in Brazil, Greece, Latvia, Luxembourg, Mexico, Portugal, the Russian Federation and Spain. In terms of countries Scotland was ranked seventh in terms of the percentage of students operating at this level.

The percentage of students operating below level 1 throughout the OECD was just over 6%. In Scotland the percentage was 3.3% and we were ranked joint seventh lowest of the countries in terms of this percentage. The three countries with the greatest percentages of students below level 1 were Brazil, Mexico and Luxembourg.

Table 1
Percentage of students at each level of proficiency on the combined reading literacy scale

* Standard Errors

The statistics in this report represent estimates of performance based on samples of students rather than on full populations. Each estimate has an associated degree of uncertainty which is expressed through a standard error. This provides a confidence interval with which to make inferences about population statistics. It can be inferred that the observed result would lie within the confidence interval in 95 out of 100 replications.

Table 2
Mean scores in reading literacy *

* As can be seen from Table 2, the precision of the results for Scotland, expressed by means of standard errors, is somewhat less than for a number of other countries, reflecting the relatively small sample size in Scotland. This, together with the difference in the arrangements for administering the tests means that Scotland's specific rankings need to be interpreted with caution. Small differences in scores should not be regarded as significant.

The top three performing countries on the combined reading literacy score were Finland, Canada and New Zealand. The mean combined score for reading for the UK shows that it is ranked sixth of the 31 countries included in Table 1 and was significantly above the mean for all OECD countries. The UK performance varied across the three sub domains assessed. In retrieving information the UK was ranked eighth, in interpreting it was ranked tenth and in reflecting and evaluating it was ranked second. Scotland was ranked sixth in terms of score on the combined reading scale and our score (526) was significantly above the OECD average.

In every country involved in PISA female students scored higher than male students on the combined reading literacy scale. The differences ranged from 14 points in the Republic of Korea to 53 points in Latvia. The score difference in Scotland was 30 points.

Student Proficiency in Mathematical Literacy

The test items used to assess mathematical literacy had a wide range of difficulty. The hardest items required creative mathematical thinking and insight, the less difficult ones required students to bring together and process information and the easiest ones required only a single processing step in a familiar context.

Due to the relatively small number of mathematical literacy test items mathematical literacy was not broken down into sub domains.

The three top performing countries on the mathematical literacy scale were Japan, the Republic of Korea and New Zealand. The UK was ranked joint seventh in mathematical literacy and the UK scored above the OECD mean score. Scotland's score (533) was ranked joint fifth.

In only three countries did female students score higher than male students - Iceland, New Zealand and the Russian Federation - but the differences were not significant. In all other countries male students scored higher than female students, the greatest differences (27 score points) being in the Republic of Korea, Austria and Brazil. In Scotland the score difference was 5 points in favour of male students.

Table 3
Mean scores in mathematical literacy

Student Proficiency in Scientific Literacy

The scientific literacy test items varied widely in terms of their difficulty. The hardest items required complex conceptual skills, the less hard ones required sound scientific thinking and the easier ones required recall and use of simple scientific knowledge.

Due to the fact there were relatively few scientific literacy test items no sub domain scores are available.

The three top performing countries on the scientific literacy scale were the Republic of Korea, Japan and Finland. The UK was ranked fourth on scientific literacy and scored significantly above the OECD average. Scotland's score (522) was ranked ninth. It is expected that the science strategy and the revised 5-14 curriculum guidelines developed in Scotland will help to redress this relative shortfall in student performance.

In 12 countries male students scored higher than female students by between one and 19 points. In two countries male and female students scored the same and in 17 countries female students scored higher than male students by between 2 and 23 points. In Scotland the score difference was 1 point in favour of male students.

Table 4
Mean scores in scientific literacy

< Previous | Contents | Next >

	Home | About | News | Publications | Consultations | Search | Links | Contacts | Site Map | FAQs | Help