Progress in International Reading Literacy Study (PIRLS) 2006: Highlights from Scotland's results

Listen

Introduction

PIRLS is an important way of finding out how children's reading in Scotland compares to the rest of the world. It measures the reading literacy achievement of nine and ten year olds and gathers information about home and school factors associated with learning to read. PIRLS is run by the International Association for the Evaluation of Educational Achievement ( IEA) every five years. Scotland also took part in the first cycle of the study in 2001.

A total of 40 countries and five Canadian provinces took part in the study in 2006. Around half of the countries were within the OECD, with the remainder being non- OECD, including a number classified as 'developing' 1.

The Scottish fieldwork was undertaken by the National Foundation for Educational Research ( NFER). The assessments were administered in Spring 2006 and contained a mixture of multiple choice and constructed response items (requiring pupils to generate and write their answers). One hundred and thirty two Scottish primary schools agreed to take part in the survey. One P5 class within each school was then randomly selected, from which a total of 3,775 pupils took part.

This paper highlights the main Scottish results, sets them in the international context and, where relevant, looks at results of key interest from other countries.

What does the study assess?

PIRLS assesses the reading performance of pupils at 4 th Grade (generally nine and ten year olds), so the sample participating in the survey is chosen based on stage rather than on age. This stage is chosen because it is the time in most pupils' lives when they have learned to read and are beginning to use reading to learn.

As Scottish pupils begin school earlier than those of many other countries, it was agreed with the IEA that P5 pupils should be tested in Scotland to ensure that subjects were appropriate to their age (this also applies to England and New Zealand). The average age at the time of testing in Scotland was 9.9 years. The average age across all countries ranged from 9.7 years in Italy to 11.9 in South Africa, putting Scottish pupils at the lower end of the age range.

The assessments and questionnaires used in the study were developed by an international consortium and approved by all participating countries. In addition, participants needed to meet stringent criteria (including rigorous sampling requirements, quality controlled survey procedures, multiple marking of tests, and rigorous data-cleaning) to ensure that the results were comparable between countries.

PIRLS adopts the following definition of reading literacy:

'The ability to understand and use those written language forms required by society and/or valued by the individual. Young readers can construct meaning from a variety of texts. They read to learn, to participate in communities of readers, and for enjoyment'.

To reflect this broad range of literacy requirements, the PIRLS assessment includes two different types of reading passage - literary and informational.

  • Literary texts comprise short stories with one or two episodes of problem/ resolution and two central characters
  • Informational texts include sets of short informational materials involving texts, maps, illustrations, diagrams and photographs

Results for PIRLS 2006 are presented as mean scores with the international mean reading literacy score set to 500. Results are also split by gender, achievement by reading purpose and achievement against international benchmarks (see below). Where a difference is referred to as 'significant' this is at the 95% confidence level.

PIRLS also collects background information from pupils, teachers and parents which can be examined to assess the effects of home, school and national influences on learning to read. Each pupils completed a questionnaire about their home and school experiences related to reading. Teachers and head-teachers completed questionnaires about instructional practices. Parents were also asked about activities undertaken to encourage early reading activities.

High response rates (over 80%) were achieved for the pupil, teacher and head-teacher questionnaires. In contrast, only 52% of parents responded to the 'Learning to Read' questionnaire (sent home with pupils for completion). Due to the strong likelihood of bias in the achieved sample of parents, results from the 'Learning to Read' questionnaire are not included in this initial report.

How well do Scottish pupils read?

Scotland's mean overall score of 527 was significantly above the international average (500). Mean scores in the OECD countries ranged from 498 (in Norway) to 557 (in Luxembourg). None of the OECD countries scored significantly below the international average.

Pupils in 19 countries or regions had a mean score that was significantly higher than Scotland's, (including England, Italy, Germany, Denmark, the Netherlands, Belgium (Flemish) and the United States). However, Scottish pupils scored more highly than those in 17 other countries or regions (including Poland, Spain, Iceland, Belgium (French) and Norway) and on a par with four others (New Zealand, France, Slovak Republic and Slovenia). The three highest achieving countries (the Russian Federation, Hong Kong and Singapore) had scores significantly higher than all other participating countries.

Figure 1: Scotland's PIRLS 2006 results in the international context

image of Figure 1: Scotland's PIRLS 2006 results in the international context

*Countries within the OECD
+
This represents the 5 Canadian Provinces of: Alberta; British Columbia; Nova Scotia; Ontario; and Quebec.
Shaded countries had scores significantly below the international average.

Reading for different purposes

As in 2001, Scottish pupils' average scores were very similar for both literary and informational reading purposes.

In nine of the countries and regions (including Scotland, England, the Netherlands, Austria, Italy, Luxembourg and Belgium (French)) pupils performed to a similar level for both reading purposes. The other countries and regions were divided roughly equally into those which scored significantly more highly in reading for informational purposes, and those which scored more highly in reading for literary purposes.

PIRLS also looked at differences in performance between the different reading processes of 'retrieving and straightforward inferencing' and 'interpreting, integrating and evaluating'. Scottish pupils performed, on average, significantly better on the second of these processes. The same result was apparent in most other English-speaking countries.

The 'achievement gap'

Of countries within the OECD in PIRLS, Scotland had the third widest gap between high and low achievers.

This means that our most able pupils rank amongst the highest achievers in the OECD in PIRLS, whilst those with poor results rank amongst the lowest. Figure 2 compares the distribution between high and low achievers across countries and regions within the OECD. It shows that Scotland, along with Poland, had the third highest range between the 25 th and 75 th percentiles, of 102 scale-score points. A similar distribution is found in Canada (Nova Scotia) and the United States. England and New Zealand had the widest gap between high and low achievers. Most countries or regions within the OECD had a narrower distribution. In particular, the Netherlands and Belgium (Fl) had a much narrower gap between the highest and lowest achievers. In terms of the top three scoring countries overall (all non- OECD), Hong Kong SAR had the lowest gap between the 25 th and 75 th percentiles, of 77 scale-score points, followed by the Russian Federation (89 points) and Singapore (101 points).

Benchmark scores provide another way of looking at this issue (see below). Scotland has a significantly larger proportion of pupils at the highest level of performance (10% reached the advanced benchmark) than the international average (7%). However, we have around the average proportion of pupils at the lowest level of performance (7% of pupils failed to reach the lower quartile benchmark score compared with an international average of 6%). In general, those countries which performed best in PIRLS overall had the lowest proportion of pupils failing to reach the lowest benchmark (18 of the top 20 performing countries had 4% or less of their pupils failing to reach the lowest benchmark). 'Developing' countries had the most pupils failing to reach the lowest benchmark score, ranging from 16% in Romania, to 78% in South Africa.

Benchmark Scores

To make it easier to interpret the results of the PIRLS tests, international benchmarks were developed (using a 'scale anchoring' method) to describe pupils' reading literacy skills related to different scores on the assessment. The benchmark descriptions are based on percentile performance. All pupils are ranked according to their scores so, for instance, those above the 90 th percentile scored higher than 90% of the international sample so are in the top 10% of performance internationally, and those at the 50 th percentile represent mid-range performance. Here is an example of the top 10% benchmark description for literary reading:

Given short stories with one or two episodes of problem/resolution and two central characters, pupils can:

  • Integrate ideas across the text to provide interpretations of a character's traits, intentions and feelings, and give text-based support
  • Integrate ideas across the text to explain the broader significance or theme of the story

Benchmarks scores in 2006 were set at: 'advanced' (625 points); 'high' (550 points); 'intermediate' (475 points); and 'low' (400 points).

Figure 2: Distribution of Reading Achievement (countries and regions within the OECD only, gap between 25 th and 75 th percentiles in parenthesis)

image of Figure 2: Distribution of Reading Achievement (countries and regions within the OECD only, gap between 25th and 75th percentiles in parenthesis)

How does this compare with 2001?

Scotland's results have remained very stable when compared with results from the 2001 survey, but a number of other countries have seen a significant change in their performance.

In particular, some previously equal-ranking countries have significantly improved their performance, and some new, higher performing countries have joined the survey. In 2001, only 12 countries had a mean score significantly higher than Scotland, compared with 19 in 2006. However, five of these 19 countries were new to the study in 2006. The three top scoring countries in 2006 (Russian Federation, Hong Kong SAR and Singapore) were ranked equally to Scotland in 2001 but have since overtaken all other countries in the survey. The remaining 11 countries were already above Scotland in 2001, but their relative positions have changed. In particular, England, Sweden, Netherlands and Lithuania have all experienced a significant drop in their scores, whereas Italy, Germany and Hungary have all recorded significant improvements. Other countries whose performance has remained stable ( i.e. no statistically significant increase or decrease) include: Canada (Ontario and Quebec); New Zealand; Norway; Iceland; United States; and France.

Figure 3: Trends in Reading Achievement 2001-2006 (countries which took part in 2001 and 2006 only)

image of Figure 3: Trends in Reading Achievement 2001-2006 (countries which took part in 2001 and 2006 only)

Shaded=significant change, un-shaded=no significant change

In terms of the achievement gap, there has been no significant change between 2001 and 2006 in the proportions of Scottish pupils reaching the advanced, high, intermediate and low benchmark levels. Interestingly, those countries which have made the biggest improvements overall have tended to increase the proportion of pupils reaching all four benchmarks ( e.g. Hong Kong, Singapore and Slovenia, although the Russian Federation only recorded significant increases in the top three). Whereas countries with significant drops in performance ( e.g. England, the Netherlands and Sweden) have seen falls in the top two or three groups with no significant change at the lowest benchmark.

Scottish pupils' relative performance in reading for the different purposes (literary and informational) is also unchanged since 2001. However, the scores of pupils in England, Sweden and the Netherlands have fallen significantly in reading for literary purposes (by 20, 13 and 8 points respectively since 2001). Scores in reading for informational purposes have also fallen significantly in Sweden, Lithuania and England (by 10, 10 and 9 points respectively).

Is there a difference between boys and girls?

The achievement gap in reading between boys and girls in Scotland was significantly higher than the international average, but the gap has not widened since 2001.

Across the world girls tended to have higher achievement in reading than boys, and for each of the two reading purposes. The same was true in Scotland (a 22 point gap between girls and boys compared with the international average of 17 points). The only exceptions to the rule were Luxembourg and Spain, where there was no significant difference in performance between boys and girls. A further eight countries/ regions (Belgium (French and Flemish)), Hungary, Italy, the Netherlands, Germany, Canada (Alberta), and Austria) had a significantly smaller gender gap than Scotland. The biggest gender gaps were found in the non- OECD countries of Qatar and Kuwait. The gap between boys and girls has not significantly widened in any individual country since 2001.

Overall, the international average scale score for boys has increased significantly (by 5 points) since 2001, whilst scores for girls have remained stable.

There is no obvious relationship between the size of the gender gap and overall performance in PIRLS. For instance, Luxembourg and Spain, which had no apparent gender gap, ranked 4 th and 26 th respectively in the overall scoring. Whereas Singapore, which had the average gender gap of 17 points, had the third highest overall achievement score. Nevertheless, eight out of the top 10 scoring countries overall did have a gender gap which was lower than the international average.

Figure 4: Scotland's gender gap in reading in the international context

Significantly smaller gender
gap than Scotland
(10 countries/ regions)

Gender gap not significantly different
from Scotland 2006
(32 countries/ regions)

Significantly bigger gender gap
than Scotland 2006
(2 countries)

Luxembourg
Spain
Belgium (Fr)
Hungary
Belgium (Fl)
Italy
Netherlands
Germany
Canada (Alb)
Austria

Canada (Brit Col)
United States
Hong Kong SAR
France
Slovak Republic
Canada, Quebec
Canada, Ontario
Chinese Taipei
Denmark
Moldova, Rep of
Iran, Islam Rep of

Romania
Israel
Russian Fed
Singapore
Poland
Georgia
Morocco
Sweden
Lithuania
Iceland
Norway

England
Slovenia
Indonesia
Bulgaria
Canada, Nova S
Macedonia
Latvia
New Zealand
Trinidad & Tob
South Africa

Qatar
Kuwait

The impact of socio-economic status

In Scotland, as elsewhere, the average performance of pupils in schools which had relatively high levels of pupils coming from economically disadvantaged homes was lower than schools which had relatively few pupils coming from deprived homes.

PIRLS 2006 shows a clear relationship between head-teachers' reports of the proportion of their primary pupils coming from economically disadvantaged homes and average achievement. The average score of Scottish pupils in classes in schools where 0-10% of pupils were reported to come from disadvantaged homes was 547, compared with 497 amongst pupils in schools where more than 50% of pupils were reported to come from disadvantaged homes.

As this data comes from head-teachers' responses on the proportion of pupils in their schools from economically disadvantaged backgrounds, it is only indicative of a relationship between deprivation and achievement. For instance, their opinion will be based on relative deprivation within the country and not standardised across countries. However, the pattern in Scotland was repeated in most other countries, to a greater or lesser extent (see Figure 5). In respect of those countries with PIRLS scores on or above the international average, Israel, Germany and New Zealand had the most marked difference (80 points or more) between the scores of pupils attending schools where a high proportion of pupils came from deprived homes and those where a low proportion were from deprived homes. The gap between the pupils attending schools with the highest and lowest proportions of pupils from deprived backgrounds in Scotland was around the middle of the range, at 50 points, similar to France, the Russian Federation, Austria, Canada (Alberta), and the Netherlands. The countries with the least obvious relationship ( i.e. less than 20 points difference between schools with pupils from the most and least deprived backgrounds) were Italy, Poland, Moldova and Slovenia. However, given the subjective nature of the data these findings should be treated with caution.

Figure 5: Relationship between deprivation and achievement (selected countries)

image of Figure 5: Relationship between deprivation and achievement (selected countries)

What do pupils think about reading?

Scottish pupils are becoming less enthusiastic about reading and less confident in their reading ability.

Generally, pupils with the most positive attitudes towards reading had the highest reading achievement. In Scotland in 2006, the proportion of pupils who enjoyed reading and expressed an appreciation of books 2 was 42%. This was significantly lower than the international average (49%) and a significant fall from 2001 when 47% of Scottish pupils expressed positive attitudes towards reading.

Other countries where positive attitudes towards reading are lower than average, and falling, include: England; the Netherlands; Sweden; Canada (Ontario); Iceland; Latvia; and Singapore. The first three of these countries have recorded significant drops in reading performance between surveys, and the next three are stable (see Figure 3). However, Singapore's performance has improved dramatically between 2001 and 2006 despite less positive pupil attitudes.

There has also been a significant increase between surveys in the proportion of Scottish pupils who had negative perceptions of their reading ability 3. About half (49%) of pupils, on average had high perceptions of their reading ability. In Scotland, the proportion was lower, at 43% (unchanged from 2001). However, in 2006 6% of Scottish pupils fell into the category with negative perceptions (a significant increase of two percentage points from 2001). Only England had a higher percentage of pupils in this category (7%), and the international average was 3%.

There has also been a significant fall in the percentage of Scottish pupils who report reading stories or novels outside of school every day or almost everyday. In 2006 35% of Scottish pupils reported reading this frequently outside of school, compared with 40% in 2001. Significant falls were also recorded in 11 other countries, including England, the Netherlands, Sweden (whose performance in PIRLS fell between 2001 and 2006), but also the Russian Federation, Singapore, Slovenia (whose performance had significantly improved between surveys). Nine countries/ regions recorded a significant increase in this measure, including Hong Kong, Italy, Germany and Hungary.

Similarly, there has been a significant increase (from 8% to 11%) between surveys in the percentage of Scottish pupils who report never, or almost never, reading for information outside of school. This was apparent in 19 other countries, including England, the Netherlands, and Lithuania (whose PIRLS scores had fallen), but also Singapore, Slovenia, Hong Kong and Italy (whose performance in PIRLS had improved).

Teachers and schools

How much time is spent teaching reading?

Scottish teachers estimated that 27% of their teaching time was spent on language work (30% internationally) and 16% on reading (20% internationally). They spent an average of two and a half hours per week on formal reading instruction, which was the same as the international average. Almost half of Scottish pupils (44%) were in classes where teachers said they taught reading on a daily basis (compared with 56% internationally).

However, there was no obvious relationship between frequency of reading teaching and achievement in PIRLS. A more important determinant of achievement is how often pupils read independently. As figure 6 shows, there is a positive association between the frequency of opportunities to read independently and reading achievement. Scottish pupils were more likely than pupils internationally to be given daily opportunities to read independently.

Figure 6: Pupil reports of independent reading, Scotland and internationally

Every day or almost every day

Once or twice a week

Once or twice a month

Never or almost never

% of pupils

Average achievement

% of pupils

Average achievement

% of pupils

Average achievement

% of pupils

Average achievement

Scotland

71

534

24

523

3

504

2

~

International

65

508

27

495

5

483

3

462

~ Insufficient data to report achievement

Classroom organisation and teaching materials

Teachers in Scotland were likely to report spending more of their time working with individual pupils or small groups (40%) than the international average (24%); and less likely to report spending time teaching the class as a whole (44% compared with 57% internationally). Over half of Scottish pupils (54%) were in classes where the teacher reported always or almost always creating same-ability groups for reading compared with the international average of 8%. Only 1% of pupils were in classes where teachers reported always or almost always creating mixed-ability groups (compared with 7% internationally). There is no obvious relationship between the use of pupil groupings and achievement.

In Scotland, 89% of pupils attended schools that used reading series as the basis of their reading programmes. This was significantly higher than England which matched the international average of 42%. Only New Zealand had a higher proportion of pupils in this category than Scotland (93%).

The use of text books in teaching reading was relatively uncommon in Scotland compared with internationally. Only four countries reported less use of text books in reading than us: New Zealand; England; Belgium (Fr); and Canada (except Quebec). The use of a variety of children's books was more common in Scotland than internationally. Only 4 countries (England, Canada, France and Sweden) reported significantly higher use of these materials.

Assessment

PIRLS 2006 asked teachers about the range of options which they use to monitor pupils' progress in reading (see figure 7). In Scotland, 89% of pupils were in classes where teachers put major emphasis on their professional opinion (compared with 68% internationally), and only 15% were in classes where major emphasis was placed on classroom tests - a significant drop from 25% in 2001 (and compared with 53% internationally).

Figure 7: Trends in emphasis on sources to monitor reading progress in Scotland 2001-2006, International (2006 only)

image of Figure 7: Trends in emphasis on sources to monitor reading progress in Scotland 2001-2006, International (2006 only)

There has been a very significant (27 percentage point) drop in the percentage of Scottish pupils in classes where teachers reported putting major emphasis

on national or regional achievement tests. As in 2001, a relatively low percentage of Scottish pupils (12%) were in classes where teachers reported using multiple-choice questions at least weekly, compared with an international average of 42%. Much more popular methods of assessment were: listening to pupils reading aloud (86%); oral questioning or summary reports (94%); and constructed response questions (50%).

A new question in 2006 asked teachers to identify the various uses to which they put classroom assessment in reading (see figure 8). In Scotland, the highest proportions of pupils were in classes where their teachers reported using assessment: to adapt instruction (98%; to inform parents of pupil progress (97%); and to identify pupils needing remedial instruction (100%). These three uses were higher in Scotland than internationally (where they were 91%, 92% and 91% respectively).

Just over a third (34%) of Scottish pupils were in classes where teachers reported using assessment to 'assign marks or grades' compared with an international average of 72%. Only two other countries surveyed (Norway and Denmark) had a significantly lower proportion of pupils in this group.

Figure 8: Teachers' reports on uses of classroom assessment in reading, Scotland and International

Percentage of pupils whose teachers reported using assessment:

To assign marks or grades

To adapt instruction

To inform parents of pupil progress

To identify need for additional support

To group pupils for instruction

To provide data for monitoring

Scotland

34

98

97

100

95

56

International

72

91

92

91

66

36

Data: Teachers' questionnaire

How will the results of PIRLS be used?

Combined with other national and international studies of pupil achievement, these results will help to inform and monitor developments in the curriculum and improving learning and teaching in reading in Scotland. Benchmarking Scotland's performance internationally is also important. Comparing Scotland's education system and performance with other countries gives us an insight into our strengths and weaknesses in an international context. This helps us to identify areas where we need to focus our resources and acknowledge and celebrate our successes, as well as improve our performance.

Want to know more?

PIRLS

For more information about the PIRLS 2006 study in Scotland, please contact Fiona Fraser (tel: 0131-244-5310; e-mail: Fiona.fraser@scotland.gsi.gov.uk). For more information about the PIRLS 2006 International Report please go to the International Study Centre website: http://isc.bc.edu

Other international studies

If you would like to learn more about international studies, that include Scotland, please visit, www.scotland.gov.uk/Topics/Education/Schools/Excellence/IE

The Scottish Survey of Achievement

The Scottish Survey of Achievement ( SSA) uses a sample survey to find out how well pupils are learning in Scotland as a whole. The information is used to help plan for improvement to support quality learning and teaching. For more information about the SSA and results, please visit Learning and Teaching Scotland's assessment website, www.ltscotland.org.uk/assess/of/ssa

HMIE

HM Inspectors of Education ( HMIE) promote sustainable improvements in standards, quality and achievements for all learners in Scottish education through independent evaluation. If you would like information about inspections of Scottish schools, or are interested in knowing more about good practice in Scottish education please visit, www.hmie.gov.uk

Your child's progress and achievements

If you would like to know more about how your own child is progressing, or you have concerns about their learning, you should get in touch with the school and talk to your child's teachers.

You will find more information about education in Scotland, and advice on supporting your child's learning on the Parentzone website, www.parentzonescotland.gov.uk

If you have any enquiries about this web only education research report please contact:

Dissemination Officer
The Scottish Government
Education Information and Analytical Services
1 B South
Victoria Quay
Edinburgh
EH6 6QQ

Tel: 0131 244-0894
Fax: 0131 244-5581
Email: recs.admin@scotland.gsi.gov.uk
Website: www.scotland.gov.uk/insight