« Previous | Contents | Next »
Listen
Programme for International Student Assessment (PISA) 2003: Initial Report on Scotland's Performance in Mathematics, Science and Reading
Appendix A
Results for the United Kingdom.
The England PISA 2003 sample fell short of the pre-agreed school and student level response rates. According to the technical standards established by OECD Member countries for PISA, this meant that for England (and therefore the United Kingdom) data to be fully included in the PISA report it was necessary to demonstrate that the England sample was free from bias.
Analysis of the England sample detected significant bias at student level. OECD analysts have concluded that at present the uncertainties surrounding the England sample are such that it is not possible to make reliable comparisons between the performance of the United Kingdom and that of other countries.
Annex 3 of the PISA 2003 international report contains the following note explaining how the United Kingdom did not meet in full the PISA technical standards and how subsequent bias analysis found that although there was no evidence of significant bias at the school level there was potential bias at the student level.
Extract from Annex 3 of the international report
In order to ensure that PISA yields reliable and internationally comparable data, OECD Member countries have established a process for the validation of all national data submissions. As the basis for this process, PISA established technical standards for the quality of datasets which countries must meet in order to be reported in OECD publications. These standards are described in detail in the PISA 2003 Technical Report. One of the requirements is that initial response rates should be 85% at the school level and 80% at the student level.
The United Kingdom fell significantly short of these standards, with an achieved initial response rate of 63% at the school level and 78% at the student level. The Technical Standards include an approved procedure through which countries with an initial school-level response rate of at least 65% could improve response rates through the use of designated replacement schools. For the United Kingdom, a school-level response rate of 95% was required but only 77% was achieved.
The results of a subsequent bias analysis provided no evidence for any significant bias of school-level performance results but did suggest that there was potential non-response bias at student levels. The PISA Consortium concluded that it was not possible to assess reliably the magnitude, or even the direction, of this non-response bias and to correct for this. As a result, it is not possible to say with confidence that the United Kingdom's sample results reliably reflect those for the national population, with the level of accuracy required by PISA. The mean performance of the responding sample of United Kingdom students was 508, 507 and 518 in mathematics, reading and science respectively. In the mathematics subscales the mean performance was 496 on the space and shape scale, 513 on the change and relationships scale, 520 on the uncertainty scale and 499 on the quantity scale. If negligible to moderate levels of bias are assumed, the United Kingdom mean performance would lie between 492 and 524 on the mathematical literacy scale, between 491 and 523 on the reading literacy scale, and between 502 and 534 on scientific literacy scale. The uncertainties surrounding the sample and its bias are such that scores for the United Kingdom cannot reliably be compared with those of other countries. They can also not be compared with the performance scores for the United Kingdom from PISA 2000.
The results are, however, accurate for within-country comparisons between subgroups (e.g. males and females) and for relational analyses. The results for the United Kingdom have, therefore, been included in a separate category below the results for the other participating countries. (And) In the main body of the report, where countries significantly above the OECD average in a given indicator are mentioned, the United Kingdom will be included where relevant.
It should be noted that Scotland and Northern Ireland carried out an independent sample that met the PISA technical standards. Results for Scotland and Northern Ireland are reported in Annex B2 and are fully comparable with results from other OECD countries and with results from PISA 2000.
The material not in the OECD main report will be set out in a table and will appear on the OECD PISA website, to enable researchers to reproduce the international means and models.
« Previous | Contents | Next »