The Scottish Government

« Previous | Contents | Next »

Listen

C: Science literacy results

C.1 Assessing and reporting Science literacy

For the first time, the 2007 SSA attempted an exploratory formal assessment of pupils' skills in Science literacy. This was piloted with a view towards the Curriculum for Excellence aim of increasing the recognition of literacy across all school subjects and in line with the commitment in the Science Strategy for Scotland to 'promote the awareness, appreciation and understanding of Science across society'. The approach, which comprised pencil and paper testing and practical assessments, was based on the following Nuffield ¹ definition of Science literacy:

• appreciate and understand the impact of science and technology on everyday life
• take informed personal decisions about things that involve science, such as health, diet, use of energy resources
• read and understand the essential points about matters that involve science
• reflect critically on the information included in and (often more important) omitted from such reports
• take part confidently in discussions with others about issues involving science.

The items contained in the Science literacy written test booklets can be mapped to the first four statements of the Nuffield definition (the fifth is covered by the practical assessment of Science literacy). Three tests were newly developed at each of Levels A to F. These took the general form of reading tests (source material plus test questions), but here the source material focused on some aspect of Science, and could involve graphical information as well as text. In this approach to assessing Science literacy, no prior knowledge of the particular scientific issues was required (the written assessment of pupils' Science knowledge and understanding is discussed in Chapter B). The tests were designed to use the scientifically based source material to assess 'scientific thinking' within the spirit of the Nuffield definition. Some scientific knowledge may help pupils to answer the questions but a lack of specific knowledge should not be a barrier. Full details of the assessment materials and procedures are available in Annex II.

At P3, pupils were assessed at two levels, the expected level and the one above, i.e. Levels A and B. At P5, where there is no expected level, assessment focused on Level B (the expected level for P4), Level C (the expected level for P6) and Level D (the expected level for P7). At P7, the levels assessed were C, D and E, while at S2 they were D, E and F. At every stage, individual pupils were randomly allocated two tasks, at two different levels: Levels A and B at P3; Levels B and C, or C and D, or B and D at P5; etc.

The results of the survey for Science literacy are reported using the following criteria for test performance:

• 'very good skills' at a level, indicated by 80 per cent or more of the questions at that level answered correctly;
• 'well-established skills', at least 65 per cent of questions answered correctly but fewer than 80 per cent;
• and 'made a good start', at least 50 per cent of the questions answered correctly but fewer than 65 per cent.

The results should be considered in the context of the pilot nature of the Science literacy assessment. The figures are based on a small number of tests, and therefore of topic variety and the results should therefore be treated as indicative rather than as definitive national estimates of pupils' levels in Science. However, their importance to Science educators should not be discounted and it is hoped that the results will be used to provoke professional discussion and debate. They should also be used to give context to, and provoke reflection on, the experiences of those teaching Science. Further details of the pilot assessment process are available in Annex II.

In total, just over 4,700 pupils across the stages participated in the Science literacy assessment. All estimates of achievement shown in the following sections are adjusted for the over-representation of opted-in authorities in the national sample, and also for sample imbalance in terms of gender and deprivation. As with all sample surveys there is a level of uncertainty inherent in the results. Where appropriate, standard errors are quoted alongside each achievement estimate and the size of the standard errors should be taken into account when interpreting the data. Further details of the sample design, standard errors, etc, are available in Annex I.

C.2 National Science literacy achievement estimates

Chart C1 shows the percentage of pupils in each of P3, P5, P7 and S2 categorised into achievement bands at Levels A to F for Science literacy. Detailed results are available in Table C1.

Chart C1
National Science literacy achievement estimates at P3, P5, P7 and S2
(% pupils classified into achievement bands at the given level: 750-850 pupils per stage)

*expected level for the stage

** there is no expected level for P5; Level B is expected at P4, Level C is expected at P6

The chart shows that over 80 per cent of pupils at P3 demonstrated well-established or better Science literacy skills at the level expected of them (Level A) and over 45 per cent of P5 pupils had already reached Level C (the expected level for P6). However, achievement levels are lower in the later stages. At P7, just over 25 per cent of pupils demonstrated well-established or better skills at Level D and in S2 around 40 per cent of pupils did the same at Level E.

Considering the levels below those expected for the stage, the proportion of pupils who demonstrated well-established or better skills also decreased through the stages, from almost 90 per cent at P5 Level B to just over 45 per cent at S2 Level D. On the other hand, many pupils were working at levels above those expected for their stages. For example, over 60 per cent of P3 pupils were already showing well-established skills at Level B, and in S2 over 30 per cent of pupils were already doing the same at Level F.

C.3 National Science literacy achievement estimates, by gender

Chart C2 compares the achievement of boys and girls within the survey.

Chart C2
National Science literacy achievement estimates, by gender
(% pupils showing 'well-established' skills or better at the level: 750-850 pupils per stage)

The confidence intervals (at the 95 % level) for each of the achievement estimates are shown by the black bars on the chart.

There are no consistent differences in achievement between girls and boys across stages and levels and in most cases the size of difference is small. At P7 we see relatively large differences with proportionally more girls than boys showing well-established or better skills at Levels C and D. Detailed results for achievement in Science literacy by gender are available in Table C2.

C.4 National Science literacy achievement estimates, by deprivation

Chart C3 compares the achievement of pupils from the 20 per cent most deprived areas against those from less deprived areas.

Chart C3
National Science literacy achievement estimates, by deprivation
(% pupils showing 'well-established' skills or better at the level: 750-850 pupils per stage)

The confidence intervals (at the 95 % level) for each of the achievement estimates are shown by the black bars on the chart.

'Most' means the pupil lived in one of the 20 per cent most deprived areas in Scotland based on the 2004 Scottish Index of Multiple Deprivation ( SIMD), the remainder of pupils are classified as 'less' deprived.

There was consistently higher achievement in Science literacy amongst pupils from less deprived areas than pupils from the most deprived areas at all stages and levels, with the exception of P5 Level B. In most cases, the size of the difference is statistically significant. This deprivation gap is consistent with the results we have seen in other SSA subjects from previous years and with the Science knowledge and understanding results presented in Chapter B. Detailed results for achievement in Science literacy by deprivation are available in Table C3.

C.5 Science literacy discussions

Around 200-300 pupils per stage were individually assessed in Science literacy on the basis of one-to-one discussions with field officers. Conversations were based on a topic introduced by the field officer using material developed specifically for the SSA. The prompt material comprised a picture and a series of statements relating to topical Science issues such as child obesity or climate change.

Field officers rated pupils using best-fit criteria within the five areas set out in the Nuffield definition of Science literacy. Full details of the task development and administration are available in Annex II.

It should be noted that due to logistical constraints, these practical assessments involved relatively small numbers of pupils, drawn from a sub-sample of the survey schools with typically 3-4 pupils being assessed in each school. Judgements about pupils' abilities are also necessarily subjective, and although the field officers were provided with some training to help improve the consistency of their assessments, their judgements were not subject to moderation.

The summary results are shown in Charts C4a to C4f. It should be noted that these results are sample statistics and not formal national estimates. Detailed results are available in Table C4 and Table C5.

Chart C4a
Results of the discussion-based Science literacy assessment:
Appreciates and understands impact of Science on everyday life

(% pupils rated as indicated for various aspects by field officers:200-300 pupils per stage)

Pupils were first rated on their awareness of Science issues. In general the pupils' awareness of the scientific issue under discussion increased with age, although it should be noted that not all issues were discussed at all stages. Issues were selected to be relevant to pupils to avoid asking them to discuss an issue that there would no expectation of them knowing about at their particular stage. Almost 45 per cent of S2 pupils were aware of the issue they discussed and its impact on everyday life, dropping to just under 20 per cent of P3 pupils.

Chart C4b
Results of the discussion-based Science literacy assessment:
Takes informed personal decisions about matters that involve Science

(% pupils rated as indicated for various aspects by field officers:200-300 pupils per stage)

Pupils were also rated on the level to which they displayed informed decision making relating to Science matters. Older pupils again show more evidence than younger pupils of appreciating and understanding the impact of Science on everyday life, with around one-third of P7 and S2 pupils showing 'clear evidence' of this (see Chart 4b).

Chart C4c
Results of the discussion-based Science literacy assessment:
Reads and understands short text on topical scientific issue in discussion or with support

(% pupils rated as indicated for various aspects by field officers:200-300 pupils per stage)

In another aspect of the discussion, a short topical text was read to pupils by the field officer and they were assessed on their understanding of it. Again older pupils performed better at this. Almost 40 per cent of S2 pupils understood all the essential points in their texts, dropping to just under 10 per cent of P3 pupils (see Chart C4c).

Chart C4d
Results of the discussion-based Science literacy assessment:
Reflects critically on the information included in or omitted from reports

(% pupils rated as indicated for various aspects by field officers:200-300 pupils per stage)

Finally, pupils were asked to critically reflect on the short topical text that had been read to them. Once again older pupils performed better in this. Just under 10 per cent of P3 pupils could make judgements on the information included or omitted from the text (Chart C4d), rising to over 20 per cent at P5. At P7 and S2 we again see similar rates of highest performance, at almost 40 per cent of pupils.

Chart C4e
Results of the discussion-based Science literacy assessment:
Takes part confidently in discussion with others about issues involving Science

(% pupils rated as indicated for various aspects by field officers:200-300 pupils per stage)

Pupils were then assessed on their level of contribution to the discussion with the field officer. In line with the picture described above, older pupils again performed better at this (see Chart C4e) with over 40 per cent of P7 and S2 pupils contributing several ideas to the discussion compared with around 30 per cent at P5 and just under 20 per cent at P3.

Chart C4f:
Results of the discussion-based Science literacy assessment:
Overall assessment

(% pupils rated as indicated for various aspects by field officers:200-300 pupils per stage)

Combining the results based on the three category scoring system, with category c) being most positive, we see that overall (as would be expected given the results in each section) there is clear evidence of age-related skills development in Science literacy. Over 40 per cent of S2 pupils were rated most positively, decreasing to just under 40 per cent at P7, under 30 per cent at P5 and under 15 per cent at P3.

C.6 Science literacy - scientific awareness enquiry

Alongside the written and practical assessments, each pupil in the survey was asked to complete a questionnaire exploring attitudes to, and experience of, learning in Science. Alongside other enquiries was one that explored awareness of scientific issues in society, as part of the broad investigation of Science literacy. Pupils were presented with a list of Science topics 'that are often in the news', and were asked to indicate whether they had heard of the topic, whether they thought it affected them, and if so why. The detailed results are given in Chapter E as part of the full report of questionnaire results.

However, the following is a brief summary of the results relating to Science literacy;

• awareness was generally high for the majority of topics, especially at S2
• 'carbon trading' was the least familiar topic
• awareness of most topics increased with stage with the exception of 'exercise and diet' which decreased, and 'climate change', 'pollution' and 'electronic communication' which remained steady
• boys tended to be more aware of topics than girls at P3, P5 and S2, with no differences between genders at P7
• among the pupils who indicated that they had heard of the various topics, the proportions who thought the topic affected them were generally very low in most cases at all stages. The three exceptions, where up to a third of pupils thought the topics did affect them, were 'exercise and diet', 'climate change' and 'pollution'
• deprivation was not a particularly important influence as regards media-based Science learning.

C.7 Science investigation

Science investigation skills were also assessed within the small-scale practical component of the survey. The pupils selected for this particular type of assessment were asked to carry out a scientific investigation during normal class time, and to write a report about it in preparation for the field officers' visit. The report was used as the basis for the discussion with the field officers, who used performance descriptors to arrive at a best-fit judgement of pupils' investigation skills. Investigation is part of the Science curriculum, and the findings of this exercise are presented here as the skills involved are related to many of those associated with Science literacy.

Again, it should be noted that due to logistical constraints, these practical assessments involved relatively small numbers of pupils, 200-300 per stage, drawn from a sub-sample of the survey schools with typically 3-4 pupils being assessed in each school. Judgements about pupils' abilities are also necessarily subjective, and although the field officers were provided with some training to help improve the consistency of their assessments, their judgements were not subject to moderation. That said, Chart C5 summaries the field officers' judgements of pupils' levels.

Chart C5:
Level profiles for Science investigation skills, by stage
(% pupils judged by the field officer to be at the given level: 200-300 pupils per stage)

The chart shows a wide spread of levels at each stage. At P3, 98 per cent of pupils were judged to be at Level A or better based on their investigation. Almost 15 per cent of these pupils are judged to be at Level C. At P5, 90 per cent of pupils were judged to be at Level B or better, and just over 70 per cent were judged to be at Level C or better. At P7, just over 50 per cent were judged to be at Level D or better. At S2, just under 50 per cent were judged to be at Level E or better.

C.8 Detailed results for Science literacy

Tables C1 to C5 provide detailed results to support the information provided in this section.

Table C1
2007 SSA - National Science literacy achievement estimates
(% pupils categorised into each achievement band on the basis of testing ^*)

* '< 50%' means fewer than 50% of the items in the level-based test answered correctly, 'good start' means between 50% and 64%, 'well-established' is 65% to 79%, and 'very good' is 80% or more. ' SE' is the standard error for the 'well-established or better' estimate.

Table C2
2007 SSA - National Science literacy achievement estimates, by gender
(% pupils categorised into each achievement band on the basis of testing ^*)

* '< 50%' means fewer than 50% of the items in the level-based test answered correctly, 'good start' means between 50% and 64%, 'well-established' is 65% to 79%, and 'very good' is 80% or more. ' SE' is the standard error for the 'well-established or better' estimate.

Table C3
2007 SSA - National Science literacy achievement estimates, by deprivation** category
(% pupils categorised into each achievement band on the basis of testing ^*)

* '< 50%' means fewer than 50% of the items in the level-based test answered correctly, 'good start' means between 50% and 64%, 'well-established' is 65% to 79%, and 'very good' is 80% or more. ' SE' is the standard error for the 'well-established or better' estimate.

**Most Deprived means the pupil with addresses in one of the 20% most deprived postcodes in Scotland based on the 2004 Scottish Index of Multiple Deprivation ( SIMD), the remainder of pupils are classified as less deprived.

Table C4
Results of the assessment of Science literacy skills, by stage
(% pupils rated as indicated for various aspects by field officers)

(approx 250-300 pupils per stage)

Table C5
Level achievement results for Science investigation, by stage
(% pupils judged by the field officer to be at the given level)

« Previous | Contents | Next »