The Scottish Government

Listen

Assessment of Achievement Programme:
Sixth Survey of Science 2003:
Summary

This document is also available in pdf format (92k)

1 Survey overview

The 2003 Science Survey was the sixth survey to be carried out in this curriculum area within the Assessment of Achievement Programme (AAP), and the second subject survey to incorporate the assessment of core skills - the first being the Social Subjects survey of 2002.

The stages assessed were P3, P5, P7 and S2.

Testing took place in May and June 2003, and almost 10,000 pupils in just under 600 mainstream schools across Scotland took part: just under 2500 pupils in around 155 primary schools at each primary stage, and just over 2000 S2 pupils in 130 secondary schools.

All the survey schools participated in the written assessments - science, numeracy, reading and writing - and half also participated in the practical assessments - science investigation skills and ICT skills.

In this survey, for the first time and as a result of the recent change to the National Guidelines for Environmental Studies, pupils' Knowledge and understanding in science is reported with reference to 5-14 levels. Also for the first time, the survey included an exploratory assessment of pupils' informed attitudes in science.

To provide a context for the attainment and attitude findings, participating schools at P7 and S2 were invited to complete a questionnaire about the resources available to them for science and about their science provision, and just under half their sample pupils were invited to complete questionnaires about their science learning experience.

The National Assessment Bank provided large numbers of the assessment tasks used in the survey, supplemented by tasks developed by staff in the University of Aberdeen ('Knowledge and understanding' in science), the University of Strathclyde (science investigation tasks), SQA (reading and writing, numeracy) and individual consultants (ICT, with CDs prepared by LTS). School and pupil questionnaires were developed by SEED.

The SQA was responsible for printing the test booklets and distributing these to schools. The survey schools organised and supervised their own written test sessions within the period mid-May to mid-June.

Practical assessments were the responsibility of 148 itinerant field officers - practising primary teachers or secondary science teachers released by their 28 education authorities for survey involvement.

Completed test booklets were received and processed by SEED, and pupil response data keyboarded by Lindata Services and analysed by Assessment Europe.

2 Science attainment results

Knowledge and understanding

Just under 6000 pupils in around 600 schools participated in the written science assessment, that is around 1300-1500 pupils at each stage, and in total 360 Knowledge and understanding tasks were administered.

On the basis of their performance, pupils were classified as being 'secure' at the level (using the criterion of 65% or more of the marks achieved on tasks at the same level), or as having shown 'basic' knowledge and understanding at the level (at least 50% of marks achieved, but not as many as 65%), or as having shown 'considerable strengths' at the level (80% or more of the marks achieved).

• Three-quarters of the P3 were classified as being secure at Level A, and over half were also secure at Level B. Similarly, three-quarters of the P5 pupils were secure at Level B, and around a quarter were secure also at Level C.
• The P7 and S2 pupils showed a less positive picture. Fewer than 10% of the P7 and S2 pupils were secure at the target levels for these stages, ie. Level D for P7, Level E for S2; indeed the majority (75-80%) failed even to show evidence of 'basic' attainment at these levels (fewer than half marks scored).
• Just over a third of the P7 pupils were classified as secure at Level C and just under a fifth of the S2 pupils were similarly classified at Level D. No S2 pupils were secure at Level F, and virtually no P7 pupils were secure at Level E.
• There is no evidence in the survey data of any difference in the Knowledge and understanding attainment of boys and girls.
• On the basis of a very modest set of common tasks, ie. tasks from the 1999 survey that were re-used here, there is no evidence of any change in pupil attainment since 1999.

Investigation skills

Just under 2500 pupils in around 330 schools participated in the practical assessment of investigation skills in science: around 600 pupils were tested at each stage in 80-90 schools.

Nine investigation tasks were newly developed for use in the survey, each with an accompanying rating checklist containing attainment targets at two adjacent 5-14 levels. Sets of three tasks were administered at each stage.

The tasks were administered in the schools by the 148 field officers, who completed rating checklists as they observed and questioned pupils at work.

• Across the stages, and averaging over attainment targets and tasks, a quarter to a third of the pupils had to be given assistance by the field officers before they could show enough evidence of relevant skills for them to be judged at one level of the other for each of the attainment targets included in the checklists.
• On a similar basis, 10-25% of the pupils could not be judged at a level at all, because they did not show evidence of the behaviour concerned with or without prompting from the field officer.
• The attainment data provided by those pupils who could be judged show clear evidence of stage progression in investigation skills.
• Again averaging over attainment targets at a level within and across tasks, around one-third of the S2 pupils were rated as having demonstrated the attainment target skills at Level E, with just under half having shown evidence of skills attainment at Level D.
• Around one-third of the P7 pupils showed target skills attainment at Level D, and 40% of the P5 pupils did the same for Level C. Over half the P3 pupils were deemed to have shown appropriate evidence of target skills attainment at Level B.
• There was no evidence in the attainment data of any gender difference in investigation skills.

Core skills attainment results

Communication - Reading

• Reading attainment was good at all stages, particularly at P3 and P5.
• Over 80% of the P3 pupils were 'secure' at Level A (ie. gaining at least 65% of the marks on their Level A task) and over half were secure at Level B; around 70% of the P5 pupils were secure at Level B and around 40% were secure at Level C.
• Roughly 60% of the pupils at P7 and at S2 were secure at Levels C and D, respectively, and around 40% were secure at Levels D and E, respectively.
• The results at P7 and S2 are in line with those reported for reading attainment in the 2001 English Language survey.
• In contrast with expectations, there were no apparent gender differences in reading in this science survey.

Communication - Writing

• The pupil attainment picture that emerged revealed evidence of relatively poor writing skills at all stages. In particular, fewer than 10% of the P7 and S2 pupils produced writing that was rated as being at the target levels for these stages (Levels D and E, respectively), and fewer than 15% of the P5 pupils produced writing deemed to be at Level C or higher.
• For most of the pupils involved in the reading and writing assessment, writing attainment was lower than reading attainment; just 25% of the pupils assessed in reading and writing were classified at the same level in both.
• While proportionally more of the boys than of the girls failed to produce any writing in the survey (20% versus 14%), among those pupils that did produce writing there were no gender differences in quality.

Numeracy

• Pupils' numeracy skills appear to be quite well established. On average, more than half the pupils at every stage successfully answered their numeracy items, involving information handling and computation.
• The numeracy tasks used here were derived from those used in the previous Social Subjects survey, and were used in the science survey in identical form or with simple changes to wording to render the contexts more 'scientific'.
• A comparison across surveys revealed slightly better performances across all stages and tasks in the social subjects survey than in the science survey, the lower achievements in the science survey reflecting generally higher non-response rates for all tasks in the science test booklets (in which the numeracy tasks were embedded).
• There were no gender differences in numeracy attainment in either survey.

Using ICT

Despite a variety of problems associated with the exercise, the practical ICT assessment produced evidence of well-established skills in some areas and of clear progression elsewhere from one stage to another.

• Over 90% of the pupils at all stages, even P3, were skilled at opening documents and navigating through web pages.
• Around a third of the P3 pupils were also able unaided to add information to existing spreadsheet tables, compared with almost 60-70% of the pupils at other stages - at all stages lower proportions of pupils could centre information within columns.
• At P5, P7 and S2, where these skills were tested, rather low proportions of pupils managed within their spreadsheets to produce graphs to illustrate their tables of data, and none, or almost none, knew how to use the formula facility in Excel to carry out simple calculations on their data.
• Around a third of the P3 pupils were at ease simulating web searches while a quarter were able unaided to prepare to send an email; at the other stages the proportions were higher, ranging from 50% to 80%, depending on the activity and the stage.
• As was the case for the other core skills, there were no gender differences evident in the ICT attainment data.

Informed attitudes

• Over 1200 pupils at P5, P7 and S2 completed questionnaires inviting information about their science lessons, and exploring their familiarity and views about topical issues in science and medicine.
• The enquiry revealed a high degree of awareness of current issues among pupils, particularly at S2. Among twelve targeted issues, the internet, cancer research and weapons of mass destruction were those that most interested the pupils, while designer babies, cosmetic surgery and genetically modified foods held little interest at all for them.
• There were no gender differences in awareness, but quite strong and expected gender differences in topic interests: the boys were more interested than the girls in nuclear power, space exploration, robotics and weapons of mass destruction, while the girls' preferred issues were cancer research, designer babies and cosmetic surgery.
• When asked to offer an opinion on whether the issue concerned was 'good for society' or 'bad for society', the internet, cancer research and organ donation emerged as those most positively viewed in this sense, while weapons of mass destruction and global warming were the most negatively viewed.
• The majority of pupils declared themselves unsure of the value to society of designer babies, cosmetic surgery and genetically modified foods.
• Pupils' reasons for their opinions were generally valid and relevant, showing evidence of informed attitudes.
• In an innovative complementary enquiry, over 2500 pupils participated in focus group discussions about given scientific issues.
• Almost 650 focus groups were observed and rated for informed attitudes by the field officers. A quarter of the groups, rising to a third at S2, were reluctant to engage in discussion and were difficult to animate and hence to assess.
• Pupil groups were generally more highly rated for showing evidence of 'Respect and care for self and others' than for 'Commitment to learning' and 'Social and environmental responsibility'.
• The least well demonstrated attributes, within 'Commitment to learning', were 'Recognising the need to find out more facts and/or ask questions about the issue' and 'Asking about evidence'.

3 Pupils' learning circumstances and views about their learning

• High proportions of pupils at P5, P7 and S2 had a quiet place to study at home, had home access to a computer and the internet, and had a calculator and dictionary available to them at home. There were no gender differences in terms of home resource access.
• The most common family pastimes were watching TV and videos, and listening to music. Reading or talking about books were much less popular activities, as were cultural outings of various types.
• There were some gender differences in engagement in family activities: girls more frequently than boys talked with their families about their hobbies, their school work and progress, and books, while boys played sport or games with family members, or otherwise kept fit with them, more often than did girls.
• In the science classroom whole class teaching predominated at all stages, and the most frequent individual activities were writing in a jotter or file, and completing worksheets.
• Using computers in class, working in the school grounds and visiting places outside school were less common, particularly at S2.
• On their own reports, the S2 pupils had more science homework than did their younger peers.
• Science lessons were very interactive, according to the pupils, with a high level of questioning, answering and explaining, and teachers usually gave help when needed. Teachers expected pupils to work hard, and the pupils claimed that they found this easy to do in lessons.
• The majority of pupils wanted to do well in the subject, and worked hard on their topics and investigations, which they found interesting in general.
• Pupils were more evenly divided on whether they looked forward to lessons and enjoyed reading books about the subject, enthusiasm for science learning decreasing with increasing age.
• Most pupils considered science important for later learning in other subjects and for jobs, and thought that their families did too, although perceptions of importance decreased with increasing age.
• The majority of the pupils found science an easy subject, were happy with the pace of work, were rarely left behind, and rarely found it difficult to catch up if they missed a lesson.
• Higher proportions of pupils at S2 than at P5 or P7 agreed that they were often or always given teacher feedback on progress in the form of levels, grades or test scores, and that short tests were common.
• Corrected work generally showed pupils where they had gone wrong, and pupils were typically shown by the teacher how to improve their work; these activities were more frequent at P5 than at P7 and S2. However, a relatively low proportion of pupils at each stage agreed that their teacher involved them in planning their next steps in learning.
• There were no differences between boys and girls in terms of their reports on their science learning.

4 Teachers' reports on science in schools

• Questionnaire enquiries were incorporated into the survey at P5, P7 and S2, to gather information from teachers about the provision and resourcing of science in the schools, and to invite their views about the quality of different aspects of science experience in classrooms.
• Questionnaires were completed by 84 primary head teachers, 82 secondary principal teachers, 95 primary class teachers and 116 S2 subject teachers.
• A very high proportion of the primary head teachers reported that their current science programmes had been introduced or revised in 1998 or later, the majority since 2000. Among secondary principal teachers the proportions were much lower.
• Programmes were currently under revision in the majority of the primary schools; and just under half the principal teachers also agreed that their programmes were under revision at the present time.
• The most frequently mentioned bases for science programmes among the principal teachers were the department's own materials, the national 5-14 guidelines and commercial textbooks or resource packs.
• A high proportion of the senior teachers in each group noted that teachers' own comments featured in school reporting to parents on pupil progress, with 5-14 levels following fairly closely behind, particularly at S2. Comment banks were mentioned equally by both groups.
• A higher proportion of principal teachers than primary head teachers mentioned 'marks or grades for effort'. A fifth of the secondary principal teachers also indicated '% marks in end-of-unit tests', whereas only one single primary head teacher did so.
• Among the primary head teachers the most positive ratings for resource quality were given to computer and internet access for teachers and for pupils, high ratings were also given for pupil attendance at, and behaviour in, subject classes. The secondary principal teachers were markedly less positive in their response to these same issues.
• Parental support for their children's learning was also significantly more highly rated by the primary head teachers than by the S2 principal teachers.
• There was no difference in the rating patterns of the two groups for teacher expectations of pupil achievement in science, but marked differences in their evaluations of pupil motivation to learn, the primary head teachers rating this higher than the secondary principal teachers.
• There was a strong gender difference in the S2 science teachers' subject qualifications, with significantly higher proportions of female than male teachers having a biological science or chemistry qualification with the reverse holding for physics.
• At all stages, number of years' teaching experience varied widely, from one year to 35 years or more, and there was a similar wide variation in years in current post.
• Class sizes at all three stages were similar, at an average of around 17 pupils, but class contact hours increased with stage. Teaching preparation time at S2 was also greater than at the primary stages.
• A majority of the primary teachers claimed that they met with colleagues to discuss science subject teaching 'hardly ever' or at most once or twice a year. This compares with a fifth of the S2 subject teachers, one-third of whom claimed to meet of such discussions at least weekly if not more frequently.
• Opportunities for professional development were also greater among the S2 subject teachers than among their primary colleagues, and greater at P7 than at P5.
• The S2 subject teachers, like their principal teachers, gave significantly less positive ratings to their pupils' motivation to learn than their primary colleagues.
• Teachers' reports about pupils' learning activities in lessons mirrored those of the pupils themselves.
• According to the majority of the teachers in both sectors, pupils at all three stages are taught as a class in most lessons, and at S2 spend most lessons writing in their jotters or files. Pupils also worked in pairs in most lessons in the majority of the S2 classes.
• The evidence is that S2 pupils more often used tools and instruments in investigations than did their younger peers, and they also used maps, drawings and diagrams more frequently. Use of computers in class was relatively rare in both sectors, but particularly at S2.
• Among the S2 subject teachers the most frequently used resource was the department's own materials. Just over half the teachers in both groups used the national 5-14 guidelines in most lessons, and around half used their own local authority guidelines this often. Commercial textbooks and/or resource packs were also used very frequently by substantial proportions of teachers in both groups: over a third of the primary teachers and around half the S2 teachers.
• Around a quarter of the teachers in both groups frequently used materials they had developed themselves. Materials produced by a teachers' group or association, and materials produced by another schools, were the least used resources in both sectors. Materials produced by another local authority were not much used in the secondary schools.
• At least half the S2 teachers felt themselves 'very well prepared' to teach all the science outcomes, with the highest proportion - three-quarters - feeling this way with respect to investigation skills in science, followed by Knowledge and understanding - Energy and forces among the physics teachers.
• The primary teachers were much less confident on all counts, the highest proportion - at two-fifths - considering themselves very well prepared for developing informed attitudes.
• The primary teachers were more negative in their opinions about how well they were resourced to teach science than were the S2 teachers, especially for Knowledge and understanding - Energy and forces and skills in science.
• There were also marked differences in the response patterns of the two groups for Knowledge and understanding - Energy and forces and for investigation skills with respect to coverage during the session. Almost half the S2 teachers who responded about coverage of energy and forces' thought they had covered the various strands 'very well' compared with around a quarter of the primary teachers, with a similar difference for investigation skills.
• While the proportions of teachers considering they had covered Knowledge and understanding - Earth and space and Developing informed attitudes 'very well' were similar in both groups, higher proportions of the S2 teachers than the primary teachers answered 'not very well' and 'not at all well'.