The Scottish Government

« Previous | Contents | Next »

Listen

Assessment of Achievement Programme
Report of the First AAP Survey of Social Subjects Enquiry Skills (2002)

7. Summary and Issues

7.1 Pupils' enquiry skills attainment in 2002

The 2002 AAP survey measured pupils' attainment in terms of the strand and level descriptors for skills for enquiry in the social subjects set out in the National Guidelines for Environmental Studies ⁶. The three skill strands within social subjects are Preparing for tasks, Carrying out tasks and Reviewing and reporting on tasks (for various reasons this strand was not assessed on this occasion).

7.1.2 Carrying out tasks

Attainment for Carrying out tasks was assessed primarily through extensive written tasks . There were 45 different tasks, nine per level (5-14 levels A-E) covering each of the three social subjects in a relevant context. Each of the tasks used the same format: three subtasks requiring pupils to respond to a small number of short-response items by retrieving information from given sources, with each subtask featuring a different type of source. From Level B up, supplementary questions required pupils to pull relevant information together from all three sources. The three subtasks were presented in a single test booklet. Pupil attainment on these tasks has been reported in Chapter 2 in terms of the proportions reaching various cut-off scores:

• the 50% criterion (half of the available marks) is taken to mean that pupils have demonstrated basic skills at the level concerned.
• the 65% criterion (around two-thirds of the available marks) is taken to mean that pupils are secure at the relevant level.
• the 80% criterion is taken to mean that pupils have considerable strengths at the level.

Attainment for Carrying out tasks was also assessed through practical activities: watching a video and answering questions about it, and/or attempting a referencing skills task based on extracts from stage-relevant books. Three video tasks, each related to one of the social subjects, were used with P3 and P5 pupils and a different set of three were used with P7 and S2 pupils. Two reference tasks were used at P7 and S2, a different pair at P5 and a third pair at P3. Pupils were asked to select a book to find out about a topic and then went on using the contents page, index, glossary and the actual page to locate specific information. Pupil attainment was reported in different ways for each type of task (see Chapter 3).

Summary of attainment: written tasks

• 60-70% of the pupils at each stage demonstrated at least 'basic' performance at the 'target' level for their stage: Level B at P3, Level C at P5, Level D at P7 and Level E at S2.

More able pupils

• Around 40-50% of pupils at each stage were classified as working 'securely' or 'with considerable strengths' at the target level for their stage.
• Over 25% of the pupils in P3 showed 'considerable strengths' at Level B with around 15% of pupils doing so for the target level at each of the other stages.

Less able pupils

• Around 25% of the pupils at each stage showed only 'basic' skills at the target level for their stage.
• Around 10% of pupils failed to demonstrate even 'basic' skills at the level below their target (i.e. the lower of the two levels assessed) in P3 (Level A), P5 (Level B) and P7 (Level C), rising to 20% for pupils in S2 (Level D).

S2 Dip in performance

• Slightly fewer pupils in S2 attained the target level for that stage than in the other year groups.
• Fewer S2 pupils attained the expected level 'securely' or with 'considerable strengths' than pupils in P7 (38% compared to 47%).
• More S2 pupils failed to attain even 'basic' skills at the target level for their stage than at other stages, but this was especially marked in comparison with P7 (38% in S2 but only 28% in P7).
• A larger percentage of S2 pupils failed to attain even 'basic' skills at the level below their target (20% of pupils in S2 at Level D compared to 11% of pupils in P7 at Level C).

Gender differences

• While there were no overall statistically significant gender differences apparent in the attainment data, as might be expected girls did tend to produce better performances than boys on most of the individual written attainment tasks. There was clear evidence of topic effects in the attainment data.

Summary of attainment: video tasks

• Within stage groupings - P3/P5 or P7/S2 - pupils at the higher stage performed better in the video task than those in the lower stage.
• The majority of pupils at all stages showed at least a 'basic' level of performance in their video task, but P3 pupils, as might be expected, performed slightly less well than pupils in any of the other stages (66% of pupils in P3 showed this level or better compared to 74% of pupils in P5; and, for a different video task, 68% of P7 pupils compared to 89% in S2).
• There was a stage difference of around 20 percentage points in the proportions of pupils showing 'considerable strengths' (14% of pupils at P3 compared to 36% of pupils in P5; and, for a different video task, 28% of pupils at P7 compared to 49% in S2).
• More S2 pupils than P7 pupils demonstrated they were working 'securely' or with 'considerable strengths' in these activities (88% of pupils in S2 compared to 68% of pupils in P7).
• There was no evidence in the data of any gender differences in performance for the video-based tasks.

Summary of attainment: reference skills

• For the reference skills activities used to assess Carrying out tasks, the findings indicate that pupils' reference skills were already fairly well established even by P3 and that improvement from P7 to S2 was marginal.
• At all stages pupils were less competent in the reference skills tasks when using indexes and glossaries than when they were using contents lists.
• Only at S2 was there any evidence of a gender effect, with the girls producing a better overall performance than the boys.

7.1.3 Preparing for tasks

Pupils' abilities to prepare for tasks were assessed at every stage through one to one interviews during which a series of relevant questions were put to the pupils as they studied stimulus photographs. The pupils' responses were judged by the interviewers as 'confident', 'with prompting', 'minimal' or 'inadequate/incorrect' (see Chapter 3).

S ummary of attainment

• The proportion of pupils answering 'confidently' in Preparing for tasks, increased with increasing age: from a quarter of the pupils in P3 who showed such strengths to just over half the pupils in S2.
• Stage progression appeared most marked between P3 and P5 (23% and 39% respectively answered 'confidently').
• At P7 and S2, there was a tendency for proportionally more of the boys than the girls to answer 'confidently'.

7.2 Core Skills attainment

Reading and numeracy

Pupils produced similar performances when tackling test items requiring them to extract and use information from texts as they did when tackling items based on other types of information source. On a relatively small number of test items demanding the application of computation skills, 50-75% of the pupils on average answered the items successfully.

Writing

There is clear evidence of progression in level attainment in writing for pupils between P5 and P7. However at S2, there is slightly greater dispersion in attainment in writing than at P7. Slightly higher proportions of pupils in S2 showed very low or very high attainment than pupils in P7.

Gender differences were also evident in writing. Among pupils in P5, fewer girls than boys were judged to have produced work below level A. For pupils in P7, significantly more girls than boys were judged to have produced writing at level D. In S2, the girls produced work at a higher level than the boys, with significantly fewer girls having produced work at Level A or below and significantly more girls having produced work at Level E.

Problem solving

Two group activities were used to assess Problem solving and aspects of Working with others. One activity, 'Rescue', was used for the P3/P5 groups and a second activity, 'Survival', was used with P7/S2 pupils. Various aspects of the behaviour of individual pupils were rated during the group discussions (see Chapter 4).

• The findings clearly show no difference in the discussion behaviour of pupils at P5, P7 and S2, nor indeed in the pupils' abilities to evaluate their own and their group's work, but the discussion skills of P3 pupils were significantly less well developed.
• The majority of pupils at all stages showed some evidence of Problem solving ability. Generally, 'Reviewing and reflecting on progress' showed lower ratings than other aspects of Problem solving.
• The results indicate a significant improvement between P3 and P5 in performance in Problem solving.
• More pupils in P7 indicated 'clear evidence' of the various observed behaviours than did pupils in S2 (57% compared to 48% respectively)

Working with others

Observations during the group tasks plus an interview with a group member after completion of the task were used to assess Working with others.

• Pupils mainly exhibited positive rather than negative behaviours when Working with others.
• Pupils showed a significant improvement between the P3 and P5 stages in positive behaviour (26% in P3 compared with 43% in P5 showing 'clear evidence'); there was no difference between P7 and S2 (around half showing 'clear evidence' at each stage).
• As might be expected, proportionally fewer P3 pupils were rated as having given 'full responses' in their evaluations of their group's work than pupils at the other stages.

ICT skills

Three different ICT tasks were used at P7 and S2.

• Pupils at both stages did relatively well on the ICT tasks, given that they were using laptop computers that they were not used to working with in the normal course of events.
• Proportionally more of the S2 pupils compared with the P7 pupils were able to complete the tasks unaided.
• Pupils at both stages found chart creation within spreadsheets a more challenging activity than editing text, adding information to spreadsheet tables, using web searches or sending emails.

7.3 Pupil questionnaires

7.3.1 The Pupils

A sample of P7 and S2 pupils who completed the written tasks also completed a questionnaire that explored various aspects of their subject learning experience both inside school and at home (see Chapter 5). The samples were evenly divided by gender, and the large majority were native English speakers. While the pupils' job aspirations were many and varied, they remained heavily gender typed.

7.3.2 The classroom learning experience

According to the pupils, social subjects lessons were more content-based than enquiry-based, but they were also very interactive, with a high level of questioning and answering. Teachers usually gave help when needed and expected pupils to work hard.

The majority of the pupils found the subject easy, were happy with the pace of work, were rarely left behind, and rarely found it difficult to catch up if they missed a lesson. Most pupils, and higher proportions at S2 than at P7, agreed that they were often or always given teacher feedback on progress in the form of levels, grades or test scores, and short tests were common. Corrected work generally showed pupils where they had gone wrong, and pupils were typically shown by the teacher how to improve their work. However, a relatively low proportion of pupils agreed that their teacher involved them in planning their next steps in learning.

The majority of pupils wanted to do well in the subject, and worked hard on their topics and investigations, which they found interesting in general. Pupils were more evenly divided on whether they looked forward to lessons and enjoyed reading books about the subject. Most pupils, and their families, considered the subject important for later learning in other subjects and for jobs.

As to the nature of learning activities in the classroom, whole class teaching predominated at both stages. The most frequent individual activities were writing in a jotter or file, working quietly alone, completing worksheets and reading text books and reference books. Using computers in class, using tools and instruments in investigations, handling objects and artefacts, working in the school grounds and visiting places outside school were less common at both stages, but particularly at S2, by which time some of these activities were rare.

Two gender differences emerged, in that higher proportions of the boys than the girls admitted to getting behind with the work and higher proportions of girls than boys at both stages claimed to find the subject easy to understand.

7.3.3 Learning out of school and at home

On their own reports, the S2 pupils had almost twice as much homework (all subjects) as the P7 pupils. High proportions of pupils at both stages had a quiet place to study at home, had home access to a computer and the internet, and had a calculator and dictionary available to them at home.

While there were clear gender differences in job aspirations, there were no differences between boys and girls in terms of their reports on their social subjects learning, nor in general in access to important learning resources at home, particularly a quiet place to study and access to a computer and the internet. There were some gender differences in engagement in family activities, with boys tending less often than girls to have cultural outings with their adult family members or to read or talk about books with them, but more often to play sport with them.

7.4 Teacher questionnaires

The P7 and S2 teachers, principal teachers and primary head teachers for schools in the written task sample were asked to complete a questionnaire to collect baseline data on qualifications, experience, learning activities used, and perceptions on teaching preparedness and adequacy of resources for the social subjects (for details see Chapter 6). While the overall response rate was disappointing, at 30-35% of the schools invited to collaborate, the S2 teacher sample represented quite closely the whole secondary school population. The P7 teacher sample under-represented the smallest schools, schools with the lowest free meals entitlement and the smallest education authorities.

7.4.1 Teachers' backgrounds and experience

The P7 teachers were predominately female, while the S2 teachers were evenly gender representative. In terms of degree subject, the secondary teachers had a similar representation of geography and history (one third each), along with a variety of other social subjects and languages. The majority of geography and history graduates were teaching their degree subject at S2; all the language graduates were teaching history; and, given the SEED requirements to train to teach modern studies, graduates with degrees in a variety of social subjects were teaching that subject or social subjects.

Years of teaching experience and years in post varied widely in both groups from a year or two to decades. As a group, the S2 teachers tended to have slightly more teaching experience than the P7 teachers but slightly less time in their current posts. A substantial minority of teachers in both groups stated they had had no professional development during the previous two years, while among those that had benefited from such experience the median duration was a matter of days.

7.4.2 Courses and programmes

A large majority of schools are currently revising their programmes of work for P7 and S2 (88% and over 75% respectively) to update them in line with the recently revised Environmental Studies Guidelines ⁷. This links to HMIE inspection comments for social subjects in secondary schools where for S1/2 courses, 45% in History, 37% in Geography and 60% in Modern Studies were noted to have important weaknesses or were unsatisfactory ⁸. The Guideline s plus the school/department's own materials are the two main sources for course development. Among S2 teachers, some expressed frustration in the differences between the Guidelines and Standard Grade courses/assessment. This was also evident in the desire expressed by some S2 teachers to report pupil attainment in S2 in line with Standard Grade. There were also critical comments related to the perceived lack of exemplification of the national standards for the 5-14 levels for the social subjects.

7.4.3 Quality of resourcing and other issues

Differences in the rating patterns of primary head teachers and S2 principal teachers reached statistical significance in the case of the availability of computers for teachers and for pupils, internet access for pupils, and pupil behaviour in classes. The most poorly rated aspects were computer availability and internet access for pupils at S2, with around one-third of the S2 principal teachers claiming both to be 'unsatisfactory'.

Parental support for their children's learning was also significantly more highly rated by the primary head teachers than the S2 principal teachers, even though the proportions giving the highest rating ('very good') were relatively low in each case, at 17% and 8% respectively.

7.4.4 Learning activities

The teachers were asked to rate 18 different learning activities, in terms of how often they estimated that their pupils were engaged in each one. At both stages the learning activities most frequently engaged in by pupils, according to their P7 or S2 teachers, were being taught as a whole class and writing in a jotter or file. Just over 50% of the S2 teachers and 60% of the P7 teachers agreed that their pupils were taught as a class in 'most lessons', and over 50% of the S2 teachers and 40% of the P7 teachers agreed that their pupils spent time in 'most lessons' writing in their jotters or files.

A pattern of greater divergence in the views of the S2 teachers, compared to the P7 teachers, is seen for many of the learning activities. While a number of the activities would seem to be more frequent at S2 than at P7 in some classrooms, they were also simultaneously less frequent in others. This perhaps suggests that some S2 teachers place greater importance on their personal preferences for choosing different styles of learning activities.

There were high inter-correlations when the teachers rated their preparedness to teach the social subjects outcomes and enquiry skills, and to develop informed attitudes in their pupils, their ratings of how well resourced they were in each case, and their ratings of the extent to which they had covered each aspect in the current session. In general, the P7 teachers gave higher ratings than the S2 teachers for most aspects. Perhaps the most interesting finding is that for all outcomes except People in the past (generally taught by history graduates) a substantial minority of the S2 teachers responded that they were 'not at all well' prepared in terms of resourcing and coverage, the proportion being highest for P eople in society. Just 4% of the P7 teachers, but fully a third of the S2 teachers, admitted to feeling 'not at all well prepared' to teach People in society.

This finding is particularly interesting when compared to the findings in the HMIE Standards and Quality report for 1998-2001 ⁹. In this report, HMIE indicated that of the three social subjects, People in society "was the most common weakness within the social subjects" (p.7) and that attainment in it should be raised in 60% of primary schools. Overall, 55% of such courses were judged to be weak. On the other hand, in 40% of primary schools attainment in People in society was good compared to 60% for Geography and 70% for History. In an area related to the core skills aspects of the current AAP survey, Problem solving and enquiry was said to be weak overall in 50% of primary schools, and in secondary schools only a few schools had clear plans "for ensuring that all pupils developed core skills" (p. 19).

7.5 Key issues

The findings from the survey raise two key issues relevant to future developments in learning and teaching in skills for enquiry in the social subjects in the primary and lower secondary school:

• evidence of progression from P3 to S2 in skills for enquiry in the social subjects
• attainment in core skills in the context of social subjects.

7.5.1 Evidence of progression from P3 to S2

Using the model of progression of the 5-14 Guidelines, evidence from this survey suggests that pupils in P5 and P7 (and in S2 to a lesser extent) have made good progress in developing skills for enquiry in the social subjects in comparison with the performance of pupils at the stage below. A number of other studies of student performance, in a range of subjects, indicate that there is commonly a break in the steady progression of performance in primary school following transition to secondary school ¹⁰.

This pattern is repeated to some extent within this survey. Detailed analysis of performance in relation to these skills for enquiry suggests that Level E, as it is currently understood, is not being attained by a large proportion of pupils in S2. This was particularly true for the written tasks.

On the other hand, in terms of several of the practical activities, differences between the P7 and S2 pupils were either marginal (such as for the reference skills tasks) or the S2 pupils performed considerably better than pupils in P7. For the earlier stages, a similar pattern was evident in the practical activities, with the P5 pupils showing higher attainment than the pupils in P3. This would especially have been expected for the group activities and the one to one interviews for the younger pupils given that they were meeting with a previously unknown adult.

At the time the survey was conducted, teachers' understanding of pupil performance at each of the 5-14 Levels A to F was based mainly on the descriptors set out in the Revised Guidelines and the exemplification in the accompanying Guide for teachers and managers ¹¹. The AAP survey provides a more detailed picture of actual pupil performance at each level than previously existed. Since this survey was conducted, an additional resource specifically exemplifying performance at Levels A-F for all three social subjects has been produced as part of SEED's Assessment is for Learning project ¹² (LTS, 2003).

The evidence from this survey indicates teachers would be well advised to use a variety of assessment strategies including practical activities, discussion and observation and not only written tests, to build up a realistic and well rounded profile of pupil performance in skills for enquiry in the social subjects.

7.5.2 Attainment in core skills in the context of the social subjects

Assessing performance in skills-based enquiry activities in the social subjects such as were included in this survey, also offers realistic contexts for the assessment of core skills (elsewhere known as key skills or transferable skills). Pupils generally responded well to the opportunity to be involved in discussion activities.

There is some evidence that the youngest pupils included in the survey found some of these discussion activities particularly challenging, for instance the interviews mentioned above.

For the group activities used for Problem solving and Working with others, the expected progression between P7 and S2 was not evident and indeed at times S2 pupils performed slightly less well than pupils in P7.

This dip in performance may, of course, have been due to other confounding factors. For example, the same group activity 'Survival was used to assess pupils in both P7 and S2. Pupils in S2 might have perceived this activity or the context as inherently less interesting than the pupils in P7. A previous AAP survey (the 1995 survey in English language) used a similar activity to assess pupils' discussion skills and found a similar dip in performance in S2.

Pupils generally performed very well in the tasks used to assess ICT core skills. Initial teething problems with the CDROM meant that this activity was only used in this survey with a very restricted sample of pupils in P7 and S2. The later 2003 AAP survey in science has been able to extend ICT core skill assessment across the stages.

7.6 Conclusions

This survey provides evidence that provision for skills in social subjects in the primary and lower years of secondary school is enabling most pupils (60-70% or more) to achieve at least basic competence at the expected level for enquiry skills, and some (around a quarter) to achieve more than this. Secondary pupils do not, however, seem to maintain the progress that is made in primary schools, especially in relation to written tests. To build up a profile of pupil performance in these skills, teachers might consider using a variety of learning activities and assessment strategies. The variety of activities and sources used within enquiry skills for social subjects also provide additional contexts and activities for assessing other core skills areas of the 5-14 curriculum.

Given that this was the first survey conducted for this area of the curriculum, the findings reported here may be most usefully regarded as a benchmark against which future cohorts might be measured. More detailed evidence of the nature of student performance, making reference in particular to the examples of what pupils can do at each of the 5-14 levels will help teachers to assess whether or not their pupils are at a similar stage of development, and in what ways they might move on from where they now are.

« Previous | Contents | Next »