The Scottish Government

« Previous | Contents | Next »

Listen

Assessment of Achievement Programme
Report of the First AAP Survey of Social Subjects Enquiry Skills (2002)

Appendix B: School and pupil sampling

The 2002 Social Subjects survey was designed to assess pupils at P3, P5, P7 and S2 in mainstream schools in Scotland - including educational authority, self-governing, grant-aided and independent schools. Special schools and Gaelic medium schools were excluded from the sampling frame.

Representative pupil samples were selected for the written testing of social subjects enquiry skills using two-stage stratified sampling. In the first stage, school samples were drawn. The population of schools was first stratified by education authority, school size and percentage free school meals entitlement, and schools were then selected from within strata with probability proportional to stage size. The sample pupils were selected in a second stage of sampling, a total of 18 pupils being randomly selected for testing within each survey school (lower numbers of pupils were available in small schools). In composite classes, only pupils at the relevant stage were selected. Where pupils with special educational needs were selected in school samples, these were included in the actual assessment at the head teacher's discretion.

For the purposes of population stratification, primary schools were grouped into the following roll size bands: <190, 190-279, 280-379, 380+. Secondary schools were grouped on the basis of their numbers of S2 pupils: <150, 150-189, 190-234, 235+. In both sectors, schools were grouped in terms of percentage free school meals entitlement: <10%, 10-14%, 15-25%, 25%+. School size and free school meals entitlement data were derived from the most recent school census, taken in 2001.

The 32 education authorities were classified into four groups for the purpose of school population stratification (maintained schools only), based on their population densities. Table B.1 identifies the four groups.

School participation in the survey was voluntary, and not all the invited schools felt able to participate. As Table B.2 shows, at each stage over 170 schools were initially selected for survey participation, of which all but three or four were authority maintained schools, but fewer than 160 actually participated at each of the younger stages and fewer than 140 at P7 and S2. Participation rates were 80-90% at the primary stages and 75% at S2.

In addition, even when a school that had agreed to participate actually did so, not all the selected pupils in that schools were necessarily tested, since a number will have been absent on the day the schools organised test sessions. The numbers of pupils who participated in the pencil and paper assessment of social subjects enquiry skills are as follows: 2536 P3 pupils, 2668 P5 pupils, 2340 P7 pupils and 2341 S2 pupils. At each stage these figures represent around 4% of the pupil population.

Tables B.3, B.4 and B.5 compare the compositions of the achieved samples at each stage with that of the pupil population, with respect to each of the stratification variables (the tables show the percentage distributions over stratification bands, for authority maintained schools only). The tables show that the smallest schools were slightly under-represented and the largest schools slightly over-represented in the P7 sample, that the smaller authorities were under-represented in the P7 sample, and that the larger authorities were under-represented in the S2 sample. As far as free school meals entitlement is concerned, the samples were all relatively balanced.

The 45 different enquiry skills tasks, nine per 5-14 level, were allocated to sample pupils in pairs within test booklets, so that each pupil took one task at each of two different adjacent levels and from two different enquiry skills outcomes (see Appendix C). In total, the numbers of pupils who attempted particular tasks were around 285 at P3, 300 at P5, 260 at P7 and 270 at S2, with an even representation of boys and girls in every case. At most two pupils in any particular survey school would have attempted the same enquiry skills task, so that the issue of any possible design effect due to pupil clustering at the task level is not relevant here. Margins of error for the percentage of pupils with 'basic skills', 'secure' or 'showing considerable strengths' (after applying the appropriate cut-off scores) at a level based on a single task are between five and six percentage points. Margins of error for % attainment scores averaged across the nine tasks at a level are around two percentage points.

The subsamples of schools that took part in the various practical activities were by design non-representative, in the sense that schools were deliberately selected for their convenient locations with respect to field officers' homes or own schools. Given the exploratory and innovative nature of the activities concerned, and their informal reporting, this phenomenon was considered acceptable on this occasion. The same comment applies to the questionnaire samples.

« Previous | Contents | Next »