The Scottish Government

« Previous | Contents | Next »

Listen

SECTION 3: DESIGN AND DATA SOURCES

Comparative samples

3.1 The evaluation compares the EMA pilots, with "control areas" based on schools with similar intake characteristics (see Appendix 2). Comparing trends in the pilot areas with "control" areas enables some progress to be made towards distinguishing the impact of the EMA from the effects of other initiatives and trends.

3.2 The control group for the evaluation of the Phase 1 EMA Pilot in East Ayrshire consisted of selected schools with similar intake characteristics from neighbouring authorities in North and South Lanarkshire. ²

3.3 The selection of a control group for the evaluation of the Phase 2 EMA Pilots in Glasgow, Dundee and West Dunbartonshire proved more difficult because there are no individual education authorities with sufficiently similar characteristics to provide obvious comparators. The EMA Pilots were deliberately located in areas of socio-economic deprivation with low levels of educational participation, and there are greater concentrations of poverty in Glasgow than elsewhere in Scotland. To overcome this difficulty, a comparative sample of 48 schools was selected from various non- EMA areas of Scotland on the basis of their similarity to schools in the EMA pilot areas in terms of:

• % living in areas of low socio-economic status;
• local area deprivation;
• % with free-meal entitlement;
• % with low Standard Grade attainment;
• average number of credit-level awards at Standard Grade;
• S4 Standard Grade point-score.

Data on National Qualifications

3.4 The first evaluation of the East Ayrshire pilot included a postal survey of young people, and case studies. By contrast, this second stage of the evaluation is entirely quantitative, and is based exclusively on qualifications records provided by the Scottish Qualifications Authority ( SQA). The SQA data consisted of administrative records of entries and awards in national units and courses. Background data are limited to the information held by SQA for the purposes of examination and certification: examination centre (school, college or other provider); sex; date of birth; and postcode of home address (See Appendix 3).

3.5 The data on national qualifications relate to sequential cohorts of young people, defined in terms of the year in which they sat their Standard Grade examinations and completed the S4 year stage. The analysis focuses on subsequent participation and attainment by each cohort in the first year after S4 (S5 or its college equivalent) and the second year after S4 (S6 or its college equivalent).

3.6 The data used for the evaluation of the East Ayrshire pilot relate to five cohorts of young people who completed S4 Standard Grade examinations in 1997, 1998, 1999, 2000 and 2001 respectively (Table 3.1).

Table 3.1: Summary of EMA cohorts in East Ayrshire pilot and control areas

3.7 The first two cohorts, (where compulsory schooling ended in sessions 1996-7 and 1997-8) provide baseline data on young people before the introduction of the EMA pilot in East Ayrshire. For both baseline cohorts, the key transition from the S4 stage at school to post-compulsory education or other destinations pre-dated EMA. However, the transition from S5 to S6 (or equivalent stages at college) of the second of these two baseline cohorts coincided with the start of the East Ayrshire EMA pilot. The three later cohorts, which ended compulsory schooling in sessions 1998-99, 1999-2000 and 2000-01, provide data on young people whose post-compulsory transitions took place during the period of the East Ayrshire EMA pilot.

3.8 The data used for the analysis of Phase 2 pilots relate to three cohorts of young people who completed S4 Standard Grade examinations in 1999, 2000 and 2001 respectively (Table 3.2).

Table 3.2: Summary of EMA cohorts in Phase 2 pilot and control areas

3.9 The Phase 2 EMA pilots in Glasgow, Dundee and West Dunbartonshire started in autumn 2001, and therefore members of the third cohort, who completed S4 in 2000-01 and entered S5, (or their college equivalents) in 2001-02, could have been eligible for an EMA. The first two cohorts, (where compulsory schooling ended in sessions 1998-9 and 1999-00) provide baseline data on young people before the introduction of the Phase 2 EMA pilot. For the first two cohorts, the key transition from the S4 stage at school to post-compulsory education or other destinations pre-dated EMA. However, the transition from S5 to S6 (or equivalent stages at college) of the second cohort coincided with the start of the Phase 2 EMA pilots.

Issues arising from use of SQA data

3.10 Investigation of the SQA data on national qualifications formed a large part of the preparatory work of the evaluation. Measures derived from SQA data are necessarily very complex because the qualifications themselves comprise a complex system of national units and courses, and the complexity was increased for this evaluation by the overlap of Higher Still with previous units and courses.

3.11 Whereas schools tend to present students for national courses (such as Higher and Intermediate 2), colleges tend to provide programmes made up of national units. In view of the need to compare the effects of EMA on participation and attainment in both types of institution, it was necessary for the purposes of this study to estimate equivalence between both types of qualification. Calculations of equivalence assume that 4 national units equalled 1 national course at the same level (and 3 Access units equalled 1 Access cluster).

3.12 Two measures of participation are used in this analysis: "all" is a comprehensive measure including all students who were entered for one or more national units or courses at the relevant time, and "full-time" which represents students who were entered for three or more national courses, or 12 or more national units. We must be clear that these measures represent "participation in national qualifications", and do not necessarily coincide with measures of "participation in education" used in Statistical Bulletins.

3.13 For this evaluation, measures of attainment derived from SQA data have been defined in terms of levels within the Scottish Credit and Qualifications Framework ( SCQF) (Figure 3.1). Measures of attainment include highest SCQF level of attainment, and a point-score which is the product of SCQF level and the volume of courses and units passed.

Figure 3.1: The Scottish Credit and Qualifications Framework

Socio-economic data about local areas

3.14 No data on household income are available for this research, so we cannot directly identify young people who would be eligible for EMA. As a proxy for household income we used the MOSAIC classification to derive indicators of the socio-economic status ( SES) of the local-area in which young people lived. MOSAIC is normally used for profiling neighbourhoods as a marketing tool. It is a classification system that divides the UK into postcode-based neighbourhood types, based on Census data, electoral rolls, commercial behaviour data, credit information and other marketing data.

3.15 Two of the mosaic groups describe areas of public housing:

• Low Rise Council;
• Council Flats.

3.16 In view of the links between rented housing and low socio-economic status, these two groups provided the main definition of areas with low SES. To these were added the small category of "Victorian low status" and "institutional areas". Applying this definition of low SES to the whole population of young people from the national SQA dataset we found that some 44 per cent came from low SES areas.

Figure 3.2: Average number of credit level awards at S4 Standard Grade by Mosaic categories

3.17 Figure 3.2, shows the relationship between the MOSAIC housing categories and Standard Grade attainment. On average, young people living in "Council Flats" achieved 1.8 credit level awards at the end of S4, compared with those in "High-Income Families" who gained five credit-level awards on average, and "Mortgaged Families" who gained 3.8.

3.18 Figure 3.3 shows that almost half (46 percent) of young people living in low SES areas gained no credit level awards at Standard Grade, compared with just 22 percent in other areas. On the other hand, not all young people in low SES areas had low attainment - some 20 percent of young people in low SES areas gained five or more credit level awards - but this proportion is much lower than for young people living in other areas.

Figure 3.3: Percentage of young people achieving credit level awards at S4 Standard Grade

3.19 An additional measure of SES used for the analysis is the Scottish Area Deprivation Index that has been calculated at post-code sector level (Gibbs et al 1998). The disadvantage of this deprivation index is that post-code sectors cover quite large areas and may conceal considerable internal differences. MOSAIC, on the other hand, is based on much smaller areas covered by individual postcodes.

3.20 Information about the context of schools young people attended in S4 is provided partly by the measure of free-meal entitlement ( FME) recorded in the 2000 school census. FME is commonly used as an indicator of low household income, but it is not wholly satisfactory for this purpose (Croxford 2000).

Characteristics of EMA Pilot Areas PRIOR TO THE INTRODUCTION OF EMA

3.21 In all four pilot areas the proportions of young people living in areas of low SES were substantially higher than average: 63.6 percent of those from Glasgow, 60.3 percent of those from West Dunbartonshire, 53.4 of those from East Ayrshire and 52.1 percent of those from Dundee were living in low SES areas, compared with 40.4 percent of young people in the rest of Scotland ⁴ (Table 3.3). Similar patterns of relative poverty in the EMA pilot areas are shown by the Scottish Area deprivation Index and average free-meal entitlement (Table 3.4).

Table 3.3: Socio-economic composition of EMA Pilot areas compared with rest of Scotland (% of S4 cohorts 1999 - 2001 combined)

Table 3.4: Other indices of poverty (means)

3.22 Most young people in Scottish secondary schools sit Standard Grade examinations at the end of fourth year (the S4 stage) before they reach the end of compulsory schooling at age 16. Their performance at Standard Grade is one of the most important factors in determining whether they will stay-on in full-time education, since it is usual for schools to require students to have a pass at Standard Grade credit level in any subject they wish to pursue at Higher Grade. Nationally, one third of young people in S4 in session 1998-9 achieved five or more awards at credit level, one third achieved between one and four awards, and just one third gained no awards at credit level. For the purposes of this research we have taken this latter group, with no credit level awards as the group with "low Standard Grade attainment". Average levels of S4 Standard Grade attainment (Table 3.5) were substantially lower in Glasgow, Dundee and West Dunbartonshire than elsewhere in Scotland. In these three pilot areas less than a quarter of young people gained five or more credit level awards, compared with over a third in the rest of Scotland.

Table 3.5: Standard Grade attainment in 1999 (% of cohort 1)

3.23 Before Phase 2 of EMA, young people in Dundee and West Dunbartonshire were very similar to those elsewhere in overall levels of participation in national units and courses in the first year after S4, but participation in Glasgow was substantially lower than elsewhere, at 75.1 percent (Table 3.6). In Dundee the proportion of young people studying at college (15.7 percent) was much higher than elsewhere in Scotland (8.4 percent), while in East Ayrshire participation at college (6.9 percent) was lower than elsewhere.

Table 3.6: Participation in any national units or courses in the first year after S4 by cohort 2 ⁵ in 2000/1

3.24 Before the introduction of EMA, overall participation in national units and courses in the second year after S4 was lower in all the EMA pilot areas than in non- EMA areas (Table 3.7: Glasgow 50 percent, Dundee 53 percent and West Dunbartonshire 55 percent vs. 59 percent in the rest of Scotland).

Table 3.7: Participation in any national units or courses in the second year after S4 by cohort 1 in 2000/1

3.25 If we shift the focus to the destinations entered by young people leaving school, we find further differences between the pilot areas. Statistics from the Careers Service on the destinations of school leavers in 2000/01 (Table 3.8) show that the proportion of young people entering full-time education in East Ayrshire was similar to the national figure, but that the proportion in Glasgow, Dundee and West Dunbartonshire was lower than elsewhere. In East Ayrshire more school leavers entered training and fewer entered employment than was the case elsewhere, while in Dundee and Glasgow more young people were not in education, employment or training ( NEET).

Table 3.8: Destinations of school leavers 2000/01

Source: http://www.scotland.gov.uk/stats/bulletins/00131-04.asp

« Previous | Contents | Next »