The Scottish Government

« Previous | Contents | Next »

Listen

SECTION 4: THE PHASE 1 PILOT IN EAST AYRSHIRE OVER 3 YEARS

4.1 The Phase 1 EMA Pilot in East Ayrshire began in autumn 1999. In this section, we look at the impact of the East Ayrshire Pilot over three years. The analyses compare the participation and attainment in national qualifications of young people who attended schools in East Ayrshire with that of a control group of young people who attended schools in north and south Lanarkshire (Appendix 2).

Trends in Participation in National Qualifications

The 1st year after S4

4.2 Figure 4.1 shows trends in participation in national courses in the first post-compulsory year, based on SQA data. ⁶ Bars on the left of Figure 4.1 show the percentage of each cohort recorded as attempting at least one national course or unit, and bars on the right show the percentage attempting three or more national courses, or twelve or more national units, which indicates full-time participation. In each case, levels of participation in the pilot (East Ayrshire) and control areas (Lanarkshire) are broadly similar, but full-time participation rose to a greater extent in the pilot area than the control area in 2001 and 2002.

Figure 4.1: Participation in national qualifications in the 1st year after S4 in East Ayrshire pilot and Lanarkshire control areas

4.3 However, it is evident from Figure 4.1 that there are discontinuities in the data arising from the change in national qualifications in 2000 as a result of the Higher Still reforms. Participation in national qualifications by the two baseline cohorts, which pre-dated the Higher Still reforms, is much lower than for the three later cohorts. This apparent difference is misleading because some members of these cohorts were probably studying National Certificate modules, which are not part of the current national qualifications database.

4.4 Once again we must note that the coincidence of the introduction of Higher Still with the start of the first EMA pilot poses some difficulties for the evaluation. As already noted, Higher Still introduced a flexible system of units and courses to provide appropriate levels of difficulty and progression for students of all levels of ability. Prior to 2000, students with low Standard Grade attainment had a limited choice of courses - Highers were too difficult for them, and other provision tended to be based on National Certificate modules. Higher Still provides such students with a range of national units and courses including Access 1-3, and Intermediate 1 and 2. Thus, changes in participation must be seen in the context of the extension of national qualifications to a wider group of students.

4.5 The student group most affected by the introduction of Higher Still courses is also the group most likely to be affected by the EMA pilot. Poverty has a strong negative influence on Standard Grade attainment, and therefore many of the potential recipients of EMAs may have relatively low Standard Grade attainment, and their participation in post-compulsory education (if any) is likely to be in courses below the level of Higher Grade. The two reforms may be mutually supportive for students from low-income households, and we needed to bear this in mind throughout the evaluation.

4.6 A further issue for the evaluation is the apparently inflated record of participation in 2000, the first year of Higher Still. SQA experienced major difficulties with its administrative and computer systems in 2000, and this has evidently led to discrepancies in the data.

Figure 4.2: School and college participation in the 1st year after S4 in East Ayrshire pilot and Lanarkshire control areas (at least one national course or unit)

4.7 The change in national qualifications poses difficulties for analysis of participation at college compared with school (Figure 4.2). Whereas schools have tended to present students for national courses such as Higher Grade, colleges have tended to provide programmes made up of National Certificate modules, and these have now been transformed into courses made up of national units. Clearly, the absence of data on National Certificate modules for the two baseline cohorts is likely to mean that college participation is underestimated prior to Higher Still. However, Figure 4.2 shows that college participation was rather low in both pilot and control areas after the introduction of Higher Still. Most students studying national qualifications did so at school, with a small proportion studying at both school and college. Participation at college was slightly higher in the Lanarkshire control areas than in the East Ayrshire pilot.

4.8 We used statistical models to test differences between the pilot and control areas in full-time participation in national qualifications. The details are given in Appendix Table A4.1. The first model confirms that the likelihood of full-time participation:

• is higher for those with higher levels of Standard Grade attainment;
• is lower for young people who attended a school with high FME;
• is lower for those living in low SES areas;
• increased in 2000, 2001 and 2002 compared with previous years.

After controlling for these factors, there was no evidence of difference in full-time participation between the pilot and control areas in the years 1998-2000. However, there was a greater increase in the likelihood of participation in 2001 in the pilot area than the control area.

4.9 A further statistical model tested these factors among a sub-sample of students with low Standard Grade attainment. It confirmed that the likelihood of full-time participation by low attainers increased substantially from 2000 onwards, and this increase was over and above the general increase in full-time participation during this period. Low-attaining young people in the EMA pilot shared in the general increase in full-time participation, but did not show any additional increase compared to those in the control area.

The 2nd year after S4

4.10 Figure 4.3 shows trends in the proportion of young people who attempted national courses or units in the second post-compulsory year (the S6 stage at school or its equivalent at college). All participation in the two areas increased from an average of 37% in 1999 to 59% in 2003. The increase in participation from 2001 onwards was slightly higher in the Lanarkshire control area than in the East Ayrshire pilot area. However, full-time participation was higher in the East Ayrshire pilot area than in the control area throughout the period.

Figure 4.3: Participation in national qualifications in the 2nd year after S4 in East Ayrshire pilot and control areas

Figure 4.4: School and college participation in the second year after S4 (at least one national course or unit)

4.11 Figure 4.4 shows the percentage of young people who undertook at least one national course or unit at school and college in the second post-compulsory year. Although there was a slight increase in participation at school among the later cohorts compared with the baseline cohorts, the upward trend was very small compared with that seen for the first post-compulsory year. There was a substantial increase in the percentage of young people studying national qualifications at college, but levels of college participation increased to a greater extent in the Lanarkshire control area than in the East Ayrshire pilot area in 2002 and 2003.

4.12 We used statistical models to test differences between the pilot and control areas in full-time participation in national qualifications in the second post-compulsory year. The details are given in Appendix Table A4.2. The models confirm that the likelihood of full-time participation:

• is higher for those with higher Standard Grade point score, but is less closely related to number of credit level awards at Standard Grade than was the case for the first year after S4;
• is lower for young people who attended a school with high FME;
• was no different in 2001 compared with previous years, but increased in 2002 and 2003.

After controlling for these factors, there was some evidence that full-time participation was higher in the East Ayrshire pilot than the control areas, but this margin did not increase over the period to 2003.

4.13 A further statistical model tested these factors among students with low Standard Grade attainment. It confirmed that among low attainers the likelihood of full-time participation in the second year after S4 increased from 2001 onwards, and this increase was considerably greater than the general increase in full-time participation during this period. Low attaining young people in the EMA pilot were significantly more likely to study full-time in 1999-2000 than their counterparts in the control areas, and they also shared in the general increase in participation up to 2003. The gap between pilot and control areas was reduced in 2003.

Attainment

The first post-compulsory year

4.14 Nationally, the average number of national courses and units attempted and passed in the first year after S4 has increased substantially following the Higher Still reform (Appendix 1); these trends are evident in the pilot and control areas (see Appendix Table A4.3). Increasing average attainment is influenced by increasing levels of full-time participation in national courses, as well as increasing success-rates in these courses.

4.15 Part of the explanation for rising participation and achievement lies in the introduction by Higher Still of courses at different levels, providing appropriate progression routes for young people with different levels of prior attainment. Around one half of each cohort attempted at least one course at level 6 (Higher Grade). After 2000 there was a slight decrease in the proportion studying level 6, and a substantial rise in the proportions studying SCQF level 5 (Intermediate 2). These trends are evident in both pilot and control areas (Appendix Table A4.4).

4.16 Around 40% of each cohort achieved an award at SCQF level 6. In most years the proportion of pilot and control group achieving level 6 was very similar, but 2000 was an unusual year in which the proportion of the control group achieving level 6 was higher than that in the EMA pilot area. In both areas trends in achievement of level 5 (Intermediate 2) rose substantially from 2000 onwards, with the proportion in the EMA pilot area achieving level 5 outstripping that in the control area. (Appendix Table A4.5)

4.17 Overall attainment has risen substantially in both the East Ayrshire pilot and the Lanarkshire control areas as a consequence of increasing levels of participation and improving success rates. An overall point-score ⁷ is used to illustrate this (Figure 4.5). The point score is calculated from both volume of courses and units passed and level of those courses and units. It can be seen from Figure 4.5 that there has been an upward trend in attainment in the first post-compulsory year. Among the two baseline cohorts, the overall average attainment score was around 25 points, which is about one award at level 6 (Higher Grade). By 2002 the average attainment score had risen to 68 points, which is about 2.8 awards at level 6.

4.18 The upward trend in attainment point score is evident in both the pilot and control areas. However, the upward trend in the EMA pilot area between 2001 and 2002 is greater than that in the control area.

Figure 4.5: Average attainment score in the 1st year after S4, in East Ayrshire pilot and control areas

4.19 The lower part of Figure 4.5 shows trends in attainment by young people with low Standard Grade attainment (no credit-level awards). Although, as one would expect, average attainment by this group is lower than the overall average, the upward trends in attainment are broadly similar to trends for the whole sample. This suggests that young people with low Standard Grade attainment shared to some extent in the increased attainment from 2000 onwards.

4.20 We used statistical models to test differences between the pilot and control areas in the attainment point-score. The details are given in Appendix Table A4.6. The models confirm that attainment:

• is higher for those with higher levels of Standard Grade attainment;
• is lower for young people who attended a school with high FME;
• is lower for those living in low SES areas;
• increased in 2000, 2001 and 2002 compared with previous years.

After controlling for these factors, there is no statistically significant evidence that attainment in the East Ayrshire pilot increased in 2001 and 2002 to a greater extent than in the control areas, although this effect is close to being statistically significant.

4.21 A further statistical model tested trends in attainment among students with low Standard Grade attainment. There was a dramatic increase in attainment by this group in 2000, 2001 and 2002. In addition we find that in the EMA pilot area the average attainment of low-attainers increased to a significantly greater extent in 2001 and 2002 than did that of their peers in the control areas.

4.22 It is clear from Figure 4.2 that fewer young people studied NQ at college in East Ayrshire than the Lanarkshire control areas. To take account of this difference, a third statistical model controlled for differences associated with school and college (Model 3 in appendix table A4.6). It found that:

• young people attending college for all or part of their NQ studies had lower average attainment than their peers (note: the opposite effect is found in the analysis of phase 2 pilots: see section 5);
• there was no evidence that the lower attainment by students at college varied between pilot and control areas, and no evidence that the effect changed over time.

After controlling for the lower attainment of college students, we can identify more clearly the effect of EMA in increasing attainment by young people who stayed on at school. Attainment increased to a significantly greater extent in 2001 and 2002 in the EMA pilot areas than in the control areas after controlling for school/college attainment differences. In other words, the EMA effect is stronger if we focus on the effect of EMA on attainment in schools.

Attainment in the second post-compulsory year

4.23 Average numbers of courses/units attempted and passed in the second post-compulsory year increased between 1999 and 2003, with similar trends in control and pilot areas (Appendix Table A4.7). The success rate in national courses also increased over the period: in 1999 the average pass rate in East Ayrshire was 60% (compared with 65% in the control areas), and this rose to 80% in both pilot and control areas in 2003.

4.24 Level 6 courses were the most commonly studied in both pilot and control areas throughout the period (Appendix Table A4.8). In East Ayrshire there was a slight downward trend in participation in level 6 courses, from 27% in 1999 to 21.5% in 2003, but this was offset by the small increase in the proportion of young people attempting courses at level 7 or level 5.

4.25 In both first and second post-compulsory years (the S5 and S6 stages at school and their equivalents at college) there have been upward trends in participation and pass rates in national courses and units. Figure 4.6 shows the overall effect of these trends on young people's attainment, as measured by the attainment point-score. It shows a marked upward trend in attainment between 1999 and 2003. Although trends for the pilot and control areas between 1999 and 2001 were very similar, the upward trend in attainment in East Ayrshire between 2001 and 2003 is greater than in the Lanarkshire control areas. This may indicate a positive effect of EMA once the scheme had become securely established and embedded in local authority and school practice.

Figure 4.6: Average attainment of each cohort by the end of the second post-compulsory year

4.26 The lower part of Figure 4.6 shows trends in attainment by young people with low Standard Grade attainment (no credit-level awards). As one would expect, average attainment by this group is lower than the overall average, and the upward trend in attainment is less pronounced. Nevertheless, in the EMA pilot area young people with low Standard Grade attainment increased their attainment of national qualifications to a greater extent than their peers in the control areas.

4.27 We used statistical models to test differences between the pilot and control areas in the attainment point-score combined over the two-year period. The details are given in Appendix Table A4.10. The models confirm that attainment:

• is higher for those with higher levels of Standard Grade attainment;
• is lower for young people who attended a school with high FME;
• is lower for those living in low SES areas;
• increased substantially in 2001, 2002 and 2003 compared with previous years.

After controlling for these factors we found that for the two baseline cohorts attainment in the East Ayrshire pilot did not differ from that in the control areas. However, there is strong evidence that in 2002 and 2003 attainment in the East Ayrshire pilot increased to a greater extent than in the control areas.

4.28 A further statistical model tested trends in attainment among young people with low Standard Grade attainment. It found that in both pilot and control areas young people with low Standard Grade attainment increased their attainment in 2001 - 2003. The increase among low attaining young people in East Ayrshire was significantly greater in 2003 compared with their counterparts in the control areas.

The effects of the Phase 1 EMA Pilot: Summary

4.29 There are clear positive effects of the introduction of EMA combined with new national qualifications. However, the introduction of new national qualifications in the first year of the EMA pilot in East Ayrshire complicates assessment of the independent effects of EMA in 1999-2000.

4.30 If attainment in the first and second post-compulsory years is combined it is evident that attainment increased steadily between 1999 and 2003. While young people in both East Ayrshire and the control areas shared in the overall increase, by 2002 and 2003 average attainment in East Ayrshire had risen significantly higher than in the control area. Young people with low Standard Grade attainment in East Ayrshire shared in this EMA advantage. In other words, there is evidence of a positive EMA effect on attainment over the two years after S4, both overall and also among low-attainers.

4.31 In the first post-compulsory year overall levels of attainment increased dramatically after the introduction of the Higher Still courses, and young people in both East Ayrshire and the control areas shared in the increase. A slight attainment advantage in the first year after S4 was evident in East Ayrshire in 2001 and 2002. In particular, average attainment by young people with no credit level awards at Standard Grade increased to a greater extent by 2002 in East Ayrshire than in the control areas. In other words, there is evidence of a positive EMA effect on attainment in the first post-compulsory year, both overall and especially among low-attainers.

4.32 There is some evidence of a positive EMA effect on overall full-time participation in the first post-compulsory year. Young people in the East Ayrshire pilot and control areas shared in the general increase in full-time participation in the first post-compulsory year after the introduction of Higher Still courses in 2000. The increase in full-time participation was stronger among young people with low Standard Grade attainment.

4.33 Levels of full-time participation in the second year after S4 were generally higher in the East Ayrshire pilot than in the control areas, but although participation increased steadily from 1999 to 2003 the difference between the East Ayrshire pilot and the control areas was evident in 1999, before the start of the EMA pilot, and did not change. In other words, there is no evidence that the EMA has had either a positive or a negative effect on overall participation in the second post-compulsory year.

4.34 The likelihood of participation in the second post-compulsory year by young people with no credit level awards at Standard Grade was lower than for other young people, but increased steadily between 1999 and 2003 in line with overall participation rates. On average low attaining young people in the EMA pilot area were more likely than their peers in the control area to be studying full-time in the second year after S4. However, the difference between the East Ayrshire pilot and the control areas was evident in 1999, before the start of the EMA pilot, and although the difference was to some extent maintained, it was reduced a little in 2003. In other words, the analysis provides no evidence that the EMA has had either a positive or a negative effect on participation by low attainers in the second post-compulsory year.

4.35 On average, young people who were entered for national qualifications at college for all or part of their studies tended to have lower attainment than those whose national qualifications were all at school. The attainment difference arises mainly from the different curriculum focus of colleges compared with schools: students at college tend to study programmes of national units rather than national courses, and progression is often "horizontal" to include study of new vocational subjects compared with the "linear" progression within academic subjects that is most common in schools. If we take account of differences in attainment by college students we find that attainment at school in the EMA pilot area rose to a significantly greater extent than attainment at school in the control area. In other words there is a very strong positive EMA effect on attainment at school.

« Previous | Contents | Next »