The Scottish Government

Listen

Insight 11
An Evaluation of the Higher Still Reforms

ISSN 1478-6788 (Print)
ISSN 1478-6796 (Online)

This document is also available in pdf format (137k)

Welcome to Insight

Insight is a publication of the Research, Economic and Corporate Strategy (RECS) Unit, one of four units in the Information, Analysis and Communication Division, which is responsible for providing analytical services within the Scottish Executive Education Department (SEED). RECS is a multidisciplinary unit (consisting of researchers and economists) which undertakes and funds economic analysis and social research in the fields of: school education; children, young people and social work; architecture; and tourism, culture and sport.

The Scottish Executive is committed to the use of sound evidence in the development
of policy and practice as well as in the evaluation of policy and its implementation. We therefore want to disseminate the results of research that SEED has undertaken and funded, in a manner that is accessible, interesting and attractive.

Insight aims to present the essence of research projects in a format that will be useful and informative for practitioners, policy makers, parents, academics, and anyone else who has an interest in economic and social research in these areas.

The views expressed in this Insight are those of the author(s) and do not necessarily reflect those of the Scottish Executive or any other organisation(s) by whom the author(s) is or are employed.

Copyright © December 2003, Scottish Executive Education Department

Insight may be photocopied for use within your own institution.

A limited number of additional copies can be obtained from the Dissemination Officer, Research, Economic and Corporate Strategy Unit, Scottish Executive Education Department, Victoria Quay, Edinburgh EH6 6QQ (telephone 0131-244 0092). Copies of Insight and our other publications can also be downloaded from our website: www.scotland.gov.uk/insight

An Evaluation of the Higher Still Reforms

Roger Mullin (Inter-ed Ltd)

In 1990 the Secretary of State for Scotland established a Committee, chaired by Professor John Howie of St. Andrews University, with the task of reviewing the aims and purposes of courses and of certification in the fifth and sixth years of secondary school education in Scotland.

The Howie Committee reported on 5 March 1992, producing an analysis of weaknesses in the system and articulating a set of aims for the Scottish education system which attracted much support in the subsequent consultation conducted by The Scottish Office. The aims are:

1. The system of qualifications should be easy to understand and use.

2. Schools should effectively develop core skills such as communication, numeracy, information technology, problem solving and working with others.

3. High standards of attainment should be sought.

4. Education should encourage breadth of learning as well as depth for individual students.

5. There should be even progression in moving from one level of qualification to the next.

6. Across different types of qualifications, curriculum and assessment arrangements should be as compatible as possible.

7. There should be a process of expanding and rationalising qualifications to meet changing labour market requirements.

8. Recognised qualifications should be available for all levels of ability.

9. New qualifications should build on the existing system.

In March 1994, Higher Still - Opportunity for All was published by the Government, setting out its views on how reforms should be tackled in light of these aims. These have been known collectively as the Higher Still reforms. The report stated that a framework of levels to assist students to make progress should be established to satisfy the aim of having appropriate qualifications for all levels of ability. Eventually, this led to the current system of Access, Intermediate 1, Intermediate 2, Higher and Advanced Higher, and in turn these have become styled as National Qualifications.

The aim of this study is to evaluate the impacts of the reforms from the perspective of key professional groups and stakeholders with a view to establishing the extent to which the Howie aims are being met. The study has involved two phases. The first, undertaken in 1999, captured views of the system and the likely impact of the reforms prior to the implementation of the Higher Still reforms, and the second, undertaken in 2003, captured views of the new system after all the reforms had been put in place. Each study was based on two surveys of opinion: the first obtaining the perceptions of school and further education college staff, and the second obtaining the views of employers, careers guidance staff, higher education admissions officers and vocational trainers.

Both in 1999 and 2003, more than 1100 school teachers and 700 further education college lecturers took part in the surveys. In addition, a series of key "stakeholder" surveys was undertaken in both years. These involved over 200 employers, admissions staff from each of Scotland's Higher Education institutions, regional representatives from Careers Scotland in 2003, and careers companies in 1999 (referred to in the following tables as careers guidance staff), and a group of over 20 training providers.

The analysis of this data is presented thematically. The eight major themes addressed are:

• The importance of the aims of the Higher Still reforms.
• Support issues (including resources and staff development).
• Core skills development.
• Progression and choice.
• Attainment and standards.
• Assessment process and accuracy.
• Esteem and recruitment values.
• Overall judgements and satisfaction levels.

It should be borne in mind that this study presents snapshots of opinion at different times, the most recent being early 2003. The work of developing National Qualifications however is a continuous process, with much recent work being undertaken to streamline and simplify the new system. It is recognised that some of the criticisms reported are already being addressed through such continuous development work.

Findings

The importance of the aims of the Higher Still reforms

All groups were asked to rate the nine Howie aims on a scale from 0-10; the larger the number, the stronger the rating. The results showed that in both 1999 and 2003, all respondents (school, college, employer, HE admissions, careers guidance and training provider respondents) considered the nine Howie aims to be of considerable importance. Table 1 provides the mean ratings given by college respondents, which are typical of the ratings received across all groups. The aims are presented in rank order of ratings in 2003, but as can be seen all aims are rated highly.

Table 1: College respondents' rating of aims for importance in 1999 and 2003

Aims 1 to 3 above attracted particular comment. Employers and Higher Education institutions stressed the importance of communication skills and literacy in particular within core skills, and many commented that any system has to be usable by a range of stakeholders, thereby emphasising the importance of having a system that is easy to understand and use. There was also agreement from all respondent groups that education systems prepare individuals for work, but they also do much more than this.

Support issues

Support issues, such as equipment and materials resources, and staff development, were matters creating much comment in 1999 and therefore merited specific consideration in the surveys of opinion. In 1999, expectations about staffing, equipment, materials and in-service training needs varied among respondents. For example:

• Both school and college staff expected the introduction of Higher Still to increase the amount of time spent on assessment;
• English teachers within schools, when compared to other subject specialists, expected to see the greatest overall impact of the Higher Still reforms;
• There were low expectations among school and college respondents that adequate resources in terms of staffing, equipment and materials would be made available to support the Higher Still curriculum;
• English teachers were more pessimistic than other subject specialists regarding adequate resources being made available;
• Both school and college staff had high expectations that the National Assessment Bank (NAB) would be well used and would provide high quality assessment instruments; and
• Both school and college staff expected to receive adequate training in preparation for the introduction of Higher Still.

In 2003, staff in both schools and colleges were more positive about resources actually received compared to their expectations in 1999. School staff, for example, felt that investment in physical resources had outstripped original expectations in 1999 (Table 2). English teachers were particularly wary of resource availability in 1999, but by 2003 had become more positive and had moved closer to the views of other subject specialists, while still remaining critical. In 2003, school and college staff were generally positive about having access to a good range of assessment materials, and were content with the quality of both internal and external assessment.

Table 2: School respondent mean ratings of statements on resources in 1999 and 2003

However, in-service training was poorly rated in both 1999 and 2003, and school staff were less content than college staff with the adequacy of training in assessment skills. In 2003, two concerns were expressed about NAB instruments of assessment, namely that they were not always good predictors of ultimate performance and that they ran the risk of "spoon-feeding" pupils.

Core skills

The Howie Committee believed that schools should effectively develop core skills. This is a belief shared by all those responding to the surveys. Indeed, the issue of core skills attracted considerable comment. According to employers, Higher Education institutions, careers guidance specialists and training providers, information technology skills are the most usefully developed of the core skills among school leavers and, furthermore, all bar Higher Education institutions see some improvement in 2003 compared to the 1999 position (Table 3). Employers and Higher Education institutions rate students' proficiency in communication skills lowest among the core skills in 2003, while students' numeracy skills are perceived as lowest by careers guidance specialists. Training providers rate communication and numeracy skills equal lowest with problem solving skills.

Table 3: Stakeholder ratings of school-leaver core skills

The main concern among stakeholders in both 1999 and 2003 was the weakness of communication skills. Comments received from employers, HE admissions staff, careers guidance staff and training providers suggested that literacy standards among school leavers were, in many cases, inadequate for either the world of work or Higher Education study. School and college staff also share these concerns and provide some explanation for the weakness. In the 2003 study in particular, school and college staff expressed concerns about the treatment of core skills as a cross-curriculum focus. They felt that this results in a general lack of priority being afforded to them. In addition, school and college staff felt that the reforms placed too much emphasis on assessment at the expense of teaching and learning.

In sum, while all groups agree that new National Qualifications provide a basis for the development of core skills, at the same time they recognise that there is scope for further improvement, especially in raising literacy standards in preparation for work or further study.

Progression and choice

One of the aims of the Higher Still reforms was to ensure that qualifications provide a strong basis for moving on to either higher levels of study or vocational training. For most respondents, Highers remained the gold standard for preparing students for work or further study. However, ratings among all groups demonstrated that, in general, save for one important matter we report on below, the system was not seen to have enhanced the smooth progression of pupils since 1999.

The one area where change for the better was clearly identified was in providing a platform for the progression of less academic students. Tables 4 and 5 report on ratings of a single, but telling, statement in both 1999 and 2003 among all groups.

Table 4: School and college ratings of statement on qualifications for all ability levels

Table 5: Stakeholder ratings of statement on qualifications for all ability levels

With the exception of Higher Education institutions, which deal predominantly with higher achieving school leavers, all groups in the study feel that the new system has delivered considerable improvements on the previous system in terms of providing recognised qualifications for all levels of ability.

In 2003, all survey groups rated Highers and Advanced Highers particularly strongly in terms of providing a good stepping stone to higher levels of academic study. School staff rated Intermediate 2 qualifications on a par with Standard Grades, and college staff rated Intermediate 2 qualifications above Standard Grades in terms of providing a good stepping stone to higher levels of study.

While Higher Education admissions staff and careers guidance staff rated Intermediate 2 qualifications on a par with Standard Grades, employers and training providers rated Intermediate 2 considerably lower than Standard Grades in terms of providing a stepping stone to higher levels of study. Employers also rated Intermediate 2 qualifications below Standard Grades in terms of providing a stepping stone to vocational training.

Some school and college staff expressed concerns that there is not always even progression between levels, and that this is making progression difficult for some students.

A further issue raised by stakeholders was that pupils leaving school are not ready for the workforce. However, employers recognised that some of the deficiencies of school leavers which inhibit progression, such as behaviour and attitudes, are a function of society as a whole and not simply of the school or college sectors.

Attainment and standards

Standards of attainment were of considerable importance to all groups of respondents, and attracted much comment. Among all survey groups in 2003, Highers and Advanced Highers were seen as the exemplars for encouraging high standards of attainment. Among school and college staff, the new Intermediate 2 qualification achieved broadly similar value ratings to that of Standard Grade. Despite these positive views, however, all groups see scope for further raising attainment standards.

Concerns were expressed about the reforms and their impact on attainment levels in certain areas. For example, school and college staff commented on the inconsistencies of standards across subjects, departments and schools. Concern was also expressed that the pressure of assessments was compromising the quest for higher standards. It was argued among teaching staff that, with an increased emphasis on teaching for assessment, the quality of teaching and standards were put under pressure. A university admissions officer also agreed, claiming:

The new qualifications are too complicated, (and rely on) too much use of continuous assessment. Students require time to absorb information so that it is retained for later studies.

Respondents from both the Higher Education sector and the schools sector commented that effective learning requires time for reflection and testing out understanding. This process is not helped if a course of study becomes a series of assessment hurdles.

According to a number of focus group participants, the necessity for bi-level composite classes in some schools was an added difficulty as it increased the complexity of the teaching process and thus potentially compromised standards.

In terms of standards, the view among some stakeholders in 1999 was that they would rise with the reforms, ranging from 35% of training providers to 67% of careers guidance staff taking such an optimistic view (Table 6). By 2003, however, a much smaller percentage of respondents believed standards had actually improved, ranging from a low of 18% among Higher Education admissions staff to a high of 23% among careers guidance staff, suggesting that earlier expectations had not been fulfilled.

Table 6: Percentage of stakeholders claiming different levels of expectations (1999) and actual (2003) standards under National Qualifications

That said, of those respondents in 2003 who did perceive a change in standards, on balance more employers, HE institutions and careers guidance staff thought standards had been raised rather than lowered, and only training providers thought on balance standards had been lowered rather than raised.

In comparing standards in 2003 with those of the system in 1999, one university admissions officer commented:

While it would be difficult to argue that across the range of Higher and Advanced Higher subjects there has been a lowering of standards, there is a widely-held view that, in certain subjects, either the curriculum or the attainment level of university entrants no longer meets the requirements of particular degree courses.

Overall, the strongest perception is of standards not having changed as much as was hoped for.

Assessment process and accuracy

In 1999, school and college respondents had specific concerns about the assessment system proposed under Higher Still. These included a concern about assessment accuracy, and the amount of extra work becoming burdensome. They had positive expectations that the National Assessment Bank would provide high-quality assessment instruments, and that training would be introduced to prepare them for teaching the new curriculum.

Comments, particularly those received in focus groups held in early 2003, suggested that administration associated with assessment was burdensome, and a number of comments painted a picture of increased stress for some students because of the amount of assessment. There was, however, some recognition of recent changes being implemented to improve the situation. Indeed, recent administrative and course/unit changes have been designed to address such lingering apprehensions.

Perceptions in 2003 regarding the accuracy of assessments were much in tune with expectations in 1999. Although a large majority of respondents thought it very likely that students deserving particular awards would attain them (true positives) and also that those not deserving particular awards would not attain them (true negatives), there was a significant recognition of possible errors.

For example, in 2003 25.3% of school respondents and 31% of college respondents thought that errors were occurring that led to students achieving awards who were not, in fact, up to standard (false positives), compared to 17.6% of school and 10.9% of college respondents who thought that errors were occurring that led to students not achieving awards when, in fact, they were up to standard (false negatives).

The survey results therefore showed that there is an expectation that there will be errors in the system, and that these are more likely to result in someone who is not up to standard attaining an award, rather than a deserving student being denied an award. In short, relatively more false positives than false negatives.

If this is a fair reflection of the results of the assessment process, it is likely to continue to undermine perceptions of standards over time. However, there are no data with which to assess perceptions of the accuracy of assessment of the old system, and therefore it is not possible to fairly compare the new with the old in terms of accuracy.

One area in which pre- and post-reform systems can be compared is the kind of assessment students undertake. School respondents believed the mix of internal and external assessment provided greater benefits to pupils under the 1999 system than under the 2003 system, not least because it is now more complex and the workload is too heavy. For these reasons, school respondents do not believe the new National Qualifications system has satisfied the aim of being simple and effective.

Esteem and recruitment values

Both in 1999 and 2003, employers and Higher Education institutions placed great importance on school qualifications for recruitment purposes. Furthermore, and notwithstanding many critical comments on specific issues, some qualifications (notably Highers) continue to be highly regarded. The surveys found that some qualifications are more highly valued than others. (Note that in some points to follow we refer to qualifications such as Scottish Vocational Qualifications (SVQs) which are not part of the National Qualification framework. This is done for comparative purposes.) Specific conclusions we can draw from the surveys include:

• The usefulness of Standard Grades, Highers, SQA units, SVQs and Higher National awards has changed little between 1999 and 2003 according to employers and HE institutions.
• More than twice as many employers in 2003 rated Standard Grades as very useful for recruitment purposes compared to Intermediate awards.
• In 2003, 100% of Higher Education respondents rated Standard Grades as either very useful or of some use for student recruitment, compared to 87% for Intermediate 2.
• More than twice as many employers and more than twice as many Higher Education respondents rated Advanced Highers very useful in 2003 compared to the Certificate of Sixth Year Studies in 1999.
• In both 2003 and 1999, only Higher National awards attracted higher ratings of usefulness than Highers among employers.
• Highers were in 1999, and remain in 2003, the most useful qualification for HE entry.

One of the problems inherent in measuring the recruitment value of the new qualifications, such as Intermediate 2, is the lack of understanding of the reforms among stakeholders, especially employers. Indeed, this helps explain why many employers rated Intermediate 2 qualifications as less useful than Standard Grades. This, in part, is a function of insufficient marketing of the qualifications among employers and, until such issues are addressed, it may be disadvantageous for young people if Standard Grades are entirely replaced by Intermediate qualifications.

In general, therefore, despite the fact that teachers believe the esteem of qualifications among employers and Higher Education institutions has seriously declined in recent years, this was not found to be the case in this study. Employers and others do want to see improvements, but they continue to value key qualifications.

Overall judgements and satisfaction levels

In evaluating whether the Higher Still reforms have met the Howie aims, the study found the main strength of the new National Qualifications was in providing qualifications for all levels of ability. The main weakness, however, was that the reforms have not provided a system that is easy to understand and use. In most other cases, there had only been small movements in ratings comparing the situations in 1999 and 2003.

More generally, combining the weighted importance given by different groups to the Howie aims with the rating of the qualification system against these aims, it is possible to quantify overall levels of satisfaction between the two systems before and after the Higher Still reforms. Tables 7 and 8 provide satisfaction levels expressed as a percentage for each group.

Table 7: Comparison of school and college satisfaction ratings in 1999 and 2003

Table 8: Comparison of stakeholder percentage satisfaction ratings in 1999 and 2003

As can be seen from Tables 7 and 8, college staff, employers, careers guidance staff and training providers feel that the system in 2003 is modestly better than the system in 1999. For school and Higher Education staff, the position is reversed with falls in satisfaction levels.

These represent neither the substantial gains nor the dramatic declines claimed by some. They do suggest that, notwithstanding the real gains, such as providing qualifications for all levels of ability, much work remains to be done to improve perceptions.

TheInsightSeries

1. Classroom Assistants: Key Issues from the National Evaluation
2. The Impact of ICT Initiatives in Scottish Schools
3. Moving On to Primary 1: An Exploratory Study of the Experience of Transition from Pre-School to Primary
4. Accelerating Reading and Spelling with Synthetic Phonics: A Five Year Follow Up
5. Assessment of Benefits and Costs of Out of School Care
6. Meeting the Needs of Children from Birth to Three: Research Evidence and Implications for Out-of-Home Provision
7. Key Findings from the National Evaluation of the New Community Schools Pilot Programme in Scotland
8. Scottish Qualification for Headship: Key Issues from the Evaluation
9. The Sitter Service in Scotland: A Study of the Costs and Benefits
10. Awards in Early Education, Childcare and Playwork: A Qualifications Framework for the Future
11. An Evaluation of the Higher Still Reforms

If you have views on Insight or wish to find out more about SEED's research and economics programme, please contact the Research, Economic and Corporate Strategy Unit, Scottish Executive Education Department, Victoria Quay, Edinburgh EH6 6QQ.