< Previous | Contents | Next >
A Review into Exam Results Issues Concerning the Scottish
Qualifications Authority
1. EXECUTIVE SUMMARY
1.1 Introduction
During the first week of September 2000, Deloitte & Touche
was commissioned by the Scottish Executive to undertake a "review into
exam results issues concerning the Scottish Qualifications Authority (SQA)".
The scope of the review was to "… identify any administrative, procedural
or management weaknesses and provide a set of clear recommendations on how
to ensure that these problems do not recur in future years1.
The review was restricted to a consideration of the current year's examinations
- 'Diet 2000'.
The key challenges facing the SQA for Diet 2000 were to implement:
- the new 'Higher Still' qualification;
- 'Core Skills' and a single Scottish Qualifications Certificate; and
- a new computer system (APS2)
to support awards processing.
A number of problems were encountered by the SQA during 1999/00
which resulted in delayed and missing certificates and incomplete and inaccurate
results. The SQA has confirmed (9 October 2000) that 2.7% of results and 16,748
candidates were affected by missing or incomplete data.
The following groups were consulted in the course of the
review:
- SQA staff, management and Board members;
- suppliers of services to the SQA;
- Scottish Executive staff;
- stakeholder organisations;
- Centre staff and suppliers; and
- pupils and students.
The review findings are detailed in chapter
4, chapter 5 and chapter
6 in terms of:
- management and organisation;
- operational processes; and
- information systems.
Recommendations are given in chapter
8.
1.2 Findings
Whilst data handling issues were, ultimately, a critical
flaw in Diet 2000, it would be inappropriate to conclude that this was the
only deficiency in performance or that the arrangements for data handling
were the single cause of the problems. No one single problem lies at the root
of the SQA's poor performance this year. Rather, there were weaknesses and
difficulties in a number of interlinked management, process and information
systems areas which, cumulatively, overwhelmed the delivery of National Qualifications
for Diet 2000. The various factors are represented diagrammatically in chapter
7 of the report and the key findings are summarised below.
The review revealed a wide range of managerial and organisational
weaknesses within the SQA, some of which have existed since it was first constituted.
These include, at times, a relative lack of breadth and depth of management
skills and experience within the Senior Management Team, most significant
latterly in respect of operational management experience, and the SMT's failure
to:
- lead the effective merger of the Scottish Examination Board (SEB) and
Scottish Vocational Education Council (SCOTVEC) to create the new organisational
entity - the SQA;
- design a management and organisational structure based clearly on the
requirements of the new ‘business’ rather than rely on the inherited
structures and practices of the predecessor organisations;
- develop appropriate strategic and operational plans for the effective
and efficient functioning of the organisation.
Specifically, with regard to Diet 2000, the SQA's Senior
Management Team does not appear to have grasped the holistic nature of the
challenges facing them during 1999 / 2000 and consequently failed to:
- define the implications of the various initiatives that the SQA had to
implement and assess the ability of the organisation to achieve effective
implementation within the available timeframe;
- detail the changed circumstances, assess the increased or changed workload,
and estimate the additional resources that would be required;
- prioritise actions and deploy resources to achieve effective implementation;
- develop a detailed operational plan and risk assessment for managing the
programme of change, and the specific projects required;
- provide staff with the necessary training, support and leadership to cope
with the changes;
- treat the new systems implementation as a core part of a wider business
change process, rather than as an individual project;
- report accurately on all of the above to the SQA Board and to officials
of the Scottish Executive; and
- deal adequately with significant and recognised personnel and performance
difficulties.
With regard to the SQA's operational processes, the failure
to address control weaknesses at each stage ultimately resulted in serious
data handling problems and deterioration in the quality of the data used in
the production of the final certificates. Such was the scale and number of
process failures that it was almost inevitable that the Operations Unit would
at some stage suffer systemic data handling problems of the type encountered.
The key events that contributed to this were:
- a breakdown in operational procedures in the processing of registration
and entries data, and failure to address the significant volume of errors
which resulted. This not only delayed the appointment of markers but rendered
the Operations Unit unable at any stage to confirm that all candidate and
course information had been completely or accurately recorded on APS;
- delays in processing internal assessment and estimate grade information
such that the scale of the missing data problem was not fully acknowledged
until the end of June 2000; and
- a failure to introduce adequate data management procedures to process
the missing internal assessment and estimate grade information in July 2000,
resulting in documentation being lost within the SQA and Centres being requested
to re-submit such documentation, sometimes repeatedly.
It should be noted, however, that without the sustained commitment
and dedication of the staff, particularly within the Operations Unit, the
levels of inaccurate and incomplete results would have been greater.
The poor overall management of appointment processes by the
Appointments Section also had a significant impact on this year's examination
procedures, including:
- a failure to recruit an adequate number of markers and, consequently,
appointed markers being asked to mark higher numbers of scripts than agreed,
to tighter deadlines and with much shorter notice than in previous years;
- a breakdown in selection procedures that resulted in the appointment of
some probationary teachers and some markers being allocated scripts from
their own Centres;
- markers being requested to attend markers' meetings at short notice and
without the necessary materials available to them to prepare for these meetings;
and
- significant problems in the distribution of scripts to markers, resulting
in over 65,000 unmarked scripts having still to be allocated as late as
mid June.
With regard to the SQA's information systems:
- the decision to embark on a new systems implementation (APS) with no contingency
plan in place for Higher Still was extremely high risk. Had the new software
not been delivered in time to support most of the awards processing activities
in 1999/00, the failings in certification could have been considerably worse;
- the specification of requirements for APS was problematic. The requirements
for Higher Still were not defined by user groups within the SQA to the necessary
level of detail until late in the project lifecycle and were not sufficiently
communicated to the individuals tasked with specifying the system. As a
consequence, while the system is capable of processing the necessary levels
of data and producing accurate results, there are concerns that the delivered
solution does not adequately support all of the required administrative
processes or provide appropriate management information;
- testing and project management were not given enough emphasis and problems
with system performance further impeded the users in processing Diet 2000.
These problems arose from a variety of causes and, although steps have been
taken to address them, concerns remain;
- the quality of legacy data meant that APS held inaccurate information
even prior to registration in 1999 and this situation worsened during Diet
2000. There is evidence that this problem with legacy data has not been
completely rectified and that duplicate candidates and units still exist
in APS. In addition to this, the system does not currently hold the most
up-to-date information as the clear-up process was implemented using spreadsheets
not linked to APS; and
- the variety of ways in which the SQA accepts information from Centres
adds to the complexity of data handling, if not to actual data processing
by APS. However, Centres and their information system suppliers have, to
date, not been given enough clear guidance from the SQA on what is required
of them.
The delivery of software later than planned and associated
problems in its usage certainly did impact on key processes at a time when
the SQA was already experiencing severe operational difficulties. In addition,
the lack of management information available from the system was a crucial
failing as the SQA was not otherwise able to recognise problems at an early
stage and to identify their extent and nature. However, APS itself appears
to process data and results correctly and it cannot be seen as the sole or
principal cause of the difficulties encountered during Diet 2000.
Since mid-August 2000, the SQA has been focused on the clear-up
and appeals processes for Diet 2000 and has deliberately delayed commencing
data processing of registrations and entries for Diet 2001 with the aim of
resolving some of the existing problems. Whilst this has been necessary and
unavoidable, it means that delays are already evident in Diet 2001. Given
the many significant issues still to be addressed, immediate action needs
to be taken to ensure that the difficulties encountered during Diet 2000 do
not recur in Diet 2001.
1.3 Recommendations
It is essential that implementation of recommendations is
sensitive to the current context and business pressures, and that the timeframe
is also realistic. The successful achievement of Diet 2001 should be the priority
in the immediate future and so the recommendations in chapter
8 of the report have been separated into 'immediate' and 'secondary'.
The immediate actions that the Acting Chief Executive, unless otherwise specified,
needs to take to administer Diet 2001 successfully, and which we understand
he has begun to address, are summarised below.
Planning:
- develop a clear understanding of the requirements of Diet 2001 and a detailed
operational plan for achieving the successful administration of the Diet;
- communicate the plan, requirements and risks to the SQA Board and the
Scottish Executive and secure commitment to proceed, including assistance
where required; and
- make such changes to the organisational structure and operating process
that are essential to achieve the Diet 2001 objective.
Staffing:
- the Scottish Executive and the SQA Board need to give immediate consideration
to the current interim management arrangements;
- review the capabilities of the Senior Management Team to ensure that the
necessary skills and experience are present to lead the organisation successfully
through Diet 2001. If additional skills and experience are required they
must be sourced immediately;
- clearly define the roles and responsibilities of the Senior Management
Team against the detailed plan for the year;
- produce a detailed plan for deploying and redeploying existing and new
staff resources, ensuring that any deficiencies in the staff complement
are addressed;
- review the working practices within the Operations Unit and take steps
to ensure that there is adequate supervision of all core business activities
and, where possible, that multi-skilling is encouraged where practical;
and
- take steps to address the low morale and fatigue within the Operations
Unit.
Communication:
- establish appropriate communication channels, including regular meetings
where required with the SQA Board, the Senior Management Team, the wider
management team and the staff of the SQA;
- ensure clear and effective communications with all stakeholders to ensure
a common understanding of the developing situation, including arrangements
for feedback from stakeholders;
- produce a user guide for Centres to explain what is being done at each
stage in the process and why, and to inform Centres of the action they should
be taking. This must include firm dates for the submission of data, and
the required frequency and format of submissions; and
- participate in more open discussion with Centres and suppliers and provide
clearer statements of requirements.
Processes:
- nominate a project manager to monitor each core business activity and
immediately address any delays in the processing timetable;
- introduce robust data control and handling procedures for each business
activity, to ensure that data is received and processed in a controlled
manner;
- implement procedures to deal with errors identified during data validation
routines, to ensure that all SQA staff and Centres clearly understand their
respective responsibilities; and
- undertake a review of the physical layout of the Dalkeith site and improve
the data handling and data processing areas.
Examination Procedures:
- contact all markers employed during Diet 2000 immediately to acknowledge
their contribution this year and to ask for their continued participation
in the marking of future diets;
- re-evaluate the basis of remuneration of markers in the light of the experience
of Diet 2000;
- review the examination and marking timetables to determine whether the
reduction in the time made available for markers can be sustained;
- take steps to ensure that invitations to markers are issued no later than
January 2001. Contingency plans should be made to address any difficulties
that may occur during the recruitment process and to ensure that the process
is not delayed in the event of continued processing difficulties over registration
and entry information; and
- review the administration arrangements for marker meetings and ensure
that the difficulties encountered during Diet 2000 are not repeated.
Data:
- validate the accuracy of course code and candidate data within APS prior
to the commencement of registration and entries;
- establish clear ownership of, and responsibility for, data; and
- revisit the data input validation rules in order to produce a clear set
of checks which are agreed between the various business units and the IT
Unit, to ensure that only 'hard' or business critical errors are identified,
and then communicate these rules to the Centres and their suppliers.
APS:
- focus on high-priority APS amendments this year by limiting non-critical
changes and other systems development projects. The current large number
of requested user enhancements should be prioritised according to the business
benefit and risk involved;
- revisit system controls and checks to define more practical rules and
ensure that there are adequate checks for completeness of data in each of
the core business activities;
- improve enquiry and reporting to provide visibility of processing status
within and between units, and to the Centres;
- undertake an exercise to determine the exact management information needed
to support each stage of Awards Processing and implement the required changes
or additions to reports;
- adopt a more rigorous approach to software testing, including the development
of formal testing plans and test scripts, and the establishment of a dedicated
testing team and environment;
- specify the level of system performance and resilience needed to deliver
Diet 2001 successfully and test the system thoroughly against the specification;
- assign a project manager and ensure that that individual has responsibility
for all aspects of the APS project and has the authority to carry out that
role effectively; and
- ensure that appropriate refresher training on APS is undertaken by all
staff and that adequate training on all future enhancements is scheduled
and executed prior to such enhancements going 'live'.
Staff and management are currently under considerable pressure
and delays are already being experienced in this year's awards processing,
due to the backlog of work on Diet 2000. Although the immediate recommendations
add significantly to the current work programme of the SQA, implementation
is nevertheless essential in order to ensure that the problems experienced
in Diet 2000 are avoided in Diet 2001.
< Previous | Contents | Next > |
