< Previous | Contents | Next >

A Review into Exam Results Issues Concerning the Scottish Qualifications Authority

7. SUMMARY OF FINDINGS

Whilst data management issues were, ultimately, a critical flaw in Diet 2000, it would be inappropriate to conclude that this was the only deficiency in performance or cause of the problems. No one single problem lies at the root of the SQA's poor performance this year. Rather, there were weaknesses and difficulties in a number of interlinked management, process and information systems areas which, cumulatively, overwhelmed the delivery of National Qualifications for Diet 2000. The various factors are represented in a cause-effect diagram shown below in figure 3.

A range of management and organisational weaknesses existed, including a failure to design a management and organisational structure based clearly on the functional requirements of the SQA.

Perhaps the most crucial failings were the fact that the SMT did not appear to grasp the holistic nature of the challenges facing them during 1999/00 and did not develop an integrated plan for the organisation to implement Higher Still and related initiatives. As a result, the SQA was not in a position to scope the size of the task ahead and decide whether it was actually achievable. Consequently, staffing and procedures were not reviewed and revised in light of the new requirements and appropriate training was not in place.

This failure to plan at an overall level was echoed within a number of units, most crucially Operations. Staff in Operations Unit were not prepared for the new requirements and procedures were not in place to address the significant volume of rejected data from the new APS system at the early stages of registration and entries. Delays in processing data accumulated through the internal assessment, estimates and results stages and the lack of control checks meant that at each key stage the completeness and accuracy of data was not reported. Furthermore, due to the lack of reliable management information, SMT and Operations Unit were unaware of the scale of the missing data problem until certification.

The delay in appointing markers and the poor overall management of the appointments process was another significant factor in the overall delivery of award results. In particular, the SQA had difficulty recruiting an adequate number of markers and experienced considerable problems in the allocation and distribution of scripts.

The decision to embark on a new systems implementation (APS) with no fallback plan in place for the new qualifications was extremely high risk and the failings in certification could have been considerably worse as a result. However, APS was delivered and functioned according to the specification.

There were issues with APS due to the fact that the business specification was not always accurate and the lack of testing meant that these issues were not often apparent until the system was in live use. In addition, problems with system performance and a lack of user training further impacted on the users.

The lack of management information meant that the SQA was not able to state its position accurately or identify the extent of data problems. This lack of information was due to several reasons, including management not having specified all the reports that were required and those reports that were available were not being used due to a lack of understanding by users.

The diagram below summarises the major causes of the poor delivery of results by the SQA, grouped under the following headings:

• management;
• organisation;
• communications;
• Higher Still implementation;
• new computer system;
• operational processes; and
• marking.

Figure 3 - Cause and Effect Diagram

< Previous | Contents | Next >