< Previous | Contents | Next >

A Review into Exam Results Issues Concerning the Scottish Qualifications Authority

5. FINDINGS - OPERATIONAL PROCESSES

5.1 Introduction

The purpose of this section is to comment on the weaknesses in the operational procedures and controls that contributed to the exam results issues that occurred during Diet 2000.

Our findings address the following aspects of the operational arrangements:

• staffing;
• data management;
• examination procedures; and
• facilities;

The following sections seek to comment on the procedural and control weaknesses that contributed to the production of inaccurate and incomplete examination results, with details of the development of the software to support the awards process being discussed in Chapter 6.

As part of this investigation, process maps have been produced to define and illustrate the steps involved in the awards process and the existing controls (see Appendix VI). The points at which these processes broke down and the controls which were found to be insufficient are also highlighted on the process maps.

5.2 Staffing

As its name suggests, the Operations Unit is central to the operational activities of the SQA. However, it appears that management generally and the Head of Operations in particular failed to assess adequately the impact that Higher Still and the introduction of the new Awards Processing System would have on the Unit. Although day to day management of the Operations Unit was the responsibility of the Head of Operations, the SMT had corporate responsibility for ensuring that there were adequate resources at all stages of Diet 2000 to achieve the final certification timetable.

The key matters which had a material impact on the capability of the Operations Unit staff to cope with the problems of Diet 2000 were as follows:

• failure to assess data processing requirements. At no point in the Diet 2000 programme was there a proper assessment of the degree to which data volumes and patterns of data processing would change following the introduction of Higher Still. Consequently, the impact on staff resources at critical points in the process was never adequately ascertained nor were any contingency arrangements made;
• lack of training. Whilst training was offered to staff in the Operations Unit, they were unable to attend sessions due to the pressures caused by the volume of data processing and corrections required during the registration and entries stages. The need for adequate training should have been identified and delivered in advance of the commencement of Diet 2000 and before Operations Unit staff were required to be involved in the specification of the APS modules for which they were responsible;
• over-dependence on key personnel. The allocation of responsibilities within the Operations Unit was such that the various processes which together allow accurate and timely awards tend to be only understood by individual members of staff, with little multi-skilling within and between the sections. In terms of resolving the problems that emerged during Diet 2000, this placed a considerable burden on all staff within the Operations Unit. The allocation of responsibility to staff at supervisor grade within the Unit was such that they had insufficient time to supervise and manage staff within their respective sections;
• use of untrained temporary staff. In May 2000, the Head of Operations made a request for 56 additional temporary staff to cope with the backlog of data processing. Whilst the need for additional resource is acknowledged, the pressure on the full time members of the Operations Unit was such that insufficient time was available to supervise and support the temporary staff who were employed adequately. Consequently, there was a further deterioration in the control environment with untrained and inadequately supervised staff being asked to undertake tasks for which they were not fully prepared; and
• ineffective relationships with Centres. In terms of supporting the Operations Unit, a key failure in this year's process was the need to have links between trained and knowledgeable staff within the SQA and Centres. Given the significance of the change in administration arrangements, recognition of the importance of effective links with Centres (throughout the awards process) should have been considered during the planning process for Diet 2000.

5.3 Data Management

In terms of the overall awards process, the Operations Unit had responsibility for the management and processing of all data that would be required to provide candidates with their award certificates. An overview of the awards process is included as Appendix VI of this report.

One of the key requirements of the awards processing function is the existence of robust procedures and controls that enable the complete and accurate recording of raw data submitted by Centres at all stages in the process.

The key data sets required to be able to award certificates to candidates are:

• course and centre approval - prior to entering candidates for awards, Centres and the courses they can offer must be approved;
• registration - details of all candidates to be presented by Centres;
• entries - details of all courses (and the component elements) for which each candidate has been nominated;
• internal assessment and estimate data - a record of the results of the unit and internal assessments for the component elements of Higher Still and Standard Grade and grade estimates for each candidate. While internal assessment data is mandatory, estimate data is non mandatory for the purpose of awarding a grade to a candidate; and
• external examination results - final results of the external examination process which are provided by the Principal Assessors.

Each of these data sets, and the issues encountered with them during Diet 2000, are outlined further below.

5.3.1 Course and Centre Approval

Course and Centre approval is undertaken in advance of each examination Diet in accordance with the approval guidelines established by the SQA. Based on interviews with staff, we were able to confirm that prior to the registration and entries stage not all Centres had returned the necessary information requested by the SQA to approve all the courses they wished to offer to candidates during Diet 2000. Consequently, there were instances during the entries phase of Centres submitting data for candidates for courses where approval had not been registered on the APS system. This, as noted below, increased the volume of errors in data validation.

5.3.2 Registration and Entries

The initial stages in the awards process are the registration of candidates and recording of course entries. Prior to Diet 2000 it had been decided that the collection of registration and entries data would be separated, rather than combined as in previous years for SEB courses.

Whilst, as noted below, there were a number of key failures, the greatest consequence to the overall data quality during the registration and entries phase was the impact of the data validation procedures which resulted in a significant volume of rejected data for which no planning or contingency arrangements had been made. The key failures that occurred during the registration and entry process were as follows:

• cleansing of legacy data. Prior to the registration process the Operations Unit and IT Unit implemented an exercise to cleanse the candidate information that was to be transferred from the previous SEB and SCOTVEC systems onto APS on 1 August 1999. A primary objective of this exercise was to ensure that only one candidate number (SCN) was allocated to each candidate. This data cleansing exercise was not satisfactorily completed and resulted in multiple candidate references being transferred into APS. The existence of duplicate candidate entries had a significant impact in terms of the quality of data required for external examination and certification.

This problem was further compounded by the fact that Operations Unit staff could not easily produce (from APS) a list of multiple SCNs by candidate (though it could identify candidates with multiple SCN references). Consequently, searches for those candidates’ other SCN references was undertaken manually by Operations Unit staff;

• logging of receipt of paper based data from Centres. Whilst a log was maintained for recording data supplied on a floppy disk and by e-mail, there were no controls to log the receipt of paper-based registration information from Centres.

This need for adequate control over the receipt and processing of registration and entries data should have been identified and addressed prior to the implementation of APS;

• data entry. It was originally intended that external data preparation bureaux would be used to convert registration and entries information that was received in paper format from Centres into digital format. However, due to delays in the process, paper-based data was input by staff within the Operations Unit. This, combined with the volume of data errors, lack of scoping of the staff resources required and lack of contingency planning, placed additional pressures on staff resources at an early but critical stage of the awards process;
• data processing. Data was received in three separate formats - floppy disk, paper and electronic data interchange. Loading of the information in disk format onto the APS system was undertaken by administrative staff in the Operations Unit, with processing runs being made overnight by IT services. For example, during the early stages of registration and entries, all floppy disks were received by one person in the Operations Unit and were not submitted for processing in her absence. This was identified and corrected by the SQA. The processing runs themselves experienced significant difficulties including loss of data, delays in completing overnight runs (often reducing the availability of computer systems the next morning) and data being rejected due to processing being performed out of sequence, for example, when ‘entries’ data was processed before candidates’ registration details;
• data validation and resolution of errors. The validation checks implemented within the registration and entries module were so stringent that a significant volume of data was rejected from the initial data sets provided by Centres. Due to the poor quality of management information we were unable to quantify the level of rejected data. However, we understand from discussions with staff that the error rate was significantly higher than in prior years.

The format used for reporting errors was difficult to interpret and did not distinguish between hard (or critical) errors and soft errors. Indeed the length of the reports and the manner in which errors were recorded was such that resolution of erroneous data was a time-consuming task.

More critically, there were significant weaknesses in the processes adopted for 'clearing' the identified errors. Staff within the Operations Unit had not been provided with adequate guidance on how to resolve errors, nor were there satisfactory controls to ensure that all Centres were advised of the required data corrections and that such errors were satisfactorily cleared.

In an attempt to improve the quality of information returned to Centres for correction, Operations Unit staff tried to correct errors directly (without recourse to the Centres) and discontinued use of system-generated error reports, instead typing their own reports and photocopying incomplete or incorrect registration forms to return to Centres. This was not only an inefficient use of scarce staff resources but resulted in a serious breakdown in the control mechanisms for completeness and accuracy of data recording.

Operations staff were unable to confirm at which stage of the process they were entirely confident that all registration and entries data had been corrected;

• registration and entries guidance issued to Centres. The Procedural Guide for Registrations and Entries which was written and issued to Centres prior to completion of the respective APS modules had a number of significant deficiencies:
  • whilst telephone numbers were provided to Centres where help could be provided, no reference was made to the section within the SQA from which assistance could be sought. We understand that responsibility for resolving Centre queries effectively fell to Operations Unit staff, with limited administrative support provided to Centres from other units within the SQA. Given the level of staffing within the Operations Unit, this placed an additional burden on an already tightly stretched resource and exacerbated the processing difficulties;
  • the format, structure and content of the Guide was poorly constructed and in many cases there was a lack of clarity in terms of the procedures to be followed. For example, the arrangements for reporting and resolution of error reports generated by APS were particularly vague and would not have been easily comprehensible to administrative staff at Centre level;
  • the guidance provided deadlines for submission of data which were flexible. Whilst we acknowledge the desire of the SQA to be a 'customer friendly' organisation, the resulting level of discretion Centres had was a serious flaw which undermined the entire data collection process.

Examples of this inherent flexibility can be found in the Registration and Entries Guide published by SQA, some of which are noted in the table below.

Feedback received from visits to Centres confirmed that confusing instructions and guidelines continue to be issued;

• poor form design and complex coding. Whilst the weaknesses in the data validation process were apparent, the forms used to submit data were poorly designed and the format of entry coding was complex and difficult for Centres to interpret. These factors, combined with the quality of the registration and entries guidance issued to Centres, contributed to a number of errors in the data submitted to the Operations Unit for processing.

During the review of anomalous results for Centres there were examples of Centres entering candidates for the wrong units (within courses) and/or not entering candidates for all component elements of a course. In each of these examples this resulted in candidates being awarded lower marks than anticipated on the original certificates. An exercise is still under way within the SQA to investigate instances highlighted by Centres, where the results across all candidates are significantly lower than had been expected;

• impact on internal assessment and examination results. Due to the significant weaknesses in the registration and entries procedures and the fact that Centres were not provided with candidate lists, it was not possible to determine that all candidates had been registered and registered on the right course prior to the processing of internal assessment and external examination results. This resulted in further problems as internal assessment information was being rejected by APS and required to be resolved, causing further delay in the process.

More recently, the problem of completeness of candidate information was again highlighted in October 2000 when incomplete lists of Standard Grade candidates presented for Diet 2000 were sent to Centres. We believe the continued provision of inaccurate information further undermines the credibility of the SQA.

5.3.3 Internal Assessment and Estimate Data

Whilst internal assessment has been a component part of the Standard Grade process since its implementation, the introduction of Higher Still increased the volume of internal assessment data required to be processed during Diet 2000. This increase in the volume of data, combined with the problems during registration and entries, and the delay in inputting internal assessment or estimate grade data into the APS system, meant that the extent of missing data was not fully acknowledged until the end of June 2000.

As outlined below, the delay in completing the processing of internal assessment data had a significant impact on the completeness and accuracy of the final result process. Failure to acknowledge the critical nature of the situation as it was known in May 2000 was a serious failure in control over the internal assessment process. In respect of the provision of estimated grades, the problems encountered in data collection resulted in estimates for Higher Still not being processed in APS.

The weaknesses identified in the processing of internal assessment data were as follows:

• poor handling of assessment information. Internal assessment data (which was mainly in a paper format) was stored at the Dalkeith site pending processing. Operations Unit staff commented that there were inadequate storage facilities for this data and that overall there was a significant lack of controls to ensure that all internal assessment data was being stored in both a logical and secure manner. This failure to implement adequate data handling procedures contributed to difficulties in finding and processing information that Centres had confirmed as having been already forwarded to the SQA for processing;
• delays in processing and logging of data. The significance of delays in the processing of internal assessment data was that at no time up until the end of June 2000 did the Operations Unit have any management information to allow them to confirm whether Centres had submitted the necessary internal assessment and estimate data. Furthermore, as the SQA neither notified Centres of the receipt of data nor followed up missing data, Centres assumed that the SQA had received and processed all the information which had been provided to them;
• no firm timetable for submission of internal assessment data. Whilst we acknowledge that internal assessment occurs on a continuous basis throughout the academic year, we believe that the timetable for submission of internal assessment data needs to be more structured and should contain firm deadlines. If adequate completeness and accuracy checks are to be undertaken, all such data has to be received by a fixed date which allows sufficient time for the SQA to process the data and undertake the necessary data validation checks;
• identification of missing data. It became apparent by the end of June 2000 that there was a significant volume of missing data. Whilst the significance of the problem was identified, Operations Unit was unable to produce from APS a report that could identify for each candidate which data elements or internal assessment components were missing. The inability to produce management information at key stages was a consistent feature of the awards process for Diet 2000. Whilst APS had the capability to produce some management reports, it was not always easy for staff to use these reports (see section 6.9).

To identify which elements of data were missing, reports detailing the information processed for all candidates were generated for each Centre. Operations Unit staff then had to review these reports manually and highlight the components for which internal assessment information was missing. These reports were then used by a help desk in Glasgow to contact Centres and request re-submission of the missing data;

• resolution of missing data. As the management of 'data chasing' was undertaken outwith APS, there was a further deterioration in controls (due to supplementary manual intervention required to identify the elements of missing data and to record receipt of re-submitted information from Centres). During the period from the beginning of July until the final issue of the certificates on 9 August 2000, there were no effective controls over the completeness and accuracy of internal assessment data being processed onto APS.

  In general, the process of resolving missing internal assessment data was badly managed as the following highlight:

  • identification of missing data was undertaken by staff within Operations Unit who had received little formal guidance on the assessment requirements for Higher Still;
  • SQA co-ordinators and teaching staff were not always available in Centres to respond to requests for resubmission of missing data;
  • there was confusion among SQA staff and within Centres as to which elements of data were missing and whether internal assessment, estimated grade or both were required;
  • incorrect data, often in inconsistent formats, was submitted to the SQA by Centres. Examples included a Centre providing internal assessment marks for a unit on the basis of a total score of 30, rather than scaling up to 100, and another Centre providing estimated grades (using scales between 1-8) instead of a mark out of 100. In such instances, the internal assessment data would be processed as received and the candidates’ recorded marks would be lower than warranted.

By 31 July 2000, a Marks List report was available which could be used to help identify missing internal and external grade information. However, Operations Unit staff were still required to inspect this report manually in order to identify which elements were missing. Whilst acknowledging that missing data was still being chased, the effectiveness of any controls at this stage was seriously undermined by the lack of information and poor management of the process.

5.3.4 External Examination Results Processing

On the return of marked scripts from markers, the arithmetical accuracy of each total mark per script is verified, and results recorded on control forms (EX6) are compared to the results recorded on the candidate scripts. The control forms are then separated from the examination scripts and forwarded to an external bureau for conversion into electronic format and subsequent processing onto APS. Scripts are then returned to shelves in the Operations Unit and retained in case of appeals.

Whilst accuracy checks were undertaken on scripts themselves, no checks were subsequently undertaken to ensure that the data sent by the bureaux on diskettes was processed on APS.

Whilst there are still unmatched EX6 forms within the APS system at the time of this report, the SQA has confirmed that all "active" EX6 forms with results have been processed using manual procedures outwith APS. The reduction in effectiveness of the procedures to ensure completeness of external examination results data represented a significant weakness in the control environment.

5.3.5 Awards Reporting and Certification

The final stage in the awards process is the production and distribution of certificates to candidates. It was intended that the majority of certificates would be printed externally by a company based in Birmingham although Dalkeith also had the facility to print 5,000 certificates. However, on Thursday 3 August, Operations Unit was asked to print and despatch an estimated 50,000 certificates and on Saturday 5 August we understand that the number of certificates to be printed internally had increased to around 70,000. This led to a further requirement for resources to be drawn together at short notice to undertake enveloping and despatch.

It is clear that there was inadequate management information in the days immediately prior to 9 August 2000 to enable the Chief Executive to assess the scale of the problem of missing data accurately. Despite this lack of information, the certificates were issued as originally planned on 9 August 2000, on the basis of data which had not been adequately checked for completeness and accuracy.

Controls over the production and despatch of certificates from Dalkeith were poor. Certificates were not produced in Centre order as in prior years and, whilst batch references were utilised, it was not possible to undertake any completeness checks prior to despatch.

On Monday 7 August, the SQA identified additional data for a number of candidates for whom certificates were at that time being printed, demonstrating that those certificates were incorrect (being based on incomplete data) in their existing form. Operations Unit staff were given lists totalling 4,760 candidates for whom the original certificates should be stopped. However, this proved to be extremely difficult and time-consuming at Dalkeith as the lists produced by the IT Unit could not identify the correct batch reference numbers for the certificates.

Based on discussions with the external certificate printers and the member of Operations Unit staff who was at the contractor’s premises during production, the production of certificates based on the data set provided by the SQA was better controlled and there were adequate completeness checks undertaken. However, as with the experience at Dalkeith, requests were made to stop the distribution of certificates where it was known that data was incomplete up until the point that mail carriers left the contractors premises.

Due to difficulties in retrieving original certificates, and an error made by IT Unit which led to a batch of certificates being printed twice, a number of candidates received multiple or duplicate certificates. The exact number of duplicates is not known at present but 839 were created by the double-printing error, and it is therefore estimated that over a thousand candidates received more than one certificate. In addition to this, a number of candidates would have received more than one certificate due to the data-cleansing problems identified earlier.

5.4 Stages in the Examination Procedures

The key stages in each Diet of examination procedures operated by SQA are given in the table below.

5.4.1 Stage One: Selection and Appointment of Markers

The selection and appointment of markers is the responsibility of the Appointments Section based at Dalkeith. This is normally undertaken in two phases:

1. selection of markers to be invited, based on prior performance of markers, estimates of examination entries (November) and the structure of marking teams (as notified by Development Division); and
2. issue of invitations to accept appointment as a marker (early January).

However, due to the problems identified below, the appointment of markers for Diet 2000 was delayed, resulting in serious resource problems throughout the examination marking process.

Selection of markers

At the start of September 1999, Qualifications Managers were requested to liaise with the Appointments Section to arrange the selection of markers. At this stage the Qualifications Managers were asked to indicate who would be used for SCE and National Qualifications and to consider new applications. However, due to the uncertainty over final entry information, some Qualifications Managers did not believe that they were in a position to complete selections. As a result, the selection process had not been completed for all subjects by the end of February 2000. At this point a decision was taken by the Director - External Relations and Corporate Services that staff in Appointments Section should select markers from the approved list for all subject areas that had not been completed and to proceed to issue invitations to markers. Whilst there was no change in the eligibility criteria for marker selection, there were a number of weaknesses during the appointment phase which resulted in:

• the appointment of probationary teachers. The SQA has identified 11 probationary teachers that were used in marking Diet 2000 examinations and it is acknowledged by the Appointments Section that this was due to a failure to apply proper procedures prior to confirmation of appointment. Checks should be undertaken and if applicants do not meet the required standards then they should not be included on the appointments database.

  Following identification of this situation (after results day) the scripts for each of the probationary teachers were subject to further sample checking by the SQA. The results of the checks undertaken by the SQA confirmed that seven markers were awarded an A grade, two a B grade and one a C grade. One marker grading is under further investigation. The SQA has also confirmed that the required examination procedures had been correctly applied during external assessment moderation for the marker graded as a C and that no adjustment was required to candidates' results. We also understand that the SQA is undertaking further checks on an additional two markers who may also be probationers; and

• compliance with application procedures for marker appointments. In some subjects, Qualifications Managers and Principal Assessors informed the Appointments Section of suitable people known to them to approach for marking. It would appear that such appointments were made verbally without the necessary application forms having been completed. An exercise is currently being undertaken by SQA to check all individual application forms for the 872 markers who had not marked in either of the preceding two years. To date SQA has been able to verify all but two of these 872 markers application forms, all of whom have been deemed to be 'suitable'.

Impact of late processing of registration and entries data

The delays in processing of registration and entries data within the Operations Unit had a significant impact on the availability of estimated candidate entry numbers which are required for the selection and appointment of markers. As accurate details of entries were not known in December or January, a decision was taken to delay the issue of appointment letters. Whilst acknowledging the merit in delaying the issue of invitations to markers until the volume of the work required could be finalised, this decision subsequently had a critical impact on the entire appointment process, further exacerbated by:

• the changes in the examination timetable for Diet 2000, which reduced the period available for marking scripts from three to two weeks (although the initial script allocations were also reduced); and
• the continued low rate paid to markers per script marked.

Invitation letters were issued to markers on 8 February (Standard Grade) and 8 March (SCE Higher and Higher Still). As no prior correspondence had been issued to markers, these invitations were being received one or two months later than in previous years. This, combined with a reduction in the marking period and possible clashes with holidays, increased the rate at which invitations were declined to levels significantly in excess of those in previous years.

Resolution of marker shortage

The extent of the marker shortage was such that eight additional temporary staff had to be employed to assist in the administration of the appointment of markers. A further complicating process factor was that the APS Appointments module allocated (‘tied’) the marker code to a specific individual as soon as an invitation was issued. Where invitations were ignored, not confirmed or rejected the marker code could not be removed or reallocated until a replacement marker was identified.

In order to monitor such re-allocations of marker codes, supplementary spreadsheets had to be used by Appointments Section staff as APS could not generate a report which listed unconfirmed or rejected marker invitations. Again, the use of procedures outwith the APS system to monitor and control a core business process added further to administrative complexity.

The process of appointing markers was still incomplete at the date of the markers’ meetings, with only 85% of the number required actually confirmed at that point. To address the shortfall two strategies were proposed:

• to ask Local Authorities to release teachers from teaching duties - following a written approach by the SQA to request early release of teachers, this strategy received a poor reaction from Heads of Centre and Directors of Education and was therefore not pursued; and
• to ask appointed markers if they would increase their allocation of scripts, with a higher rate being offered for additional batches.

The second strategy, whilst more successful, still did not fully address the marker shortage. The only solution for the remaining shortfall was to apply central marking either by employing the examination teams already at Dalkeith to undertake the external assessment moderation procedures, or to recruit markers for this specific purpose. While acknowledging that central marking has been used for a considerable period of time to mark external examination papers for subjects with lower candidate presentation numbers, use on this wider scale was not originally planned for Diet 2000.

From a marker’s perspective, the appointment process for Diet 2000 was unsatisfactory in terms of substance and timeframes and may, unless addressed immediately, have an adverse impact on the recruitment process for Diet 2001.

5.4.2 Stage Two: Procedural Programme

Finalisation of the procedural programme was late in Diet 2000 because:

• the new software was not delivered until December 1999;
• entry data was often inaccurate; and
• markers were appointed late.

This made it difficult to plan the marking of external assessments, particularly as the later examination timetable delayed the key dates for moderation, which in a number of instances overlapped with the summer vacation period for teachers.

5.4.3 Stage Three: Examinations

Whilst this stage incorporates the entire examination process, the review primarily focused on the robustness of the controls over the receipt and distribution of candidates’ scripts.

The preparation and accuracy of the key control documents (EX6 forms) issued to invigilators at Centres, and which are essential for tracking scripts through the entire marking and award process, were seriously affected by deficiencies in registration and entry data and the late appointment of markers.

When the EX6 forms were printed from the APS system for distribution to invigilators it was evident that there were significant deficiencies in the underlying data. This resulted in:

• duplicate entries for candidates on the same form;
• duplicate entries for candidates on more than one form; and
• incomplete lists of the candidate population (as confirmed by invigilators).

The inclusion of duplicate entries on the same form was manually corrected by invigilators and did not have a significant impact on the processing of final examination results. However, the inclusion of duplicate candidate entries on more than one form did result in more fundamental difficulties, such as scripts being included in the wrong 'packets' by invigilators.

In instances where the candidate population was incomplete, invigilators completed manual supplementary EX6 forms to record the missing candidate information. This resulted in delays in distributing scripts to markers as updated EX6 forms had to be prepared by Operations Unit staff before they could be issued to markers.

In instances where revised EX6 forms had not been prepared timeously, Operations Unit staff had to send scripts out manually without their EX6 forms and subsequently match the batches to a printed EX6 form which was available by the time the marked scripts were returned. This was yet another work-around which introduced the potential for error and inconsistency.

The late appointment of markers did result in significant difficulties in the distribution of scripts to markers. At the date of running the packet allocation program to allocate marker codes to packets of scripts on the EX6 forms, insufficient markers had been appointed and there were numerous marker codes with markers attached who had rejected the invitation to mark. We understand that when the last packet allocation routine was run only 75% of markers were in place compared to the target of 95%.

Due to the backlog in processing the re-allocation of marker codes in APS, Operations Unit staff had to rely on hand-written notes prepared by the Appointments Section to identify packets of scripts where the marker had been altered. However, this did not prevent scripts being distributed to markers who had previously declined their invitation to participate, and resulted in unmarked scripts being returned to Dalkeith.

Evidence provided to our review included the following examples:

"I decided not to mark exam papers this year, so did not send back the acceptance form. Nevertheless a bag of nearly 100 scripts appeared at my back door. I informed them (SQA) that I still had no intention of marking scripts and was sending the bag back. One week later more scripts were delivered to my door."

"In the first week of June a grey unmarked sack was delivered to my house. I was on school sportsweek in Spain. When I returned and felt the sack, I knew it contained a marker's grade pack, strange because I had refused to mark this year and had not done so for 5 years. I left it and assumed that SQA would realise their error and contact me…On return from my summer holiday I found 3 messages from SQA to call them about missing scripts."

In instances where scripts were held prior to distribution as the original marker had declined, there were further delays as Operations Unit staff had to identify the new marker and manually prepare an address label. This not only resulted in significant additional administration but also weakened the controls over script and marker tracking.

During the interviews conducted as part of the SQA internal review, the Director - Awards Division estimated that, at mid June, there were 65,000 scripts which had not been allocated to markers. Whilst a number of these scripts were re-allocated to external markers, there were a number of instances where examination teams who were at Dalkeith had to undertake the marking of scripts at the same time as undertake marker checks, standardisation and finalisation procedures.

The delays in sending scripts to markers had a serious impact on the time available for marking. In some cases this resulted in markers returning scripts unmarked, citing the failure of the SQA to meet the timetable agreed in the marker’s appointments letter.

From discussions with markers and Operations Unit staff, we understand that there were instances where markers were asked to mark scripts from Centres where they were employed as teachers. It is not normal practice for markers to receive scripts from their own Centres and we believe that this was a further example of a failure to apply normal rigorous vetting procedures during the allocation of scripts.

5.4.4 Stages Four-Five: Markers' Meetings

Markers' meetings, at which attendance by markers is compulsory, are held for most (but not all) subjects on a date between the examinations themselves and the marking of the resultant scripts. The meetings are run by the Principal Assessors with administration support provided by SQA Appointments Section staff.

Whilst all but 13 markers (who were required to attend) attended such meetings, many markers were informed of meeting dates at very short notice and in some cases only the day before the meeting. Of the 13 markers who did not attend markers’ meetings, six were authorised absences for whom alternative arrangements were made to conduct marker briefings prior to actual marking of scripts. The remaining seven markers were not authorised absentees and no alternative briefings were provided. These markers were, however, permitted to mark scripts. This has been acknowledged by the SQA as a breach of normal procedures. In prior years such markers would have been prevented from actually marking scripts.

The late appointment of markers impacted on preparations for markers’ meetings as the following material had not been distributed in advance to all markers:

• photostats of samples of candidate scripts;
• answer books and mark sheets; and
• scripts to be marked by the marker.

As the numbers of markers to be accommodated at each meeting was also unknown, there were problems in sourcing premises for meetings. In addition, firm information on team structures was not available until markers’ meetings and support documents and materials necessary for the meetings were sometimes poorly organised or unavailable. These deficiencies did not inspire commitment and enthusiasm in markers at a key stage in the marking process.

5.4.5 Stages Six - Eight: Examination Procedures

The key stages in the examination procedures which are undertaken by examining teams (led by a Principal Assessor) for each subject are:

• marker check / standardisation. This is a review of a small sample of each marker's work to ascertain the quality of the marking and to consider the need for adjustments to be made for severity, leniency or inconsistency;
• cut-off score / passmark determination. The aim of this stage is to set the marks at which grades will be awarded and to maintain comparability from one year to the next; and
• finalisation. At this stage, scripts from those markers classified during checking as unacceptable are provided for finalisation, together with a review of candidates’ scripts that are close to cut-off scores.

In order to examine the level of compliance with the examination procedures a review was undertaken by the SQA, the findings of which are noted below.

Marker check / standardisation

These procedures were implemented in accordance with SQA guidelines, with unsatisfactory marking being identified by individuals and remarked by the examining teams. The number of unsatisfactory markers was 20 for Diet 2000, compared to 33 in the previous year. Furthermore, as noted below, the distribution of marker grades is not materially different from the previous two diets²⁶

During Marker Check and Standardisation procedures the Principal Assessor is required to record each marker’s fitness for future appointment on a Marker’s Grading Form. The following codes are used to rank performance:

The only areas to date where concerns over marker quality were identified by Principal Assessors were in respect of Higher Media Studies, Higher Management and Information Systems and Higher Business Management. In each case some required examination procedures were not applied due to significant difficulties in the administration of the marking process. We understand that the SQA's internal review concluded that whilst the procedures had not been rigorously applied the final grades awarded were not affected by this lack of process rigour.

In respect of marker check and standardisation it is the conclusion of the SQA's internal review that the procedure in Diet 2000 was less well organised than usual. In a normal year marker checks would be undertaken on a sample of scripts drawn from the beginning, middle and end of the marking period. However, the work of markers in this Diet was not sampled in this way. This was contrary to normal procedures.

Cut-off score / passmark determination

The preparation of the statistical data required for the passmark meetings was affected by the delays in the processing of examination data and the fact that data sets for each subject were not complete. From our discussions with SQA representatives, we understand that a programme was used during the passmark stage which only allowed results for candidates where all components of the data set were complete to be used for setting passmarks. However, in comparison to previous years when almost complete data was used for passmark meetings, the passmark decisions for Diet 2000 were made on the basis of a sample of results. We understand that this sample approach has been adopted on at least one previous occasion by the SQA. In instances where Principal Assessors believed that there was insufficient data, passmark meetings were delayed until a more complete data set was available.

Given that the Higher Still examinations had no prior years’ data to compare with that from Diet 2000, training was provided to examining teams to enable them to use benchmarking techniques when agreeing a passmark for each subject. Consequently, the setting of passmarks for Higher Still courses in Diet 2000 relied on qualitative factors as well as the statistical analysis of actual results.

The setting of passmarks for Standard Grade was affected by the lack of estimate grade information, normally considered a reliable basis for prediction of candidate performance. We were, however, informed by the SQA that whilst estimate information was desirable, it was not an essential component of the passmark setting process and that there should have been no material impact on the final passmarks set for Standard Grades.

Finalisation

This procedure is not required for all subjects. Interviews with a sample of Principal Assessors did not identify significant differences from previous years, other than greater problems of data completeness and accuracy.

5.4.6 Stage Nine: Issue of Results

We have commented above (sections 5.4.4 and 5.4.5) on the procedures followed to record the final examination results and issue the final certificates.

5.4.7 Stage Ten: Appeals

As the results of the urgent appeals or current appeals processes have not yet been analysed we are unable to take a view on the efficacy of the procedures adopted.

We can only comment at this stage that the appeals process is being affected by the larger than anticipated numbers of appeals and continuing problems in providing ready access to supporting information for the appeals process, such as marked scripts and grade estimates.

5.5 Facilities

The standard of facilities at the Dalkeith site is becoming increasingly unsatisfactory for the volume of data processing being undertaken. The site is characterised by an ineffective layout of the main data processing area, insufficient storage space and poor working conditions for staff.

The lack of effective storage facilities resulted in weaknesses in data logging and handling during Diet 2000, as the site could not cope with the volumes of documents being received. This significantly increased the risk of data loss. The layout of the working environment is also not conducive to effective data processing or productivity.

5.6 Payment of Marker Fees

Following the completion of marking, significant difficulties were encountered in paying marker fees and expenses. All markers were contacted in July and advised that all standard rate payments would be made by the end of August, and special arrangement payments by the end of September. The Head of Finance confirmed that these revised deadlines were achieved.

The delay in paying markers, however, resulted in a further erosion in the relationship the SQA has with the markers.

5.7 Conclusion

In respect of operational processes, it is our overall conclusion that the events that led to the examination crisis are primarily attributable to a failure to plan for Diet 2000 adequately and to identify and address the implementation problems evident from an early stage in the new Awards Processing System. More specifically, we would highlight the following:

Staffing

In addition to the failure to assess the resource requirements needed to deliver Diet 2000, staff did not receive the necessary training, support and leadership to cope with the changes and the increase in data processing volumes consequent on the introduction of Higher Still.

It is evident from the findings of our review that, without the sustained commitment and dedication of the administrative staff within the Operations Unit throughout the period from January to August 2000, the levels of inaccurate and incomplete results would have been greater. However, in the context of inadequate management supervision, weak data logging procedures and ineffective procedures throughout the process, staff within the Operations Unit were eventually overwhelmed by a volume of data that they could not cope with, particularly during June and July 2000.

Data Management

The failure to address control weaknesses at each stage ultimately resulted in a serious deterioration in the quality of the data used in the production of the final certificates.

Such was the scale and number of process failures throughout Diet 2000, it was almost inevitable that the Operations Unit would at some stage — in the event, in late July - breakdown with the loss of effective control over the completeness and accuracy of data processing.

The key events that contributed to this crisis situation were:

• a breakdown in operational procedures in the processing of registration and entries data and failure to address the significant volume of errors. This not only delayed the appointment of markers but rendered the Operations Unit unable at any stage to confirm that all candidates and course information had been completely or accurately recorded on APS;
• a delay in processing internal assessment and estimate grade information such that the scale of the missing data problem was only evident in June 2000, by which time the external examinations had been completed and the marking process had commenced;
• a failure to introduce adequate data management procedures to process the missing internal assessment and estimate grade information in July 2000 resulted in documentation being lost within the SQA, and Centres being requested to re-submit such documentation, sometimes repeatedly;
• the time pressures under which the Operations Unit staff had to process the missing internal assessment data, combined with the lack of effective data management and processing controls, had a serious impact on the quality of data required during the final results process; and
• weaknesses in the procedures and control documentation used in the processing of external examination results undermined the integrity of the process and precluded any adequate audit trail.

Given the combined effect of these events and the deterioration in controls before and during the production of final certificates, it was almost inevitable that there would be missing and inaccurate data when the awards were made.

Whilst there is no doubt that senior management were aware that there had been a serious breakdown in procedures at all key stages when the award certificates were issued, there was no reliable information available for them to assess adequately the extent of inaccurate and missing data. Nevertheless, the award certificates were despatched as originally planned on 9 August.

Problems of data management continue to hinder the ongoing appeals process, with examination teams having difficulty in gaining access to marked scripts and estimate grade information.

Examination procedures

The delay in appointing markers and the poor overall management of appointment processes had a significant impact on this year's examination procedures.

The delays in the appointment of markers, combined with the use of inadequately trained temporary staff to help with administration, resulted in:

• a higher than normal rejection rate from marker invitations;
• a failure to recruit an adequate number of markers;
• a breakdown in selection procedures that resulted in the appointment of some probationary teachers, markers being allocated scripts from their own Centres and Principal Assessors approaching markers directly without going through the normal vetting processes;
• markers being requested to attend markers' meetings at short notice and without the necessary materials available to them to prepare for the meetings;
• significant problems in the distribution of scripts to markers, resulting in over 65,000 unmarked scripts still to be allocated as late as mid June; and
• markers being asked to mark higher numbers of scripts than agreed, to tighter deadlines and much shorter notice than in previous years.

The poor management and handling of this year's marker appointments may prejudice recruitment of markers in succeeding years, unless action is taken to reassure markers. This may result in a further increase in the rejection rate.

In terms of adherence to the final examination procedures, whilst the allocation and distribution of scripts for marking was beset by problems arising from the failure to recruit adequate numbers of markers, normal procedures, with the exception of the matters highlighted below, were generally followed. The principal weaknesses that have been identified in interviews with examination teams were:

• the effectiveness of marker check selection; and
• general management of the examination procedures, which were frustrated by delays in provision of information, cancellation of meetings and the need to divert examination teams to central marking in order to address the backlog in unmarked scripts.

Diet 2001

Since mid-August 2000 management effort within the SQA has been focused on addressing the missing data problems and appeals processes for Diet 2000. In addition, the SQA has taken the decision to delay processing of registrations and entries until these problems are resolved and steps have been taken to resolve the data management difficulties.

Whilst this has been necessary and unavoidable, it has meant that delays are already evident in commencing data processing of registration and entries for Diet 2001. Given the many significant issues still to be addressed to ensure that the difficulties encountered during Diet 2000 do not recur, we believe that immediate action needs to be taken to ensure that:

• there are adequate staff and other resources in place for Diet 2001;
• key software amendments are tested and implemented; and
• potential problems in marker recruitment are assessed and addressed.

< Previous | Contents | Next >