< Previous | Contents | Next >

A Review into Exam Results Issues Concerning the Scottish Qualifications Authority

6. FINDINGS - INFORMATION SYSTEMS

6.1 Introduction

The SEB and SCOTVEC had very different systems platforms and approaches to systems development so the alignment of information technology and systems was one of the key challenges facing the SQA on its formation. The SEB had an IT team of six and tended to use external contractors to develop applications, whereas SCOTVEC had a team of 13 and undertook most of its development in-house.

The SEB's Exam Processing System (EPS)²⁷ and SCOTVEC's RET system²⁸ were built on different technical platforms. EPS had been outsourced whereas RET was an in-house development. Neither system could fully meet the requirements of the SQA as:

• the SEB's system did not support internal assessment and group awards;
• SCOTVEC's system did not support external assessment; and
• neither system could support the requirements of Higher Still and a single cumulative certificate without further development.

A SCOTVEC/SEB joint IT working group (JIT) was formed in 1995 but neither organisation was keen to migrate to the other's environment and agree to a common strategy. One of the recommendations made by JIT was that it should meet with HSDU to clarify requirements and to explore and resolve issues raised²⁹ However, no meeting was held with HSDU, and JIT was disbanded in 1996.

In December 1996, Admiral consultants were appointed to undertake an investigation into the information systems requirements of the SQA. The recommendation made with regard to the core operational systems was to migrate to a single database by developing SEB's environment towards an open platform solution. The following specific recommendations were made³⁰

• introduce middleware³¹ and new programs to integrate the existing separate data sources and fulfil the requirements of Higher Still in order to provide a single virtual database as an interim solution;
• migrate to a single solution based on an open database and platform in the longer term;
• invite competitive bids for the interim and longer term solutions, for example from CA-Ingres and Oracle; and
• further develop in-house programming skills to undertake this work.

The report also stated that the "creation of the single business tool will not be a simple matter and it is recommended that it should only be achieved in the longer term, after the implementation of the Higher Still programs".

In March 1997, the SQA appears to have agreed the development of a new computer system (see section 6.2), the Awards Processing System (APS). APS is based on the same technical platform as the SEB's EPS system³² The hardware was tendered but the database solution was not as there was a existing investment in Ingres. Recommendations for an interim solution were not implemented by the SQA.

The APS Project Board was set up in November 1997 to take the development forward. The Board met every six weeks and was attended by the project sponsor, project manager, IT manager, lead users and a number of other individuals from the business at various points. A lead business user was appointed for each module with responsibility for specifying the business requirements for that module and an IT project leader was assigned to each module with responsibility for development.

The eight APS modules, the key functions they support and the main inputs and outputs are represented below (figure 2). Note that links between modules are not shown, only links to external systems.

The planned and actual delivery dates of these software modules and their key components are given in Appendix IV.

The SQA currently has an IT Unit of 35 staff, including 15 in the development team, with all managers having split site responsibilities.

Centres use a number of different management information systems (MIS), the most common ones being:

• SEEMIS (used in the previous Strathclyde region's schools and further education colleges);
• Phoenix (schools system);
• Integris (used in schools in East and West Lothian); and
• SITS (used in further education colleges).

Of these systems, only SEEMIS users have the facility to transfer data electronically to the SQA (via a hub in Hamilton). All other MIS users submit their data on disk or by e-mail.

In March 2000, the Scottish Executive's Deputy Director of IT met with members of the APS Project Board and highlighted that systematic testing of APS was crucial including volume testing and data interchange with Centres.

Figure 2 - APS Overview³³

6.2 Strategy

The SQA has not given enough consideration to IT/IS at a strategic level, evidenced by the fact that there is:

• no definitive IT/IS strategy; and
• no one at SMT level in the organisation with significant experience of IT.

This is surprising given the nature of the SQA's operations and its reliance on information systems.

The process that the SQA followed to go down the APS route is unclear and we have been unable to establish exactly who took the decision and when. SMT Minutes of March 1997 make the following implied reference to the APS decision³⁴:

"It was agreed that the proposed action could now be carried forward without any further involvement of the Senior Management Team at this stage."

Interestingly, during the course of this review, two of the SMT Directors questioned whether the APS development was, in fact, necessary. However, there is no evidence that these views were expressed at the time.

It is acknowledged that the SQA rejected the middleware option presented by Admiral as an overly complex and expensive approach with no guarantee of success. However, no alternative interim solution, based on further development of the existing EPS and RET systems, was implemented which left the SQA exposed to risk in completing Higher Still awards and certification if APS failed. While there was a fallback position available for Standard Grades and SCE Highers by reverting to the legacy systems (which was in fact used) there was no viable alternative option for Higher Still. With no fallback in place for Higher Still, the SQA was reliant on APS being delivered in time. As the Director of Awards remarked:

"Without the new software, there could be no new examinations in 2000".

The SMT appears not to have recognised the risks associated with the decision to implement APS with no fallback position for Higher Still, and the significant impact that this new development would have on the organisation as a whole.

6.3 Project Management

The timescales for APS were ambitious but the IT Unit was staffed to take account of the increased workload and a detailed project plan was produced. However, no overall budget or measurable success criteria were set for the project.

The project was set up according to the Prince methodology and a project board, sponsor, project manager and a number of lead users were identified in line with good practice. However, the project board members varied, with up to fifteen people present at any meeting.

The project manager did not have experience of managing a large systems implementation and was not given the authority needed to manage all aspects of the project. So, while management of technical delivery was undertaken, there was no-one managing and controlling the overall project and, as a result, the quality of specification, user testing and training was not ensured.

One lead user, the Head of Operations Unit, was assigned responsibility for five of the eight APS modules and was also given the role of senior user assurance co-ordinator with responsibility for ensuring that³⁵:

• acceptance criteria and user acceptance test plans were established and applied; and
• materials for users - such as manuals and training materials - were available.

These were substantial demands on one individual, particularly as this lead user was also responsible for managing one of the key units in the SQA and there were existing concerns about this individual's performance. Consequently, the responsibilities were not carried out effectively. Although action was taken to assign some tasks to others in April 1999, the demands on this individual remained, in our view, excessive.

The project board identified risks but did not appear to assess these adequately or carry out effective risk management in all cases, for example:

• one lead user had responsibility for five modules and user testing; and
• slippage was identified as early as April 1998³⁶ but not corrected, with the result that further slippage occurred.

However, other risks were properly addressed, for example:

• a decision was taken to carry out results processing for Standard Grades on the Cobol module of the existing EPS system, in order to ease pressure on the APS project team; and
• steps were taken to hire contractors and to utilise an existing relationship with development company Edina for additional work on Results Processing when developer resource became an issue.

Although the project was set up using Prince, it does not appear to have been followed during the lifecycle of the project. While clear guidance was given on style and coding standards and change control, in practice these were not always maintained due to time constraints.

6.4 Requirements Specification

The requirements for Higher Still were not defined to the necessary level of detail until late in the project lifecycle. While the Higher Still Development Unit defined the policy, some practical aspects of implementation had not been sufficiently considered. The Joint IT working group raised this as an issue as early as 1996. The flexibility that is viewed as one of Higher Still's strengths caused additional complexity in designing a system to support the qualification.

In addition to the lack of detailed definition, the lead users who were tasked with specifying the requirements for the new system did not have adequate knowledge of how Higher Still was intended to work and not all user groups were consulted at the requirements gathering stage. However, failings in the specification of requirements could not be attributed solely to lead users as a memo to SMT and Management Team from the APS project chair in April 1999 illustrates37

"Heads of Unit should take a pro-active role in making sure that their needs have not been overlooked."

As a result of the above, the quality of the specification provided by lead users was often of a poor standard and invariably produced later than planned. The IT project members then had to devote much time with users to guide them in the development of business requirements. An increased emphasis was, in consequence, placed on software prototyping. The high-level requirements gathering was planned to take three months, but actually took nine. There were instances when specification was not completed prior to development commencing and even instances of specifications not being completed prior to testing, for example the development of Component Maintenance (Entries module) was completed in June 1999 but the specification of this element was not completed until August 1999.

6.5 Data

Centre and candidate data was transferred from the legacy SEB and SCOTVEC systems into APS. Centre details, approval records and candidate details were migrated. However, the existence of duplicate candidate records necessitated a huge data cleansing exercise. This was not completed accurately, with the result that the data in APS was flawed from the start. The following reference to the migration of legacy data is made in the IT Unit Plan 2000/01³⁸

"Often any discrepancies in the merged data sets were not being uncovered until major programs such as invoicing or certification actually took place in the live environment. The review of the data by the business via sampling was a failure. We will need to keep the content of the merged data sets under constant review for the next six months".

Data is accepted in a number of different ways from Centres:

• electronic file transfer from SEEMIS systems via the SEEMIS hub;
• on disk or e-mailed from other MIS systems including Phoenix; and
• hard copy by post or fax.

The SQA maintains that it is necessary to give Centres this choice of options as they use a variety of systems, even though it adds to the complexity of data processing. The IT Unit is currently developing "SQAnet" which will allow Centres to submit files via the SQA's extranet. This is likely to simplify data submission and handling.

Data submitted from Centres was rejected due to lack of compliance with often trivial validation checks. The Centres would resubmit this data, not having received feedback from the SQA, and further errors and possibly duplicate entries would be generated in accordance with the rules embodied in APS. The APS project manager summed up this issue as a "failure to take on Centres' requirements and to specify the Entries validation correctly".

Data processed out of sequence, due to the backlog of work in Operations Unit, was again rejected by the system. For example, if a Centre's entries were processed before the corresponding registrations, the entries would be rejected as the candidates did not yet exist on the system.

From March 2000 the Operations Unit requested that Centres submit data on paper. The reasons for this are unclear. However, the move to paper forms increased the time taken to input data to the system and would also have reduced the accuracy of input data.

A large amount of data which was missing on 9 August currently resides on spreadsheets external to APS, and is in process of being uploaded to the system in order to generate revised certificates.

6.6 Centres and Suppliers

There were several meetings held between the SQA and MIS suppliers in 1999 but, despite these meetings, suppliers consistently complained of a lack of open discussion with the SQA.

Data was not provided by the SQA for supplier testing until August 1999 and was reissued in November in a complete form. This lack of data for testing, together with changes made by the SQA to the specification, are the reasons cited by some suppliers for the late issue of software to Centres. For example, Integris software was not available until December 1999 and SITS software until February 2000. However, other suppliers delivered their software in advance of SQA's ability to process data. For example:

• SEEMIS went live with essential facilities in September 1999;
• by October schools had completed most registrations and many entries;
• by November most schools had completed entries and had started entering results; but
• the SQA was not able to accept and process SEEMIS' entry data until November.

The SQA offered test processing of supplier files but few Centres and suppliers took up this offer and the SQA did not pursue this.

Conflicting messages were given to Centres and suppliers by the SQA at different times and by different people. For example, the statement of results was specified originally as an electronic file but suppliers were later informed that this would be issued on paper instead. The SQA then decided to issue the statement of results electronically but only informed the suppliers on the day prior to issuing the results file.

In an e-mail39 from the SQA’s Head of Operations Unit to Phoenix, the confusion which was generated by the SQA is again illustrated:

"… with regard to the internal assessments which can contribute to courses. As you know, this is information which we asked for on paper… Although we didn't ask for it to be provided electronically I thought that maybe it was recorded on their Phoenix system anyway. It may be of course that because we indicated that this would be a paper-based exercise that it is not held on the Phoenix system by Centres."

The late delivery of software by at least two MIS suppliers resulted in late processing of registrations and entries by the SQA and was further compounded by some Centres' decision to delay sending any data until July 2000. However, for the majority of Centres, with the exception of a few instances that the suppliers quickly addressed, there were no key issues with the submission of Centres' data to the SQA. References have been made to problems with data transmission and blank and faulty disks. However, these were isolated cases and do not appear to have contributed substantially to data management problems.

Ineffective SEEMIS file tracking undertaken by the SQA created the situation where on at least one occasion (3 May) the SQA did not pick up the SEEMIS files for processing and this was not detected for several weeks (July in the case of the 3 May incident). There is no audit trail in place for SQA's handling of these files.

There are a number of cases where a Centre did not submit complete and accurate information to the SQA, for example:

• one high school did not enter any of its pupils for one of the internal units in Higher Still English; and
• a number of instances have been found of schools not submitting estimates for all courses, or submitting them incorrectly due to unfamiliarity with the new Higher Still grading structure.

However, better checks and reporting from APS would have allowed the SQA to identify these errors to the Centres at the appropriate stage.

The relationship between the SQA and the Centres has been stretched to the limits over the last year, with Centres reported to have suffered a complete loss of confidence in the SQA. In SQA Board minutes from as early as March 2000 the problem is noted⁴⁰

"There is no doubt that the introduction of a new qualification numbering system and new data management procedures has placed a strain on relationships between SQA and its Centres."

However, insufficient action was taken to address this critical issue.

6.7 Testing

Insufficient emphasis was placed on testing as evidenced by:

• a lack of formal test scripts or plans;
• insufficient user acceptance testing for some modules;
• staff identified as testers being withdrawn to undertake processing, leaving staff without sufficient knowledge of the business rules or testing procedures to attempt to implement the test programme;
• insufficient volume, ‘end to end’ and integration testing;
• a failure to test the system with a number of concurrent users until actual live running;
• a failure to pursue to any degree the testing of data input from suppliers' systems; and
• the abandonment of automated testing by Compuware, due to a lack of time.

Testing was rushed due to time pressures. As the Director — Awards Division, chairing the APS project, stated:

"The testing window of four weeks disappeared to nothing".

The issue of the business placing insufficient priority on testing was raised at the project board meeting in August 1999 and the following recommendation was made⁴¹

"More emphasis needs to be placed on planning testing by lead users and software should not be in phase 1 of user testing for longer than 28 days".

However, there is evidence that the business has still not recognised the importance of testing as some ‘fixes’ delivered by IT staff have remained in user test for several weeks without being actioned.

Not surprisingly, given the lack of a complete specification to test against, the quality of testing varied and when the software was delivered into the live environment a large number of change requests were raised.

6.8 Delivery

Software was delivered later than planned, particularly Assessment Support and Results Processing, largely due to delays introduced at the specification stage. Appendix IV charts when modules were actually delivered against the planned dates. IT staff typically use the phrase that software was delivered:

"in time as opposed to on time".

However, in some cases this late software delivery appears to have had an impact on the business. For example, internal assessment forms and EX6 forms were delivered late. It is difficult to ascertain which other software delays had a direct result on the business as, due to the various dependencies, the points at which the business needed to start using the modules was also changing.

The quality of the delivered software generally met the original specification, but was not necessarily the best fit to support the business processes given the issues on specification.

There were performance problems, for example with Approvals⁴² which was raised at the project board meeting in August 1999 and was subsequently addressed. There were also performance issues on certificate file generation as this took substantially longer than expected and resulted in additional certificates having to be printed at Dalkeith as opposed to the external printers at very short notice.

IT Unit has undertaken steps to address the performance issues including:

• appointing Computer Associates to carry out a health check on the Ingres database and make recommendations, the majority of which were then implemented;
• appointing IBM to carry out a similar health check on the Sequent hardware platform; and
• undertaking redesign and coding to address locking problems.

The abnormal data processing pattern and system usage that has been experienced this year also caused performance problems as the system was not specified to cope with the actual volumes of record processing and concurrent usage that have resulted.

Users are still complaining about performance of the system, though lack of availability may well be due to IT staff carrying out system maintenance during working hours, or business requests to process data urgently. What can be said is that sufficient volume and load testing has not been undertaken to ensure that performance difficulties will not arise again next year.

There is no evidence to suggest that APS does not process data correctly. Instances when users have complained of data going missing or not being processed have been the result of poor system design and a lack of user training leading to user error, as opposed to errors in the software itself.

User training was the responsibility of the lead users and again the quality of this varied. Due to the time constraints on the business, training was not given high priority and comprehensive training manuals are not available.

6.9 Information and Reporting

It has already been stated that the quality of specification was generally poor and this included specification of control checks, status reporting and management information. For example, the validation checks and error reports specified proved unusable due to the size and quality of the reports.

The lack of management information during Diet 2000 can be attributed to a number of factors:

• reports necessary for progress tracking and decision making were not available from APS due to a failure of the business to place sufficient importance on specifying the management information requirements;
• users were not always aware of available reports or how to generate them;
• available reports were not always in a usable format;
• users did not spend time examining available reports due to other workload pressures; and
• as the system can only produce reports on the data that has been entered, the necessary information wasn't always available from APS due to delays in data processing.

Members of the Senior Management Team were aware of the lack of management information. One example, quoted by a Director during interview for this review, related to the provision of reports for the uptake of vocational qualifications that had been used for monitoring purposes for several years. However, from December 1999 no such information had been available to managers. Information such as this is necessary for effective monitoring by senior management and we are surprised that more robust action was not taken to investigate its absence at an earlier stage. It certainly made it difficult for the SQA to identify the problems at an early stage and identify their extent and nature.

6.10 Conclusion

It appears that information systems were not viewed strategically within the SQA. The decision to embark on APS with no contingency plan in place for Higher Still was high risk and the failings in certification could have been considerably worse had the new software not been delivered in time to support most of the awards processing activities in 1999/00.

APS was treated as an isolated project as opposed to being part of an holistic business change process. The large amount of staff time required on the project, the delivery of software later than planned, and the introduction of a new system with its associated learning curve all impacted on key processes at a time when the SQA was already experiencing operational difficulties.

Due to the problems encountered in specifying business requirements there are some concerns that:

• the delivered solution does not adequately support all of the required processes;
• the screens are not always user-friendly;
• there is a lack of in-built controls; and
• the poor specification of reports, both in terms of completeness and quality, are having a significant impact on awards processing.

The SQA did not place great enough emphasis on testing and training with the result that users experienced problems when the system was delivered and problems with system performance further impeded the users in processing Diet 2000. These problems arose from a variety of causes and, although steps have been taken to address them, concerns remain.

The quality of legacy data meant that APS held inaccurate information even prior to registration in 1999 and this situation worsened during Diet 2000. There is evidence that this has not been completely rectified and that duplicate candidates and units still exist in APS. In addition, the system does not currently hold the most up-to-date information as the clear-up process was implemented using spreadsheets not linked to APS.

The variety of ways in which the SQA accepts information from Centres adds to the complexity of data handling, but not to actual data processing by APS. However, Centres and their MIS suppliers have not been given enough clear guidance from the SQA to date on what is required of them. For example, in the absence of any information from the SQA:

• one Centre's SQA Co-ordinator stated that they were intending to submit a disk every two weeks so that they could be sure they were keeping the SQA up to date; and

• one supplier referred to feelings of "déjà vu from last year" as the SQA had not yet issued the Approvals file and the supplier had advised its Centres not to submit any data until guidance has been given.

Ultimately, while late delivery of software and limited ease of use of the system added to the pressures on the business, APS was not the root cause of the SQA's problems and the system itself appears to process data and results correctly. While there are still concerns about the reports available from the system, the more crucial problems were that information was not always available from the system due to delays in data processing and the failure of the business to use the management information that was available from APS effectively. This meant that the SQA was not able to recognise problems at an early stage and to identify their extent and nature.

< Previous | Contents | Next >