< Previous | Contents | Next >

A Review into Exam Results Issues Concerning the Scottish Qualifications Authority

8. RECOMMENDATIONS

8.1 Introduction

Recommendations to ensure that the problems encountered do not recur in future years depend for their success on a number of factors, but most critically the ability of the individuals within the organisation to implement the recommendations effectively.

Moreover, implementation of the recommendations to address the perceived weaknesses within the SQA needs to be viewed in the context of both the current situation within the SQA and the requirement to progress Diet 2001 effectively.

At the present time, the SQA is in a fragile condition, with management and staff both feeling extremely bruised and demoralised following recent criticisms. This is likely to be compounded with the publication of this report and the reports of the two parliamentary inquiries. It is important to recognise this context in making and implementing recommendations. Equally, it is important to acknowledge that the vast majority of staff worked exceptionally hard over long days during a lengthy period of time in a desperate attempt to ensure that Diet 2000 successfully achieved its 10 August objective. That the failure was not greater is due to the diligence of the vast majority of staff who were involved in the process.

At the same time, the management and staff are already under tremendous pressure in respect of next year's Diet. Since mid-August 2000, the SQA has been focused on the clear-up and appeals processes for Diet 2000 and has deliberately delayed commencing data processing of registrations and entries for Diet 2001 with the aim of resolving some of the existing problems. Whilst this has been necessary and unavoidable, it means that delays are already evident in Diet 2001.

Given the above, it is essential that the identification and implementation of recommendations are sensitive to the current context and business pressures, and that the timeframe is also realistic. In the light of current circumstances, the recommendations contained in the sections that follow are split between those that need to be implemented immediately with a view to assisting in the successful administration of Diet 2001 and those that should be regarded as secondary objectives, in the sense only that whilst essential they are less constrained by time.

We have not proposed an allocation of responsibilities for implementing these recommendations because of current uncertainties in management structure and future remits. Indeed, the Acting Chief Executive has already recognised the need to strengthen existing management by tapping external resources and skills.

The SQA is currently implementing a number of initiatives in order to address some of the problems highlighted in chapter 4, chapter 5 and chapter 6 of this report. Appendix V summarises the action that the SQA has taken to date and these actions address some of the recommendations made below.

8.2 Management and Organisation

8.2.1 Immediate

Based on the information available to date, it is sensible to be concerned about the ability of the SQA to administer Diet 2001 successfully and appropriate to recognise achievement of that objective as the biggest challenge and, therefore, the main priority for the SQA in the immediate future. The Scottish Executive, the SQA Board and the senior management of the SQA need, therefore, to balance the need for desired management and organisational change with the ability of the organisation to implement change whilst focused on the challenge of Diet 2001.

In terms of successfully administering Diet 2001, it is recommended that the Acting Chief Executive should:

• develop a detailed understanding of the requirements of Diet 2001 and a detailed operational plan for achieving a successful administration of the Diet. This plan must prioritise actions and should:
  • detail the requirements against the timeframe, specifying the critical path;
  • assess the ability of the organisation to handle the changed / increased workload and produce a risk assessment;
  • detail the additional resources, for example people, facilities, hardware, services and funding that are required to achieve the Diet 2001 objective;
• communicate the plan, requirements and risks to the SQA Board and the Scottish Executive and secure commitment to proceed, including assistance where required, or obtain alternative instructions;
• review his Senior Management Team to ensure that he has the necessary skills and experience to lead the organisation through Diet 2001 successfully, and gain their commitment to the challenge ahead. If additional skills and experience are required they must be sourced immediately;
• clearly define the roles and responsibilities of the Senior Management Team against the detailed plan for the year;
• working with his Senior Management Team, make such changes to the organisational structure and operating process that are viewed as essential to achieve the Diet 2001 objective;
• as part of the above, produce a detailed plan for deploying / redeploying existing and new staff resources including the handling of any outstanding personnel issues;
• establish appropriate communication channels, including regular meetings where required with the SQA Board, the Senior Management Team, the wider management team and the staff of the SQA;
• ensure clear and effective communications with all stakeholders to ensure a common understanding of the developing situation, including arrangements for feedback from stakeholders;

Finally, the Scottish Executive and the SQA Board need to give immediate consideration to the current interim management arrangements. The Acting Chief Executive is on secondment for an initial period of six months, extendable by agreement of both parties. A serious crisis could be created if the Acting Chief Executive were to decide to terminate his secondment at the end of the six month period.

8.2.2 Secondary

Beyond the immediate actions necessary to achieve Diet 2001, there are a number of recommendations in respect of the management and organisation of the SQA that arise out of this review. These are as follows:

• review the Management Statement and Financial Memorandum relating to the SQA to clarify and remove anomalies, such as the respective responsibilities of the Chairman and Chief Executive with regard to overall management of the SQA;
• review the number of members of the SQA Board of Management with a view to reducing this number from 25 to a smaller number that will facilitate more effective Board management;
• notwithstanding the above, we do not recommend a radical overhaul of the structural arrangements for the corporate management of the SQA. Similar arrangements apparently work effectively within other NDPBs and we see no reason in principle or design why they should not work effectively for the SQA;
• develop a more effective operational planning regime to bridge the gap between the Corporate Plan and operational management and implementation, which was a significant weakness during 1999/2000;
• review both the management and wider organisational structures of the SQA. Both are historical amalgams of the predecessor organisations. The current directorate structure does not represent a sensible or effective allocation of responsibilities in terms of either staff numbers or functions, whilst the management structure has too large a management team (21 Heads of Unit). Similarly, the wider organisational structure is the embodiment of the predecessor organisations and does not match the changed operational requirements of the business;
• there are differing views of the effect of the current split-site arrangements on both the operational and cultural aspects of the organisation. The SQA should consider the feasibility of a reduced site, including single-site, operation and develop a business case for consideration. However, the difficulties of moving to a single-site operation need to be fully considered together with the potential benefits. The timeframe for achieving a single-site operation also needs to be considered in detail given the on-going operational expectations on the SQA;
• cultural change is a medium-to-long-term objective rather than a short-term one. However, it is essential that the SQA starts to improve the management style and individual and group behaviours within it, such that an SQA identity and culture develops from the still prevalent SEB and SCOTVEC identities that contribute to some of the organisational tensions;
• the EFQM Business Excellence Model should be considered as a template for quality improvement and change management within the SQA, and a self-assessment of the organisation against criteria within the model should be carried out to establish the current position and areas for improvement;
• ISO Quality Management Systems should be considered as a framework for ensuring that processes and procedures are documented and systematically audited, both internally by trained auditors and externally by independent assessors;
• ‘Investors in People’ accreditation should also be considered in order to co-ordinate the development of people to meet the future needs of the organisation; and
• best practice benchmarking against leading organisations should be initiated to gain knowledge of world-class systems and processes.

8.3 Operational Processes

8.3.1 Immediate

The following immediate recommendations are made in respect of staffing and facilities:

• undertake an assessment of the level of staff resources required for Diet 2001, taking into consideration the ongoing delays and the timing and volume of data processing. The assessment of staff requirements needs to take into consideration the ongoing resources required to address the outstanding issues in respect of Diet 2000;
• take steps to address the low morale and fatigue within the Operations Unit given the sustained effort needed throughout Diet 2000;
• review the working practices within the Operations Unit and take steps to ensure that there is adequate supervision of all core business activities and where possible that multi-skilling is encouraged where practical;
• if deficiencies in the staff complement are identified, take steps to ensure that new and/or temporary staff are given adequate training;
• consider the introduction of shift patterns during peak processing periods;
• establish stronger links between Centres and the SQA in order that any queries can be resolved in an efficient and effective manner; and
• undertake a review of the physical layout of the Dalkeith site and take steps to improve the data handling and data processing areas.

The following immediate recommendations are made in respect of data management:

• prepare a detailed resource and time plan to identify how and when the remaining steps required to complete Diet 2000 will be concluded and at what stage APS will be updated with all the data required to prepare amended award certificates;
• prepare a detailed plan and processing timetable for all the core business activities for Diet 2001. Nominate a project manager to monitor each stage and ensure that immediate action is taken to address any problems or processing delays;
• hold briefing sessions with all Centres to address matters where difficulties arose during Diet 2000 in respect of the provision of data;
• implement procedures to deal with errors identified during data validation routines to ensure that all SQA staff and Centres clearly understand their respective responsibilities;
• advise Centres of the dates by which internal assessment and estimation data need to be submitted. The dates set should provide sufficient time for data to be processed and missing assessment information to be followed up prior to the start of the final examinations;
• introduce robust data control and handling procedures for each business activity to ensure that data is received and processed in a controlled manner.

The following immediate recommendations are made in respect of examination procedures:

• contact all markers employed during Diet 2000 immediately to acknowledge their contribution this year and to ask for their continued support in future diets;
• re-evaluate the basis of remuneration of markers in the light of the experience of Diet 2000;
• take steps to ensure that marker invitations are issued no later than January 2001. Contingency plans should be made to address any difficulties that may occur during the recruitment process and to ensure that the process is not delayed in the event of continued processing difficulties over registration and entry information;
• review the examination and marking timetables to determine whether the reduction in the time made available for markers can be sustained; and
• review the administration arrangements for marker meetings and take steps to ensure that the difficulties encountered during Diet 2000 are not repeated.

8.3.2 Secondary

The following additional recommendations are made:

• consider options to simplify the actual data that has to be submitted by Centres for internal assessments such as:
• a default for unit assessments with exception notification of different combinations;
• one final end date for internal assessments as opposed to an end date for each unit; and
• submission of failed assessments only with the default being a pass;
• on the introduction of any new developments, ensure staff within Operations Unit are adequately briefed on the requirements;
• re-consider the job specification for the Head of Operations and give consideration to the benefits of employing a person with a background in data management; and
• improve the design of forms used to provide data at all stages.

8.4 Information Systems

8.4.1 Immediate

The following recommendations are made in respect of APS:

• focus on high priority APS amendments and SQAnet this year and limit other systems development projects;
• prioritise feedback from units on APS according to the business benefit and risks involved and only undertake changes viewed as critical this year. Examples of amendments which are likely to be a high priority for this year are:
• improved data processing controls and sense checks;
• the improvement and creation of key operational and management reports; and
• the migration of Standard Grade results processing to APS;
• ensure that APS can provide management information to support each stage of the awards procedures including appointment of markers, identification of duplicate units, and completeness of internal and external assessment data. An exercise should be undertaken to specify all online enquiries and standard reports that are required by users, unit managers and the SMT. This must include cross-unit and cross-divisional reporting requirements. As with changes to APS, these report requests should be prioritised;
• produce a formal testing plan as part of the planning stage of APS enhancements and test scripts for the complete system. A dedicated testing team should be nominated by the business and IT Unit should be responsible for briefing them with regards to good testing practice. Testing should be undertaken on the complete system, not just changes, and it is important that full end to end testing is undertaken by users;
• set up a dedicated test environment with a copy of the live data so that load and volume testing can be undertaken to a degree which actually tests the real-life situation;
• specify the level of system performance and resilience needed to deliver Diet 2001 successfully and undertake a thorough test of performance against this specification. The performance assessment should cover optimisation of the database design and queries and may require further external assistance, including the possible use of Compuware to undertake some automated testing;
• limit system maintenance to outside of normal working hours so that it does not impact on users and give the impression of an unreliable system; and
• ensure that the current versions of the APS database platform and software tools remain supported and, if necessary, agree special terms with vendors in order to provide this (as upgrades should not be attempted during Diet 2001).

The following recommendations are made in respect of project management:

• appoint a project manager with responsibility over all aspects of the APS project and the authority to carry out that role effectively. This project manager should be suitably trained and experienced in order to minimise the risk to delivery. If necessary, skills should be seconded in order to provide the appropriate level of expertise;
• undertake effective risk management and apply the Prince methodology throughout the project lifecycle;
• appoint a lead user for each process supported by APS and ensure they have the authority and responsibility for gathering feedback, specifying and prioritising requests and working with IT Unit on an ongoing basis to deliver and test improvements; and
• ensure that appropriate refresher training on APS is undertaken by all staff and that adequate training on all future enhancements is scheduled and executed prior to such enhancements going 'live';

The following recommendations are made in respect of data quality, input and controls:

• ensure the accuracy of the existing data prior to adding to it this year. A thorough and complete data cleansing exercise should be carried out to remove all duplicates and ensure accuracy of the core data;
• establish clear ownership of and responsibility for data:
• responsibility for the accuracy of candidate, entry, internal assessment results and estimates data should reside with the submitting Centres;
• data owners should be appointed within the relevant SQA departments with each having responsibility for ensuring the accuracy and completeness of their particular data set;
• revisit validation rules in order to produce a clear set of checks agreed between the business units and IT Unit to ensure only 'hard' or business critical errors are rejected. The revised validation rules need to be communicated to the Centres and their suppliers and tested prior to live use;
• revisit system controls and checks to define more practical rules, for example around the generation of duplicate unit entries, and to ensure there are adequate completeness of data checks for each of the core business activities;
• improve enquiry and reporting to provide visibility of processing status within and between units and to the Centres. For example:
  • send automatic electronic confirmation reports to Centres;
  • undertake exception reporting on data which is late in being submitted;
  • introduce better quality error messages; and
  • report back to Centres on all data submitted so that estimates and results are confirmed as well as entries; and
• establish automatic file tracking procedures for the SEEMIS data uplift so that both receipt of a file and processing of a file by the SQA are acknowledged.

The following recommendations are made in respect of Centres and their MIS suppliers:

• produce a user guide for Centres to explain what is being done at each stage in the process and why, and to inform Centres on the action they should be taking. This must include dates for data submission, and the preferred frequency and format of submissions; and
• participate in more open discussion with Centres and suppliers and provide clearer statements of requirements. Examples of where this needs to take place are on the revised validation checks, the possible simplification of course and assessment data, and the layout and functionality of SQAnet.

8.4.2 Secondary

The following secondary recommendations are made in respect of APS:

• undertake a formal review of APS, reconsidering the existing functionality and user-friendliness of the system and challenging the original specification; and
• ensure that APS is properly documented both at a user and a technical level.

Considering the strategic importance of information systems to the SQA, the following recommendations are made:

• make IS/IT strategy and systems delivery a priority for the SMT and the Board. This should include ensuring there is a reasoned business case for projects;
• consider appointing a Director of IT given the current and growing importance of this area to the SQA's business;
• develop an IT/IS strategy in line with the business strategy and maintain the strategy in order to support the business objectives;
• view systems projects holistically, considering the business change involved;
• place increased emphasis on forward planning of both strategic and operational IT/IS requirements; and
• make better use of technology, for example scanning and SQAnet in the short term. Longer-term options such as IT based assessment should involve full consideration of business benefits and risks prior to making informed decisions.

The following additional recommendations are made in respect of data input and controls:

• redesign forms so that hand-written results are not required, for example by using checkboxes with marks for pass/fail/defer unit assessment results;
• introduce scanning of forms such as the assessment results, estimates and EX6 forms to ensure still greater accuracy of data input as well as a higher throughput;
• provide an electronic option for all data and incentivise Centres to submit data using this option as opposed to using paper forms; and
• provide an electronic delivery mechanism (SQAnet) for all Centres and, again, incentivise Centres to use this method for submission as opposed to providing files on disk.

The following additional recommendations are made in respect of Centres and MIS suppliers:

• all Centres should be able to access SQAnet, perhaps with the exception of some smaller employers, but certainly all schools should ensure that they have the facility (i.e. a computer and an internet connection) to use this; and
• service level agreements should be considered with Centres, for example the undertaking to process and report within one week of electronic data submission, and two weeks for hard copy data submission. This will assist in establishing standards of service and confidence in the SQA.

< Previous | Contents | Next >