The Scottish Government

« Previous | Contents | Next »

Listen

EXECUTIVE SUMMARY

BACKGROUND

The Scottish Executive has an aim of continuous improvement in the quality of services for children and young people. The flow and presentation of information between schools, local authorities, the Scottish Executive and SQA is an important element in achieving this aim.

In August 2001, the National MIS Project was relaunched as ScotXed. This was to create a consistent means of electronic data exchange. ScotXed is a partnership programme involving all parties in the Scottish school education system. It aims to support and promote the electronic exchange of information through effective and secure systems for the exchange of data and by providing access to high quality information for educational partners in Scotland.

As part of its commitment to continuous improvement in the quality of services, the Scottish Executive identified a need to conduct an evaluation of the ScotXed Partnership. There were a number of key questions for the evaluation to address and these were:

• The extent to which the ScotXed partnership has supported and promoted the electronic exchange of data;
• The extent to which the partnership has minimised the bureaucratic burden;
• To ascertain views and perceptions of partners in the partnership in terms of their contributions to the Programme, their understanding of the Programme and so on;
• To assess available evidence on publishing benchmarks, the monitoring of the impact of educational policies, etc;
• To examine evidence to ascertain the role of ScotXed in policy development in Scotland.

METHODOLOGY

A staged approach was adopted for this evaluation. An initial quantitative phase provided robust statistical data that have been used to measure and monitor awareness and usage of ScotXed, attitudes towards ScotXed, the ScotXed Unit and the ScotXed Partnership. A subsequent qualitative phase allowed for the exploration of a range of issues in an in-depth manner. In the initial quantitative phase of semi-structured telephone interviews, a total of 319 interviews were conducted with schools and local authorities (126 headteachers, 127 school admin / support staff and 66 local authority staff). This was followed by 15 in-depth interviews with a range of stakeholder organisations with an involvement or interest in ScotXed such as HMIE, LTS, SQA and the Scottish Executive, and 20 focus groups with local authority staff, headteachers and school admin / support staff from across Scotland.

SUMMARY OF STAKEHOLDER VIEWS'

Of the stakeholder organisations included in this evaluation, some were involved in working directly with data from ScotXed, some indirectly, some were involved in policy development and were likely to drive requests for new and different data, and some had been involved in ScotXed from a co-operative or developmental viewpoint.

Almost all of those interviewed were positive about ScotXed, the types and amount of data generated and the use to which it was being put. Stakeholders also had an appreciation of why data requests are made and their fit with policy developments. In general, ScotXed was viewed more as a data collection framework that reacts to and evaluates policy rather than as a mechanism for driving policy forward.

Where stakeholders had had direct contact with the ScotXed Unit, views were positive. Those who had worked with the team had found them to be professional and easy to work with and responsive to requests.

Most stakeholders did recognise that secondary schools benefit more from ScotXed than primary schools, primarily because of the availability to secondary schools of Standard Tables and Charts ( STACs).

Most stakeholders did not view schools, local authorities and the ScotXed Unit as forming an equal partnership. Indeed, most would not have described this relationship as a partnership. Most perceived the Scottish Executive as being a dominant partner. They viewed the ScotXed Unit as driven by, and reactive to, policy developments and regarded the role of schools and local authorities as providers of data when requested. There was awareness that data would be analysed and interpreted at a local level, but stakeholders tended to focus on the data available at a strategic level and the process by which data were collected and analysed. There were concerns from one or two stakeholders about the ability of some school staff to interpret data and most were aware of, or assumed, that schools worked closely with local authority staff in interpreting ScotXed data.

Looking to the future, almost all stakeholders perceived interconnectivity and the sharing of information across different children's services to be an important development, although there were some queries as to how this would be funded and how issues such as the sharing of information can be overcome. One stakeholder noted that ScotXed offers a standard model that could be adopted by other agencies.

MAIN FINDINGS

The following paragraphs report the main findings of the research primarily from a school and local authority perspective.

ELECTRONIC EXCHANGE OF DATA

• The concept of electronic data exchange is supported by schools and local authorities across Scotland and use of MIS has increased in recent years. Furthermore, the quality of data collected for, and available from, electronic MIS is perceived to have increased hugely in recent years.
• A range of different MIS is currently utilised across Scotland, although the two key MIS used are Phoenix and SEEMIS.
• A majority of respondents claim to be familiar with the MIS systems used in their schools or local authority. The qualitative findings show that there is still a degree of reliance on MIS providers or local authority staff to provide support, training and guidance.
• A number of bespoke systems and applications have been developed across different local authorities in order to produce data specific to a school or local authority.
• The wide range of different MIS used, has led to some comments that there are inconsistencies in MIS across Scotland as a whole and that this can lead to inconsistent provision of information in schools and local authorities.
• There were some calls for the Scottish Executive to take the lead in introducing a common MIS across all schools and local authorities.

USAGE OF AND INVOLVEMENT WITH SCOTXED

• A range of roles is attributed to ScotXed. There is highest awareness of its roles among local authority staff and less understanding among school staff, particularly those in an admin / support role. Key roles attributed to ScotXed relate to the collection, collation and / or dissemination of data, although notable proportions of school staff are unable to attribute a specific role to ScotXed. While stakeholder organisations acknowledge the role of ScotXed in data collection, they primarily view ScotXed as a tool utilised for policy development.

• There is little perception of a ScotXed Partnership, particularly among schools where the focus of relationships is with local authority staff. This means that for many users of ScotXed, there is a sense of remoteness with the website and others within the Partnership as well as an inequality between different partners in the Partnership. Some local authority staff and many school staff see themselves as purely being providers of data.
• ScotXed is one of a number of channels used to provide or receive data, and the qualitative data show that there is a degree of confusion over MIS used, how ScotXed sits with other MIS and how the two interrelate. School staff accessing or providing data to ScotXed via SEEMIS or Phoenix are likely to attribute data exchange to SEEMIS or Phoenix rather than ScotXed.
• Staff in primary schools primarily interface with ScotXed to provide data or to download information for school handbooks. Within secondary schools, there are higher levels of usage, particularly for STACs.
• Local authority staff play a significant role in working alongside schools with STACs data. Among secondary school users some issues were highlighted regarding STACs in terms of its presentation and comprehensibility, particularly for those who do not have a statistical background.
• There appear to be some tensions between different users of ScotXed. For example, some local authority staff may disagree with policy makers as to what data to prioritise and some school staff perceive a contradiction between statistical measurement of performance and introducing more flexibility and innovation into the curriculum.

THE IMPACT OF SCOTXED

• Views from stakeholders, local authority and school staff on the impact of ScotXed are largely positive about the concept of electronic data exchange and respondents comment that data collection has become much easier in recent years, although this is not attributed purely to ScotXed. Stakeholder organisations tend to focus on the strategic role played by ScotXed in providing data for policy development. Many stakeholders are particularly enthusiastic about its achievements.
• There was a perception that ScotXed has helped to make data collection easier than previously, but that requests for data have increased. So while the bureaucratic burden of data collection and provision is seen to have decreased, many respondents perceived workloads in relation to additional data collection and provision to have increased, albeit that some data requests will come from other sources and not ScotXed. The quantitative and qualitative data also show that the bureaucratic burden on teachers has decreased and this has shifted to school administration staff.
• Views from those commenting on the impact of ScotXed are largely positive, although relatively high proportions of respondents - particularly those in schools - are unable to comment constructively on its impact across a number of dimensions, including its impact on the effectiveness of decision making within local authorities, its impact on policy development at a local authority level, its responsiveness to changing policy requirements, its usefulness in generating new types of data and so on. This is largely because these respondents view ScotXed purely as a mechanism for the collation and provision of data.
• When assessing a range of potential impacts of ScotXed, notable proportions of admin / support staff were unable to provide comment.

BARRIERS TO USAGE OF SCOTXED

• ScotXed's overall achievements regarding the quality and ease of data exchange and use - about which many, although not all, expressed positive views - are at the same time shadowed by a number of considerations.
• There are a number of resource based barriers which manifest themselves in a lack of available training (for example, on how to interpret ScotXed data or how to use ScotXed) and / or a lack of time (for example, to take up training, to find information or to input data), a lack of cover, and a lack of IT support. For many respondents, there is a lack of understanding of how they can optimise use of the data available via ScotXed and this can be reinforced by the use of external parties to provide MIS.
• One cluster of barriers relates to direct usage of ScotXed by some schools and local authorities and concerns comprehension of information and the consequences when comprehension is limited. These include:

• Some reports of poor understanding as to how to use the data accessible via ScotXed, and a related reliance on others to help interpret and analyse data;
• Irregular usage of and a degree of ignorance as to the capabilities of the website.

• Other more general barriers are the consequences of:

• ScotXed's weak branding among school users and how this can sometimes become confused with other applications of MIS. Therefore ScotXed sometimes fails to gain credit for its achievements;
• Some educators' attitudes towards making statistics an important focus in education;
• IT outsourcing, which can serve to exacerbate the disconnect between ScotXed and school users;
• Within schools the local authority is the conduit for news and information, and so there may be little or no direct connection made between ScotXed and its outputs or communications.

• Ways barriers can be overcome include:

• Increased resources such as more time to provide data to ScotXed as well as more time to work with the data available via ScotXed;
• Increased training opportunities, for example, on how to work with ScotXed data or time for greater numbers of staff to become familiar with the data available via ScotXed. Training also needs to be delivered at a point in time when it can have greatest impact on understanding and usage of ScotXed;
• Increased support to schools - both from local authority staff and the ScotXed Unit at the Scottish Executive;
• Promotion of the role played by ScotXed, the ScotXed Unit and the ScotXed Partnership - as well as a more pro-active role for the ScotXed Unit;
• Widening out of the data held by ScotXed to create more benefits (rewards) for primary schools in particular;
• Ensuring that the debate on the relative importance of statistical collections within education continues and is welcomed as part of the process of educational development.

INFORMATION AND COMMUNICATION

• For schools, connection to ScotXed can be indirect as local authority staff are the main channel of communication with and about ScotXed. All ScotXed guidance is tailored by local authorities to reflect local circumstances.
• Local authority attitudes towards ScotXed are more positive as they have direct involvement with the ScotXed Unit, ScotXed outputs and so on. The evaluation emphasises the importance of local authority staff to schools as the channel of communication.

• Local authority staff are generally happy with the information they receive from the ScotXed Unit and via meetings or involvement in Task Groups.
• Views on existing forms and channels of communication are largely positive, although there is a need to ensure that information reaches all those who need to receive it.
• Some school headteachers and admin / support staff would like different information channels to be used, depending on the nature of the information being provided.

TRAINING

• This evaluation indicates a number of areas where more training could be provided to help further ScotXed's aims, to help increase awareness and usage of the available outputs and increase understanding as to how collections inform educational policy and practice.
• A wide range of different forms of training was suggested, although all should be written in plain English and be easy to understand and comprehensible.
• Many respondents within schools express a preference for face-to-face training in an interactive environment. There are also some requests for training on how to interpret data produced by ScotXed. Opportunities for interactive questioning or examples of practical application of ScotXed, such as those provided by roadshows, would be welcomed, although attendance may not be a priority for many.
• The time at which training is delivered is important as lack of usage directly after training means that familiarity with ScotXed outputs can be lost.

THE FUTURE FOR SCOTXED

• The primary role of ScotXed is perceived to be the provision of data, and with the exception of stakeholder audiences, there is little emphasis on the strategic role it plays. Many in schools and local authorities associate ScotXed with statistical data alone, and this makes it less salient in some minds than some other educational initiatives such as A Curriculum for Excellence.
• Furthermore, ScotXed is perceived by some to lack coherence with other educational initiatives because of its emphasis on statistical data, although some stakeholders do perceive this to be complementary to current educational initiatives.
• In addition to those already mentioned above, a number of other potential improvements were suggested by respondents including:

• Possible adoption of a more direct advisory role for the ScotXed Unit, although there would be a need to agree this with local authority staff and stakeholders, so as not to alienate audiences that promote use of ScotXed;
• A need to ensure that data is more useful to schools and that schools understand both how to use it and why they are asked to provide it;
• Changes to the data that can be produced within ScotXed;
• Provision of a STACs equivalent for primary schools.

• There are also suggestions for a need to raise the profile of ScotXed.
• Longer term, many respondents see ScotXed playing an important role in joining-up children's services as a whole. Most support the concept of interconnectivity, although some have concerns over how practical this will be, particularly in the short term. Data protection, ethical and professional issues are all seen to impact upon the potential for joined-up children's services.

CONCLUSIONS

This evaluation indicates that ScotXed is achieving many of its objectives. Of particular note is:

• The support and promotion of electronic data exchange between schools, local authorities and the Scottish Executive;
• The contribution that regular ScotXed collections are playing in increasing overall MIS use and familiarity within schools;
• How many of those involved in the partnership such as stakeholders and local authority personnel are finding the data, and range of data provided, increasingly useful, especially for providing help with internal and external benchmarking, self-evaluation, and monitoring and improving pupil and staff performance;
• The appeal to educators of STACs, despite some issues with its format and difficulties in understanding its use and output by some users;
• Increasing confidence in ScotXed data itself, which users and most providers consider very reliable and solid.

Less success is evident in relation to:

• Delivering a concept of partnership within ScotXed, as many of those involved in collections, particularly at school and local authority level, perceive themselves as servicing ScotXed requirements, and frequently have little insight as to how their collections are directly affecting policy debate and decisions. ScotXed is not seen to be a driver in policy development, rather providing the necessary data in response to policy makers and their requirements;
• Communicating the full range of possible applications for the data, so that schools particularly may begin to see themselves as users, not just sources, of data.

There is mixed evidence as regards:

• ScotXed's success in reducing the bureaucratic burden on schools. The burden has clearly shifted to admin / support staff from teaching staff, and it is easier to collate data for ScotXed. However, increased demand for data, and for new and different kinds of data, from a range of different sources, appear to be cancelling out some of these benefits for admin staff.
• There is a need for the remoteness from ScotXed felt by some to be overcome and a need to improve communication with end users in order to ensure that all relevant parties understand the benefits of working with ScotXed.
• The role of the local authority as the main or often the only conduit, interpreter and controller of ScotXed information, data and advice to schools appears to amplify the impressions of some that ScotXed and the Scottish Executive operate far from the everyday business of a school.

« Previous | Contents | Next »