The Scottish Government

SCOTTISH EXECUTIVE EDUCATION DEPARTMENT

ANALYTICAL STRATEGY 2006

Also available in downloadable PDF format

Prepared by SEED Information and Analytical Services division and placed on Scottish Executive website October 2006

Introduction

1. The Education Department (ED) aspires to making the best possible use of information and evidence in carrying out its core business, developing and promoting the effective implementation of policy relating to education and young people, tourism, culture and sport.

2. This document sets out:

• Some broad principles which all ED staff will seek to embed in their working practices in pursuit of this aspiration; and
• Some key features of the practical working arrangements within the department, which are designed to ensure that policy and analytical specialists work effectively together to realise these principles in practice.

3. The document's purpose is to provide a clear and explicit framework for the Department's strategic approach to evidence-based policy-making and analytical work, thereby guiding and influencing the work of both policy advisers and analysts.

Background

4. ED policy advisers are organised into three broad policy groupings, the Schools Group, the Children, Young People and Social Care Group and the Tourism, Culture and Sport Group, currently comprising 13 policy divisions in all.

5. The staff of ED also includes three types of professional analysts: statisticians, economists, and social researchers. These three sets of analysts are brigaded together to form the core of the Information and Analytical Services (IAS) Division, a free-standing division within the Department. IAS Division provides specialist analytical support for all the policy divisions in the department.

6. In addition to informing the process of developing and introducing new policy, information and evidence plays a key role in monitoring and reporting on the delivery of the Department's objectives. The corporate processes of reporting on, for example, the delivery of Partnership Agreement commitments, place specific requirements on information and, in some cases, other forms of evidence. Analysts need to work effectively with policy colleagues and with the business management function (also within ED-IAS) to ensure these processes work effectively.

Guiding Principles

7. Six key guiding principles have been adopted by the department to help guide the development of the relationship between policy-making and the analysis of evidence:

• High quality information and evidence are crucial to effective policy-making. The most basic questions of policy-making depend on information and evidence for their answers: what is happening in the real world? What outcomes are being achieved? What do we think the effect of the options available to Ministers might be? What has been tried elsewhere? Are our actions and interventions having the desired effect - and, if not, why not? How do we get the best value out of the Executive's limited resources?
• Wherever possible, significant policy decisions should be taken with the benefit of a clear, comprehensive and robust body of relevant information and evidence. This principle is at the heart of evidence-based policy making. It does not mean to say that information and evidence should automatically determine the decision which will be made. Equally, it is important to recognise that situations will arise when good information and evidence are not available when a decision has to be made. But the more that relevant information and evidence are available when key decisions are made, the more successful the policies and programmes consequently taken forward are likely to be.
• The primary purpose of analytical work conducted within the Department is to inform future policy decisions. There are a few exceptions to this: sometimes work will be carried out as part of monitoring arrangements to which the Department is committed (for example, on the delivery of Partnership Agreement commitments); and some work will be carried out, or carried out in a particular way, because it is agreed it will thereby be of use to those working in the field (that is, people who wouldn't normally be described as "policy-makers"). But, for the most part, the value of analytical work conducted within ED is expressed though the extent to which it informs policy-making within the Education Department.
• Analysis should be undertaken and delivered impartially and should provide challenge as well as support. The process of discussing the analytical contribution to policy will at times involve analysts in asking difficult questions of policy-makers - but all as part of the process of maximising the value of analytical work to policy-making. At other times, analysts may be given a more formal role of checking, validating and where appropriate querying the conclusions of policy-makers - sometimes described as a 'challenge function'. In doing so, it is important to be clear where the authority for this challenge function has come from; usually it will be the case that analysts are carrying out this function on behalf of the Head of Department, in relation to returns being made by the Department to 'the centre' (for example, in Pre-Expenditure Assessments). It is important in those circumstances that analysts are perceived positively and receive full cooperation in carrying out that role.
• Evidence needs of policy should be planned for and met in an integrated, coherent way and matched flexibly to the demands of the policy cycle. We need to recognise that different types of evidence are needed at different points in the policy cycle and plan to ensure that it is delivered at the right time. For example research evidence and economic analysis may be particularly important when assessing the range of initial policy options and when evaluating the impact and value for money achieved by a policy after a period of implementation. Statistical evidence may be important in clarifying the nature and extent of the issues that a policy is addressing and in monitoring the success of a policy as it is being put into practice. Whatever the mix, it is likely that all policies will require a variety of different types of specialist analytical input and we need to ensure that is delivered in a seamless fashion.
• information and evidence should be seen as a public resource. This is already a principle of the operation of the Government Statistical Service; and reports of commissioned research work are routinely published at present. With the advent of Freedom of Information, we can be required to release the information and evidence which has informed a decision. In this context, the Executive as a whole is committed to taking an open, proactive approach to making our information and evidence base publicly available. This is in addition to guidance applying to professional statisticians, for example, regarding public release of statistical data.

Principles into practice

8. These principles have a number of direct practical implications for policy-makers and analysts:

• We need to give policy makers quick, flexible access to a clear and reliable body of information and evidence, supported by authoritative professional advice from relevant analysts. Given the unpredictable and sometimes fast-moving nature of policy decision-making, the analytical function needs to be ready to provide quick advice on the body of information and evidence available at any given point given time, while taking steps, where appropriate, to improve what will be available for the longer term. And sometimes it will be the deployment of relevant professional skills itself which will be as important - for example, in applying economic skills to an analysis of policy options.
• policy-makers and analysts should work together to identify future evidence needs and ensure they are met in good time to inform key policy decisions . The analytical function needs to work with policy colleagues to identify the specific pieces of analytical work which are likely to be of most value, based on a shared understanding of the key policy decisions which are expected to be made, and of the lead times involved in developing particular analytical outputs (e.g. statistical collections). In order to achieve best value from the use of the limited analytical resource available to the department, IAS Division may sometimes need to agree relative priorities amongst competing demands for analytical work through discussions at Group Head and departmental level.
• policy-makers should endeavour to keep analysts informed of the ongoing development of policy, whether or not there happens to be direct analytical involvement in the policy process at that stage. The capacity of analysts to contribute to discussions with policy-makers on the potential value of analytical work depends on a strong and continuous understanding of the policy process on the part of analysts. Simple actions on the part of policy-makers, like routinely copying analysts into key communications (e.g. submissions) in their policy field, can be of great help in maintaining this understanding.
• Analytical work has to be fit for purpose. If it to be sensibly relied upon, each element of the information and evidence base has to be of an appropriate quality. In not all circumstances will it need to be as perfect as in others, but analytical work should always be of as a high a professional standard as it is useful for it to be (also taking into account possible wider uses of the data in the future).
• Analysts have to be able to combine evidence from different specialisms to present an integrated evidence base to policy makers. Information and evidence which will inform a policy decision doesn't necessarily organise itself along professional boundaries. What is important is that the analytical function has the capacity to deploy information and evidence which addresses the questions of the day, whichever professional group has been involved in its production, and whichever professional skills have been deployed in the process.
• The department should encourage its agencies and other relevant public bodies to promote effective use of evidence in their practice and seek to maximise the potential for synergies in the production and sharing of analytical work. Through its links with sponsored NDPBs and in linking with relevant Executive Agencies (particularly Historic Scotland, HM Inspectorate of Education and the Social Work Inspection Agency) and other relevant public bodies, the department should encourage the effective use of analytical evidence and, where appropriate the development of capacity to undertake high quality analytical work. IAS Division should link actively with relevant bodies to ensure mutual awareness and where relevant promote active co-ordination and/or collaboration on analytical activities. Wherever appropriate IAS should draw on evidence and analysis generated by such bodies, taking full account of it in providing advice to policy divisions and Ministers.
• The department should seek to maximise productive links with other departments and their analytical divisions to ensure 'joined-up' and fully-rounded evidence is fed into policy. There are many specific policy areas, for example youth justice, social care, health, enterprise and lifelong learning to name but a few, in relation to which the department has common interests with other departments and needs to liaise very closely with other departments over the generation and interpretation of evidence. To ensure this happens, policy divisions will foster appropriate links with their colleagues in other departments and IAS Divisions will pro-actively link with other ASDs as appropriate. In addition the department and IAS Division will seek to contribute fully to corporate, cross-cutting work aimed at improving the evidence base for policy in Scotland led by relevant parts of the Executive such as the Analytical Services Group, including the Offices of the Chief Statistician, Chief Researcher and the Chief Economic Adviser.
• Forward programmes of analytical work should be agreed between policy-makers and analysts prior to jointly presenting major elements of the planned programme for Ministerial approval. Once policy-makers and analysts have worked through the evidence needs of policy, and identified the pieces of analytical work which are most likely to be of value in making future decisions on policy, this programme should be captured in an analytical work plan. Ministers should then be invited to comment on and approve the major elements of the planned programme, particularly in relation to proposed statistical collections (which is specifically required for the Scottish Executive's Statistical Planning work) and major pieces of externally commissioned research work. Policy-makers and analysts should jointly own and be accountable to Ministers for the proposed work programmes they have agreed.

Getting maximum public value from the analytical work we undertake

9. There are a number of ways in which we can take steps to extract maximum benefit from the analytical work that is undertaken in the department, beyond the direct interaction that occurs between analysts and their colleagues in policy groups and divisions:

• Ministers welcome direct reporting to them on key aspects of the information and evidence base. Experience has shown that Ministers value direct reporting to them of information and evidence relating to their policy priorities, in a way which is not very well achieved by simply reporting on individual publications as they occur. Developing the information and evidence base and communicating it in a form which is helpful to Ministers is an important way in which analysts can add value to the work of the Department.
• IAS maintains appropriate links with the Department's agencies and relevant NDPBs with regard to their analytical needs and activities and seeks to share relevant analyses with them. It is recognised that there is clear public benefit in the department co-ordinating its analytical activity with that undertaken within agencies and NDPBs to maximise synergies, avoid inefficient duplication and to support these bodies in delivering their core objectives where possible. In particular there is a need to maintain working relationships with the Executive Agencies linked to the Education Department. A Memorandum of Understanding has been developed with HMIE, where the range and scope of direct collaboration is greatest and regular links are also maintained with SWIA and Historic Scotland.
• Where it can help drive improvement we are pro-active in making information and analyses prepared for policy purposes available to those delivering public services outwith the Executive. The wider potential value of analytical work should be taken account of in developing and agreeing programmes of analytical work, taking account of the expressed wishes of external stakeholders. And, in some areas, specific pieces of analytical work may be carried out with the express purpose of supporting management and performance improvement within the public services for which the Department is responsible (for example, through the preparation of the STACS product, which supports self-evaluation and quality improvement in schools). Where such work is taken forward, it should be on the basis of a clear statement of departmental policy on the purpose and nature of such outputs.
• We more generally make our information and analysis openly available to the broader stakeholder community and users of services where appropriate. Through publishing and disseminating statistics, research and economic analyses, we seek to promote open and informed debate about the nature, quality and performance of services in the Department's areas of interest. We engage directly with relevant academic and professional communities where this can significantly increase the positive impact of our work, and, through relevant publications, we promote appropriate levels of accountability for the quality of services to the public at large.

10. Obtaining maximum value out of ED's analytical work through the sort of approaches outlined above represents one way in which the Department can support Ministers' broad objective of promoting continuous improvement in public services across Scotland.

How we will recognise success

11. Taking into account all the above, we can envisage what success should look like for the operation of the analytical function within ED-IAS, and for its relationships with policy-makers.

12. We should have analytical services which have the capacity to:

• understand the Department's policy positions, and how they are being developed;
• work with policy-makers to identify the ways in which information and evidence can best inform the development of policy;
• manage and deploy a clear and reliable body of information and evidence in relation to the key policy areas of the Department;
• apply professional skills in a way which most effectively supports policy-making;
• advise on how the information and evidence base should be developed, working flexibly across professional boundaries where required;
• coordinate and collaborate with other parts of the core Executive and with Executive Agencies, NDPBs and other relevant bodies, drawing appropriately on evidence that they have produced whilst also sharing analysis produced centrally with them
• deliver specific pieces of analytical work to appropriate professional standards;
• publish a range of high quality, authoritative and well-respected analytical outputs; and
• support the reporting of delivery of the Department's objectives.

13. And relationships between the analytical function and its policy colleagues should be characterised by:

• a readiness on the part of policy-makers and analysts work together to identify that potential contribution of information and evidence to policy development;
• ready access for policy-makers to intelligent and useful advice on the information and evidence base drawing broadly on relevant sources;
• regular and open discussions between policy-makers and analysts about the priorities for developing the information and evidence base;
• a clear joint understanding between policy-makers and analysts of the anticipated value of each piece of analytical work being taken forward;
• clear reporting by analysts of progress in delivering programmes of analytical work;
• regular approval by senior managers of the overall shape and priorities of programmes of analytical work; and
• understanding on the part of policy-makers and analysts of the context in which each other work, including the observance of professional standards and codes of practice.

How we will organise ourselves to achieve success

Structure

14. As indicated above, the Department is committed to delivering an integrated analytical service to its policy makers. To facilitate this process, IAS Division has adopted an integrated team structure, combining statisticians, economists and researchers in each unit, through which it delivers its support. It has three integrated Analytical Support Units (ASUs) each of which is responsible for meeting the analytical needs of one of the three policy groups in the Department. Each of these ASUs is led by a senior analyst who is responsible for managing the work of the team across its full range of work.

15. Whilst these integrated ASUs are the primary organisational structure within IAS, there is also a need to ensure the Division maintains a high quality professional service in each of three distinct analytical disciplines. This requires clear professional leadership structures to coordinate professional support and development and to provide a professional lead on cross-cutting, generic issues. To achieve this, each of the three ASU heads has been selected from a different professional background and each has been allocated a clear role as the head of profession for their specialism in the department. Thus one ASU head is the Education Department's head statistician, one is its head economist and one is its head social researcher.

Planning

16. Each ASU maintains a workplan which is reviewed annually. Analysts of all disciplines work together to develop an integrated programme of analytical work in relation to a particular policy Division or Group, led by the senior analyst for the ASU. The main planning process takes place around March and April, with the aim of producing a plan targeted on policy divisions' priorities for the coming business year.

17. The centrepiece of the main planning process is a series of detailed discussions involving policy-makers and analysts of all disciplines, organised on a division-by-division basis. The basic purpose of these meetings is to establish a joint view of the likely information and evidence needs of the Division concerned, given the policy questions they expect to arise in the next 12 to 18 months. Policy-makers have a clear role in going into these discussions ready to describe, as far as possible, the policy process they expect to be engaged in over that period, and the questions which may arise within that process. Based on that, both policy-makers and analysts should consider possible ways in which information and evidence - the outputs of analytical work - can provide maximum value in helping answer these questions.

18. Following these meetings the ASU Head of Unit will agree a summary of the evidence needs of each policy area with the division head, and then proceed to draw together a draft analytical work programme for each Group as a whole, based on:

• the specific evidence needs identified by each policy division in the Group;
• wider departmental commitments to monitoring in specific areas;
• the views of external stakeholders where relevant; and
• any relevant policies on provision of outputs for the purposes of performance management and improvement within relevant public services.

This work programme should then be agreed with the relevant policy group, with the Head of Group ultimately signing it off.

19. Arrangements should thereafter be made by IAS to discuss with relevant Ministers the main elements of the planned programmes, in particular the proposed set of statistical collections and the main pieces of externally commissioned research proposed. These proposals will be summarised in a draft Business plan for IAS Division, covering the work of all analytical units. In finalising the Business Plan, further discussions may take place with specific policy divisions and groups to address issues raised by Ministers. The final Business Plan will then be presented to ED's Departmental Management Board for ratification.

20. Once agreed, the final Business Plan and work programmes will form the basis of two external publications: the Statistics Plans for the year ahead (one for each portfolio); and the planned external research programme (which invites expressions of interest to tender).

21. Approximately mid-way through each year, there will be a lighter round of meetings to allow any necessary refreshing of the evidence needs statement and analytical work programme. In the light of these mid-term reviews, it may be necessary to seek Ministerial approval for changes to analytical activities and priorities.

22. Throughout the year, ASU Heads will use the work programme as the basis for monitoring and reporting on progress with analytical work. In addition, the evidence needs statement will be referred to both in the specification of specific pieces of analytical work, and in communicating the outputs of specific pieces of work back to policy, to ensure that work is taken forward in a way which maximises its potential policy value.

23. ASUs, individual policy divisions and groups should also use the work planning process to identify practical actions which either or both together could undertake to improve the effectiveness of internal working practices relating to analytical services. As part of their overall workplan, ASUs will identify on an annual basis a short set of internal improvement activities, designed to improve their impact and effectiveness.

Review of this Strategy

24. This strategy sets out broad principles for the analytical work undertaken within the Education Department and describes a range of ways in which the department intends to put these principles into practice. The Strategy will be reviewed annually by the Departmental Management Board to ensure that its principles remain relevant and to monitor the progress being made in applying them in practice. Further versions of the Strategy will be produced in the light of this annual process of review.

Scottish Executive Education Department

August 2006