The Scottish Government

« Previous | Contents |

Listen

This evaluation explores the experience of the CitiStat pilot in Scotland. CitiStat is a performance management system which originated in Baltimore, Maryland, USA which uses recent operational data on key performance indicators as the basis for discussions between the Leader of an organisation and service Directors. The CitiStat model has been adapted and piloted by four case study organisations during 2005-06 within the Scottish public sector.

Key lessons

CitiStat brings a number of distinctive elements to performance management.

• The high level involvement of Chief Executives and senior non-Executives focuses the agenda on improvement, scrutiny and accountability and gives a clear sense of organisational and partnership priorities.
• The analytical approach supports more effective accountability through the Panel. The challenge process is an important driver, provides opportunities for praise in recognition of achievements and gives service managers confidence to implement required changes.
• Enhanced data quality and analysis are key to the process and set it apart from existing performance management models.
• CitiStat provides a focus on actionable data, rather than available data. It enhances the accuracy, validity, quality and utility of data and strengthens the links between the data, performance and agreed actions. The process has highlighted the distinction between data designed for performance reporting rather than for driving forward continuous performance improvement.
• CitiStat creates a dialogue that enhances mutual understanding and communication between participants, enhances corporate memory and can be a tool to promote closer working within and between Departments, agencies and partnership structures.
• CitiStat can be a driver for organisational and cultural change. It challenges established cultures and ways of working in the public sector.

Key findings

There is no single model of CitiStat that has been piloted in Scotland. The four CitiStat case studies have all had a distinctive structure and have developed in different ways. However, each has used the basic processes of the CitiStat model as shown in Figure 1. This illustrates the links between the analysis of agreed performance indicators, the CitiStat brief to the Panel, the Panel process itself in which service Directors account for their performance and agree further actions, the evaluation process and the follow up brief which feeds into the next cycle.

The case studies have generated some useful lessons for any further implementation of CitiStat in Scotland. Despite the relatively short duration of the Pilot, there are a number of positive signs of change that illustrate the potential value of the model.

There was consensus that the Pilot process had largely been a success and the case studies would wish to retain most of the elements of the model. Particular elements that have worked well include:

• The flexibility of the model to fit into existing governance and accountability structures and be structured in ways to suit different organisational contexts.
• The high level involvement of Chief Executives and senior non-Executives.
• The challenge process of the CitiStat Panel, including praise and recognition of achievements.
• Enhanced data quality and analysis.
• Stronger links between the data, performance and agreed actions and the focus on generating action and change.
• Dialogue between key stakeholders that enhances mutual understanding and communication.
• The use of CitiStat as a driver for organisational and cultural change.

Two case study organisations are to continue using CitiStat after the Pilot phase has ended, one is to pursue the issues raised through a series of local meetings and the fourth is to consider recommendations on the future use of CitiStat during the early summer 2006.

One of the most significant developments has been in terms of using the process to develop actionable data, rather than available data and to develop indicators that are meaningful for service Managers rather than simply to meet performance reporting requirements.

There is no conclusive evidence that any one CitiStat model is better than any other. The approach to the Pilot illustrated the value of allowing CitiStat to develop according to the spirit of the Baltimore model, reflecting different perspectives and local contextual possibilities, rather than through strict adherence to formal structures and procedures. This might be termed 'reinvention' rather than 'replication' of good practice; it allows for local customisation and ownership of quality improvement approaches and facilitates the integration and use of evidence in practice. It is thus an important principle that any future developments should allow the model to be adapted to suit the context.

This has been a relatively short-term Pilot in which it was not expected that there would be substantial tangible improvements in service delivery or identifiable savings which could clearly be attributed to the Pilot. There were some early, limited indications of improved performance, including improvements in sickness absence in at least two of the case studies, which participants have ascribed to the CitiStat process. It has not been possible to undertake a full assessment of the value for money of the approach. This is due to the short duration of the Pilot, where the costs of establishing the process are concentrated in the early months and lack of full cost data. The evaluation is able to identify what elements of CitiStat worked well in specific contexts and which the case studies feel are of sufficient value that they and others might continue to use the model.

Key issues for the future of CitiStat and the implications for further implementation

The report identifies key issues for the future of CitiStat and the implications for any further implementation within the Scottish public sector. These include discussion of the specific elements of the CitiStat approach that the case study organisations would wish to retain; whether enough is known to 'roll-out' CitiStat across Scotland and the sustainability of the process in the longer term.

Any further implementation of CitiStat will need to recognise the distinctive governance and accountability structures within different parts of the public sector (including health services, local government, government agencies and NDPBs) and ensure that CitiStat dovetails with existing requirements, rather than adding to reporting burdens.

Most features of the Baltimore model should be retained, although the model should not be imposed but implemented flexibly to allow its reinvention in different contexts. Whilst the spirit of the Baltimore model may be applicable in Scotland a key distinctive contextual factor in Baltimore is that there are no well developed audit processes or external scrutiny as in Scotland. This raises issues for any further implementation of CitiStat in Scotland about where pressure of performance reporting requirements might be reduced and how resources devoted to performance management and reporting might be focused more strategically to reflect the audit process and external scrutiny environment. Any further implementation of the CitiStat approach should ensure good communication about the primary purpose of CitiStat as a local performance management tool. CitiStat needs a dedicated resource if it is to be sustained.

Other key implications include ensuring sufficient resources are available to support the analysis and briefing process; making improvements to information systems to support efficient data provision and analysis; developing the scope of the model to focus both on hot spots and whole system working; developing an outcome focus; embedding on-going evaluation into the CitiStat process and continuing the more reflexive and learning driven approach to policy implementation demonstrated by the CitiStat Pilot.

About the Project

The CitiStat model has been piloted by the City of Edinburgh Council, Aberdeen City Council and NHS Tayside Health Board between September 2005 and February 2006 and NHS Ayrshire and Arran Health Board between November 2005 and March 2006.

The two local authority areas initially focused on environmental quality of life. This issue was chosen by the Scottish Executive because it was felt to be a measurable activity, important both to the public and to policy makers. The Pilot areas were selected to reflect varied council structures and were selected in consultation with the authorities themselves and the Convention of Scottish Local Authorities ( COSLA). The two Health Boards were selected by the Scottish Executive Health Department ( SEHD). The Tayside case study addressed the delivery of access and treatment services. NHS Ayrshire and Arran examined delayed discharge from hospital. Access and treatment and delayed discharge are both part of wider SEHD targets in relation to Health Improvement, Efficiency and Governance and Access and Treatment (known as HEAT targets) included in Health Boards Local Delivery Plans from April 2006.

The evaluation was commissioned in June 2005 by the Scottish Executive, Office of the Permanent Secretary, Performance and Innovation Unit ( PIU) and Office of the Chief Researcher and concluded in March 2006. The evaluation was carried out by Research for Real. The research team used an action research approach to the evaluation. This aimed to actively engage the Pilot area teams in the evaluation process generating useful, practical findings as it went along. Emerging findings were fed back into the on-going CitiStat process in each area. The participants in each of the case study areas decided their own indicators of success to best reflect the way the Pilot was developing in their area and local priorities.

If you wish further copies of this Research Findings or have any enquiries about social research, please contact us at:

Tel: 0131 244-7560
Fax: 0131 244-5393
Email: socialresearch@scotland.gsi.gov.uk
Website: www.scotland.gov.uk/socialresearch

If you wish a copy of "What do we Measure and Why? An Evaluation of the Citistat Model of Performance Management and its Applicability to the Scottish Public Sector", the research report which is summarised in this Research Findings, please send a cheque for £5.00 made payable to:

Tel: 0131 622 8283 / 0131 622 8258
Fax: 0131 622 8258
Email: business.edinburgh@blackwell.co.uk

This document (and other Research Findings and Reports) and information about social research in the Scottish Executive may be viewed on the Internet at: http://www.scotland.gov.uk/socialresearch

The site carries up-to-date information about social and policy research commissioned and published on behalf of the Scottish Executive. Subjects covered include transport, housing, social inclusion, rural affairs, children and young people, education, social work, community care, local government, civil justice, crime and criminal justice, regeneration, planning and equalities issues. The site also allows access to information about the Scottish Household Survey.

« Previous | Contents |