The Scottish Government

« Previous | Contents | Next »

Listen

Section Six: A Future for the Citistat Process in Scotland

Assessing the value of CitiStat

6.1 The focus of this evaluation of CitiStat has been the experience of the case studies in using the CitiStat model and the extent to which they perceive that it adds value to their current performance management and accountability structures. This has been a relatively short-term Pilot in which it was not expected that there would be substantial tangible improvements in service delivery or identifiable savings which could clearly be attributed to the Pilot. There was a strong interest in the value for money of the process at the CitiStat Exchange Inquiry event, although it was agreed that it was too soon to undertake a full assessment of value for money. The evaluation is able to identify what elements of CitiStat worked well in specific contexts and which elements the case studies feel are of sufficient value that they and others might continue to include them in the model.

What is the added value of CitiStat?

6.2 The CitiStat Pilot has generated some useful lessons for any future wider implementation of CitiStat in Scotland. Despite the relative short duration of the Pilot, there are a number of positive signs of change that illustrate the potential value of the model. CitiStat has been tested differently across the four case studies but in each case it has acted as an arena for problem definition, problem solving and strategic learning in relation to performance management and collaborative working.

6.3 The approach to the Pilot illustrated the value of allowing CitiStat to develop according to the spirit of the Baltimore model, reflecting different perspectives and local contextual possibilities, rather than through strict adherence to formal structures and procedures. This might be termed 'reinvention' rather than 'replication' of good practice. Reinvention allows for the local customisation of quality improvement approaches, increases the absorptive capacity or receptivity within organisations and so facilitates the integration and use of evidence in practice ¹. It is thus an important principle that any future developments should allow the model to be adapted to suit the context.

6.4 Given this approach, it is worth identifying what appears to be distinctive and different about CitiStat, in contrast to the existing performance management and accountability approaches familiar to the case studies, including traditional committee and Board processes.

Leadership

• The high level involvement of Chief Executives and senior non-Executives focuses the agenda on improvement, scrutiny and accountability and gives a clear sense of organisational and partnership priorities. The challenge process is an important driver and gives service managers confidence to implement required changes. It influences behaviours by highlighting respective responsibilities as managers for the implementation of policies.

Data Quality and Analysis

• Enhanced data quality and analysis are key to the process and set it apart from existing performance management models. CitiStat provides a focus on actionable data, rather than available data. It facilitates a review of existing performance indicators and data sources to enhance the accuracy, validity, quality and utility of data. The use of real-time data brings a new immediacy to the process and provides quicker feedback on actions. The analytical approach supports the more effective process of accountability through the Panel. The process has highlighted this tension between data designed for performance reporting rather than for driving forward continuous performance improvement.

Ownership and Actions

• Review of the data used for performance management enhances ownership of the indicators. CitiStat strengthens the links between the data, performance and agreed actions and whilst challenging, provides a strong and broadly welcome focus on taking action which has been empowering for some managers.

Dual Approach: Hot Spots and Whole System Working

• CitiStat creates a dialogue that enhances mutual understanding and communication between participants. It is applicable both to the identification, analysis and action around hot spots and also to a whole system approach. These are not mutually exclusive approaches. There is scope for using the leverage of CitiStat to develop a long term and wider perspective, rather than simply focusing on 'quick fixes'. CitiStat can be a tool to promote closer working within and between departments, agencies and partnership structures.

Cultural and Organisational Change

• CitiStat can be a driver for organisational and cultural change. It challenges established cultures and ways of working in the public sector.

Key issues for the future of CitiStat

6.5 The CitiStat Exchange Inquiry event discussed a number of questions about the future potential of the model. These included discussion of the following issues:

• What has worked well? - the elements of the model that the case studies would wish to keep.
• Whether enough is known to implement CitiStat further across Scotland.
• How CitiStat could be sustainable over a longer period.

What has worked well?

6.6 There was consensus that the Pilot process had largely been a success and that participants would wish to retain most of the elements of the model. Particular elements that have worked well include:

• The flexibility of the model to fit into existing governance and accountability structures and be structured in ways to suit different organisational contexts.
• The high level involvement of Chief Executives and senior non-Executives.
• The challenge process of the CitiStat Panel, including praise and recognition of achievements.
• Enhanced data quality and analysis.
• Stronger links between the data, performance and agreed actions and the focus on generating action and change.
• Dialogue between key stakeholders that enhances mutual understanding and communication.
• The use of CitiStat as a driver for organisational and cultural change.

6.7 Two case study organisations are to continue using CitiStat after the Pilot phase has ended, one is to pursue the issues raised through a series of local meetings and the fourth is to consider recommendations on the future use of CitiStat during the early summer 2006.

6.8 The main concern about adopting CitiStat more widely was the adversarial or confrontational nature of the strict Baltimore model. There were also concerns that CitiStat may not be as effective at long-term, complex, slow moving and strategic issues, for example, around health inequalities or death rates. There were also concerns that it may be seen as biased towards quantitative measures. Nevertheless, as these concerns were not tested fully during the Pilot, it was felt that CitiStat has the potential to be developed further so that it can be used to examine these longer term complex issues and develop more of a focus on service outcomes.

Potential to implement CitiStat further across Scotland

6.9 Consideration of the scope to implement CitiStat more widely across the public sector in Scotland highlighted a number of concerns. Importantly, further implementation was often seen to imply the mandated adoption of a standardised model across agencies. This idea was strongly resisted; the diversity of the case studies and short duration was felt to provide inconclusive evidence about the value of any one version of CitiStat which might be widely applicable. The value of the model was felt to be in adopting the spirit of the approach and allowing further exploration of its potential. This would require a more reflexive and learning driven approach to policy implementation. It was also suggested at the CitiStat Exchange Inquiry event that any further implementation should be supported by the adoption of the CitiStat model by the Scottish Executive itself.

6.10 The tension between data designed for performance reporting rather than driving forward continuous performance improvement fuelled concerns that further implementation may imply additional accountability requirements and there was a plea for simplification and integration of performance reporting requirements of the Scottish Executive.

"There's an awful lot going on. I think the Scottish Executive needs to make greater progress on the integration of inspection and auditing functions. There's a disparate range of approaches in place at the moment. This makes it difficult, in my view, for a smaller authority to achieve an integrated approach to performance management."

6.11 In this context, a number of important principles for any future implementation were identified. These were that performance should be measured against targets that matter to local delivery agencies such as local authorities and Health Boards; that the Scottish Executive should not ask for information on things that are not as useful or meaningful as they could be for performance management and that duplication of effort should be avoided by streamlining the timetables and demands of different Government Departments. There should also be a greater focus on service outcomes, with local flexibility and freedom to decide how best to deliver those outcomes.

6.12 Discussion of 'roll-out' illustrated differences in understanding of the purpose of CitiStat. The Scottish Executive has always seen it as a local performance management tool and had provided resources to assist agencies in developing the Pilot in quite a new way, as discussed in section 5. However, case study organisations had a concern that 'roll-out' would see CitiStat used as a national management tool and that there would thus be a tension between local influence over targets and the ability to use CitiStat for benchmarking performance across different agencies.

6.13 Indeed, the CitiStat Pilot process illustrated the tension experienced by case studies of using the model as a local performance management tool in the context of continuing performance reporting requirements to the Scottish Executive. The implications of this are that the model needs to be fused with the information requirements for Ministers and with existing internal organisational scrutiny processes and be streamlined so that it does not add to bureaucracy. Any pre-existing commitment to performance improvement should not be underestimated in any roll out.

6.14 The model has the ability and flexibility to introduce measures for improvement that can sit alongside Ministerial targets or Performance Indicators. Genuine accountability could be improved where there is greater ownership of improvement areas, rather than working to imposed targets. The Scottish Executive could set expectations of improved performance management, but the Health Boards or local authorities could problem solve their own improvement areas in order to deliver better results. Such an approach would combine both a compulsion to improve, with the freedom to choose both the focus of improvement and the process.

"In thinking about driving organisational change, it's back to this thing about 'feeding the beast'. Presumably the most powerful approach is if you can get your baseline data to actually reflect what's happening in services and that then drives the CitiStat and the performance management data so you're going into the Scottish Executive, rather than the other way about. Whereas now, it's 'you know we've got all these ministerial targets to meet' and therefore we need to set up information systems to drive that."

6.15 The wider implementation of CitiStat raises questions about the need for better communication about the primary purpose of the model; whether it is a model for national modernisation in public services, a local performance management tool or a mechanism to develop more effective partnership or collaborative working or indeed, some combination of these elements. The Health Board case studies in particular had concerns about how CitiStat might be used.

"I think there are really important lessons about what the Scottish Executive are going to do in creating a policy structure and what Ministers want out of what's going on in local authorities and health. What does that mean and is it about management, or is it about signing off a government structure that they trust us to implement, or is it about falling all over the minutiae? So I think there is an important stock-take at the end of the Pilot…where they'll decide what they're going to do to us."

Sustainability of CitiStat

6.16 There were a number of concerns about the sustainability of the model.

"It's kind of stating the blatantly obviously that if you get really senior people in an organisation in a room and they bang heads together and ask for things to change, miraculously they do! And that's why I do worry about the sustainability - can we continue to make that happen? I'm not sure that that is a sustainable model for very senior people in big complex organisations."

6.17 The Pilot has exposed some of the wastage of resources that are expended in collecting data that is old, collected for historical reasons and is therefore not necessarily useful for performance management purposes. CitiStat itself requires considerable investment of resources and would only be sustainable if there is strategic alignment between audit, scrutiny and accountability arrangements to allow resources to be focused more strategically and take pressure off other areas. Whilst the spirit of the Baltimore model may be applicable in Scotland, a key distinctive contextual factor in Baltimore is that there are no well developed audit processes or external scrutiny. In Scotland this raises issues for any future implementation of CitiStat about where pressure of reporting requirements might be reduced and how resources devoted to performance management and accountability might be focused more strategically.

6.18 The evaluation processes has been an embedded part of the CitiStat Pilot and an important element of the CitiStat process in Scotland. They have enabled reflection and discussion on how the model could be refined for the local context and have contributed to the 'reinvention' and refinement of the model in each context. On-going evaluation should be encouraged as part of the CitiStat process to continue and cascade a more reflexive and learning driven approach to policy implementation as demonstrated by the Pilot.

Summary of implications for wider implementation

6.19 This section summarises the implications for any wider implementation of CitiStat within the Scottish public sector.

Implications for the Scottish Executive

• Ensure any further imlementation of CitiStat retains the flexibility of the different models used in the Pilot process, rather than imposing a single model.
• Recognise the distinctive governance and accountability structures within health and local government and ensure than CitiStat dovetails with existing requirements, rather than adding to reporting burdens.
• Ensure clear communication about the primary purpose of CitiStat as a local performance management tool.
• Retain the spirit of the Baltimore model without the confrontational element. Do not impose the adoption of CitiStat but be flexible and allow for the reinvention of the model in different contexts.
• Continue the more reflexive and learning driven approach to policy implementation demonstrated by the CitiStat Pilot.
• Consider the use of CitiStat within the Scottish Executive to support any wider implementation of the model.

Implications for public sector agencies

Purpose of CitiStat and integration

• Consider how the model may be used to enhance collaborative working and formal partnership arrangements and to provide performance management of partnership agreements.
• Ensure there is clarity of purpose for CitiStat in each context. Identify clear strategic drivers of the process and link CitiStat to Community Plans and service plans.
• Align Panel and Podium membership with the strategic purpose of the process.
• Consider how the model might align with existing scrutiny processes to avoid duplication.
• Ensure alignment with existing accountability and governance structures in local authorities, health boards and other public sector agencies.

Establishing the process

• Explore potential roles for elected members and non-executives.
• Consider the best use, number and role of observers.
• Provide support for pre-planning of CitiStat to ensure agreement on the most appropriate and manageable focus.
• Consider how best to retain the spirit of the Baltimore model within the specific organisational and cultural operational context.
• Change the labels if necessary - adopt appropriate terminology for the context.

Implementation and focus

• Retain the key emphasis on enhancing data quality and analysis, the use of 'real time' data, senior level involvement and the link between data, performance, scrutiny and action. Ensure relentless follow through of actions.
• Develop key indicators in collaboration to ensure ownership.
• Ensure appropriate data is collected including qualitative data to enable fuller understanding of management problems.
• Ensure sufficient resources are available to support the analysis and briefing process.
• Consider use of critical friends in the process to provide technical understanding of service issues and support more effective and informed questioning.
• Review how staff at all levels might be brought into the process to enhance motivation and ensure that CitiStat is more widely understood.
• Ensure information systems support efficient data provision and analysis. Consider methods to capture 'real-time' data from service delivery contexts.
• Focus on longer term and systemic issues as well as hot spots.
• Develop a focus on outcomes, rather than solely inputs and outputs.
• Explore how the model could incorporate the views of service users, patients and the general public.

Evaluation and learning

• Embed on-going evaluation into the CitiStat process
• Continue the more reflexive and learning driven approach to policy implementation demonstrated by the CitiStat Pilot.

« Previous | Contents | Next »