« Previous | Contents | Next »
Listen
Section Three: The use of Data to Improve Performance
3.1 There is no single model of CitiStat that has been piloted in Scotland. The four CitiStat case studies have all been structured differently and have developed in different ways. However, there are a number of common themes and distinctions highlighted here based on evidence from the case studies and from discussions at the CitiStat Exchange Inquiry event.
Actionable, rather than available data
3.2 The adage that 'you can't manage what you can't measure' has been used a number of times within the CitiStat Pilot process. CitiStat has raised the further question; 'what do we measure and why?' This is where some of the most significant developments have taken place in terms of using the process to develop actionable data, rather than available data and to develop indicators that are meaningful for service managers rather than simply to meet reporting requirements.
"We collect tons of data and we don't do anything with the majority of it. And it takes time and effort to do that and it's a case of 'well we might need it one day or someone may ask a question'. Through this whole process, [if it becomes widely adopted]… I would certainly advocate collecting the data that we require and getting rid of everything else."
"… and let that be an organic process and just dump everything else and that will give us some time back. Let's collect data on what's the theme and in a following month or another cycle something else crops up, let's start collecting data on that until we resolve that and then dump something else that we don't need."
"I think that's really come out strongly for us . . . we collect our data to submit it to the Scottish Executive and it is of no use at all to people running the service because we're measuring the wrong thing and we're not measuring enough. It takes two weeks to validate it, so by the time we know the [correct] answer… [it's too late]. One of the strongest things [to come out of this Pilot is] that the data are not useful in managing the system"
Improvements in data quality and validity
3.3 In all the case studies CitiStat has facilitated a review of existing performance indicators and data sources within the participating services. The process has often improved the consistency and accuracy of the data. It has also refined the indicators to more accurately reflect different elements of the issue they are measuring; for example, sickness absence has been a common issue and been broken down to the component parts, separating out short and long term absence.
3.4 This has led to the design of a new, more relevant and meaningful set of indicators and targets in which participants have been involved; this enhances the effectiveness of that data and makes it more suited to effective performance management. In one of the case studies, bringing forward the availability of data not only highlighted performance management implications locally but also showed that there had been inaccuracies in data previously used.
3.5 CitiStat has introduced a degree of rigour into the data collection and analysis process that wasn't there before. Data is frequently more consistent and based on sounder definitions and understandings. There is also a new degree of timeliness. The use of 'real-time' data has generally provided greater confidence of the utility of the data for decision making and brought a new immediacy to the process and more responsive feedback on actions taken. There is of course, a balance to be struck between accuracy and having data available quickly and there have been particular concerns about the use of unvalidated data within the Health Board case studies. However, the use of more recent data does allow a cycle of actions and testing to see what improvements result from service redesign, process change or re-prioritisation.
3.6 The importance of using 'real-time' data does vary across service areas and for many indicators trend analysis over a longer period of time will also be necessary to ensure that changes are real and not simply variations in the data. The use of time-series data and presentation in charts and graphs has encouraged discussion and action around prospective management issues. Whilst this process of data validation and review is not yet complete and comprehensive across all the case studies, the experience to date shows the real benefits of this aspect of the process.
3.7 The importance of data quality as a driver for the rest of the CitiStat process was a common theme across all the case studies; getting this element right is almost a pre-condition of an ability to engage in the full CitiStat process. It is an essential element of the CitiStat process, but not sufficient on its own. Many of the actions requested by the panels involved in CitiStat related to the provision of new or different data to refine or improve the data available. This produced better information on inputs and outputs. Some participants saw this as a largely bureaucratic exercise, unlikely to lead to change in its own right. Other case studies took a more collaborative approach to this process; for example the hot spots in Tayside were identified by senior managers from the acute and primary care divisions working together more closely and involving their service managers, as appropriate, to identify new indicators to monitor improvements. Most importantly, this process also promoted local ownership of targets.
3.8 A focus on the data contains a number of risks for CitiStat if the process is used to narrow perspectives and restrict the indicators used or alternatively, if the indicators are imposed and are not meaningful in a local context or linked to other related indicators.
"Data can easily be manipulated to be seen to achieve short term targets - these indicators can pull us in different and unhelpful directions."
3.9 The use of more accurate and valid data may still not be sufficient to enable understanding of the management problems. Further work to capture more relevant data may be required. The accuracy of data and its use both for management reporting and service redesign and management is crucially linked to the responsiveness of the process of developing the data and developing ownership of the data.
3.10 The model as developed has also been seen as biased in favour of quantitative measures and this may foster a perception that CitiStat is not suitable for smaller and rural authorities that don't have the same volume of data as larger, urban authorities or health boards.
"CitiStat needs a high volume of data - in the absence of this it doesn't know what to do with itself. Perhaps you need visible problems? You need real problems for senior managers not to think that their time might be best spent elsewhere."
3.11 However, a focus on quantitative rather than qualitative indicators is not necessarily an inherent feature of the CitiStat model despite the suggestion in the name. The experience of the case studies shows that qualitative data is also essential to develop a fuller understanding of the management issues and may also help to validate the significance of 'exceptions' or statistical outliers.
3.12 Qualitative data was generally not used in a systematic way in the Pilot but was certainly introduced to the discussions in a more impromptu way; for example, the views and experience of customers were referred to a number of times in the Edinburgh sessions. Service managers would give anecdotal evidence of operational issues and their impact on the public. Public perception of services was also a major underlying concern of the Panel. In Aberdeen the experiences of local wardens was discussed in connection with fixed penalty notices and resulted in a new performance indicator being introduced. Following their decision to continue with the CitiStat process, NHS Tayside now plans to incorporate qualitative data including tracking a few patients to map the process of delayed discharge and get a fuller picture of what is happening at local level. It also plans to use information from public and patient involvement events to influence CitiStat improvement indicators and targets.
How data has contributed to improving performance
3.13 The data analysis process has been shown to be crucial to the effectiveness of the CitiStat Panel.
"What in fact happened is the Panel became more intelligent and more educated and started asking more pertinent questions. The first round of questioning, the questions were a bit dull and woolly . . . and the awareness of the Panel grew and they're able to ask more searching questions".
3.14 The panels have provided the impetus and identification of priority areas for change. The model has been used both to tackle 'hot spots', or areas where there are agreed localised issues or problems that require urgent action; and to develop a longer term, collaborative whole system perspective that looks at wider issues and connectedness between different parts of the system in which the service is located. These approaches are not mutually exclusive. The act of identifying hot spots and then tackling them collaboratively across the wider system provides both focus and promotes integrated actions to tackle wider causes of trends.
3.15 Evidence about the contribution of data to performance improvement shows that the very act of scrutinising available data in this way can expose issues of poor performance; however, it also shows the limitations of focusing on narrow targets that may, on the face of it, appear to have been achieved, at the expense of the bigger picture. Data that appear to demonstrate good performance in terms of 'hitting the target' may in fact be masking a number of issues that might merit investigation if performance is to be maintained, and may also be contributing to other problems elsewhere in the wider system. The process has highlighted this tension between data designed for performance reporting rather than for driving forward continuous performance improvement.
3.16 The process has also shown that since no one person or organisation has an overview of all the data relevant to the operation of the whole system, then it is useful to develop a collaborative overview. This may be across teams within an organisation as, for example, in NHS Tayside or across organisations in a more formal partnership arrangement as in NHS Ayrshire and Arran.
3.17 The NHS Ayrshire and Arran experience of refocusing the CitiStat approach highlights the importance of data quality as a driver for the rest of the process; it suggests that if the desire is to develop a whole system perspective from the outset then the accompanying partnership arrangements have to be in place and all partners signed up to a joint and genuinely shared agenda at a strategic level. In the absence of this, or as a deliberate alternative approach, it may be that by tackling a series of hot spots, in some kind of priority order, that respective partners may be brought more readily into the process over time as the data may illustrate graphically to each partner their specific locus on the problem and underline the urgency of tackling it. In this way there may be scope for using the leverage of CitiStat to develop a long term and wider perspective, rather than simply focusing on 'quick fixes'.
3.18 The CitiStat process has been piloted through six cycles in three of the case studies and two cycles in the fourth; the case studies acknowledge that this is a very short timeframe in which to expect to see tangible, consistent improvements in performance. What has been achieved to date is that the process has exposed significant weakness in the knowledge base and has begun to develop process improvements to ways of working both within and across organisations. In the longer term, there is an expectation that this process will improve outputs and ultimately lead to performance improvements in measures that demonstrate outcomes.
« Previous | Contents | Next »