« Previous | Contents | Next »
Listen
6 Tracking progress and assessing performance
6.1 The third and final question in the TAMFS paper was: How can progress be tracked and performance assessed?
6.2 Before presenting a summary of respondents' replies to this question, it is important to note that, in general, respondents strongly agreed with the premise that progress should be tracked and performance should be assessed in this area. None of the respondents questioned, or disagreed with, the need to measure progress, and indeed a number of respondents called for the development of more robust accountability processes between local areas and the Scottish Government.
6.3 However, two individuals pointed out that any system to track and assess performance must be able to take into account the complexity of the issue.
… Large numbers of people coming for help with bereavement issues may look bad statistically, but actually it is better that they come for help rather than languishing in the misery of loss. (Individual respondent)
Things may get worse before they get better, which doesn't mean changes in direction are required, e.g., if GPs make referrals for early intervention, then uptake of these services could increase, but the long-term impact may be that referrals to acute services will reduce. (Multi-agency consultation event)
6.4 Although respondents generally agreed that it was important to measure outcomes for people, some suggested that it might be difficult to measure outcomes in certain areas - for example, in relation to preventing self-harm. In this case, it may be necessary to rely on indicators which measure the quantity and quality of local services.
6.5 There were a range of views about what form performance monitoring should take, and in some cases, there were opposing opinions on the matter. However, the vast majority of respondents suggested that progress should be tracked and performance assessed through one of three methods:
- Through the use of existing agreed indicators and targets
- Through evaluation of projects and initiatives
- By seeking the views and feedback of service users
Use existing agreed indicators and targets
6.6 The majority of respondents felt that, as much as possible, local area performance in relation to mental health improvement should link into existing, nationally agreed indicators and targets such as, for example:
- Single Outcome Agreements (local authorities)
- HEAT targets ( NHS)
- Scottish Recovery Indicator
- National mental health indicators (under development by Health Scotland)
6.7 There was a less common view which argued for a move away from "a target-focused mentality." However, the much more prevalent view was that local areas should not be burdened with additional targets and reporting structures.
6.8 Many felt that the Warwick Edinburgh Mental Well-being Scale ( WEMWBS) was useful, and should be used across Scotland to measure changes in population mental health and evaluate the impact of projects and other initiatives. One individual specifically requested whether it might be possible to include WEMWBS in community health profiles. Another asked whether national data from WEBWBS could be made available at local authority level, as well as health board level, to help local authorities assess their performance and progress.
6.9 It should be noted that some respondents expressed concern that local authorities may not include relevant mental health indicators in their single outcome agreements. One individual said that:
"While supporting the setting of objectives and actions for delivery at a local level, we believe it is unsatisfactory that the question of targets for mental health and social care should be left to each individual local authority." (Voluntary sector respondent)
Evaluation of projects and initiatives
6.10 Evaluation was proposed as another method for measuring progress and assessing performance, and several respondents suggested that evaluation should be built in to all planned activities. One of the advantages of evaluation was that it would allow more qualitative data to be collected on outcomes, and it could present a more rounded picture of the process of change, using a variety of methods.
6.11 Respondents suggested that the Evaluation Guides developed by Health Scotland were useful for this purpose.
Seeking the views and feedback of service users and carers
6.12 There was widespread agreement among respondents across all sectors that it was important to involve service users (and their carers) in the ongoing development of services that affect them. Some suggested that much more needed to be done to seek honest feedback from service users and carers when measuring the impact of services - and to act on that feedback in a positive way.
6.13 It was suggested that seeking the views and feedback of service users and carers could be done in a variety of ways:
- Before and after surveys
- Use of self-assessment tools
- Measuring other behaviour change (smoking, drinking)
- User satisfaction questionnaires
- "Stress questionnaires" (to measure wellbeing at work)
- Applying user ratings to services (in the same way that accommodation and restaurants are graded)
- Asking service users and carers what they think
6.14 One respondent praised the work of Heads-up Scotland in meaningfully involving children and young people in issues that affect their mental health and wellbeing. This individual called for similar work to be undertaken across Scotland particularly if children and young people and their families are going to be a focus of mental health improvement interventions. Another respondent echoed this point, and pointed out that engaging with children and young people needs resources, and longer timescales than those generally used in Government consultation processes.
Other suggestions
6.15 In addition to the three methods described above, respondents suggested that a wide range of outcomes and indicators could be used to track progress and assess performance in this area. Some of these suggestions are listed below. In some cases, the relevant data is already being collected, but may not be used as a measure of mental wellbeing. In other cases, the development of new tools would be required.
- Measure levels of staff absence (in the NHS) due to stress, depression, bullying and harassment.
- Collect and publish information on waiting times for psychological therapies.
- Gather evidence of the impact of volunteering on mental health and wellbeing. [Suggested using the Volunteering Impact Assessment Toolkit for this purpose.]
- Set targets to address the needs of young carers.
- Set targets for a reduction in the prescribing of drugs / anti-depressants. (Note, however, that other respondents argued that targets to reduce anti-depressant prescribing were unhelpful.)
- Gather statistics on the number of referrals to Housing Investigation Teams due to chaotic lifestyle disturbance, identify how many of these referrals relate to people assessed as having a mental health problem, track what support was given, and the outcomes of those interventions.
- Monitor unemployment rates.
- Measure levels of hazardous drinking.
- Develop national indicators to measure the mental health and wellbeing of children and young people, and in particular looked-after and accommodated children.
- Develop national systems (via ISD) to record the use of counselling and other psychological therapies.
- Develop indicators of better multi-agency working (for example, better referral pathways for people who present to A&E or out-of-hours services due to self-harm or suicide attempt).
- Develop primary prevention measures, including indicators of whether people are being connected to preventative and early intervention services.
- Develop indicators of community participation and social activity ( e.g., levels of volunteering, youth club participation, library usage, use of community and leisure facilities).
Need to develop qualitative measures
6.16 Some respondents argued that it is not enough to simply collect statistics and report on numerical targets. There was a strong feeling that more qualitative data ('soft indicators') was also needed to assess the progress of the mental health improvement agenda. One local authority respondent said that:
"There needs to be a greater movement towards a social model of health, rather than solely a medical model…. The challenge for the social model is the qualitative aspect of change generated." (Local authority respondent)
6.17 Another individual - the same respondent who suggested measuring levels of staff absence due to stress, depression, bullying and harassment (see point 1 above), also suggested that additional qualitative information was needed to truly measure improvement:
"We note that NHS Boards currently only report a single, annual staff sickness absence figure to the Scottish Government. We would like to see Boards report on certain reasons for absence in the future so that the NHS can demonstrate the improvements it is making for its own staff." (Representative of NHS professional body)
6.18 Similarly, another respondent asked for national support to develop a framework that would capture information about the factors that contributed to suicide - in addition to simply measuring the number of suicides in an area. This would assist local planning committees to commission better prevention services. Another individual argued that it was necessary not only to measure whether people move on from services (throughput), but what they move on to.
« Previous | Contents | Next »