The Scottish Government

« Previous | Contents | Next »

Listen

CHAPTER THREE: RESULTS

Introduction

3.1 This chapter provides a synthesis of all of the data collected during the evaluation. The chapter is structured to largely reflect the approach to the analysis of the data which was to analyse each main stage of the SRI development process separately, including:

• Preparation
• Administration
• Collation and scoring
• Feedback
• Action planning
• Re-administration.

3.2 The chapter also incorporates an element of cross-over between the stages, allowing the reader to gain a sense of the use of the SRI as a development process within a range of 'live' settings. A specific section focuses on the content of the SRI.

3.3 Much of the findings are reported generally because they were relevant across most or all of the pilot sites and areas and respondents. Where a particular finding is specific to a particular pilot area, site, setting, participant group or individual, this will be referenced.

Response

3.4 Five of the SRI pilot areas were able to participate in the evaluation. Of the five participating pilot areas, a total of 16 completed SRIs were included in the evaluation and mapping information was provided for each of these. For the remainder of this report, these 16 SRIs will be referred to as individual pilot sites within pilot areas.

3.6 A total of 40 interviews were carried out across the five participating pilot areas.
This included 20 administrator interviews, 11 service user interviews and nine service provider interviews. One focus group was held in each pilot area and each focus group included key individuals from various pilot sites who were/would be involved in taking the SRI action planning processes forward. This included service-level staff, managers, service-user representatives and strategic managers.

Table 3.1 provides a detailed breakdown of the evaluation data collection activity.

Table 3.1: Response

3.7 Table 3.2 below provides further detail on the settings in which the SRI was piloted.

Table 3.2 Settings of SRI pilots by pilot area

Limitations

3.8 Service users who had been involved in the SRI pilot in Ayrshire and Arran, Forth Valley and Grampian did not consent to be interviewed as participants in the evaluation. Similarly in Ayrshire and Arran and Grampian, no service providers who had been part of service provider groups for the SRI pilot consented to interview. Reasons for non-consent for service users included that an evaluation interview as well as participation in the SRI groups would be too stressful. When service providers did not respond, the only reasons provided by pilot leads were that the service providers were too busy to take on the interview within the scheduled timescales. This does represent a gap in the completeness of the data for the evaluation, and therefore findings based on service user and service provider data should be interpreted with caution.

Findings: preparation

Preparation for administration of SRI

Preparation day

3.9 Administrators were asked to rate how well the SRI preparation day prepared them to use the tool on a scale of 1 to 5 (5 being that the preparation day prepared them completely and 1 being the preparation day did not prepare them at all). On average, interview participants scored the preparation at 2.5 in terms of preparing them to administer the SRI. Reasons for this varied; however, the dominant themes that emerged are outlined below.

3.10 The most common comment was that a formal training environment can only prepare you so much and the practical implementation of the tool provided people with the most learning. Of the 20 interviews conducted, 10 advised that nothing could fully prepare people to administer the SRI. Six participants advised that no amount of formal training could prepare or explain the practicalities of what is involved with the SRI ( e.g. ethical concerns, service user involvement, time commitment and timescales).

3.11 All of those who attended the preparation day advised that there was too much information included in one preparation day. There was widespread appreciation that training like this is extremely dense in information and that this may be inevitable.

3.12 Administrators gave a wide range of answers when asked what was particularly helpful and what could have been improved in relation to the SRI preparation and there was no distinction between areas. Table 3.3 below, summarises the positive and negative responses of administrators and the national support function.

Table 3.3 Participants' perspectives on SRI training

3.13 Some of the positives identified by participants were directly contradicted by negatives identified by others such as the extent to which recovery as a concept was included or opportunities to participate. This highlights the diversity of needs of individuals attending the training. Familiarity with the concept of recovery varies, and there were indications that those who were already familiar with recovery found the preparation day to be more helpful as they shared an understanding of recovery language and principles. On the point of involving service users more in the training, local pilot leads were responsible for deciding who should attend training.

3.14 A number of suggestions for improving the preparation day for SRI were made, including:

• Including case study scenario workshops to run through the practical implementation of the SRI, providing trainees with a more accurate indication of the time and resource commitment required, alongside other practicalities such as understanding questions and sampling assessments and care plans
• Interviewing and group facilitation skills (required for Parts 3, 4 and 5 of the SRI)
• More focus on recovery in its broadest sense to ensure universal understanding of the background and concepts of recovery, alongside the language that administrators will come across during the administration of the SRI
• More guidance on the scoring of the SRI (this was considered by many as the most challenging part of SRI administration)
• Inclusion of service users at preparation day (acknowledging local responsibility for nominating preparation day participants) and, in terms of content, more on the service user involvement aspect of the SRI (one participant explained that the ethos of the SRI is the centrality of the service user to the process and this should be reflected in the training)
• Reduce the intensity of the preparation by allocating over more than one day (although there was acknowledgement that attendance at training can be challenging in terms of being released for training, backfill, etc)
• Increase opportunities for participation with smaller groups
• Provide locally-based preparation days

Achieving local 'buy-in' to the SRI

3.15 Buy-in from local stakeholders is imperative to the effective implementation of the SRI. Most participants claimed that there was support for the SRI pilot at operational and strategic management levels in their local area. This support often stemmed from an existing strategic commitment to implementing a recovery agenda throughout services.

National support

3.16 The level of national support, demonstrated through the formal commitment in Delivering for Mental Health to the implementation of SRI and the support to the process provided by the Scottish Recovery Network, was perceived to enhance local buy-in. National support was perceived to increase the impetus to drive the SRI forward locally and enable better local understanding and appreciation of the use of SRI and the recovery agenda more generally.

Close project management

3.17 All areas have had a central driver for the SRI locally, however, where these individuals have been involved in the actual administration of the SRI this was perceived by evaluation participants to have been the key to achieving buy-in from others. These dedicated project managers were able to develop a structured approach to building local engagement and support for the SRI process and recovery, and co-ordinating the administration of the SRI pilot.

Formal accountability

3.18 Another key approach to achieving local buy-in was to build SRI into official strategies, operational processes and service requirements. For example, in Grampian there are lines of accountability for the services that participated in the SRI to the Integrated Mental Health Services Manager who has joint responsibility (in partnership with the local authority) for commissioning mental health services locally. The Grampian mental health strategy emphasises the use of SRI to promote recovery-based services which has translated down to formal service level agreements with the use of SRI as a requirement of the commissioning process.

Implementing SRI with a supportive ethos

3.19 One of the issues that came through strongly in the interviews with the consultant and those administering the SRI in all the areas in the evaluation was a concern of staff and management of being judged and of the tool being primarily a means of performance management. The consultant commissioned to assist local implementation of the SRI, the Scottish Government, the Scottish Recovery Network and the evaluation team, reiterated that the SRI pilot was a trial and that services would not be judged by their scores or the outcome of the SRI. This seemed to provide some ease and relieve some apprehension.

3.20 There has also been considerable work undertaken at ward/service level by each of the managers of the services participating in the SRI pilot at pilot sites. Local meetings to discuss the use of the SRI and allocation of tasks within the SRI were particularly helpful. Local activities included discussions with staff and patients to make them aware that the SRI was not being used as an audit or judgement of services but instead as a developmental tool to indicate areas of strength and areas for improvement. However where this has been the only approach to build engagement and support for SRI, it appeared to be less effective than where a range of approaches were used to achieve this.

Access to helpful publications

3.21 National publications were also mentioned by the administrators of the SRI as helpful in terms of reinforcing the aims of SRI and gaining local commitment, specifically the Scottish Government publications Delivering for Mental Health (2006) and Rights, Relationships and Recovery: The Report of the National Review of Mental Health Nursing in Scotland (2006).

Reservations amongst psychiatrists

3.22 One key finding was of a common perception amongst interview and focus group participants of a lack of active involvement in and support for the SRI from psychiatrists. These participants acknowledged that psychiatry has a lot to contribute potentially to the SRI development process and the need to commit further attention to involving this group in the future.

Preparation for action planning

3.23 Feedback from the focus groups suggested that the somewhat limited timescales for the SRI pilot meant that in the rush to train and prepare for the administration of the SRI, planning for the process of action was not always given the time it required. Following the administration of the SRI, some pilot areas felt that in retrospect, if they had planned for the whole SRI development process, they would have ensured more momentum in moving on the feedback and action planning activities. In one pilot area, participants claimed that they were told that they were to focus on the administration and content of the SRI for the pilot only, and they were not required to think forward to action planning.

Administration

Preparation

Confidentiality

3.24 Administrator and focus group participants in each pilot area raised concerns about issues associated with protecting confidentiality and anonymity whilst administering the SRI, however, this concern was not shared by the service users or service providers interviewed. Confidentiality issues centred on accessing patient-related documentation and conducting interviews and focus groups.

3.25 In three pilot areas there were concerns around SRI administrators accessing confidential assessment and care plan documentation, (some of which had been awarded patient/doctor confidentiality status) to complete Parts 1 and 2 of the SRI. In one of these pilot sites, each ward manager administered the SRI in a ward that was not their own to ensure an amount of objectivity in completing the SRI, causing some concern. Where the ward or service manager administered the SRI in their own service, issues around recording confidential information on the SRI were important but less controversial, as these individuals have access to the assessments and care plans as part of their professional role. The national support individual was consulted on this issue by each of the three pilot sites and each ward/service manager was supported to come to their own decisions on this. In all areas, the emphasis was placed on the data being recorded anonymously with no identifiable data recorded on the SRI.

3.26 In the inception discussions with pilot areas, some people raised concern around the involvement of service users in Parts 1 and 2 of the SRI. It was suggested that only health service employees who are bound by internal codes of confidentiality, or those with honorary contracts, should administer and complete Parts 1 and 2 of the SRI.

3.27 SRI administrators, in planning the administration of the SRI, and on reflection having completed the process, felt that there were a number of important issues around the interviewing of service users for the purposes of the SRI. In many cases key workers or ward staff who knew the individuals conducted their SRI interviews. A main concern was whether a service user being interviewed by an individual, who also provided mental heath care, was ethical in relation to the service user's ability to provide their genuine opinions of the service, which the individual interviewing them represents. Another concern was whether staff charged with providing an equal, accessible, approachable and supportive service could also be involved in contributing to an objective criticism of the same service.

Sampling

3.28 Some administrators expressed uncertainty around the selection of documentation samples for the SRI due to a lack of guidance on sampling techniques including the size of the sampling frame. The national support consultant was contacted on this issue by a number of individuals and supported local areas to come to their own conclusions in this matter after discussion.

3.29 For Parts 1 and 2 of the SRI, the selection of 10 assessments and 10 care plans seemed, for some administrators, to be a low number to achieve accurate representation across the service and also time constraints meant they chose to sample less than 10.

3.30 There was concern from some administrators that the service users who would agree to participate in the Part 5 group interviews would have some motivation for taking part ( e.g. an outstanding issue/complaint in regards to a service) and so bias the results of this part of the SRI. However, there was also acknowledgment that the key aim of the SRI is as a developmental tool to identify areas for improvement and administrators were aware that even if, by chance, data was predominantly negative than this would still provide them with an insight into areas for improvement in the service.

3.31 The following five sections focus on the administration of specific parts of the SRI and scoring.

Parts 1 and 2: Assessment and care plan data

3.32 The overall consensus amongst the SRI administrators spoken to in the evaluation was that the process was found to be extremely time-consuming.

Selecting cases

3.33 Whilst most areas sampled the recommended 10 assessments and care plans, a significant number of local pilots used smaller samples due to time constraints: Three areas sampled five and one area sampled three. Sampling frames selected by pilot sites varied including:

• Those who had been in the service for a specific amount of time (between seven and 10 days or more in an inpatient setting, one year or more in community-based services etc)
• Geographical location ( e.g. spread over a number of CMHTs)
• Those with a specific worker ( e.g. three assessments and care plans taken each from three workers) or from within a specific team ( e.g. three assessments and care plans taken each from three teams)
• Those who had not been in another service previously
• Available to participate in the SRI

3.34 The overall method of selecting the 10 case records for Parts 1 and 2 of the SRI was a random sample of current clients of each service. However, whilst this was the dominant sampling method, other administrators used stratified sampling ( e.g. an equal split of men and women, split between long and short-term clients in a ward setting, informal and detained clients etc).

Administration experience

3.35 The main issues administrators mentioned during their interviews in relation to completing Parts 1 and 2 of the SRI are outlined in Table 3.4 below.

Table 3.4 Dominant issues in completing Parts 1 and 2

3.36 Despite initial difficulties, there was overall agreement amongst those who administered Parts 1 and 2 of the SRI that the process did become easier as they used and became familiar with the tool. One community-based service in Glasgow and one inpatient ward in Tayside used two people to complete Parts 1 and 2, another local pilot used three administrators to complete Parts 1 and 2. This seemed to be an effective way of splitting the workload.

3.37 Some challenges arose during data collection. Two administrators advised that discrepancies in the way different staff members record information and their writing was challenging when trying to extract information. Another advised that some information was electronic whereas other parts were hard copy and referring between the TWO was time consuming, and another advised that some of the records were incomplete.

Language and terminology

3.38 Six administrators (in Ayrshire & Arran, Forth Valley and Grampian) suggested that the language of Parts 1 and 2 - 'assessment' and 'care plan' - could be unreliable as not all services use this language. For example, two administrators advised that there was confusion locally about whether this referred simply to the official care plan, or all the case notes in an individual's file because together, they all comprise the care plan. The health-based language in these parts of the SRI was perceived to be particularly problematic for voluntary and social care-based services. Voluntary sector organisations, for example, do not have care plans for individuals but may use other terminology, such as personal plans. Additionally, there was confusion locally around what was considered assessment ( i.e. was the SRI focussing only on initial assessment documentation or on ongoing admission and assessment). It was suggested by a number of interviewees that the title of these parts should be changed, for example, to 'patient file' to broaden out the applicability of these parts of the SRI to different service sectors.

Part 3: Service information, policies & procedures

Selecting relevant information

3.39 There was inconsistency throughout local pilots in perceptions of what constituted service information, policies and procedures, with many administrators taking a very broad approach and examining 'everything they had'. In this sense, administrators overwhelmingly advised that it would be useful to have more specific guidance on what type of service information, policies and procedures to include when completing Part 3.

3.40 All local pilots placed an emphasis on:

• Local information and local policies ( e.g. equalities policy, operational policy, missing persons policy, psychiatric emergency plans, vulnerable adults policy, etc)
• Information that was available and accessible in the pilot sites ( e.g. user choice, involvement and information leaflets)
• Broad NHS information and specific service information

3.41 All areas reported easy access to the information required for completing Part 3 as the information was broadly accessible to service users and service providers.

Experience of administration

3.42 Administrators found Part 3 to be the most straightforward. Despite reports that the process of collecting and collating the information required was very time consuming, administrators did acknowledge that they became more familiar with the process the longer they spent completing Part 3.

3.43 Approaches to data collection varied across pilot areas and sites:

• In Forth Valley, the local lead for the SRI pilot completed Part 3 on 'service information, policies & procedures' and extrapolated the information across the three local pilots, freeing up time and resources amongst the ward-based administrators to focus on other parts of the SRI and service delivery commitments
• In Ayrshire & Arran, Part 3 was completed by the local Leading Change team to ensure an unbiased snapshot of information provision (this proved specifically useful in accessing strategic policy and procedural documents that local pilot sites may not necessarily have ready access to)

3.44 The task of administering Part 3 of the SRI was considered by interviewees as an unusual and useful audit to help ensure that the information provided to patients and clients of services is focused on their needs and requirements. Administrators from all local areas advised that the work involved in administering Part 3 of the tool highlighted gaps in information provision locally. In the two pilot areas that used the SRI in voluntary sector organisations (Grampian and Augment in Tayside, which used it independently of the pilot and evaluation), voluntary organisations were found to perform better than statutory services in providing information for service users.

3.45 Administrators felt that much of the statutory service information, policies and procedures available locally were:

• Not concise enough to fulfil the requirements of the SRI
• Available but not recovery-orientated
• Characterised by jargon and medical terminology

3.46 The point was raised, particularly in Ayrshire & Arran that if the SRI finds information is provided locally, this does not mean that clients use it; should the SRI be about what information is available or evidence that the information is used by service users?

3.47 Each participant that was asked about Part 3 of the SRI felt that the language was unclear and open to interpretation leaving potential for inconsistencies in individuals' interpretations of questions. The SRI Content section of the results provides further detail on this.

3.48 A common finding for local pilots in relation to Part 3 of the SRI was that it was difficult to find any information or policies and procedures that related specifically to a ward or health centre. Documents that were used tended to be generic across all wards or settings. When service-specific information could not be found there was acknowledgement (specifically in Greater Glasgow & Clyde and Tayside) that this was a gap in service provision that the SRI could pick up on to help improve and develop local services.

Part 4: Service provider group interviews

Preparing participants

3.49 The level of information provided to potential participants prior to the group interview varied, from in-depth information in Forth Valley, where it was provided to all sites by the local lead, to limited information in other areas. The amount of information seemed to depend on the time available for the administrators to prepare.

3.50 Ayrshire & Arran administrators provided the tool prior to participation in the service provider group interview, so that the participants could familiarise and prepare themselves for the interview.

Selecting participants

3.51 The number of participants in service provider interviews ranged from one to nine between pilot sites (with no pattern across pilot areas). In all of the local pilot sites service providers were selected to participate in a group interview if they worked in or with the service piloting the SRI. The dominant approach to recruiting service providers for interview was verbally, either by the person administering the SRI or by the service manager. Most pilot sites adopted a multi-disciplinary approach to sampling for the service provider group interview. All sites invited service providers to attend a group interview on a pre-arranged date, time and venue due to time constraints, given there was a deadline for completing the SRIs in this pilot.

3.52 Whilst some pilot sites reported rather last-minute ad-hoc recruitment for Part 4 of their SRIs, other pilots were more structured in their approach to inviting service providers to participate.

3.53 The Forth Valley service provider group interviews were all conducted by the local lead and a ward manager. Information was prepared prior to the interviews taking place and distributed to all those eligible staff in the three wards taking place in the pilot. Individuals from a range of backgrounds were invited to interview ( e.g. administrative staff, qualified and unqualified nursing staff, occupational therapy, allied health staff, psychiatrists, etc) with information about the process and the date and time set aside for the interview. Using this method, no service providers refused to take part but administrators reported that attendance at the interviews was based on who was available to attend at the time.

3.54 In Tayside, the service provider group interview was added to the end of a regular allocation meeting, resulting in nine service providers in attendance.

3.55 In Tayside and Greater Glasgow & Clyde, administrators also interviewed certain strategic managers individually, in addition to conducting a group interview with other service providers. This was to ensure that managerial input was achieved when time commitments would not allow them to attend a group interview. Two administrators conducted all of their service provider interviews individually and reported that these were productive and informative meetings. However, another administrator who had only one person attending their 'group' interview found the interview intense and time-consuming and often the individual did not have all the information required to answer the questions.

3.56 The multi-disciplinary approach adopted by the majority of the pilot sites proved to be beneficial, with administrators in areas with this approach reporting that the discussion ensured that a range of different angles were covered and questions were answered by a range of different professions. They also reported that where questions were confusing (described in more detail in the section on content below), a multi-disciplinary group helped to ensure that a rounded and balanced understanding of the questions was employed.

3.57 Where low numbers were involved in the Part 4 group interviews, the administrators concerned felt that this was because of a low general 'buy-in' due to poor local planning, drive and organisation. In one site, the administrator had not made the numbers required to make the process meaningful or clear in their recruitment process. It was perceived that if local processes aimed at achieving 'buy-in' were improved, alongside an improvement in feedback and communication between those administering the SRI, management and strategic planning and service provider staff, then participation in the SRI may have been improved.

Administrators' experiences

3.58 Table 3.5 below outlines the dominant issues that arose when questioning administrators of their experience of Part 4 of the SRI.

Table 3.5 Dominant issues for administrators in completing Part 4

3.59 Three administrators (in Ayrshire & Arran and Grampian) reported participants in the service provider group interviews as initially being quite reserved and suspicious of the questioning for the SRI. They put this down to a natural concern around the service providers feeling that they were being judged or questioned over their practice and re-emphasised by them the importance of explaining that the SRI is a developmental tool and was not being used to audit.

3.60 Grampian was the only area where a service user was involved in the administration of Part 4. The service user felt that the service providers who took part in this interview seemed to appreciate service user involvement and following some initial resistance to answer questions, her involvement was beneficial as she was seen as unthreatening. This individual felt that involvement in the administration of the tool was beneficial to their own recovery.

3.61 Whilst there was some confusion around specific questions and the terminology and language used in Part 4 (discussed in more detail in the content section below), administrators reported that most service provider groups were able to answer the majority of questions in detail.

3.62 Where questions were not answered, the primary reason for this was running out of time, and this was relatively common. Most pilot sites set aside between one and two hours to complete Part 4 and one pilot site in Tayside set aside one morning. Those who reported providing one hour found that this was too short and questions tended to go uncompleted.

Service provider participants' experiences

3.63 Service providers interviewed for this evaluation predominantly advised that the recruitment process was handled well by the local pilots. Most service provider participants felt they could say 'no' if they did not want to participate, although one participant felt they could not say 'no' was because they felt 'duty-bound' to participate to set a good example. One criticism was the 'closed' nature of the recruitment process in most pilot sites where service providers were identified by the local pilot leads and then approached to participate at a set time rather than advertising and inviting any interested parties to join in on an agreed date. This was perceived by some as a lost opportunity to involve all those who were interested.

3.64 Six interviewees felt the support provided prior to participation in the SRI was adequate and the remaining three advised that it was not. When asked what was helpful about the support provided prior to participating, the service providers stated:

• Basic information on what questions/topics would be included in Part 4
• Explanation of what participating would entail ( e.g. time commitment and expectations)
• Pre-existing knowledge of recovery and the context of the SRI.

3.65 The service providers who advised that the support provided was not adequate gave the following reasons:

• Too little information to prepare thoroughly, seeing Part 4 questions prior to interview would have been more helpful
• The interview was a considerably more time consuming and complex than anticipated.

3.66 It was suggested that facilitators of Part 4 interviews should explain briefly at the start of the interview what it would entail. Two participants in the evaluation also advised that local recovery awareness discussions/workshops prior to the group interview would have been beneficial. Another felt that a group discussion prior to the interview to discuss answers and views would minimise the time commitment required for the SRI.

3.67 Four of the service providers interviewed for this evaluation advised their experience of the Part 4 interviews was 'very good' and five advised it was 'quite good'. All service providers participating in the evaluation reported the interviewer's facilitation as being helpful to the process.

3.68 When asked how they found being interviewed as a group, participants all agreed that this method of interviewing was beneficial to them and elicited the most relevant information, as they could 'bounce off' others. The group interview was helpful when it was multi-disciplinary as participants found that when they thought a question was not as relevant to their profession and/or it was more appropriate for someone senior on the group, this took some of the pressure off them as individuals. One new staff member found the group interview particularly helpful in learning more about the service they had joined.

3.69 In inpatient settings, service providers from three pilot areas reported little support to participate in the interview on the day, with many reporting the time commitment as a key barrier. Seven participants advised that some form of support to release them from their daily duties would be beneficial to allow them to participate fully in the SRI process.

3.70 All nine service providers spoken to advised that the use of a local venue in close proximity to their usual place of work was helpful in terms of making the SRI process accessible to them as they could still attend work emergencies if required and felt relaxed.

3.71 All service providers interviewed reported that the group interview was participatory with wide and varied discussion where everyone had a chance to put forward their own stance on questions. All service providers spoken to advised they had found the group interview for Part 4 of the SRI an interesting process and that the discussion generated was helpful for them professionally.

3.72 When asked to rate their opinion on how important their own contribution (from 1, not important, to 5, very important) was to the SRI process, service providers gave their contribution an average rating of 3. When asked to rate other people's opinions of their contribution to the SRI process (using the same scale), service providers gave an average rating of 3.22. There was some variation in scores between pilot areas, ranging from 2.5 to 4.

Part 5: Service user group interviews

Selecting participants

3.73 In all of the local pilot sites participants for Part 5 service user interviews were given information on the purpose and content of the SRI (to elicit their opinions on their experience of the service) and then asked to volunteer to take part in a group discussion. In all areas it was emphasised that participation was voluntary and that people could withdraw at any time if they wished. In some areas, individual services recruited participants; in other areas, more systematic approaches were adopted with information (leaflets, consents and copies of questions) for service users provided centrally from the local lead and distributed by each of the wards participating in the pilot.

3.74 Whilst some pilot sites recruited widely, making the process accessible for all service users in the service ( e.g. Grampian), others were more targeted in their recruitment ( e.g. Ayrshire & Arran, Forth Valley and Greater Glasgow & Clyde) where the administrators provided information on the SRI to those service users they felt were well enough and able to cope with the interview or to those who were able and willing to engage. One administrator advised that there was less enthusiasm for participating in the group interview in inpatient settings as many service users in these services are in crisis.

3.75 The majority of administrators interviewed felt that they achieved quite low numbers of service users participating in the SRI and that they would have liked more service users to have taken part to gain a more accurate and representative picture of the service.

Preparing participants

3.76 Most potential participants for Part 5 were provided with information on the SRI in written format, with five pilot sites providing the Part 5 questions to potential participants. However, six pilot sites provided information to clients verbally only (this was more common in inpatient settings).

3.77 One administrator reported that they were not entirely convinced that some of the service users who participated knew what they were consenting to. Other services gave more notice to service users, especially community services who tended to give an average of about 10 days notice to potential participants.

Approaches to administration

3.78 A range of approaches to the administration of Part 5 of the SRI were used during the pilot, including:

• Administration by a service user who attended a voluntary sector service that was undergoing the SRI, where it was felt that this encouraged participation due to familiarity and trust
• Administration by a ward manager and staff nurse with the assistance of advocacy services within the ward
• Administration by statutory sector employees from other services to ensure people could talk freely

3.79 Where those who were administering Part 5 of the SRI were directly involved in a service user's care, the question of being objective enough so that any comments made by service users during the SRI interview would not have repercussions on their care at a later date was raised. It was suggested by administrators that it would be more beneficial for someone from another ward to conduct the service user group interview in each ward to ensure that people could talk openly and honestly about the service.

3.80 Whilst the majority of the pilot sites conducted the service user interviews as a group interview, four pilot sites (three in Greater Glasgow & Clyde and one in Ayrshire & Arran) opted to conduct these as individual interviews because of the personal nature of the questions being asked in Part 5 (specifically the questions around trauma). Another pilot area's focus group participants felt that it in retrospect it was not appropriate to hold the service user interviews in focus groups as it involves talking about personal situations and some questions may not be appropriate to ask in groups.

Administrators' experiences

3.81 Table 3.6 below outlines the dominant issues for the administrators involved in conducting the service user interviews.

Table 3.6 Dominant issues for administrators in completing Part 5

3.82 Administrators reported the questions in the service user interviews as being not user-friendly. In Forth Valley, the questions were altered prior to the service user interviews in an attempt to make them more user-friendly, although the facilitators of the group discussion still found themselves having to paraphrase and explain questions further.

3.83 The time set aside for group interviews with service users was between 20 minutes (when conducted as individual interviews) and two and a half hours. All the administrators who were involved in this part of the SRI reported this as being a long time for service users to participate in such a discussion. This was certainly perceived to be the case within inpatient settings, where some of the service users were acutely unwell. However the majority of the service user evaluation participants did not feel that their interviews were too long.

3.84 In terms of eliciting all the information required for completing the SRI, administrators reported varying levels of success for a range of reasons, including administrators' views that some questions were not relevant to specific service users ( e.g. drug and alcohol services if the service user had never had an issue with this).

3.85 Comments from administrators in relation to the experience of conducting the interviews were all positive, with five reporting that they were amazed and surprised at how open and honest service users were during the discussion. This, they believe, provides added weight to Part 5 of the SRI. Administrators also reported the benefit of participating in the SRI for service users, with one advising that it opened participants' minds and highlighted the services to show service users what it actually is that they do.

Service user participants' experiences

3.86 Some service users reported being uncertain about how they had been selected to participate in the SRI and that they would have liked to have known this. All service users said that it was explained that participation was entirely voluntary and would be kept confidential. Ten of the service users spoken to advised that they felt they could say no if they did not want to participate in the SRI interview. One service user in Greater Glasgow & Clyde advised that they felt as if they had to take part and could not say no, whilst another in Tayside advised that they felt they should take part to 'give something back' and to help others who might be at a different stage in their recovery.

3.87 Most service users felt that they were provided with enough information prior to their interview but a few service users advised they would have liked more information on what the SRI is and recovery more generally. One service user felt that they did not have adequate support to participate and would have liked more information on the expectations and the background of the SRI to help prepare for the interview.

3.88 All of the service users reported being given enough time to consider participation and none felt that they needed longer to decide.

3.89 When asked of their overall experience of the interview, two service users advised their experience was 'excellent', seven advised it was 'very good' and two advised the experience was 'quite good' (one from Greater Glasgow & Clyde, one from Tayside).

3.90 When asked to rate how important their contribution was in their own opinion (from 1, not important, to 5, very important) to the SRI process, service users reported an average rating of 4. When asked to rate (using the same scale) how important other people felt their contribution was to the SRI process, service users reported an average rating of 4.

3.91 Table 3.7 below outlines the dominant issues for service users who participated in the service user interviews for Part 5 of the SRI.

Table 3.7 Dominant issues for participants in completing Part 5

3.92 Service users who were interviewed individually had mixed views on the value of this approach. Half felt comfortable talking one-to-one due to the personal nature of issues being discussed, confidentiality and the similarity to clinical discussions which occur on a one-to-one basis. The remaining three service users advised that they would have preferred a group discussion as this would allow them to share experiences and listen to how other people recovered. One service user who did not speak much English advised that a group discussion would have been beneficial as it would have promoted her understanding of the questions.

3.93 Service users advised that the interview method of gathering information from them for the SRI was the most appropriate method for two reasons:

• It did not assume a certain level of literacy
• It enabled service users to give their true opinions without placing any restrictions on them ( e.g. in terms of length as would be the case in a written questionnaire)

3.94 Of the five participants who took part in a group interview (all from Tayside), they all advised that this was helpful to them because:

• They received input from others and could provide more detailed in-depth answers as a result
• They gained knowledge of how other people recovered

3.95 The most helpful aspect of group discussion was considered by all participants to be peer support. One service user from Tayside was interviewed both in a group and also individually, in view of the personal nature of some of the questions. This person felt that some questions were more suited to group discussions and that individually it felt quite daunting. Another service user felt that it can be easier to discuss mental health generally in a group because in an individual interview it can feel as if you are 'being put on the spot'. The group discussion was felt to be more of an overview of experiences giving more reality. It was felt that group discussions encouraged a more informal and relaxed atmosphere, which service users really appreciated.

3.96 Despite the time commitment required, raising concerns about those with impaired concentration or memory, the majority of service users appreciated that to gain such in-depth information takes time and for this they were willing to give up their time to give their views and experiences. The average time commitment required was an hour, with service users advising the interview took between half an hour for an individual interview and two hours for a group interview.

3.97 Service user interviewees explained that their decision to take part was greatly influenced by the interview environment. Flexibility of interview venue was considered helpful, especially by those who were home-bound or who preferred to be in familiar or their own surroundings (their 'own territory') to ensuring they were comfortable throughout the interview process.

3.98 In terms of the interviewer, four service users advised that they were more comfortable with being interviewed by someone they had a pre-existing relationship with ( e.g.CPN, Group Leader). In Tayside, the use of Augment seemed to be beneficial in terms of promoting a relaxed and informal atmosphere in the service user group interview. In this respect, it made the interviews more accessible for service users.

3.99 Of the questions asked, the majority (n=9) of service users advised they were able to answer all or most of the questions asked of them (all of the service users from Tayside and the remaining four from Greater Glasgow & Clyde). The wording of the questions and terminology used proved difficult for eight service users. Whilst this did pose a problem, paraphrasing, further explanation and examples from those who were administering the SRI seemed to assist. The questions where people felt they could not answer were predominantly questions which they thought did not apply to them ( e.g. housing issues when they had never had an issue with housing and so could not provide an answer). However, one service user did not speak much English and there were real difficulties in completing the SRI because of language barriers and understanding of complex terminology and jargon. They were not offered any additional interpreters to assist in the interview, which she advised would have been helpful for her to understand the questions and provide a truthful answer.

3.100 Service users described the support they received to take part in the SRI, as centred on the explanation of questions and emotional support. Two service users felt that they did not receive emotional support, which they believe would have been useful. Additionally, a service user claimed that there was no option given to bring a carer along to the interview, neither were any of the more practical issues considered ( e.g. interpreter, childcare, transport). In Tayside, free transportation was provided to ensure that service users could participate.

3.101 One service user reported feeling valued as a result of being asked to participate. Others felt that the SRI process to them was beneficial in terms of letting them get their views across and their recovery.

Collation & scoring

3.102 Approximately two-thirds of the queries and questions from local areas to the national support consultant were in relation to the scoring element of the SRI. This, coupled with the completed SRIs received from local areas, suggests that further guidelines and training are required on scoring. Some of the completed SRIs received from services across all the local areas had not been scored at all, and those that had been scored were inconsistent in completeness and therefore incomparable. The overall method of scoring individual items and providing a total score was not consistent across local pilot sites (some areas have provided an average score of between 1 and 5, others have provided an aggregate score of all questions) making any further analysis of the scores for the purposes of this evaluation redundant.

3.103 In some service elements, pilot sites scored highly based on documentation evidence and service provider interviews but scored low in service user interviews. The average score then was quite high, but the important information for the pilot site was that service users did not know about the service elements and therefore they were not performing as well as they could. This left the administrators doubtful about the utility of using average scores to identify areas for development and for benchmarking.

3.104 Not all pilot sites completed the scoring exercise, with the administrators advising that the focus in the preparation day was not on the scoring but on the process of administering the SRI. In this sense, they felt that the scoring was not important and was secondary to the pilot. Conversely it is the detail in the tool that helps to pinpoint where things are wrong, the score can indicate that something's wrong, but it does not tell you what.

Helpful approaches to collation and scoring

3.105 Administrators found the scoring very time-consuming but appreciated that this could not necessarily be avoided due to the nature of the SRI. Some administrators felt that administrative support for scoring would have been helpful in speeding up the process. Three administrators found the scoring experience very interesting.

3.106 Whilst individual pilot sites tended to score individually, in two pilot areas a group completed the scoring, with each administrator being able to comment and discuss their views and opinions on how this should be done and what the overall scores should be, although one administrator also suggested that the person who collected the data for the SRI does not, and perhaps should not, be present for the scoring.

3.107 Of those administrators who were involved in group scoring, all felt this was an efficient, helpful and beneficial process; more so than attempting to score alone. The national support consultant was involved in the scoring group in one pilot area which they felt was beneficial as it gave the process a degree of objectivity and in another area a service user was involved.

3.108 The scoring process became quicker and more straightforward as a result of practice. However some mixed views on group scoring were expressed, ranging from 'tedious' and 'a laborious process requiring too many people', to 'very participatory and productive' and 'ensuring everyone got their say'.

3.109 Administrators found that the more notes they took during the administration process, the easier it was to score the SRI, especially where more than one person was involved in administering the tool. In one pilot area, notes were considered central to scoring as they allowed scorers to cross-reference to get more accurate scoring, and also improve the opportunity to follow-up on issues identified.

3.110 Of those spoken to, those who scored the SRIs soon after they had administered the tool found it the easiest, compared to those who waited some time after administration to score.

Scoring challenges

3.111 A number of challenges were faced by those attempting to score the SRI including:

• It is difficult to give scores for areas that were not relevant to a service ( e.g. housing support in an inpatient unit)
• Question 7c on the scoring sheet is not represented in the tool itself
• Confusion when the administrator thought that some of the scores for certain questions fell between two scores ( e.g. 3.5) and there was uncertainty as to how these should be scored (in the event, administrators tended to score down)
• Lack of clarity on how to produce an overall score for questions included elements of different parts of the SRI. For example, the score for question 5a involved data gathered in Parts 3 and 4 of the SRI, resulting in confusion around how you would provide an overall score for question 5a given the scores in Part 3 question 5a and Part 4 question 5a might be different. Some areas provided an average score of both Part 3 and 4, others provided a total score and others listed two scores for the question.
• The scoring and administration guidance in the SRI contained some errors ( e.g. parts of the guidance stated there were five basic needs when there were six)
• Perception that some questions were subjective, making scoring difficult, if not inappropriate
• Too much cross-referencing was required to make any sense of the data.

Benefits of scoring

3.112 All of the administrators who did the scoring exercise advised that the scoring process benefited the service as it pointed out areas where they were good at practice and those where improvements were required ( e.g. ensuring paperwork is recovery-orientated and strengths-based) and put the recovery agenda at the forefront of service provision.

3.113 However, those administrators who did not score their SRI did not feel that they had missed out on any opportunities for service development. They felt that the utility of the SRI as a development tool was to be found in participating in the SRI process and the detailed data, not an overall aggregate score. In this sense, these administrators identified areas for improvement just by sifting through the data gathered for the SRI. Interviewees and focus group participants stated that it is the detail in the tool that helps to pinpoint where things are wrong; the score can indicate that something's wrong, but it does not tell you what, so it only has limited utility for action planning and benchmarking.

Reflections on administration of the SRI

Resource required

Manpower and time required

3.114 Due to the variation in approaches adopted in local pilots to administer the SRI ( e.g. some administrators did not have to complete Part 3 as this was done centrally and some did not complete Part 5 as this was done by service user representatives), useful comparisons cannot be drawn between individual administrators in terms of the time spent on administering the SRI. All of the administrators reported two or more people being involved in the administration of the SRI in their pilot site. However, when combining the time spent by all administrators involved in the SRIs in any given site, an accurate picture of time spent in each pilot site can be obtained. The timescales to administer one SRI were varied, ranging from seven hours to 42 hours, and the majority of pilot sites (n=9) reported that the process took between 15 and 20 hours. The average amount of time spent on administering each SRI was 21 working hours.

3.115 Administrators were asked how they found the task of administering the tool, as the guidance states that it is 'onerous'. When asked if they thought this was the case, 14 administrators advised that they did not agree with this assertion and all of the administrators advised that the process is 'worth it', as long as there is a continuing drive for service change and development.

Support to administer the tool

3.116 Support provided to administrators to use the SRI, centred predominantly on the provision of time to complete the tool and support from management and/or peers during administration. The support provided in regions where an area-wide, strategic approach to administration ( e.g. Grampian and Forth Valley) was adopted was reported as being more productive for administrators than the areas where pilot sites were largely left to complete the SRI independently.

Additional resources used

3.117 The majority of administrators (n=15) reported using no additional resources to complete the SRI, apart from the service information, policies and procedures used to complete Part 3. This documentation was located either in hard copy format by visiting services, or by searching the internet and local intranets.

3.118 Five administrators reported using administrative assistance (either from administrative staff or nursing staff, e.g. staff nurse) to prepare information for the SRI, set and arrange appointments and meetings and collate the information ( i.e. typing it up). The rest of the administrators undertook the above administration tasks within the team who were involved in administering the SRI.

3.119 Some local areas involved other groups and/or organisations in certain parts of the SRI ( e.g. in Ayrshire & Arran the Leading Change team completed Part 3 and in Tayside, Augment conducted all the service user interviews). In-kind support was provided in two of the local areas (Grampian and Tayside) by service users who administered Part 5 of the SRI. In-kind volunteering was an additional and important resource in these two areas although paying service users for their involvement was raised in Grampian as an important development issue.

3.120 At the outset of the evaluation, not all of the pilot leads or administrators knew that they had access to a national support consultant during the pilot, although each pilot site had been informed about this by the Scottish Government. Local pilot participants tended to use peer support to resolve problems, but when they did consult the national support consultant they were appreciative of the resource.

Service user involvement

3.121 In pilot sites where service-user groups were involved in the local SRI pilots (Grampian, Lanarkshire and Tayside), a number of issues pertaining to service-user involvement were raised throughout the evaluation, in the inception interviews, the interviews with administrators and the final focus groups. Each of these areas utilised service-user involvement in differing ways, with varying levels of success.

3.122 In Grampian, a service user interviewed other service users for Part 5 of the tool and the service provider groups for Part 4 of the tool. They were also involved in the group scoring process of the SRIs for each pilot site in their area and action planning discussions. In Grampian, the SRI was uniquely piloted in a voluntary sector organisation which places strong emphasis on service-user involvement.

3.123 The Tayside pilot sites worked in partnership with Augment (Scotland) Ltd, a voluntary sector user-led organisation. Augment (Scotland) Ltd was involved in local discussions from the outset and representatives from Augment who would be involved in the administration of the SRIs attended the preparation day. However, unlike Grampian, service user involvement in Tayside comprised staff and service users from Augment administering Part 5 of the tool (the service user interviews) and attending the initial inception interview. Whilst the manager of Augment was involved in action planning, it seems at this point that service user involvement in the formal pilot ceased in Tayside.

3.124 In Lanarkshire, the local SRI team was developed in partnership with the local voluntary service user organisation 'Lanarkshire Links'. The idea behind this was similar to that of Grampian and Tayside, in that a service user representative from Lanarkshire Links would be trained in, and involved with, administering the tool for Part 5. However, the pilot had not been implemented at the time of the evaluation.

3.125 The following key learning points regarding service-user involvement in the SRI were raised in the evaluation:

• There was concern amongst the service-user groups that service-user involvement in the SRI should be meaningful and equal to that of NHS or local authority staff taking part in the pilot
• In Tayside there was some concern amongst staff around the use of service users to administer the tool when it was considered an 'internal' developmental tool
• Where service users were involved in the administration of the SRI ( i.e. not just as participants in Part 5 group interviews) this method of administration worked well
• Service users involved in the administration process reported feeling equal to other professionals involved and important to the local pilots

SRI Content

3.126 This section provides a detailed account of the evaluative comments of pilot participants (mainly administrators and those involved in focus groups) and their suggestions for improving the content of the SRI.

Parts 1 and 2

3.127 Table 3.8 below outlines the comments on specific questions that were made by administrators.

Table 3.8 Evaluation participant comments' issues with Parts 1 & 2 questions

3.128 The content of Parts 1 and 2 was generally considered to be appropriate and administrators were usually able to source the relevant information in the sampled assessment documents and care plans.

Part 3

3.129 Administrators were at times confused over what was to be included in Part 3, in relation to what service information, policies and procedures should be considered for inclusion. It was suggested by a minority of administrators that some of the questions in Part 3 (5a and 5b) are currently geared more towards management level and may be difficult for non-management staff to answer. There was a suggestion from one administrator that Part 3 of the SRI should be completed by senior strategic management, similar to the approach adopted by Forth Valley.

Part 4

3.130 The feedback from the administrators, focus groups and service-provider interviewees in relation to the questions asked in Parts 4 and 5 of the SRI, was that the language used was very medical, described by some as 'health speak'. This was by far the most common comment from the administrators and some focus group participants in relation to administering Part 4 of the tool. This was specifically the case in relation to administering the SRI in voluntary sector settings, where medical terminology and 'health speak' are not used. As a result of this language, some administrators and service providers reported finding it difficult to understand some of the questions. Administrators reported having to interpret the questions and paraphrase them to make them understandable to the participants.

3.131 There was a sense that some wording in the Part 4 questions needs to be more clearly defined to ensure collective understanding of the questions. It was felt that by leaving some words undefined ( e.g. 'contact' in Question 6a, 'principles' in Question 8b.ii) was leaving the question open to interpretation, thus allowing the potential for inconsistencies between areas.

3.132 Table 3.9 below outlines the questions which administrators had issues or concerns over, alongside the concern they expressed.

Table 3.9 Evaluation participants' issues with Part 4 questions

3.133 Overall, the feeling amongst evaluation participants was that the questions in Part 4 of the SRI were too long, convoluted and complex; six administrators advised there were too many questions in Part 4 of the SRI (across all local areas). The administrators and service providers reported that some of the questions were repetitive and they felt that questions appeared to be duplicated, thus making the process longer than it should have been. This was considered a key issue given the time commitment required in amongst service providers' duty to provide efficient services. It was suggested that some questions could be broken down to make them more manageable and that they need reviewed to ensure that repetition is avoided.

3.134 The nine service providers interviewed for this evaluation provided an average score of 3.6 when asked to rate the relevance of the questions asked in the service provider group interview from 5 (completely relevant) to 1 (not relevant at all). The dominant answer provided when asked was 4, but a low score of 2 from one participant from an administrative professional background brought down the average score provided.

3.135 Some participants felt that the questions asked in Part 4 were more relevant and pertinent to nursing and allied healthcare staff than to administrative professions. There was an overall feeling amongst service providers that the questions in Part 4 of the SRI were meaningful and all service providers interviewed for this evaluation advised that, as a group, they were able to answer all of the questions asked (unless they ran out of time).

3.136 Service providers also advised that recovery is a relatively new concept and there could have been confusion around the relevance of some of the questions in Part 4 as a result of this.

Part 5

3.137 Table 3.10 below summarises the issues that administrators, focus group participants and service-user interviewees raised concerning the questions in Part 5 of the SRI. As a result of many of the questions being similar to those of Part 4, some issues are similar to those listed in Table 3.8.

Table 3.10 Evaluation participants' issues with Part 5 questions

3.138 Overall, evaluation participants felt that the questions in Part 5 were too long-winded and not user-friendly. They felt that they were full of jargon and health service language, which is difficult for service users to understand. Similar to the comments on Part 4 questions, it was generally felt that questions in Part 5 were too vague and that they needed more description and guidance to define the specific information required. As was the case with the comments for Part 4 questions, participants advised that some of the questions were too long and needed to be broken down to make them more manageable ( e.g. 8b), especially for service users who may have difficulty with concentration. The questions in Part 5 were considered to be repetitive and participants advised that cutting down the number of questions would be more practical and effective.

3.139 In their evaluation interviews, when service users were asked to rate the relevance of the content of the questions asked from 5 (completely relevant) to 1 (not relevant at all), they gave an average score of 4.3. Three service users interviewed during the evaluation advised that the reason that the content was so relevant to them was that it was asking about their experiences and opinions. Similar to service providers, they also advised that whilst some questions may not have been relevant to their experience, they may be relevant to others and whilst they themselves may not have been able to answer, these issues were all still relevant to recovery more generally and so should still be asked.

"There were some [questions] where I could give an opinion, but it didn't relate to my situation." (service user, Tayside)

3.140 The overall feeling of service users was that the questions in Part 5 were very broad and all-encompassing and in this sense, they viewed the SRI as an inclusive process. Therefore, service users advised that they valued the opportunity to provide their experiences and opinions.

Comments on overall content

3.141 Evaluation participants also reported some additional concerns about the SRI content in general specifically:

• That Parts 4 and 5 are not identical (they believe that to get an accurate picture of services the questions asked of service providers and service users should be identical)
• Guidance does not assist in advising of the approach to adopt in Parts 4 and 5 ( e.g. should interviews be conducted individually or as a group)
• Questions are too 'politically correct' and are not meaningful
• There should be scope to include service-user quotes in the SRI which could be used to provide more directive and powerful feedback.

Gaps in the SRI

3.142 Evaluation participants provided some suggestions on additional items that should be considered for inclusion in the SRI including:

• For the older people inpatient context, one administrator advised that for this group of service users, questions on vocation and employment might not apply and they advised that there should be questions on dementia support and other aspects of socialisation and support that are more geared towards older people, as opposed to service users of working age
• Communication needs ( e.g. sign language, requirement for interpreter) should be included in the assessment data required
• Service users' goals should be included in Part 1 of the SRI as well as in Part 2 as goals and aspirations are often covered during assessment and this is not picked up in the SRI as it currently stands
• Carers' and relatives' perspectives are currently omitted from the SRI
• The use of peer support in service delivery
• Education and awareness around illness and medication
• Formal assessment of recovery-orientated training within services
• How services deal with patients who harass or who are abusive to other patients (considered key to how recovery-focused the service is and how much it promotes recovery)
• Lack of availability in other languages excludes some communities
• Just because you can tick all the boxes in the SRI doesn't mean you can work in a recovery-focused way. There's nothing in the tool about individual interactions and practices of individuals. The SRI doesn't reflect the skills in recovery coaching.

3.143 It was suggested that the SRI should be tailored towards different service settings, with one SRI available for inpatient settings, one for community-based services and one for voluntary and non-statutory sector services. This is contradictory to other evaluation feedback suggesting that the tool should be used across inpatient and community services to reflect mental healthcare delivery as one seamless service.

Extent to which the SRI is sensitive and recovery-focused

3.144 All SRI administrators were asked to rate how sensitive they thought the SRI was to the strengths and weaknesses of their service, from 1 (not sensitive at all) to 5 (very sensitive). The average rating provided by administrators was 3.

3.145 When asked to rate (from 1, not at all, to 5, very useful) the extent to which the SRI provides useful information on service sensitivity to different user groups and needs, administrators provided an average rating of 3. Differences did appear depending on which local area the administrators were from. The highest average score was given by administrators in Tayside (4.3) followed by Grampian (4.0), Ayrshire & Arran (3.8), Greater Glasgow & Clyde (2.7) and then Forth Valley (2.6).

3.146 In terms of how recovery-focused the SRI is, the administrators, service users and service providers were all asked to rate from 1 (not at all) to 5 (very) how well the SRI captures their own opinion of how recovery-focused the service is. The average score provided across all local areas were relatively consistent, with the average score from each local area falling between 3 and 4. The overall average score (4), suggests that participants in the SRI pilot process do believe that the SRI captures the majority of the information on whether services and organisations are recovery-focused.

3.147 Service-user interviewees commented that they were able to answer all the questions in the SRI which covered issues which assisted with their own recovery. For them, this was an indication that the questions asked were recovery-orientated and could measure how recovery-focused a service is. It was emphasised by one service user that people participate in their own recovery and just because certain things ( e.g. housing and financial support and advice) exist, this does not mean to say that that will promote someone's recovery as they have to work at it themselves.

3.148 Service provider interviewees felt that the SRI puts a focus on the issues that are important in people's recovery and brings these to the forefront of practice. However, some ward staff felt that it can be difficult for a service to be recovery-focused if it is an acute admission ward where people are detained under law.

Feedback

Process

3.149 As indicated at the beginning of this Chapter, most pilot areas were at the stage of providing feedback on their SRI results and beginning the action-planning when data collection for the evaluation came to a close. Therefore there is little detail available on the different strategies that were used to feed back SRI data to those involved in SRI administration, working in the services that were pilot sites or strategic-level managers and groups. However, in most areas the administrators, pilot leads and those managing the pilot sites had informal access to the SRI scores and were aware of the key findings to the extent that they could feed back this information to the focus groups (see below).

3.150 Seven administrators reported feeding information from the SRI back to the staff within the service, however, no service users and only three service providers (all from Forth Valley) advised that they had received any form of feedback. Service users were more likely than service providers to express a desire for feedback to be provided.

3.151 Administrators and pilot leads who had fed information back had done so through various channels and, particularly at local level, through service managers ( e.g. Charge Nurses, Ward Managers, etc) to cascade to staff and service users. The most common way for administrators to report feeding back information was verbally and relatively informally, through discussions with staff and management. It was less common for administrators to produce formal reports, with only one administrator advising that this would be the format in which the information from the SRI would be fed back to management. Some pilot sites claimed that this type of feedback to small groups takes time and resources, but it is necessary to discuss the findings and implications at this level before developing feedback for senior managers and strategic planning groups.

3.152 All pilot sites had also fed back to, or were planning to feed back to, strategic planning groups such as the Delivery Plan Partnership groups responsible for the development of mental health services via the pilot lead. Information was not yet available regarding the extent to which feedback 'upwards' had impacted, due to the time constraints of the evaluation.

Feedback as a learning opportunity

3.153 Of those who had fed back the SRI findings, there was a general feeling that the SRI provided a learning opportunity for services and service providers. In this sense, the SRI was seen as a positive process whereby it provided opportunities to identify gaps in services and skills. As a result, administrators advised that the people they had fed information back to had been interested and keen to improve on the areas where they had scored low.

3.154 The overall consensus from those who had fed back the information to services and service providers, was that services had scored a lot better than they thought they would and that the SRI had not highlighted issues that they were not already aware of. A number of learning opportunities were identified, including:

• The outcome of the SRI has been to open up the minds of patients and staff in terms of how the service performs in relation to recovery and what the concept of recovery more generally is about
• Staff working in statutory service pilot sites have learned from the voluntary sector pilot site about how they provide a recovery-focused service (in one area the voluntary sector service performed better than the statutory service in the SRI)
• SRI has provided managers with good insight into equality, social inclusiveness and recovery, as they can be far removed from service delivery, helping them to realise how things are doing at ground level
• In one area, service managers were "shocked" that all services do not routinely gather information on strengths in assessments and voiced concerns about the change in mindset required to address this
• Some pilot site staff were surprised to find that service users were not able to describe why they attended different parts of the mental health service ( e.g. inpatient, community mental health team, day centres) at the same or different times, and what they should expect to gain from each of these.

"The SRI lets us look at where we are now; we have come away with new information on what is good and bad in our services." (Focus group participant)

"The SRI picked up that the newer service was more inclusive and recovery orientated which is good as this is the way it should be." (Focus group participant)

Impact of the SRI and Action planning

Approaches to the process of action planning

3.155 At the end of the data collection stage of the evaluation, some pilot sites had already started on a journey of change, usually at an individual pilot site level. Most were grappling with what they felt was the difficult part, which was planning change at a strategic level. However, pilot areas had some idea of the proposed formal approach they would take to action-planning.

3.156 Although all participants valued the necessity of a top-down approach to achieving buy-in to the SRI, the evaluation captured the way in which the SRI gave staff and management at service level, the confidence and ability to identify and make changes from the bottom upwards. One focus group felt that this was consistent with the ethos of Leading Change.

SRIs fit with other current development processes and tools locally and nationally

3.157 In all pilot areas it was considered essential to join up the use of the SRI with other recovery initiatives currently underway in local services, such as recovery-awareness training. In the pilot sites, SRI was considered to fit well with a number of initiatives, development tools and patient information systems already available nationally and being implemented locally such as:

• Realising Recovery (it was reported that the SRI and recovery awareness training worked in conjunction to help service providers to better understand recovery)
• Mental Health Integrated Care Pathways ( ICPs)
• Recovery Oriented Systems Indicators ( ROSI)
• Tidal Model
• Specialist Single Shared Assessment
• Clinical Outcomes in Routine Evaluation ( CORE)
• Wellness Recovery Action Planning ( WRAP)
• Reclaiming Lives

3.158 Further information on the above can be found in Appendix 3.

3.159 Focus group participants felt that it was important to demonstrate how SRI connects with and complements these tools so that SRI is not seen by staff to stand in isolation.

3.160 In one pilot site, there are inequality champions and it was suggested by focus group participants that these people could be recovery champions too. In another pilot area, those working in wards pioneering the SRI and other recovery-related initiatives are already delivering talks to other staff groups in their area. In each pilot area, the pilot leads were making efforts to raise the profile of the SRI across their local services.

3.161 It was suggested that those in the Scottish Government who are responsible for rolling out the SRI nationally, should link with other national organisations that mental health services have to report to, such as Quality Improvement Scotland ( QIS), the Mental Welfare Commission ( MWC) and the Scottish Commission for the Regulation of Care ( SCRC) to look at gaining their buy-in to SRI data. It is important to note here that QIS and the MWC are represented on the Scottish Government's SRI Steering group and have expressed their commitment to the pilot.

Perceptions of the SRI's ability to influence change

3.162 Administrators, service users and service providers were asked how much they thought the SRI had the potential to influence change, from 1 (no potential at all) to 5 (high potential). Administrators and service users were more likely to think that the SRI has a higher potential to influence change in services than service providers. The average local area scores are all relatively high, with Forth Valley providing the lowest average score (4) and Grampian providing the highest average score (5). The overall average of rating (4) highlights that there is an overall general belief that the SRI has the potential to influence change within organisations and services.

3.163 When asked to explain why they had given the rating they had, service users advised that if the findings of the SRI process highlighted deficiencies in services then staff and management would have to change service-practice accordingly. Service users felt that services would listen more to the experience and opinions of people who use them, and so become more patient-focused:

"It [the SRI] will identify what people really think - the strengths and weaknesses according to the people who use them [services]." (service user, Greater Glasgow & Clyde)

3.164 No barriers were identified within the focus groups, in terms of implementing changes to services and practices as a result of SRI findings. There was a sense of acceptance amongst most staff and management that moving towards recovery-focused services was a laudable aim; however, a need to better involve psychiatry was identified. Service providers felt that the SRI's potential to influence change depended on effective feedback to management, commitment at all levels and the SRI (and recovery) being high on the national and local development agenda, all of which were present in this pilot.

3.165 A key aspect of the SRI that encouraged evaluation participants to believe that it had potential to influence change was that the SRI, unlike many other more problem-focused tools, provides as much, if not more, positive feedback than negative. In most pilot sites, participants were encouraged to find that they were doing well in many aspects of delivering a recovery-focused service. A further positive way in which the SRI promotes change is by, unlike other tools, providing a useful guide to exactly what changes need to be made, and providing a structure for those changes due to the amount of detail provided on the various elements of recovery.

3.166 There was a sense amongst the service users that the SRI places those who use services at the centre of any change processes, and 'looks at things from our perspective'. The SRI was perceived by focus group participants as a good approach to change because it involves a strong element of self examination by those involved in delivering the service under scrutiny.

3.167 In one pilot area, the focus group participants described the SRI as an intervention in itself, which resulted in an immediate impact on individuals and enabled quick and simple changes to practice to come about. For many, it was not the scores in the SRI that facilitated change, but the experience of participating in the SRI process itself that seemed to have a strong and positive impact on an individual's ability to identify areas for change, and their motivation to take action to make the changes. In Tayside, two service users reported that during the group interview for Part 4, issues around social anxiety arose. As a result of this discussion, a local workshop for service users on social anxiety has been arranged and issues associated with social anxiety are now taken into account when planning social activities.

Impact of SRI process on knowledge, attitudes and behaviour of evaluation participants

3.168 All service users and service providers were asked to rate from 1 to 5 (with 1 being no impact and 5 being complete impact), how much participating in the SRI had had on their knowledge and attitudes of the factors that assist with recovery. Service users provided an average rating of 3. When service providers were asked the same question, they also provided an average rating of 3. The total overall average rating for all evaluation participants was 3.

3.169 Whilst these scores may seem low, the majority of participants felt that the SRI did not teach people anything about recovery that they did not already know. However, they felt that it enhanced their insight into recovery by helping them to understand what a recovery-focused service looks like by identifying the service components necessary for such a service.

3.170 Some service users reported a change in their behaviour following their participation in the SRI process:

• One service user advised they were now doing voluntary work as a result of participating in the SRI (this was because the discussion in the SRI highlighted the opportunities that were available)
• Another service user advised that they were now more aware of what the service they attend can offer them, so their participation in the SRI process had widened the support available to them
• Another service user advised that the discussion for the SRI that she participated in, had made her want to participate in activities again, such as gaining vocational qualifications and taking part in courses offered ( e.g. health & safety)
• Two service users advised that the discussion for Part 5 of the SRI had made them want to become involved in things again, outside of the service they attend
• Seven service users advised that the process of being listened to, and their views being taken into account, really helps their recovery, with one describing it as a 'great compliment' even to be asked his views and opinions on the service he attended (this was something that service users believed should be measured in the SRI in some way)
• Service users reported that the very fact they had been asked to participate in the SRI had changed their own attitudes towards recovery. For example, one service user advised that they were now more aware of the things in life which they struggle with and their own strengths and weaknesses, helping them to focus
• Importantly, one service user advised that the process of participating in the SRI has made them realise that service users are able to question and participate in their own treatment and care.

3.171 The SRI was also perceived to have had a positive impact on the attitudes and behaviour of staff working in pilot-site services in the following ways:

• Following involvement in the SRI, teams were more enthusiastic and passionate about recovery and recording information in a recovery-orientated way
• Recovery has moved form being a buzz-word into everyday language
• Staff notice that service users now talk about 'recovery' and are adopting the same language as service providers
• Communication between staff and service users has improved, the SRI was considered as a mechanism to help improve communication if nothing else
• The SRI was perceived to have broken down barriers and increased trust amongst staff and different professionals and partnership working had increased as a result of the SRI.

Service development challenges raised by the SRI

3.172 The evaluation focus group discussions focused very much on the areas the SRI has highlighted for service development and the associated challenges this would bring, although, at the time of collecting this data, no pilot areas had produced formal action plans. This section provides a summary of the key areas where action for change is anticipated in pilot areas as a result of the SRI pilot.

Investing in cultural change

3.173 In the 1990's, mental health nurses were trained and taught to focus on problems and strengths-based, solution-focused therapies were not really supported. However, the Rights, Relationships and Recovery (2006) report signalled a change of direction and emphasised the need for culture change. All pilot areas were aware of the development challenges ahead in achieving real culture change to alter this approach to practice.

Assessing and meeting basic needs

3.174 The SRI was perceived by focus group participants to have directed them to reflect on the depth to which they address basic needs assessments. Participants in three pilot areas questioned the evidence-base behind the six basic needs identified in the SRI, and suggested that some reference to the rationale for selecting these basic needs be included in the SRI guidelines and training.

3.175 Pilot sites seemed to do less well in addressing the religious and belief needs of their service users, with many finding that they don't really explore this issue in any depth with service users to identify ways in which they can support them to meet their needs (often people are just asked what religions if any they ascribe to). One focus group discussed the potential to broaden out this need to include spiritual needs which are not always about organised religion.

3.176 Another basic need that was not so well addressed was that of shelter, with some sites finding that they could explore whether service users' accommodation was suitable for their needs and what actions they could take to support them in this (some inpatient ward staff felt this was not within their remit, but this was not the consensus across pilot sites).

Employing people with experience of mental health problems

3.177 One pilot area focus group explained how the SRI had challenged people's beliefs about the way in which they work in a recovery-focused way. For them, the SRI exposed the difference between what people in their statutory services say and do in relation to their not being exemplars in employing people with mental health problems, and not routinely exploring opportunities to maximise service users' existing capabilities to increase their social inclusion.

3.178 Some pilot areas found questions about employing staff with mental health problems problematic in view of the fact people may not disclose their experience of mental health problems in the recruitment process. As a result of the SRI, one area has put 'positive discrimination within recruitment' on the human resources agenda.

Increasing the status of service-user involvement in organisational development

3.179 In one pilot area, it was anticipated that the SRI would raise the status of service users' views and increase their ability to influence recovery-focused service development. It was argued that the SRI is about the centrality of the service user and this has highlighted the need for service users to be central in terms of driving forward the recovery agenda in their local area and at a national level.

Improving assessment, care planning documentation and policies

3.180 In many areas, care-documenting systems and policies were found to be out of date and not enabling the recovery agenda. In one pilot area, focus group participants felt that staff use these policies as a guide to, and defence of, their practice. Evaluation participants felt that changing people's awareness of and attitudes to recovery, was only part of changing their practice and behaviour. To enable staff to move away from old procedures the appropriate changes to policies and documentation need to be made to set the new practice parameters.

3.181 Administrators reported finding that the assessments and care plans in their services were problem-focused and not recovery-orientated. There was a perception that this documentation tended to reflect what they described as a 'medical model' or treatment-focused model of mental healthcare. The SRI was deemed very useful in terms of measuring how recovery-focused the documentation was. Many pilot sites were planning to change their problem-based assessment and care planning documentation, to a strengths-based one, incorporating hope and aspirations.

3.182 One pilot area found that the SRI uncovered the duplication of information recorded in different patients' case notes (there was little to differentiate between individuals in the sample of notes selected), this has led to action plans to adapt the assessment and care plan documentation to ensure that person-centred care is provided and evidenced. This is an important point about the individualisation of assessment and care planning, which is key to one of the central tenets of recovery i.e. that everyone's recovery journey is unique.

3.183 The SRI also highlighted that although much good practice is undertaken, a great deal is not recorded. Many participants stated that they knew they were carrying out certain aspects of care planning, or asking questions in assessments that were in keeping with recovery principles, but were not evidencing this adequately in notes and records.

Educating patients and carers about recovery-focused services

3.184 Two pilot areas identified the need to not only educate staff, but service users and carers too, about what a recovery-focused service means.

Improving patient and carer information

3.185 In three pilot areas, focus groups reported that they would be re-writing new patient and carer leaflets to ensure they are recovery-focused and provide the amount of detail required by the SRI.

Awareness of equality

3.186 The process of administering the SRI itself was perceived to have helped to make staff more aware of equality and diversity issues relating to their practice. In one pilot area, the SRI demonstrated a limited awareness or uncertainty amongst staff of diversity issues, highlighting a need for basic awareness training on equality and diversity.

Re-administration

3.187 Focus group participants debated the potential uses and practicalities of the
re-administration of SRI and a number of potential approaches were identified, these are summarised below.

Using SRI scores as benchmarks

3.188 In most pilot areas, participants envisaged some value in re-administering the SRI to measure whether the impact of the organisational changes they have implemented as a result of the previous SRI administration can be demonstrated through improved scores. However, participants were not sure whether a change in scores is the best way to demonstrate improvement through the SRI results.

3.189 All focus group participants thought that SRI scores should not be used to compare services locally or nationally for two main reasons. Firstly, the SRI was considered by many as a guidance tool rather than an outcome or audit tool. Using the SRI to create recovery league tables would go against the positive, participative and developmental ethos of the SRI pilot which has had a positive impact in terms of the awareness, attitudes and behaviour of those involved, and their motivation to improve. The second reason is that participants are not confident enough in the validity of scores as an ultimate measure of how recovery-focused services are because the real information is in the detail of the tool, and the ambiguity in some questions and guidelines provide potential for inconsistency in approaches to administration.

Frequency of SRI re-administration

3.190 A number of factors were considered by focus group participants when considering how often to re-administer the SRI including:

• The natural turnover of staff would mean a decreasing pool of those trained in SRI, and trainers, and create a need to reinvest in training for re-administration
• One team of staff trained to use SRI (a 'recovery team') could administer the tool across a range of services
• The costs to administer SRI across all services might be too high, criteria for selecting a sample of services would have to be introduced
• Certain elements of the tool, for example those where a service has performed poorly, and focused improvements could be re-administered alone
• Too much re-administration could prove too intense for the service users involved.

3.191 Some pilot areas planned to use the SRI as a yearly audit tool in the pilot sites to measure progress. It was anticipated that the need to re-administer would decrease in frequency as recovery-focused service delivery became more naturalised. Another pilot area planned to use the SRI six to 12 months after a change, then again after a year to see if they have achieved change, then bi-annually to see if change maintained. They considered that this would involve planning so it would be ready to administer when the time came.

SRI as a tool to assist contract monitoring

3.192 In one pilot site, the SRI will be written into an annual contract monitoring for mental health statutory and voluntary services, as part of their performance-management framework. It was anticipated that the SRI would be complemented by outcome data.

SRI as an impact assessment for new services

3.193 It was suggested by focus group participants that the SRI could be used to assist planned changes to services, such as to examine new operational procedures, new hospitals and policies in the way that other health impact tools are used. One focus group thought that the SRI would provide an excellent impact assessment tool for equality.

SRI as a reference for everyday practice

3.194 Another pilot area planned to use the SRI as a reference tool to assist staff and multi-disciplinary groups in structuring care planning and supervision sessions.

« Previous | Contents | Next »