The Scottish Government

« Previous | Contents | Next »

Listen

CHAPTER TWO: METHODOLOGY

Introduction

2.1 The evaluation explored and reviewed the developmental process of the SRI from the perspectives of all the groups of individuals involved. Information was collected at each key stage including:

• Preparation
• Administration
• Collation and scoring
• Feedback
• Action planning
• Implementing change
• Re-administering.

2.2 The evaluation included 3 linked stages, each of which are detailed below.

Stage 1: Preparation

Local pilot site group discussions

2.3 Each local pilot site lead and representatives of the local DfMH project team/those involved with the project, were invited to a group discussion with the SDC evaluation team at a local venue with the following purposes:

• To introduce the evaluation, answer any questions and identify a main local contact
• To identify how and why they chose the services they did to pilot SRI
• To gather details of the proposed use of SRI in each contact and any other materials that will inform the mapping exercise
• To jointly plan and agree local case study data collection with each pilot team

Mapping SRI use

2.4 During the local group discussions, throughout the pilot evaluation, the research team developed a detailed map of the planned and actual use of the SRI during the pilot in each pilot site. The map was based on the key stages in the SRI development process (see Figure 1.1) with the following information collated onto a database:

• Setting(s), service type, target group(s), delivery model, size
• Participants involved, backgrounds, where and when involved (administrators, practitioners, service users, service providers and commissioners)
• Structural links i.e. where the SRI pilot fits within local service planning and development
• Data sources used ( e.g. case notes, service users input) including approaches to sampling
• SRI scores
• Method of feedback of SRI and to whom
• Action-planning processes and action plans
• Progress of implementation of action plan
• Details of re-administration of SRI
• Timescales and resources ( e.g. manpower) for all of above

2.5 The research team sourced the mapping information mainly from the local main contacts and from completed SRIs.

Stage 2: Local case studies

2.6 A case-study approach was taken to the bulk of data collection, with the following approach applied within in each pilot area.

Review of SRIs

2.7 The review of SRIs involved a documentary analysis of the completed SRIs and interviews with SRI administrators, service providers and service users. If pilot sites were planning to administer multiple SRIs, SDC worked with the local teams to identify which SRIs should be included in the evaluation to provide a manageable sample of a maximum of 3 SRIs per area. Criteria used to assist selection of SRIs included those that:

• Were administered early on in the pilot (to increase opportunity to follow through to action planning/implementing change/re-administration)
• Focused on areas where it anticipated that there would be most learning

Documentary analysis

2.8 SDC invited each SRI administrator to provide the evaluation team with a copy of their SRIs as and when they were complete. Scores of each element of the SRI and any comments were entered into the mapping database.

Interviews with SRI administrators

2.9 Each SRI administrator was invited in writing to participate in a face-to-face interview with the SDC researcher following the administration of the SRIs they were assigned to complete (a maximum of 5 SRI administrators per site were interviewed). The interviews were held locally at a location convenient to the interviewees and lasted for an hour. The interviews covered the following areas:

Preparation and Set-up

• Quality of training and extent to which it prepared them to use the tool
• 'Buy-in' and ownership of the SRI process and how this was approached.

Administration

• Experience of the process of administration including each section of the SRI and each data collection method section:
• Assessments and care plans - access, content, extracting data, sampling
• Service information, policies and procedures
• Service user interviews - recruitment and involvement
• Manager interviews - recruitment
• Support from managers/practitioners and service users to carry out data collection
• Process of collation and scoring
• Timescales - how long to complete and with what resource?

Content

• Is the SRI sensitive to strengths and weaknesses of different services?
• Does the SRI provide useful information on service sensitivity to different user-groups' needs?
• Does the SRI capture the administrator's perception of how recovery-focused the service is? If not, why not?
• Are there any gaps?
• Should additional items be included in the Indicator? Should anything be removed?

Developmental

• How were SRI scores and other details fed back, by whom, to whom and in what format?
• What level of interest was there in the SRI results? What key issues were raised by whom?
• Extent to which SRI has the potential to influence change
• The SRI administration task is described in the guidance as onerous- is it? If so in what ways? Is it worth it?
• Impact on their own attitudes/knowledge/confidence regarding the factors that influence recovery - has/will involvement with the SRI project lead them to do anything differently?
• How could the administration of SRI be improved?

Interviews with SRI service user and service provider groups

2.10 At each pilot site all those who participated in service provider and service user groups for the SRI were invited to participate in a face-to-face interview. Offering an initial invitation to service users and providers to participate in the evaluation and obtaining consent to be contacted by SDC was undertaken by the SRI administrator, during or following the SRI interview. SDC provided information sheets and consent forms. SDC then contacted those who had consented to participate to arrange an interview.

2.11 A maximum of six service providers and six service users (and carers where appropriate) were interviewed per pilot site.

2.12 The service user and service provider interviews covered the following areas:

Preparation

• In what way were they supported to prepare for the interview? Was this support adequate? What other supports would be helpful?
• Opinions on the recruitment process for SRI interviews

Administration

• Experience of the interview
• Support offered to participate in the interview e.g. interpreter, hearing loops, advocacy, emotional support
• Appropriateness of venue
• Interaction/dynamic with other group members
• Timescales - how long to complete and opinion on this

Content

• To what extent were the SRI questions relevant to the individual service user/ service provider? Were they able to answer all questions?
• Does the SRI include questions that are relevant to gathering information on service sensitivity to different user groups/needs and person centred care?
• How well does the SRI capture the service user's/service provider's perception of how recovery-focused the service is?
• Are there any gaps?
• Should additional items be included in the Indicator? Should anything be removed?

Developmental

• In what ways, if any, did they feel they benefited from being involved in piloting the SRI?
• How important do they feel their contribution to the SRI is?
• How were SRI scores and other details fed back, by whom, to whom and in what format? How useful was this?
• Extent to which SRI has the potential to influence change
• Did being involved with the SRI impact on their own attitudes/knowledge/confidence regarding the factors that influence recovery - has/will involvement with the SRI project lead them to do anything differently?
• How could the administration of SRI be improved?

Review of action planning

2.13 The review of SRI action plans included documentary analysis of any action plans and focus groups with those involved with action-planning and implementing change.

Documentary analysis

2.14 SDC was supplied with SRI action plans as and when they are completed by the main local contact in each site. Due to time constraints, only one action plan was submitted and this was after the data collection cut-off date. Therefore, any thorough analysis of the action plan itself was not possible. Instead, the action plan has been included as an appendix to the report (see Appendix 2).

Focus groups with those involved in action planning/implementing change

2.15 One focus group with those involved in action-planning was conducted in each pilot area. It was initially planned that the focus groups would take place following the production of SRI action plans where possible, allowing time for implementation of changes and possibly re-administration. In reality, no pilot sites had reached this stage and so those who were involved in beginning the process of action-planning were invited to attend. The focus groups were facilitated by the research lead, lasted for approximately 2 hours and were held at a location convenient to the participants.

2.16 The main contact for the evaluation in each pilot site provided the SDC research team with contact details of those involved in the action-planning process within their locality. In some cases this proved to be the same people who were involved in the administration of the SRIs or in the service user or providers interviews.

2.17 The focus groups were concerned with exploring the action-planning, implementing change and re-administration aspects of the SRI development process (see Fig 1.1), and the following areas were covered:

Preparation

• Extent of preparation to use the tool to plan system level change
• Approaches to achieving 'buy-in', extent of buy-in and influence on the SRI process as a whole

Relevance and utility for action planning

• How useful was SRI in identifying areas for system level change, were some items in the tool more helpful than others, specific to the particular setting of the project?
• How well did SRI facilitate insight into and reflection on service-delivery values and practice in terms of equality, social inclusion, recovery and person-centred care?
• What were the key learning points in terms of the extent to which services are sensitive to recovery-related values? What did they learn that they did not know already from other approaches to service development?

Content

• Are there any gaps? Should additional items be included in the indicator and what should come out?

Utility in terms of promoting change

• What progress has been made in terms of implementing change?
• To what extent does the tool influence/facilitate/promote change in terms of practice and wider service development?
• Facilitators and blocks to change
• Impact on their own attitudes/knowledge/confidence regarding factors that influence recovery

Developmental

• What might improve the process of action-planning in order to ensure SRI can be useful as a means of promoting change in service culture
• How often should SRIs be used and within what timescales - what is the required investment of resource to complete the SRI process and is this realistic?

Consent and confidentiality

2.18 Informed consent was sought from all of those invited to participate in evaluation interviews and/or focus groups. Evaluation information sheets were provided to all those invited to participate with contact details for the research team. In all cases, interviewees and those participating in focus groups were assured confidentiality in that no personally identifiable information would be included in the draft and final reports unless the interviewee concerned had agreed to the content.

Stage 3: Analysis

2.19 The analysis brought together the mapping and interview data to allow a full analysis of the use of the SRI within the range of settings provided by the pilot sites in order to provide evidence-based, realistic and thorough guidance for the effective use of SRI within mental health services.

2.20 Qualitative data were analysed using a staged-content process of identifying themes. Quantitative data from SRIs were cross-tabulated to compare scoring between the elements of the SRI and across pilot sites and settings. Each component of the SRI process was analysed separately as well as combining the components to gain a picture of the SRI as a development process in a range of 'live' settings. In particular the analysis included:

Assessing the completeness and relevance of the SRI in terms of:

• Sensitivity of the tool to the recovery, equality and social inclusion needs of the individuals and groups who use the range of services to which the tool was applied
• The extent to which the tool identifies how well the services are recognising and responding to inequalities, e.g. poverty, marginalized groups
• Views of participants on the relevance of the SRI to them and the service in question and any content gaps or surpluses identified.

Exploring the potential of the tool as facilitator for system level change by assessing:

• The range of action points raised by SRI and their potential to influence real change in terms of promoting social inclusion, equality and recovery
• The relationship between SRI scores and action plans and how strongly they are linked and what the SRI adds to current service development plans
• Influence of participation in the SRI process on the attitudes, knowledge and confidence of individuals, service providers, planners and users
• Progress made by projects towards implementing changes identified in action plans.

Identifying how best the SRI should be used by:

• Categorising the different approaches in which SRI has been used (including setting, target group, process, who is involved, timescales) and why
• Comparing the relevance and utility of SRI as a facilitator of change between the identified variations in approach
• Collating views on the thoroughness of the preparation training and preparation for action-planning.

Assessing the SRI as a measure of change by:

• Comparing differences in scores between first and second administration of the SRI where this has taken place
• Comparing differences in SRI score between different settings/wards and the perceived relevance of the tool to that setting to identify where the tool is sensitive to difference in levels of recovery promotion.

2.21 From this, conclusions were drawn regarding:

• The preparation required for the use of SRI
• The content of SRI
• Selection of appropriate service settings for the use of SRI
• Factors necessary for the successful use of SRI as a development tool that can effect relevant and desired change (including administration, action-planning and change implementation processes, readiness factors to do with the service in question, local context, organisational structures, buy-in , and leadership)
• The potential the SRI has to promote and facilitate service change and improvement
• The added benefit gained from being involved in the SRI process, from all perspectives but especially for service users.

« Previous | Contents | Next »