« Previous | Contents | Next »
Listen
2.0 Evaluation Design
The evaluation was conducted in three stages.
Stage 1: Overview A scoping of the literature on organisational change, models of service delivery for primary care mental health, and effective interventions to treat mild and moderate depression
Mapping of (a) appropriate elements of the whole service system in each geographical area and (b) locating where the Doing Well project would sit within the system
Stage 2: Detailed case studies of services from the perspective of service users and health care professionals (including front-line staff, middle and strategic managers as well as service planners) covering issues of pathways to care and sustainability of initiatives.
Stage 3: Interactive feedback workshops designed to feed learning back to local health systems, drawing on the literature review, local mapping, case studies and monitoring of change.
The final stage of the evaluation involved a reflective approach to learning for local NHS Board areas and their national partners to achieve a synthesis of the evaluation findings. This focused on what the local areas tried to achieve, how they went about this and what worked or did not work and why. Attention was paid to articulating changes to service systems that demonstrated good practice and to identifying how these changes could be rolled out and rendered sustainable.
The evaluation of Doing Well was conducted from the start of the implementation phase of the projects and sought to capture what sites set out to achieve, why the chosen approach was selected, how this worked in practice and what lessons were learned. The evaluation was undertaken within a rapidly changing environment. Consequently, evaluation outcomes reflect the situation at the time of completion of data collection. Therefore, this report does not reflect the full final outcomes for each site but presents the main stages of implementation, the barriers and facilitators and the key learning that Doing Well sites can share with others who wish to develop similar services.
2.1 METHODS
Data collection
Three data sets were collected during the course of the evaluation.
1) Mapping exercise
The mapping exercise involving each of the seven Doing Well sites consisted of baseline site visits, local workshops in each site (with a variety of stakeholders from within projects and across wider local health, social and voluntary care systems) followed by feedback in the form of mapping reports. These reports consisted of summary maps of each Doing Well project and a comparative analysis across all sites in terms of management arrangements, target groups and needs, system issues, aims and objectives, use of evidence, anticipated impacts and measures. The mapping produced an overview of the projects, and highlighted key issues arising out of the Doing Well programme across geographical sites.
A final survey and end-of-evaluation site visit captured changes that had occurred during the project and included key lessons learned.
2) In-depth case studies
In order to examine sustainability and other pertinent issues emerging from the implementation of the Doing Well initiatives, users, front-line staff, middle and strategic managers in five of the seven Doing Well sites were interviewed. The case studies were selected to ensure that the sample captured differing levels of complexity and system change anticipated by projects (see table 1). To keep the user experience in focus, service users, service providers and managers in each area were asked to describe how they perceived the patient pathway created by each project and to identify barriers and facilitating factors.
Case studies offered an in-depth view of the dynamics of change introduced by the Doing Well projects during the course of the Programme. Argyll and Clyde and Grampian were the two sites not included as case study sites as both had additional research and evaluation activities underpinning their developments.
3) Analysis of routine outcome and activity data
As part of their local evaluation, each project routinely collected outcome and activity data. The national evaluation team used the data to enrich and inform other sources. Routine data produced a picture of ongoing activity and outcomes from each project as it unfolded. Throughout the evaluation, the national evaluation team worked with each Doing Well project and the CCI team to examine and learn from the projects' developing profiles and from trends within the Doing Well programme as a whole. Activity data, waiting times and contact time have provided useful insights into the functioning of the different interventions and service models.
However, individual sites differed in both their interventions and their target populations. There was some variability in clinical outcome measures used across the participating sites. Clinical outcome measures used by different sites included the Patient Health Questionnaire ( PHQ-9), the Hospital Anxiety and Depression Scale ( HADS) or the Clinical Outcomes for Routine Evaluation ( CORE). As a result, the quantitative data collected by the individual sites were not sufficiently homogeneous to allow direct comparisons of outcomes between sites, and indeed this would not be wholly appropriate given the diversity of interventions and their stages of implementation. Nonetheless, the data provide indications of the impact on patient outcomes, and identify the general direction of this impact. More information and reflections on the outcomes and activity data is provided in Section 4 of this report and Appendix 3 contains a summary on interpretation of change associated with each of the main outcome measures reported.
Analysis
The qualitative interviews were conducted by three researchers, recorded on mini disc/digital recorders and transcribed verbatim. For the data analysis a systematic framework approach was adopted (Ritchie and Spencer, 1994). This allowed for identification of patterns in the data. Literature provided key themes that formed an initial framework for analysis. Coding categories were revisited, expanded, and refined independently by the research team throughout the period of analysis.
The analysis and reporting of all three data sets progressed in an iterative fashion:
- The case study material was indexed according to following codes derived both from the mapping report and initial reading of all interview transcripts:
- Access and referrals
- Interventions planned and delivered
- Outcomes for patients and system
- The following additional codes reflecting the study's objectives were added:
- Origins of project
- Advertising, networking and marketing of services
- Management arrangements
- Sustainability strategies
- The five case study sites were then described and compared in terms of:
- Origins of project
- Aims and core functions
- Interventions planned and delivered
- Patient pathways (access and referrals)
- Organisational structure including management arrangements and the projects' location within the wider mental health service system
- Sustainability
- The two sites that were not used as case studies were then described using the same framework, although in less depth
- Outcomes and activity data from each setting were included in the final description of each site.
Presentation of evaluation findings
From this analysis, the following key questions were identified to structure the presentation of results in the sections which follow:
- What progress was made and what has been achieved within the seven Doing Well projects?
- What are the key themes relating to the introduction of self help interventions for people with mild to moderate depression?
- What system changes are needed to sustain these interventions?
- What key pointers for development at national and local levels can be identified?
- What are the recommendations for future action to implement learning from the Doing Well programme?
Ethics and consent
Ethical approval for the interview study was obtained from the Multi-centre Research and Ethics Committee ( MREC) and management approval was obtained from the five Research and Development departments within the case study health board areas. Individual consent was obtained from staff and all service users who took part in this research.
« Previous | Contents | Next »