The Scottish Government

« Previous | Contents | Next »

Listen

CHAPTER THREE METHODOLOGY

3.1 Introduction

The purpose of this chapter is to provide a summary of the overall approach and methodology used for the evaluation. It also highlights any changes made to the proposed original methodology and discusses strengths and limitations of the data collected.

3.2 Overview

The research process encompassed three key stages: scoping and mapping; conduct of case studies; and, development of recommendations/production of outputs. A longitudinal component was built into the evaluation study design; this provided some opportunity to determine whether the local (and national) priority given to suicide prevention had changed over time and to capture and reflect upon such changes.

3.3 Theories of change

The overall approach taken to the evaluation was theory driven. The increased attention to theory driven approaches has arisen as a response to the challenges of evaluating complex interventions for health improvement that target significant social problems. A key strength is that this approach does not seek to judge a programme in its entirety, but attempts to determine whether a programme has been delivered as intended, and what aspects of programmes work, for whom and in what circumstances (Weiss, 2004).

Within the context of Choose Life the specific theory-based approach that was utilised was Theories of Change (ToC) (Connell et al 1995; Fulbright-Anderson et al 1998), in which the evaluator, in conjunction with key stakeholders, seeks to identify prospectively the underlying rationale or 'theory' of the planned programme. Further strengths of the ToC approach are its value as a tool to improve programme planning and in addressing some of the problems associated with efforts to establish causal attribution.

In the Choose Life evaluation different models of how best to implement a suicide prevention programme were explored at both national and local levels, with a particular focus on why particular actions and activities were anticipated to lead to which kinds of goals.

A theory-driven approach does not exclude or encourage the use of any particular research method. Research methods used (and outlined below) included qualitative interviews, analysis of documentation and action plans (including NIST templates), workshop discussion and survey questionnaires.

There were two main ways in which our use of the ToC approach differed from that recommended by its proponents. The first was that data were not gathered until after the overall programme was commissioned and the local partnerships established (rather than in the early stages of programme planning). This limits the potential of the approach to influence the planning process and is a common feature of its use within the UK due to the way in which evaluations are funded (Sullivan et al 2002; Mackenzie and Blamey 2005). On the other hand, there is growing evidence that the process of planning extends well beyond the initial set up phase of policy programmes; indeed planning was an integral part of the first phase of Choose Life.

Second, the majority of theories articulated within the project were not probed in sufficient detail to allow the emergence of robust and testable pathways linking resources, activities, processes and outcomes. As has been demonstrated across a range of programme evaluations (Blamey et al 2005; Mackenzie et al 2005) such an approach is extremely resource-intensive for both evaluators and key stakeholders within the programme.

In addition to the above limitations, there is a growing body of evidence that the Theories of Change approach itself has serious limitations in practice. Two key challenges are: the extent to which stakeholders are prepared or able to specify precisely the level of change anticipated within a complex policy setting to allow for their theories to be adequately tested (Judge and Bauld 2001; Mackenzie and Blamey 2005); and the degree to which the relatively linear approach adequately captures complexity (Barnes et al 2003).

There is currently no consensus about which evaluation approach is most suitable for which purpose (King's Fund 2004) and there are limitations with any approach favoured. A multi-method approach such as ToC is of clear benefit in the evaluation of a complex intervention such as Choose Life in its early phase , particularly in determining whether the programme has been delivered as intended and in understanding rationales underlying the programme's activities.

3.4 Review of national strategies

An early component of the evaluation was the conduct of a review of national strategies for suicide prevention ( annex 1). The purpose of this review was to consider suicide prevention strategies across the world and to gather evidence about effective policy and practice at both national and local levels. (The review criteria restricted selection of countries to those that had a national strategy.)

There are some limitations in an exercise focused upon what is contained in published strategy documents. It was not possible to explore how policy statements were implemented in practice. Additionally, there was little information available on the process and pace of implementation or on the relative influence of non-government organisations or community groups.

However, key strengths of this exercise were that the review enabled Scotland's suicide prevention strategy and action plan to be compared to suicide prevention activity underway internationally.

3.5 Coordinator surveys

Invitations were issued to all Choose Life coordinators in the 32 Local Authority areas to complete a questionnaire in stage one of the evaluation (November 2004) and at follow up in stage two (November 2005).

The questionnaires covered the following areas:

• Local vision for change
• Progress towards vision for change
• Progress in the development of the local infrastructure
• Resource allocation and generation
• Examples of innovative and effective practice
• Monitoring & Evaluation
• Sustainability and Mainstreaming
• Collection of data on suicide and deliberate self-harm
• Reflections on national support and on local progress.

Both open and closed questions were used in the questionnaires, although open questions predominated. Rating scales were also developed to measure satisfaction with national action towards the achievement of the national milestones set out in Choose Life, and to review local implementation progress towards the achievement of the local Choose Life milestones.

For the first survey, responses were received from 27 local authority areas (26 fully completed questionnaires and one partially completed questionnaire). Five local areas did not respond, of which three gave an explanation for inability to participate in the survey and two gave no reason. A complete response was also received from the SPS. For the second survey, 28 responses were received from the 32 local areas. Only two local areas failed to respond to either survey, although very sparse data were received from one area for both surveys.

A limitation of the survey resulted from changes in the coordinator post. In six areas the questionnaire was completed by a new coordinator at follow up. The comparison of scores on rating scales between surveys should therefore be treated with caution.

3.6 Interviews with national informants (including National Implementation Support Team)

Interviews were undertaken with members of NIST and a sample of key national informants, including the National Programme for Improving Mental Health and Well-being, ChildLine, Penumbra, the Samaritans, the Scottish Association for Mental Health ( SAMH), SPS, the Royal College of Physicians ( RCP), the Royal Air Force ( RAF) and the National Union of Journalists ( NUJ). The first round of interviews was held in November 2004 and repeated 12 months later. The purpose of the interview was to develop a national perspective on the implementation of Choose Life. Key elements of the interviews involved:

• Considering progress towards the achievement of the national milestones set out in Choose Life
• Understanding the criteria applied at the national level to assess local implementation
• Considering the quality of collaboration between the NIST and major national agencies for the achievement of Choose Life objectives.

3.6.1 Changes to the original methodology

A component of data collection developed in addition to the original plan was exploration of the NIST 'story'. It was agreed to undertake data collection with NIST to gather information pertaining to the NIST perspective on key events, decisions and outputs. This involved both individual interviews with members of the team and a joint 'workshop' session.

3.7 Case studies with local areas

The case studies were the main vehicle for exploring the processes of implementation and to identify the overall theoretical framework within which local suicide prevention teams were working. It was decided that eight was the optimal number of case studies that could be undertaken, providing a representative sample of local approaches to suicide prevention within prevailing time constraints.

Geographical type (rural/remote, urban and mixed rural-urban) was the primary criterion used to categorise local authority areas. Additional criteria, which were considered to be important for understanding the diverse and innovative approaches to the implementation of Choose Life at local level, were applied, including: the local suicide rate, focus on priority groups, interaction between national and local level organisations/services and approaches to coordination. The selection of sites was intended to ensure adequate variation in these primary and secondary characteristics.

The sample of case study sites comprised East Ayrshire, Fife, Glasgow, Highlands, Inverclyde, North Lanarkshire, Shetland Isles and West Lothian. Consistent with our longitudinal approach, there were two site visits, one in the spring/summer of 2005 and a follow up visit in autumn/winter 2005.

Following endorsement by NIST of the sites recommended for selection as case studies, the relevant coordinators were approached. Access to each site was negotiated and preliminary in-depth meetings with the coordinator and other relevant personnel were arranged.

Participants for the first round of interviews within each case study site were purposively sampled. Interviews were undertaken with four key informants involved in the decision making process (e.g. members of the Choose Life partnership responsible for setting priorities) and two representatives involved in Choose Life funded activities. Six was the optimal number of interviews that could be conducted in the time available.

In identifying the four key informants the evaluation team made every effort to ensure a mix of professionals from a range of statutory and community and voluntary organisations. As part of this process, local coordinators provided guidance about the key decision makers in partnerships.

One researcher conducted all the interviews and detailed notes were taken throughout. Interviews were recorded and, when necessary, the researcher cross-referenced notes with the tapes to ensure accuracy of information.

3.7.3 Selection of project examples

Sixteen project activities were selected for more detailed examination (two per case study site). Two projects per site was the optimal number that could be explored in the time available. The evaluation team made provisional recommendations for the selection of activities in each local area, with a view to ensuring that the full set would provide a representative selection across Choose Life priority groups and objectives. These recommendations were negotiated with coordinators at the first site visits. Interviews were conducted with a representative of each project, generally the project lead or manager.

3.7.4 Observational activities and collection of documentation

Choose Life events, most commonly Choose Life partnership meetings but also training events, evaluation days and practitioner fora, were observed by the research team. Meetings were arranged with national organisation representatives involved in local implementation where this existed (for example with SPS and Penumbra).

Key documents were collected, including minutes of Choose Life partnership meetings/subgroups, information pertaining to the project examples in each area, locality reports on progress and reports of previous needs assessments or research used to inform the local action planning process.

A key challenge faced by the research team was to decide the level at which theory should be (most usefully) articulated in the case study area and the degree to which detailed information about project activity was required. As highlighted by other evaluations of complex community initiatives, it was often more feasible to explore the theory and rationale for an individual project activity than across a local area undertaking an array of diverse approaches. It was important, however, to bear in mind that Choose Life did not solely fund individual interventions but was also acting as a mechanism to stimulate other non- funded activity and collaboration and to create synergy in tackling suicide prevention. For these reasons the evaluation team aimed to understand theories of change at an overall programme level as well as undertake more detailed explorations of selected projects within the case studies.

3.7.5 Changes to approach

The evaluation team wished to ensure throughout that the collection of data was guided by the need to answer the main aims and specific objectives of the evaluation. There was some concern that repeat individual interviews would not significantly contribute new learning in respect of case studies at the second fieldwork visit and that the short time lapse between site visits would not provide a realistic opportunity to reflect upon progress. To this end workshops were held in each case study area and facilitated by the evaluation team. These replaced repeat individual interviews with stakeholders, as originally proposed.

The use of group discussion rather than individual interview provided an opportunity for joint local testing of, and refection on, the local area's theories. The workshops also involved local participants in assessing their progress towards Choose Life objectives and milestones and discussing how best this could be demonstrated. Attention was paid to considering the implications arising from phase one for future planning of Choose Life implementation, e.g. local area partnerships' ability and capacity to sustain change and to change direction, where necessary.

The wide range of stakeholders attracted to the case study workshops did create some limitation in terms of data collection. Although this interest positively reflected local awareness and participation in the strategy's implementation, workshop discussions were not limited only to stakeholders from the initial planning stages or those working at a strategic decision making level. This could be problematic when individuals were asked to consider local decision making for Choose Life but were not involved in the decision making process.

A debriefing meeting was held with the Choose Life coordinator following the workshops. The purpose of the meeting was to identify and rectify any gaps in factual information and to collect any outstanding first visit case study data (e.g. information relating to outputs and outcomes of key activities).

3.8 National workshops

The first national evaluation workshop was held in February 2005 to bring together the evaluation team, NIST, local coordinators and a number of key national stakeholders. This workshop set out to develop understanding of the different models of national and local actions and activities being put in place; explore the evolving relationship between the approaches of the centre and of local areas; and identify ways in which progress could be measured. The second workshop, in February 2006, was held with the same range of stakeholders and represented the final stage of data collection. It set out to explore learning and test out key themes emerging from the evaluation. Key elements of the day were to:

• Review progress and learning in relation to the objectives of Choose Life
• Identify future priorities for development, support required to facilitate this, measures of progress and outcome and implications for information collection and sharing and research.

3.9 Economic focus

One of the key purposes of the evaluation was to track resources allocated by the Executive to the implementation of the strategy and action plan, and to investigate whether and how the money allocated to national and local initiatives:

• stimulated local investment from CPPs
• targeted relevant priority groups
• was spent on proven effective practice and interventions
• stimulated local innovation.

The economic analysis within the evaluation did not involve a separate data collection exercise. Instruments developed for the evaluation, such as data collection tools for local case studies and the survey of coordinators, were tailored so as to address issues of relevance to the economic analysis. Much data of relevance were also routinely collected by NIST. Three key sources of information were used for this analysis. The key resource was information returned to NIST by each local area using a standard template. The NIST template provided financial breakdown of activities that received Choose Life funding in each local area and further information, such as relevance of activities to Choose Life action areas (training, community/voluntary and self-help initiatives and coordination); relevance to Choose Life priority groups and objectives; target groups; intended outcomes of activities; evaluation; partners in delivery; and in-kind support received.

Funding was intended to act as a stimulus for additional support for local suicide prevention work. The template information was augmented by information collected through coordinator surveys which included questions about whether or not projects received additional funding and on project challenges and successes. The third source of information for the economic analysis was a survey of the 16 project examples, which was intended to capture the economic costs of providing Choose Life activities, e.g. total staff and associated costs incurred through the delivery of Choose Life projects . The survey also enquired about additional funding, including support generated by local statutory agencies and charitable organisations, as well as in-kind contributions, including the unpaid time of professionals and volunteers.

The analysis of the templates was subject to a number of important limitations. A challenge was that these templates were not completed in a consistent way. Although NIST and the evaluation team made attempts to clarify sources of data, it remains the case that details of funding allocated to some activities had not been submitted to NIST by the end of the evaluation period. In particular, data for the financial year 2005-06 remained incomplete at the time this report was being finalised Data from case studies, and to a lesser extent from the survey, also suggested that most areas did not provide as much information as they could on additional sources of funding or in-kind contributions to projects.

3.9.1 Changes to original proposal

In addition to the approach set out above, discussions between NIST, the commissioners and the evaluation team identified further potential areas that would benefit from attention. The evaluation team agreed to produce two additional outputs, namely:

• A review of economic aspects of international evidence and paper on economic aspects of suicide prevention and cost-effectiveness of suicide prevention strategies
• Estimation of the potential economic benefits that may be realised if Choose Life does reach its goal of reducing the suicide rate by 20% over a ten year period.

3.10 Data analysis

Data analysis was conducted as a continuous (iterative) process throughout the evaluation. An evaluation database was designed and used to store (and retrieve) data on all 32 local areas, including surveys with local coordinators and case study areas. Findings from each element of data collection (case studies, workshops, national interviews and surveys) were written up in detailed reports which were then used for comparative analysis.

Data were analysed according to predefined themes, for example, sustainability and partnerships. Themes were also developed from inductive analysis of data and built upon analysis conducted in each phase of the evaluation.

An analytical framework was developed to guide the team throughout the evaluation; this was expanded and developed according to themes emerging from the data. Analysis of data primarily drew upon the 'charting' method of systematically handling complex datasets by drawing out the dimensions that related to each theme across all cases.

« Previous | Contents | Next »