« Previous | Contents | Next »
Listen
CHAPTER TWO METHODS
2.1 This section describes the methods used to undertake an analysis of the international literature on ASIST. The purpose of this review was to examine the ways in which ASIST has been implemented in different countries and contexts around the world (discussed in the next chapter), and to ascertain the effectiveness and impact of ASIST (discussed in Chapter 4).
Scope of the review
2.2 The review included all available and relevant literature relating to the Applied Suicide Interventions Skills Training ( ASIST) programme up to 2007. These included evaluation reports (published and unpublished), a number of small-scale Scottish evaluations, and a wide variety of other reports or articles on ASIST which were not strictly evaluations.
2.3 The majority of papers included in this review were identified by the commissioners of this study. A further search was conducted by the research team to identify any literature relating to ASIST which was not included in the original list. In addition, all Choose Life Coordinators in Scotland were contacted to confirm whether there were any other published, or soon-to-be published evaluations of ASIST in Scotland which should be included in the review.
2.4 In addition to reviewing ASIST literature, a limited review of the literature on other types of suicide prevention training was undertaken. The training programmes which provided evaluation evidence were: Skills-based Training on Risk Management ( STORM), Mental Health First Aid ( MHFA) and its Scottish equivalent, Scottish Mental Health First Aid ( SMHFA). Although MHFA and SMHFA are not suicide prevention programmes, they address the possibility of suicide in people who are experiencing mental ill health using risk review material from an earlier version of ASIST.
2.5 The aim of this review was to examine the similarities and differences between ASIST and other training programmes that address suicide intervention; and to identify any lessons for the future development and sustainability of ASIST in Scotland. The primary focus was on similarities and differences in format, targeting and implementation, rather than comparisons of effectiveness. The findings from this second review are presented in Section 5 of this report.
What counts as evidence?
2.6 In order to provide evidence of effectiveness, suicide intervention programmes would ideally be able to demonstrate a direct reduction in suicide rates. There are, however, substantial difficulties in demonstrating such an impact. For example:
- The reporting of suicidal acts is inaccurate and unreliable.
- Suicide is a statistically rare event.
- Interventions, such as training, are indirect ( i.e., targeted at helpers, not suicidal individuals).
- The effects of some interventions - training, in particular - may not be seen for many years
- Furthermore, in the case of training interventions, it is not clear how many people need to be trained - and how much contact they need to have with people who are at risk - in order to result in a reduction in suicides.
2.7 Moreover, it is generally acknowledged that suicide rates are affected by a multitude of societal and individual factors 3 - not just the suicide intervention programme. Given these complexities, it would be practically impossible to attribute any changes in suicide rates to a specific preventive intervention. In light of these limitations, most evaluation studies seek to measure changes across a broad range of outcomes (for example, reported changes in caregivers' knowledge and skills), and improvements in these areas are seen as evidence of a programme's effectiveness.
2.8 However, even when the focus is on indirect measures of programme effectiveness, community-based suicide intervention programmes are still very difficult to evaluate. This is due to their "complex, programmatic and context-dependent" nature. 4 The various elements involved in the implementation and delivery of suicide prevention programmes ( e.g. participation, targeting, role out, variation in settings, sustainability) are hard to predict and control and hence are difficult to evaluate using standard experimental designs. It is, therefore, not always possible to design and execute scientifically rigorous studies using statistical methods of analysis. In light of the challenges involved in conducting research in the area of suicide prevention, the following review incorporates findings from both qualitative and quantitative designs, and adopts a broad view of evidence which includes that which is provisional, emergent and incomplete. 5 This approach reflects a view of suicide prevention practice as "complex, dynamic and deeply rooted in context." 6
2.9 A model which is often used to evaluate training interventions is the Kirkpatrick Model. This considers the effectiveness of the training on four levels: (i) reaction; (ii) learning; (iii) behaviour change and (iv) organisational change. These four levels will be described in more detail at the beginning of Section 4 of this report.
Assessing and appraising the evidence
2.10 The initial search for ASIST-related literature produced 37 papers which were considered to be relevant and useful for achieving the aim of this review. These included:
- fifteen (15) international evaluation reports, published and unpublished
- seven (7) small-scale unpublished Scottish reports undertaken for local monitoring purposes ( e.g., findings from a single questionnaire survey involving small numbers of people)
- fifteen (15) miscellaneous papers, mostly unpublished, which comprised a broad range of other types of reports, including strategic policy documents, magazine articles, project reports on a wide range of suicide prevention activities (including ASIST), etc.
2.11 The key difference between the 15 international evaluation reports (including one Scottish report) and the seven small-scale Scottish reports was that the international reports were designed as training programme evaluations ( i.e. they addressed the various Kirkpatrick levels, typically incorporated a number of measures, attended to methodological issues etc), whereas the Scottish reports were designed as local monitoring exercises and hence were less rigorous in terms of their methodology.
2.12 A full list of papers is given in References section of this report. Note that three of the 15 evaluation papers involved evaluations of an early version of ASIST, the Suicide Intervention Workshop ( SIW). In order to meet the aims of this review, the main focus was on the 15 evaluation reports, with all other literature used as a supplementary sources of information.
2.13 For the purpose of assessing the quality of evidence, all 15 international ASIST evaluation reports were subjected to more detailed appraisal. One of the 15 reports was a Scottish evaluation commissioned by Choose Life locally in West Dunbartonshire and received intensive support from NIST. As all 15 papers were either partly or wholly qualitative, a framework developed by Spencer et al (2003) for assessing the quality of qualitative evaluations was used as a basis for commenting on the strengths and weaknesses of each study. 7 The papers were assessed in four broad areas:
- whether the evaluation's aims and objectives have been clearly stated and addressed
- whether data were adequately collected
- whether data were adequately analysed, interpreted and reported
- whether findings were contextualised into a theoretical/practical framework.
2.14 A copy of the full list of criteria is provided in Annex 1 of this report. On the basis of these criteria, the quality of each paper was judged as good, fair or poor. One-fifth of the evaluation papers were appraised by two additional independent reviewers. There was 90% initial agreement between reviewers and the small areas of disagreement were resolved through further discussion.
2.15 The quality of evidence in the different studies varied greatly, with roughly a third of the papers falling into each of the three quality assessment categories. It was decided to include all papers in the review for two reasons: (a) the small number of papers available; and (b) all papers, regardless of their research quality, had some interesting insights to offer. However, the quality of each individual paper has informed the way evidence has been considered in terms of answering the questions outlined in Section 1 of this report. (See Discussion in Section 6.)
Synthesising the evidence and writing the final report
2.16 A data extraction sheet was used to allow a full summary of each paper to be made. Data extraction sheets for all papers are provided in Annex 2 of this report. These tables contain:
- information about the authors
- publication details
- place and time of evaluation
- aim and focus of evaluation
- ASIST version used
- targeting and implementation
- sample size and composition
- time between training and evaluation
- evaluation design and instruments
- findings
- cost information
- lessons / recommendations
- strengths and weaknesses of the evaluation
- quality of evidence
- other comments ( e.g. local policy framework, geography etc).
2.17 In-house reports from Scotland have been summarised using a simplified data extraction sheet and were not graded on quality of evidence (see Annex 3), as they were not designed or intended as methodologically rigorous studies.
« Previous | Contents | Next »