Scottish Government

SERIOUS VIOLENT AND SEXUAL OFFENDERS: THE USE OF RISK ASSESSMENT TOOLS IN SCOTLAND

CHAPTER FOUR: STRENGTHS AND WEAKNESSES OF DIFFERENT APPROACHES TO RISK ASSESSMENT

INTRODUCTION

4.1 As Kemshall (2002) observes in her review of current issues associated with the assessment and management of risk among serious violent and sexual offenders, different risk assessment tools have been shown to have particular strengths and weaknesses: for a detailed discussion of these the reader is referred to that review and to Cooke (2000). In this chapter the perceived strengths and weaknesses of different approaches to risk assessment in practice are examined. First, however, the reasons why different agencies and professional groups adopted particular tools or particular approaches to risk assessment are discussed.

REASONS FOR ADOPTING PARTICULAR APPROACHES TO RISK ASSESSMENT

4.2 Respondents were asked in the audit questionnaire to indicate the reasons that their organisation had opted to use the particular risk assessment tool to which the audit return referred. The relevant responses are summarised in Table 4.1.

Table 4.1: Reasons for adopting risk assessment tools

4.3 The factors most commonly said to lie behind the choice of tool were: the ease with which the tool could be administered and/or scored; the ability of the tool to identify risk of harm; and the ability of the tool to identify the risk of sexual offending. Cost, by contrast, was least often indicated as being a factor that had encouraged adoption of a particular tool by an organisation.

4.4 The reasons for choosing particular risk assessment tools varied somewhat among the different professional groups. The factors accorded greater prominence by social workers were ease of use, known accuracy of the tool and knowledge of the tool's use in other locations. Police, on the other hand, emphasised the ability of the tool to identify risk of sexual offending and risk of harm. Tools employed by prison psychologists were most often chosen because of their ability to predict risk of violence, their ease of administration, known accuracy and cost.

4.5 Psychologists were most attracted to tools - such as the PCL-R and HCR 20 - that had been validated and for which Scottish norms were available. As one prison-based psychologist explained:

"… they are well-researched and they work and I think they are tools that everyone agrees measure what they are meant to measure."

A further advantage of the use of standardised tools was, according to this respondent, the opportunity they afforded for a consistent approach to decision making, thereby ensuring that all prisoners were treated in a similar way.

4.6 Given that the nature of the tools being used varied enormously, it is instructive to consider also the reasons underlying the choice of particular tools. Clearly this is not possible with many of the tools being used, since they were only included in the audit on one or two occasions. Instead, the comparison focuses upon the four most widely used tools. The police-developed risk assessment procedures have been included since, although they differed slightly from force to force, they represented a common underlying approach to the assessment of risk. The relevant data are presented in Table 4.2.

Table 4.2: Reason for adopting particular tools

4.7 The RAGF was most commonly selected because of its ability to predict risk of harm, its ability to identify risk of violent offending and its compatibility with approaches adopted by other agencies. In only two cases, however, was its adoption said to have been influenced by its known accuracy in predicting risk. A key factor in the use of RAGF was its introduction and assumed support by the Scottish Executive, with some local authorities having participated as pilot areas for its use:

"The fact that the Scottish Executive issued, and therefore by implication endorsed, the framework."

"This tool was adopted as a result of guidance (Red Book) issued by the Scottish Executive."

"RA1-3 was introduced by the Executive and we adopted it as the central government advice on risk assessment."

4.8 Another reason given for adoption of the RAGF was the fact that it helped to structure the risk assessment process and assisted practitioners to justify risk decisions they had reached:

"Rather than the tool's 'ability' to identify, it assists practitioners to think clearly through the risk assessment process and 'justify' the assessment which they have reached."

4.9 LSI-R on the other hand, had most often been selected for use because of its known accuracy, its ease of administration and knowledge of its use in other locations. Other reasons for opting to use the LSI-R included its ability to assist in gate-keeping and the targeting of resources, its ability to inform action plans, its positive reputation and, related to most of these factors, its ability both to predict risk of recidivism and to identify criminogenic needs. For example:

"LSI-R helps this department target its resources more appropriately to those whose need is greatest."

"Ability to identify criminogenic needs as well as likelihood of re-offending."

"To provide a tool to assist in gate-keeping. We use LSI-R as an indication of the appropriate level of intervention."

"Measures both static and dynamic risk factors. Believed to be a forerunner in field."

4.10 The cost of the LSI-R was, however, viewed as a disincentive by social work respondents, especially since a cheaper alternative - in the form of the RAGF - was available:

"LSI-R is being used pretty widely and we have had a lot of training in it and are fairly confident using it. That is not to say we think it is the best, but probably up there with the ones that are most efficient. None of them are perfect in any way. We have kind of questioned if we should use them for the cost, as it is pretty expensive to keep buying all the forms. So we have contemplated if we should shift to using OGRS or OASys or RA 1-4 because they are cheaper. The Scottish Executive are not pushing very strongly for one or the other so we are left with a judgement call in terms of the one we feel most comfortable with."

"Well, we got this guidance from the Scottish Executive so rather than pay a pound a sheet for LSI-R…"

4.11 Known accuracy and ability to predict risk of sexual and violent offending most often encouraged agencies to adopt Matrix 2000, while their ability to predict risk of sexual offending and risk of harm most often lay behind the choice of risk assessment procedures adopted by the police. One officer explained, however, that the current approach to risk assessment was likely to be an interim measure:

"Risk assessment form adopted until a recognised/tested system has been identified which will fulfil the needs of the organisation."

4.12 The Tay Project had decided to develop their own tool - drawing upon research by Hanson and Thornton - since they did not feel that existing tools covered the range of relevant risk factors. Although David Thornton had subsequently developed Matrix 2000, which incorporated dynamic risk factors, project staff did not feel comfortable using it on account of its complexity and lack of ease of administration. As one interviewee commented, " it's not user-friendly. It's almost like an academic experience rather than like a risk assessment."

4.13 The project staff were confident that their tool was reasonably effective in helping to allocate offenders to categories of risk, on the basis that the resulting score was usually congruent with the worker's subjective impression. Similarly, the proportions of offenders falling into each category on the basis of risk assessments informed by the tool appeared appropriate. Staff reported that where there appeared to be some discrepancy between professional judgement and the categorisation resulting from the tool, the scoring would be double-checked using RRASOR or Matrix 2000.

4.14 The Scottish prison Service had also given consideration to the use of locally developed tools such as the Tay Project risk assessment instrument but decided against its adoption because of its lack of validation. As one prison respondent explained:

"It didn't seem to us to have any merit in that it wasn't validated in any way and no research had been carried out into its effectiveness. It wasn't standardised in its operation."

The staff at the Tay Project were acutely aware of the need for the tool to be assessed and validated but considered this task to be beyond their ability. They were keen that the TAYPREP-30 should be validated externally and any necessary improvements made to enhance its predictive ability.

Defensible decision-making

4.15 A number of respondents in interview emphasised how their choice of risk assessment tools or approaches had been driven by a need to engage in defensible decision-making: that is, the ability to back up risk assessment decisions with empirically based evidence. For example, one police officer commented that the police had been scrutinising the Tay Project risk assessment tool for possible adoption "basically to give as a more defensible position". Another police respondent explained the position more fully:

"We have quickly come to the conclusion after a wee while of working with it [police risk assessment procedure], realising that if we were called into question, if we were asked to go to court and give an account of why we had come up with this risk grading it would only be based on opinion and, as you know, in a court of law opinion doesn't stand for anything…. We have realised for some time now that we need something that is accredited or backed up by some sort of research and we are currently looking at the Tay Project."

4.16 Social work respondents similarly believed that a more structured approach to risk assessment promoted defensible decision-making. The ability to make defensible, transparent decisions also lay behind the approach to risk assessment that had been adopted by the Scottish Prison Service. Risk assessments had to be completed for mandatory and discretionary life sentence prisoners who were having their tariffs - that is the 'punishment' part of the sentence - set by the courts. Those who had already served in excess of the 'tariff' would be referred to a tribunal who would decide, on the basis of a risk assessment, whether the prisoner could be released or should continue to be detained. Risk assessments therefore needed to be evidence-based, employing the best available tools that had, ideally, been validated against a Scottish or other UK population.

PERCEIVED STRENGTHS OF THE TOOLS

4.17 The perceived strengths of the risk assessment tools employed by different professional groups are summarised in Table 4.3 and the perceived relative strengths of different tools are presented in Table 4.4. These data are based upon audit responses but the issues generated were also explored in interviews.

Table 4.3: Perceived strengths of the tools referred to in audit returns

4.18 The most commonly perceived strength of the tools used by social workers was their ease of administration, their ability to predict risk of harm and the fact that they were widely used in other agencies. Police officers, on the other hand, identified their ability to identify risk of sexual offending and risk of harm and their compatibility with the risk assessments undertaken by other agencies the main strengths of their approaches to risk assessment. Prison psychologists most often viewed the strengths of the tools they used as their ease of administration, known accuracy and ability to predict risk of violent offending.

Table 4.4: Perceived strengths of particular tools

4.19 Just as different risk assessment tools had been adopted for different reasons, so were they perceived by audit respondents as possessing differing strengths. Its ability to identify risk of harm, its wide use in other agencies, its ease of use, its compatibility with other risk assessment procedures and its ability to predict violent offending were viewed as the main strengths of the RAGF. RAGF was also viewed by social work respondents as assisting professional judgements of risk and encouraging a more structured approach to assessment and case planning:

"Assists and records basis for professional judgements. Records risk factors and analysis based on them."

"Enables/encourages social workers to assess and implement action plans in a more structured way."

4.20 The most commonly identified strengths of the LSI-R were its known predictive ability with respect to re-offending, its wide use in other agencies and its ease of administration and/or scoring. Through its attention to both need and risk, LSI-R was said to assist in the targeting of criminal justice social work services and it was also believed to encourage greater consistency in assessment.

"Ability to identify risk of reconviction and link to appropriate services."

"Increased consistency in SER (Social Enquiry Report) standards. Efficient resource allocation."

4.21 The strength of the LSI-R was not seen, on the other hand to lie in its ability to predict risk of harm or risk of sexual or violent offending. Matrix 2000, however, was believed to be good at predicting risk of sexual offending and violent offending and was considered by some respondents to be relatively easy to use. The police risk assessment procedures were considered good at assessing risk of sexual offending and risk of harm and were believed to be compatible with approaches to risk assessment in use in other agencies. A further strength of these procedures was thought to be their ability to combine a variety of information from a range of relevant agencies and engender common ownership of the resulting risk assessments and risk management plans:

"Risk assessment contains information obtained from all agencies having involvement with the subject and is a signed document agreed between the agencies."

4.22 The HCR-20, SVR-20 and PCL-R, by measuring both dynamic and static risk factors, were said by those who used them to provide a sound basis for developing risk management plans. They were also perceived as having the advantage of being applicable in a multi-disciplinary context. HCR-20 and PCL-R were also said to have the advantage of being widely known and, therefore, providing for a common understanding across within and across disciplinary groups. HCR 20 and SVR 20 were said to be well structured for use in interviews and resulting risk assessment reports. These tools, and the PCLR, were also reported have good validity and reliability (see Kemshall, 2002).

4.23 HCR20, in particular, was believed by those who used it to encourage a focus on risk management both in the short and longer term:

"We are not just saying 'this guy is high risk, just leave him', we are actually saying 'okay, this guy is high risk at the moment, this is how you can manage it so you can lower his risk'. That's good for us from a professional point of view."

4.24 Other psychological risk assessment tools - such as the VRS and VRSSO - were viewed as having fewer strengths than those just described, but they had an additional advantage insofar as they were able to identify where an offender was placed in terms of their motivation to change.

4.25 The SPS risk assessment strategy was thought by managers and psychologists to encourage a more focused approach that nonetheless yielded a considerable amount of relevant information. It was also said to promote a more standardised approach to risk assessment, which ensured that all prisoners were treated the same way. One psychologist succinctly summarised its perceived strengths as follows:

"Consistency between raters, use of evidence based materials, transparency in decision-making, fairness to prisoners who can see why decisions are being made and when they are being made."

4.26 The TAYPREP-30 was described both by project staff and by police officers as being user-friendly and easy to use after some training. Project staff believed that it tended to encourage a more consistent approach to the assessment of sex offender risk and ensured that workers gave consideration to a wide range of risk factors. Precisely because it covered all the relevant dynamic factors, project staff believed that it was particularly useful for informing risk management strategies

PERCEIVED WEAKNESSES OF THE TOOLS

4.27 The risk assessment tools in use in Scotland were also perceived to have a number of weaknesses. These are shown in Table 4.5.

Table 4.5: Perceived weaknesses of the tools referred to in audit returns

4.28 Some of the standardised tools in use in health settings (such as the PCL-R and HCR-20) were said to be limited by the absence of normative data from samples of Scottish in-patients. For this reason, their effectiveness with mentally disordered offenders was unknown. In the case of the tools employed by prison based psychologists, the main disadvantage was perceived to be their limited use in other agencies and, hence, the absence of a common basis for risk assessment. The cost of training staff in the use of standardised instruments was also said to be high. The police risk assessment procedures were also viewed as limited because they differed from the approaches used by other agencies but they were most often criticised for their lack of validation and unknown predictive accuracy. The risk assessment tools employed by social workers were said to possess a number of weaknesses including their inability to predict risk of sexual offending, their lack of validation, their inability to predict risk of harm and their inability to predict violent offending.

4.29 The perceived weaknesses of different risk assessment tools are summarised in Table 4.6. In addition to their unknown predictive accuracy, the police risk assessment procedures were thought by one respondent to be insufficiently objective:

"Is very heavily dependent on personal judgement, with unclear guidance how to assess clinical and actuarial factors."

However there was also some reluctance among the police interviewees to adopt Matrix 2000 - the tools being used by some police forces in England and Wales - because, while it might provide an indication of risk level, it was regarded as less useful for informing risk management plans.

4.30 The lack of a consistent approach between forces and between different agencies was also believed by police to be a weakness of the risk assessment procedures they adopted:

"Many sex offenders move frequently and there is no general police risk assessment format throughout Scotland."

"Weaknesses are found where an offender moves from our area from outside and the previous criminal justice team from another local authority use their own tool for risk assessment."

4.31 One social work respondent, who regularly liaised with the police over the management of sex offenders, expressed some concern that the relatively unstructured approach to risk assessment might result in the exaggeration of offender risk. In other words, people might be categorised as high risk on the basis of what they might do in the future rather than on the risk that they currently present.

Table 4.6: Perceived weaknesses of particular tools

4.32 RAGF was criticised primarily for its unknown ability to accurately predict risk and, to a lesser extent, for its inability to identify the risk of sexual offending. Other concerns about the RAGF included the fact that it was time-consuming to complete, ran the risk of being insufficiently objective and did not produce an overall 'score' to indicate the offender's level of risk. One social work respondent also thought there was a possibility that workers using the RAGF might underestimate risk as a means of workload management. For example:

"Staff find initial risk of harm repetitive. 'Automatic High Risk' list is too wide. RA4 is too long and repetitive and still needs a risk management plan at the end of it."

"Can be subjective if administrator is either over simplistic or over pragmatic."

"Lack of a points system means a reliance on social workers' judgement, but this can be positive too."

4.33 The main drawbacks of the LSI-R were perceived to be its inability to identify risk of sexual offending, risk of violent offending and risk of harm and its cost. Other less commonly mentioned weaknesses included the perception that it is overly prescriptive and can disadvantage people from deprived backgrounds, its tendency to replicate what is already gleaned from the interview, its use of confusing language (e.g. double negatives) and the absence of validation against a Scottish sample. One social work respondent also observed that the significance of the resulting score might vary widely between rural and urban areas. Most concern, however, centred upon the limited value of the LSI-R with particular groups of offenders, notably those who are the focus of this report:

"The tool is of limited value in work with sexual offenders and most serious violent offenders."

Does not specifically address violent behaviour and should not be used with sexual offending. Caution must be used. Does not identify risk of harm."

4.34 The VRAG was suggested by interview respondents to be limited because of its purely actuarial nature and its lack of attention to dynamic risk factors. However some psychologists questioned how 'objective' some of the more favoured structured clinical measures - such as the PCR-L - actually were, stressing the importance of these tools serving as an aid to clinical assessment rather than replacing it:

"I think if, as a service, we think it [PCLR] is an absolute, reliable, hard tool then we are slipping up. It should augment your clinical practice, not replace it."

4.35 Because most tools require an element of judgement, some respondents also saw the potential for a lack of consistency in the administration and interpretation of structured clinical tools, either within or between professional groups. For instance, one prison psychologist explained:

"…other people through in health, for example, or mental health tended not to stick to the format, so it's easy in may ways for the format to be shifted or to be modified… [different professionals] come from different theoretical bases and although they try to adopt a psychological theoretical basis or orientation, it actual fact it becomes quite skewed and quite changed."

4.36 One weakness perceived to be associated with the PCL-R was not inherent to the tool itself, but rather concerned how others interpreted the scores derived from it. A high score on the PCL-R, for example, tended to lead to stereotyping and a diversion of attention away from risk management issues. As one psychologist explained:

"If you give someone a score that is high on the PCL-R, then you have immediately given them this kind of label of psychopathy which sends everyone a bit sky high. Everyone has got that great Hollywood image of Anthony Hopkins in Silence of the Lambs. You have lost them - they stop looking at risk management at that point."

4.37 One psychologist also suggested that there was too much reliance upon psychological risk assessment tools that could only be administered by trained forensic psychologists and psychiatrists and that this could also limit the level of multi-disciplinary involvement in risk management plans. Instead, s/he suggested, initial screening could be undertaken with a general risk measure such as LSI-R, which could be employed by and understood by a wider range of professional groups.

IDENTIFYING DIFFERENT LEVELS OF RISK

4.38 There was a view among social work respondents that existing tools is use were useful for providing a general indication of risk level but that they needed to be supplemented by clinical judgement, particularly since some types of offenders were not adequately 'covered' by existing tools.

4.39 Psychologists saw the strengths of the tools they employed less in terms of their ability to assign offenders to different levels of risk, but more in terms of their ability to help identify offenders' risk management needs. For example one respondent commented, " it's not important if they are low, medium or high: it is about how you manage it" while another suggested that " it is more about hazard identification and hazard situation management".

SUMMARY

4.40 Tools had most often been adopted for use in risk assessments on account of the ease with which they could be administered and/or scored, their ability to identify risk of harm and their ability to identify the risk of sexual offending. Social workers were more likely to use tools that were relatively easy to administer while psychologists were attracted to tools that had been validated, particularly if normative data for Scotland were available. One specialist project had developed its own tool which, although it had not been validated, was being considered for national adoption by the police.

4.41 A primary consideration for all professional groups in their approach to risk assessment was the ability to make defensible decisions backed up by appropriate tools. Different tools and approaches were perceived to have different strengths, however their ability to inform risk management plans was considered critical. Weakness associated with different tools included their complexity, their lack of objectivity, their lack of validation and their inability to measure the specific risks associated with the types of offenders who are the subject of this report.