Scottish Government

Effective Interventions Unit Evaluation Guide 11:
Evaluating Arrest Referral Schemes

This document is also available in pdf format (47k)

WHAT IS ARREST REFERRAL?

Arrest Referral is an intervention aimed at people who have been arrested and whose offences may be linked to drug use. An Arrest Referral service capitalises on access to arrestees at this time to help them engage with treatment and care services with the aim of reducing involvement in future drug misuse and associated criminal behaviour. It is anticipated that these services will form part of a wider integrated care system in each area. For further details on integrated care go to http://www.drugmisuse.isdscotland.org/eiu/intcare/intcare.htm . The main aim is to provide information about appropriate services to arrestees and to facilitate voluntary referral to treatment at the primary points of entry into the criminal justice system - usually police cells or court premises.

WHY MONITOR AND EVALUATE ARREST REFERRAL SCHEMES?

Arrest Referral is still in its infancy in Scotland, with many Drug Action Teams (DATs) only just beginning to put services in place. As with any new initiative, a robust system of monitoring and evaluation should be an integral part of the scheme. This helps to assess progress towards agreed objectives and identify where improvements could be made. Monitoring and evaluation are separate yet complementary methods and both are necessary to ensure the effective introduction and successful development of arrest referral in Scotland. For definitions of monitoring and evaluation (and other related concepts) please see EIU Evaluation Guide 1 at http://www.drugmisuse.isdscotland.org/goodpractice/EIU-evaluationg1.pdf .

In the first instance, schemes may want to conduct local monitoring and evaluation. However, it is anticipated that the data collected for monitoring and evaluation purposes at local level might also be used for subsequent national evaluations to enable cross scheme comparisons and transferable learning from other areas. The principles outlined in this guide are designed to facilitate both local and national level evaluation.

IDENTIFYING OBJECTIVES AND MONITORING PERFORMANCE

As already stated, the purpose of an Arrest Referral service is to offer an opportunity to drug users who have been arrested to engage with drug treatment and/or other appropriate services with a view to reducing their offending behaviour. There is some variation in practice across schemes and, therefore, some variation in what would constitute appropriate monitoring and evaluation. However, we suggest that evaluations should broadly be driven by the same principal objectives. These are:

1 - To assess whether Arrest Referral is effective at targeting drug users who offend and ensuring that they enter into needs-based treatment and care services.

2 - To assess whether entry into treatment and care services through a process of Arrest Referral contributes to reducing future drug misuse and related offending in the target population.

A multi-measure and multi-method approach to monitoring and evaluation may be needed. An important part of these activities will, of course, be to assess the extent to which schemes are meeting the aims and objectives outlined above. However, monitoring and evaluation should also examine service level outputs and outcomes e.g. the effectiveness of procedures. Whilst this guide focuses primarily on the first of these, i.e. monitoring throughput and outcomes for arrestees, it is also important that individual service co-ordinators bear in mind the need to record process level data. This is discussed briefly at the end of the guide.

DESIGNING AND TESTING DATA COLLECTION INSTRUMENTS

In the absence of any baseline against which to measure the success of Arrest Referral in Scotland, it is anticipated that schemes will also want to ensure their results are comparable with others. To enable this, there is a need for some standardisation of monitoring and evaluation tools. We would therefore recommend the following (also see Table 1):

Core Information

Common to all schemes is the need to identify suitable arrestees for interview by referral workers. It is recognised that the approach to targeting potential participants may vary across schemes. In order to monitor the characteristics of both the appropriate identified target population and those who take up referral offers, some basic demographic information is necessary. ¹ This 'core data' will essentially be epidemiological in nature and focus on numbers of arrestees screened by police or Arrest Referral workers, their characteristics and their willingness to engage in referral. Data collected at this stage of the referral process would be primarily screening information and represent the basic core details necessary to monitor throughput and uptake of referral. Data generated would be largely descriptive.

Auxiliary Information

Where suitable arrestees have been identified, and contact with an arrest referral worker has been made, more in-depth data collection will be required to identify individual's treatment needs. This may be as part of an agreed assessment process. We have called this 'auxiliary information'. Whilst the nature of desired local level data will vary greatly across schemes, ² some examples of the type of auxiliary information that we would anticipate being collected would be: detailed health history profiles; information about specific drug using profiles and information relating to housing, employment and child care needs. Such data would be more complex in nature than the core data outlined above and its collection would probably require more focused, in depth, open ended interviews. It is anticipated that this phase of data collection would be carried out by referral workers at the first meeting. However, this may vary across schemes and may be contingent on time restrictions imposed by custody officers at the point of interview.

Post-Referral Information

One of the difficulties faced by Arrest Referral schemes is how to monitor progress of referred persons after a referral has been made. If performance is measured solely by the total number of referrals made, all that is required is the information outlined above. Given the rationale and motivation behind Arrest Referral initiatives, however, it is more likely that schemes will want to engage in post-referral information gathering to gauge the uptake and subsequent success, or otherwise, of the referral route. This will also include, at the crude level, measures of retention in services and impact on future offending and other quality of life variables both in the short and long term. Post-referral monitoring is also essential in providing information about the effectiveness of Arrest Referral in comparison to other interventions. Arrest Referral schemes cannot be monitored in isolation and collecting information post referral may allow comparison of success with, for example, self referral or GP referrals. It is recognised that this type of follow-up can be time-consuming and potentially difficult. However, it may will be worth the time and effort to collect this information at the post-referral stage.

TABLE 1: Suggested data items for monitoring/evaluation of Arrest Referral Schemes

Whilst post-referral monitoring may be desirable, this will be largely contingent on access to sources of information about referred persons. Where Arrest Referral workers maintain a caseload, this information will be readily available. However, this may be the exception rather than the rule. In the majority of cases, there will be confidentiality issues in terms of access to medical and mental health practitioners' files and police records relating to future subsequent offending, especially that which is drug-related.

These can be overcome by developing information sharing protocols and procedures - see EIU Integrated Care document chapter on information sharing at http://www.drugmisuse.isdscotland.org/eiu/intcare/intcare.htm The core data below includes the item 'consent to share information'. This refers to asking the consent of the individual to share the information being collected with other agencies - principally Arrest Rreferral workers and referral agencies.

We recommend that most information should be collected by means of structured, closed-response questionnaires. This will enable collection of basic data in a standard way. However, there is scope for evaluations to include more open-ended questions about perceived usefulness of schemes, projected goals of entering into treatment and care programmes and experiences of the scheme and referral. Qualitative responses can be difficult to code and can be labour and time intensive to analyse. However, they often provide a wealth of contextual information on individuals' views and experiences. For more information on quantitative and qualitative methods and their relative merits please see EIU Evaluation Guide 3 http://www.drugmisuse.isdscotland.org/goodpractice/EIU-evaluationg3.pdf . For guidance on data coding, management and analysis please see EIU Evaluation Guide 4 at http://www.drugmisuse.isdscotland.org/goodpractice/EIU-evaluationg4.pdf

MONITORING AND EVALUATING THE 'SYSTEM'

Monitoring and evaluation of Arrest Referral schemes is not simply restricted to analysing client progress. It is also important to monitor the effectiveness of working practices - for example, the training needs of referral workers, best practice for selecting interviewees, whether on site or on call arrangements are more practicable. It is important that monitoring and evaluation of this sort is not viewed as a vetting or punitive exercise. Rather it should be viewed as contributing to the overall understanding of best practice and the ways in which all of those involved in the Arrest Referral process can contribute to its effective functioning.

Whilst every effort may be made to ensure that all bases are covered in the collection of data from the outset, good monitoring and evaluation programmes will inevitably involve an element of change. Thus, as Arrest Referral schemes evolve, one suggestion is that scheme co-ordinators develop a learning log in which changes to practice and procedures are systematically recorded. They might also be used to outline good and bad practice for the learning benefits of others. Learning logs might also be used for recording information relating to training needs of Arrest Referral workers and for recording relevant information about lessons learnt from individual cases. The learning log might also collect data on the following issues:

• Inputs, outputs and outcomes in terms of resources, staff time and people management.
• Who is responsible for data collection and any variations in data collection techniques or methods. This should include information about the point location of screening (e.g. court or police suites) and whether initial screening is carried out by a referral workers or member of police staff.
• In some cases, performance may be affected by wider systems in place for dealing with referred persons. Where treatment provisions are restricted in such a way that referrals are affected by treatment availability rather than being needs led, this should to be recorded.

Recording such information in a learning log ensures that there is a clear audit trail. This can be used as a means of supplementing more objective performance outcomes to enable a broader and more in depth evaluation of Arrest Referral schemes.

SUMMARY

• Plans for monitoring and evaluation of Arrest Referral schemes should be part of initial plans to develop and implement the intervention.
• Monitoring and evaluation should not focus solely on throughput and uptake of treatment services but also on the functioning of Arrest Referral systems and processes.
• Data collection for arrestees will be phased with core data requirements, auxiliary data requirements and post-referral data gathering.
• Clear audit trails should be maintained. This could include a learning log that details changes to procedures and practices.

FURTHER RESOURCES

Effective Interventions Unit (2002) Arrest Referral: A Guide to Principles and Practice, Scottish Executive: Edinburgh

Effective Interventions Unit (2002) Integrated Care for Drug Users: Principles and Practice, Scottish Executive: Edinburgh

Sondhi, A., O'Shae, J., & Williams, T. (2002) Arrest Referral: Emerging Findings from the National Monitoring and Evaluation Programme, DPAS Briefing Paper 18, Home Office: LondonSondhi, A., O'Shae, J., & Williams, T. (2002) Arrest Referral: Emerging Findings from the National Monitoring and Evaluation Programme, Home Office: London

Footnotes

1 For fullness of understanding, schemes may want to record information relating to number of suitable detainees. Where schemes disseminate service literature to all identified substance misusing detainees, this should be recorded.

2 Some information about arrested persons may be used to address local level needs only. In some cases, individual schemes will target a particular category of drug users, for example, people under 21, women or single parent drug using arrestees. As such, there may be information that is required for monitoring purposes at local level that is perhaps less appropriate in a national framework.

Effective Interventions Unit
Substance Misuse Division
Scottish Executive
St Andrew's House
Edinburgh EH1 3DG
Tel: 0131 244 5117 Fax: 0131 244 2689
EIU@scotland.gsi.gov.uk
http://www.drugmisuse.isdscotland.org/eiu