The Scottish Government

« Previous | Contents | Next »

Listen

Integrated Care for drug users: Principles and practice

Chapter 7: Monitoring and Evaluation

Monitoring and Evaluation

• Definitions of monitoring and evaluation and why it is important

• Describing service level evaluation and strategic evaluation

• Developing and evaluation culture

Monitoring and evaluation

This chapter discusses the monitoring and evaluation of integrated care for drug users. It examines and discusses the concepts of monitoring and evaluation, service level and strategic level evaluation and developing an evaluation culture.

The key sources of evidence that have been drawn on for this section are the EIU evaluation guides and the EIU consultation workshops. The EIU evaluation guides have been developed and produced by EIU since 2001 and draw heavily on the published evaluation literature.

What are monitoring and evaluation?

Monitoring is an on-going process involving the continuous and regular collection of key information on the inputs, outputs and outcomes of a service, intervention or partnership. The main aim of monitoring is to establish if something is going to plan, and whether any change in activity is necessary.

Evaluation is a systematic assessment of whether a service / intervention / partnership has achieved its stated aims and objectives. Evaluations commonly address questions about effectiveness, efficiency and acceptability. An evaluation can address the context and process of evaluation as well as the outcomes achieved.

For further information on the definitions and core concepts of monitoring and evaluation, please see the Effective Interventions Unit Evaluation Guide 1. Definitions of aims, objectives, inputs, outputs, outcomes and rationale are provided. This guide can be downloaded at www.drugmisuse.isdscotland.org/goodpractice/EIU-evaluationg1.pdf

The Rationale: why are monitoring and evaluation important?

Systematic monitoring and evaluation of integrated care is crucial to help assess how and why integrated care is or is not working, and to highlight areas which need improvement. Good evaluation has the potential to improve systems and services. It can help to identify what works, what could be done better and what is ineffective. This should inform decision making about service provision in an area, and therefore help to ensure that clients are receiving the best possible treatment, care and support.

How can integrated care be monitored and evaluated?

Integrated care is likely to be monitored and evaluated at two main levels: service level and strategic level.

Service level - Monitoring will generally focus on the inputs, outputs and outcomes of service activity. Evaluations will generally focus on whether the service is achieving its stated aims and objectives, i.e. that the service is providing the support and care expected and agreed as part of the integrated care process.

Strategic level - Monitoring activities will generally record the inputs, outputs and outcomes of strategic activities. Evaluations will generally focus on whether the necessary partnerships and joint working arrangements are in place to achieve the aims and objectives of integrated care i.e. providing services and maximising their co-ordination to provide person-centred treatment, care and support.

Evaluations at both the service level and strategic level will only be worthwhile if the aims and objectives of specific services and integrated care as a whole are clearly set out and 'SMART'. The 'SMART' approach works by helping to focus on exactly what has to be achieved, who will be involved, how it will happen and when it has to be done:

1. Specific

2. Measurable

3. Achievable

4. Realistic

5. Timebound

It will also be crucial that the aims and objectives are based on an underlying rationale or logic. Setting out the underlying assumptions and logic about why a service or intervention will have the desired impact will help focus the aims and objectives. These assumptions are often based upon existing evidence or previous experience with the target group.

In the case of integrated care, one of the underlying assumptions is that by improving co-ordination and promoting joint working, the pathway of care for the individual drug user will be more seamless and based on their individual needs. There is experience, and some limited evidence, to support this hypothesis. For further information on setting aims and objectives, formulating evaluation questions and planning an evaluation see EIU Evaluation Guide www.drugmisuse.isdscotland.org/goodpractice/EIU-evaluationg2.pdf

Service level evaluation

Service level evaluations will generally focus on whether the aims and objectives of a service have been achieved. It will be important that each service within an integrated care approach is meeting its own aims and objectives. These aims will, of course, vary widely depending upon the type of service provided. However, common questions for an evaluation to address include:

• Is the service successful in attracting and retaining their target client group?

• Does the service 'work' and who for?

• Why and how does the service 'work'?

• What do clients and their families think of the service?

• How can the service be improved?

• Is the service cost effective?

The methods employed to undertake the evaluation will depend upon the question(s) posed. There are a number of key considerations to resolve before a final method is agreed. For example:

• who to include in the evaluation

• when and how often data will be collected

• whether comparison with another group is appropriate

• which inputs, outputs and outcomes will be included

The choice of design will also depend on what stage of development a service is at. For example, if the service is being piloted, it may be important to focus on the process of service delivery (which staff do which jobs, where and how often clients are seen, how long clients stay in contact). However, if the service is fully developed it may be important to focus on the outcomes / impact of the service (were the desired outcomes achieved?).

Often service evaluations can make good use of existing information by conducting some systematic analysis of data. This commonly includes monitoring data, assessment data (see EIU evaluation guide 7), and management information. In this way, evaluation can be built into day to day practice.

However, there is sometimes a need to collect additional information using both quantitative and qualitative research methods. For example, it may be useful to follow up clients who have left the service, or to ask family members for their views and experiences of engaging with the service. Further information about the range of methods that can be used to evaluate services and interventions is set out in EIU Evaluation Guide 3, including a description of three core evaluation designs. This guide can be downloaded at www.drugmisuse.isdscotland.org/goodpractice/EIU-evaluationg3.pdf

Strategic level evaluation

Periodically, it will also be essential to examine more broadly how the range of services in an area are meeting the needs of drug users. The overall aim of integrated care is to combine and co-ordinate all the services required to meet the assessed needs of the individual. Integrated care promotes and supports collaborative working between agencies at each stage in the progress of the individual through treatment and care, rehabilitation and reintegration into the community. Evaluations of the integrated care approach will commonly address questions including:

• Do existing services that contribute to the integrated care approach meet the wide ranging needs of drug users in the area?

• Do existing services that contribute to the integrated care approach have the capacity to deal with the needs of drug users in the area?

• Are there appropriate and effective joint assessment arrangements (e.g. single shared assessment)?

• Are there appropriate and effective information sharing protocols and referral procedures?

• Are there appropriate and effective partnership working arrangements?

• Are there appropriate and effective joint resourcing arrangements and joint management arrangements?

• Are there barriers to providing an integrated approach and / or opportunities to improve collaboration between key agencies?

• Do current monitoring and evaluation arrangements provide sufficient information for on-going review and improvement?

An important precursor to evaluating the capacity and accessibility of services in an area to meet local need may be to conduct a systematic needs assessment. This is discussed in Chapter 3 on Accessibility and an associated Annex 3B. These data help provide a baseline against which to assess capacity, accessibility and appropriateness of services. As already mentioned, the EIU will be publishing a Guide to Needs Assessment in Scotland early in 2003.

A range of methods is likely to be required to address the questions raised in an evaluation of an integrated care approach. These may include interviews and focus groups with service users and their families, representatives of the range of both specialist and generic service providers in the area and service commissioners to elicit their impressions, views and experiences.

However, if a core principle of integrated care is to provide person-centred care it will also be essential to look at whether integrated care is delivering improved outcomes for service users and their families. It should, for example, be possible to identify how a representative sample of individuals has moved through an integrated care approach. Ideally it would be useful to compare their integrated care experience with previous experiences, or to compare an integrated care approach with a 'non-integrated' care approach. As with service evaluations, there may also be existing management information that can contribute in such an evaluation.

It may also be desirable to include an economic evaluation component into strategic evaluations of integrated care. As with all public services, only limited resources are available for tackling substance misuse. Resources need to be used wisely so that they have the maximum impact in helping individuals and the wider community. Economic evaluation allows the costs (the resources that they use) of different services and interventions to be considered alongside the outcomes or 'benefits' achieved. This can help to inform comparisons between different approaches and can allow resources to be used more effectively. For further information on economic evaluation, please see EIU Evaluation Guide 6 at www.drugmisuse.isdscotland.org/goodpractice/EIU-%20evaluationg6.pdf

Ensuring a feasible, inclusive and worthwhile evaluation

There are some important considerations when planning either a service level or a strategic level evaluation. It is essential to be clear from the outset why evaluations are being conducted, whether the evaluation is feasible and how the results of the evaluation will be used. Each evaluation will be unique, but there are some common principles to apply to ensure a feasible, inclusive and worthwhile evaluation. It is important to be clear about:

Considering these questions early in the evaluation process can make the whole process much easier. In particular, it will be important to decide whether an external evaluator should be commissioned to undertake such exercises. For strategic evaluations involving a number of different agencies it may be particularly important to commission the evaluation externally.

Implementing the evaluation

There are some special considerations when implementing evaluations. These include recruiting and training monitoring and evaluation staff (if not commissioned externally), obtaining consent from those participating, complying with the Data Protection Act and seeking ethical approval (if required). If service users and their families are asked to participate in an evaluation, it is also good practice to provide them with travel expenses, childcare expenses and a small incentive for their participation. Further information about implementing an evaluation is set out in EIU Evaluation Guide 4. This can be downloaded at www.drugmisuse.isdscotland.org/goodpractice/EIU-evaluationg4.pdf

Reporting and dissemination

Often evaluations are conducted because they are a condition of funding. In these cases, evaluation is often perceived as a burden. However, this is a missed opportunity for learning and development. Such learning will only happen if the results of the evaluation and effectively disseminated and discussed.

It is, of course, useful to set out the key findings from evaluations in written reports or feedback sheets appropriate to the audience and circulated widely to all stakeholders, including service users. However, there are other (more innovative) ways in which the results of evaluations can be disseminated and used. For example:

• using the results as the basis for seminars and workshops with service staff

• using the results as a basis for discussion with managers and commissioners

• compiling the main results into 'do's' and 'don'ts'

• preparing 'recipes for success' and building them into service procedures

• preparing case studies to bring the evaluation to life

Developing an evaluation culture

It will be important that all stakeholders involved in the evaluation process understand the purposes of evaluation and are committed to making it work. This can be a long process, particularly when evaluation is often perceived as something that is undertaken to apportion blame, rather than to help improve services and interventions. Developing an 'evaluation culture' that values the process and learns from its results will not happen immediately, but over time will ensure that evaluations are even better in the future. Often this process needs a 'facilitator' or 'champion' who will ensure that evaluation activities are developed in an area and that training is provided. As part of our activities to promote evaluation in the drugs field, the EIU have produced a series of evaluation guides and supported evaluation workshops in most Drug Action Team areas in Scotland. The guides produced to date are listed at the end of this section. If you would like a copy of the materials used as a basis for the evaluation workshops, please email us at eiu@scotland.gsi.gov.uk .

EFFECTIVE INTERVENTIONS UNIT EVALUATION GUIDES

For more information on evaluation in the drugs field, please see the EIU Evaluation Guides. The first 5 guide the reader through the evaluation process. The second 5 guides focus on specific aspects of evaluation and evaluating specific types of projects.

GUIDE 1 Definitions and common concepts www.drugmisuse.isdscotland.org/goodpractice/EIU-evaluationg1.pdf
GUIDE 2 Planning an evaluation www.drugmisuse.isdscotland.org/goodpractice/EIU-evaluationg2.pdf
GUIDE 3 Designing an evaluation www.drugmisuse.isdscotland.org/goodpractice/EIU-evaluationg3.pdf
GUIDE 4 Implementing an evaluation www.drugmisuse.isdscotland.org/goodpractice/EIU-evaluationg4.pdf
GUIDE 5 Reporting and dissemination www.drugmisuse.isdscotland.org/goodpractice/EIU-evaluationg5.pdf
GUIDE 6 Designing and economic evaluation www.drugmisuse.isdscotland.org/goodpractice/EIU-%20evaluationg6.pdf
GUIDE 7 Using assessment data for evaluation www.drugmisuse.isdscotland.org/goodpractice/EIU-evaluationg7b.pdf
GUIDE 8 Evaluating outreach services www.drugmisuse.isdscotland.org/goodpractice/EIU-evaluationg8.pdf
GUIDE 9 Evaluating employability programmes www.drugmisuse.isdscotland.org/goodpractice/EIU-evaluationg9.pdf
GUIDE 10 Evaluating community engagement www.drugmisuse.isdscotland.org/goodpractice/EIU-evaluationg10.pdf

Or you can request copies of these at EIU@scotland.gsi.gov.uk or on 0131 244 5117.

Further resources

The resources below are useful if you are planning undertake some monitoring and evaluation activities. Other resources can be found in the EIU evaluation guides.

EMCDDA evaluation instruments bank www.emcdda.org/eib/databases-eib.shtml
WHO evaluation workbooks www.who.int/substance-abuse/pubs-psychoactive-drugs.htm
W.K Kellogg Foundation Evaluation Handbook www.wkkf.org/documents/WKKF/EvaluationHandbook

« Previous | Contents | Next »