The Scottish Government

« Previous | Contents | Next »

Listen

Introduction

What is the background to the study?

DTZ was commissioned by the then Scottish Executive ¹ in 2005 to undertake a two-year evaluation of Intensive Support and Monitoring Services ( ISMS).

ISMS was introduced by the Antisocial Behaviour, etc. Scotland Act 2004 and the Intensive Support and Monitoring (Scotland) Regulations 2005, and came into force in April 2005. It is currently in the first phase and has been rolled out in seven local authorities (Dundee, East Dunbartonshire, Edinburgh, Glasgow, Highland, Moray and West Dunbartonshire).

What is ISMS?

ISMS is a new type of disposal within the Children's Hearings System that involves issuing a young person with an Movement Restriction Condition ( MRC), which by law must be monitored via an electronic tag, for a set period of time. This must be accompanied by Intensive Support ( IS) during their assessment for ISMS, while on the MRC, and in a post- MRC phase for the same length of time that the young person was on the MRC. Before imposing such a restriction, a Children's Hearing must be satisfied that Section 70 (10) conditions of the Children's Scotland Act 1995 are met. The Section 70 (10) conditions are:

(a) that the child, having previously absconded, is likely to abscond and, if s/he absconds, it is likely that his physical, mental or moral welfare will be at risk; and

(b) that the child is likely to injure themselves or some other person.

What were the aims of the study?

This research has evaluated the operation, financial costs and long-term effectiveness of ISMS. The principal aims of the research were:

1. To evaluate the effectiveness of ISMS in reducing the offending behaviour of persistent young offenders, in responding to the needs of young people who are at risk of absconding, and/or who are likely to injure themselves or others, and improving the longer-term outcomes for young people.

2. To assess the cost effectiveness of ISMS in delivering successful outcomes especially for the young people, compared with alternative measures.

What were the methods used?

The study team designed and employed a number of different research instruments to address the requirements of the research.

This study has involved:

• a policy and research review;
• interviews with other researchers working in the field on an ad hoc basis throughout the period of the study;
• a rolling programme of interviews with the phase 1 local authority areas on the implementation and operation of ISMS in their area;
• analysis of local authority monitoring and evaluation data, including cost data;
• two sets of interviews with the main external providers on ISMS;
• a survey of wider stakeholders (including social workers, teachers, addictions workers, and the police);
• a survey of Panel Members within the Children's Hearings System;
• case studies of individual ISMS and Intensive Support ( ISS) cases;
• analysis of other outcome data from a number of other sources; and
• a set of community consultations in each phase 1 local authority.

Findings

How many ISMS cases have there been?

Table 1 provides details on the number of ISMS cases and assessments.

Table 1. Headline ISMS data (as at 30 April 2007, for a two year period)

Source: Local Authorities

Has the number of ISMS cases met expectations?

The estimates were not targets, but rather represented approximate throughput levels that the local authorities believed that they would achieve. Therefore, although there have been a relatively low number of ISMS cases, much lower than initial estimates, this should not necessarily be seen as a failure. It does seem that the number of cases was over-estimated at the outset, but there have also been a larger number of Intensive Support Service ( ISS) cases, i.e.IS without the MRC. Many of these cases were not suitable for ISMS because the IS that they received during the assessment phase rendered the MRC as no longer necessary, as the young people no longer met the Section 70 (10) conditions. There were other problems in moving many cases on to a MRC, such as a lack of suitable accommodation or difficulties in the young person's family relationships. In some areas, the estimates were not achieved because insufficient priority was given to the implementation of the ISMS programme and/or because of ideological concerns about the nature of electronic monitoring, i.e. some professionals had a view that the MRC, including the electronic monitoring, for vulnerable young people was inappropriate and unduly punitive. However, other areas clearly prioritised and approved of the programme.

What does an ISMS programme look like?

What is an ISMS programme?

Under Scottish Government guidance, ISMS involves the young person receiving IS with a MRC, monitored electronically, for a period of time (which can vary). This is preceded by an assessment phase (of around six weeks, where IS is provided) and post- MRCIS of the equivalent period for which the MRC was applied. In each of the local authority areas, the average time on the ISMS package and the MRC varied, at 17-38 weeks for an ISMS package and 9-17 weeks for a MRC. The curfew hours also varied, but were normally from evening to early morning. Serco provide electronic monitoring services in Scotland (taking over from Reliance Monitoring Systems in 2006).

The main external provider of intensive support services was Includem in five of the seven phase 1 local authorities (Dundee, East Dunbartonshire, Edinburgh, Glasgow and West Dunbartonshire). Highland Council uses NCH, while Moray delivers these services through an in-house team. This support is complemented by other services such as education, social work, addiction services (where appropriate) and mental health services (where appropriate).

The support services provided in all areas vary depending on the needs of the young person and their circumstances. At the heart of the model is one-to-one support based on a professional relationship ('Pro-social Role Modelling'). This approach is about supporting and challenging a young person's behaviour through these established relationships. This usually involves providing support in the community, addressing offending behaviour and other problems the young person may have, and providing 24-hour crisis support. It is a mixture of informal befriending, e.g. trips to leisure centres and activities, in order to gain engagement with the young person, before moving on to more structured work around, for example, offending behaviour, substance misuse or relationships. This often involves work with families/carers and key support workers as well as the young person themselves.

ISMS delivery focuses on case management to fit complex packages around the needs of individual young people in a tailor-made way.

The level and type of support varies within and between areas, dependent on the model of delivery and the needs of the young person. Total weekly interventions are in a range of 20-60 hours per week. Total curfew hours varied between 29 and 84 hours per week. There is no clear evidence on what components an ISMS programme needs to have to be 'successful' or what the level of these interventions needs to be in terms of intensity.

Does ISMS work?

In answering this overall question, we considered a number of related questions, which are detailed below. This is not an easy question to answer as we have examined seven different models of delivery, all at different stages. This is exacerbated by the fact that some of the phase 1 local authorities have very few ISMS cases and that the programme has only run for a short period of time. To prove robustly the extent to which ISMS has produced better outcomes, we would need to conduct a randomised control trial ( RCT), which is seen as the gold standard in relation to testing the effectiveness of interventions. Within an RCT there would have been a closely matched control group which would have allowed us to compare the outcomes at the end of the evaluation for both groups. However, for practical and ethical reasons, it was not possible to do this in this context as young people who met the same conditions would have to be deliberately excluded from a programme from which they could benefit.

However, this evaluation has gathered a lot of information on the implementation of ISMS in each area, but outcomes data has not been routinely collected by local authorities and is, at best, patchy. The study has collected case study evidence on outcomes for a relatively small number of young people on ISMS and we have based our findings on this data and additional data that we have collected from SCRA and some of the local authorities. However, outcomes data is not available for all young people who have engaged with ISMS.

Providing young people with intensive support while electronically monitoring them in this way is a unique intervention, and therefore was not set-up with a clear evidence basis. There is some evidence that suggests that elements of the package do work in terms of tackling offending behaviour, especially the IS element ( e.g. Wraparound Milwaukee 2002, Youth Justice Board 2007). The previous literature on the effectiveness of electronic monitoring alone is more mixed, but leans towards limited or no impact on key criteria, such as reduced recidivism ( e.g. Bonta, Wallace-Capretta and Rooney 2000, Renzema and Mayo-Wilson 2005).

Has ISMS had an impact?

Evidence from the case studies and local evaluation and monitoring work indicate that the ISMS and ISS programmes have been effective for a significant number of young people in terms of reduced offending, improved attendance rates on programmes, reduced absconding and reduced substance misuse.

Is ISMS in keeping with 'what works'?

Further work would be beneficial in establishing an evidence base on 'what works' in delivering ISMS and ISS. In order to conduct robust outcome evaluation of interventions, a high degree of 'programme integrity' at the delivery level is required. 'Programme integrity' requires programmes to have clear and consistent aims and objectives which are linked to the methods being used, and to be delivered by appropriately trained, experienced and soundly managed staff. According to Hollin (1995: 196) it 'simply means that the programme is conducted in practice as intended in theory and design'.

At this stage, we cannot assume that ISMS and ISS implementation has been conducted in accordance with all of the principles of programme integrity, given the level of variation between and possibly within local areas in many aspects of delivery - and so this aspect of ISMS and ISS implementation ought to be addressed if outcome evaluation is to be conducted at any future stage.

Does ISMS reduce re-offending?

Overall it is important to bear in mind when considering the impact of this kind of intervention that meta-analysis of research conducted on interventions published between 1964-2002. This analysis found that in general programmes with a similar population to ISMS reported a 9% reduction in recidivism for those who had participated in interventions when compared to a control group (Latimer et al 2003). The research which formed part of this meta-analysis was of 'gold standard' and therefore the results of this analysis can be seen as robust.

Within our research two of the local authorities, Dundee and Glasgow, are systematically collecting data, specifically on ISMS, in relation to the effects on offending behaviour and re-offending. However, we are not able to compare these results to a control group (for the reasons outlined earlier in the paper) and therefore the results detailed below can be seen as showing the reduction only for those receiving the intervention rather than showing a comparison with those who had not received ISMS within the same period.

In Dundee, the ISMS Team has self reported significant falls in offending behaviour before and after ISMS. The reduction in offending of ISMS cases six months pre-assessment and six months post-assessment was 70%. 12 months pre-assessment and 12 months post assessment (for cases where there is information available), the fall was 52%. These large falls were experienced for a number of categories of offences.

In Glasgow, the reduction in the average monthly offending rate prior to and during ISMS was 28%. There was also a 14% fall in the seriousness of offending based on the Gravity Scale developed by the Youth Justice Board and Oxford University.

Case file information, reviewed as part of the case studies, indicated, generally, that young people on ISMS reduced their incidence and seriousness of offending behaviours. In addition, there appeared to be a similar reduction in levels of offending recorded in the files of young people placed on ISS.

Does ISMS reduce indicators of risk of offending?

In Dundee, the reduction in offending behaviour traits is also demonstrated by the reduction in ASSET scores. ASSET is a tool used to measure risk and based on a number of factors, young people are assessed and scored on the level of risk they present. For those on ISMS, the average ASSET score was 29 at assessment, 20 on coming off the MRC and 20 on ISMS exit (after the throughcare phase). This fall of 31% demonstrates that the risk factors that lead to young people re-offending are reduced by ISMS.

In Glasgow, there was, again, a more modest reduction, of 8%, in YLS risk assessment scores (from 28.5 to 26.3). YLS is another type of risk assessment tool.

Attendance rates for those on ISMS in Glasgow were high, at 67% of timetabled programmes. Education attendance improved for ISMS cases in Dundee, with a 66% attendance rate after ISMS compared to a 37% rate before it. Improved educational opportunities are linked by research with reducing the risk of re-offending.

Does ISMS improve outcomes for young people?

All of the programme managers in the phase 1 local authorities stated their support for the ISMS programme and believed that it was effective in improving outcomes for young people, although it does have its limitations, e.g. it is difficult to get favourable outcomes in cases where problems are more entrenched, and many of the young people are coming off the programme are 16 and resources may not be as readily available to help them afterwards.

Is ISMS well regarded?

In our stakeholders' survey, in comparison to the other measures used within the Children's Hearings System, ISMS was viewed relatively favourably in terms of perceived effectiveness. ISMS was viewed as more effective than home supervision (now termed a supervision requirement) by 71% of the survey respondents. ISMS was also perceived favourably when compared with secure accommodation, accommodation other than secure ( e.g. children's home or residential school), and ISS. Compared to all three of these other measures, at least half of the respondents said they perceived ISMS as more/much more effective.

A similar story emerged when we asked stakeholders to compare ISMS with measures available in the adult system, again with at least half of the participants perceiving ISMS as more or much more effective than these other measures.

In our survey of Panel Members, the majority of respondents perceived ISMS as being no less effective than other measures open to them. Two-thirds of respondents perceived ISMS to be at least as effective as secure accommodation. More than half of respondents said that they thought ISMS was more effective than ISS.

Breaches of curfew conditions were commonly reported in case files. However, such breaches are not necessarily failures if, overall, the programme is helping young people to engage with interventions, when they had not previously engaged, and if it leads to a drop in their risk factors.

Who is ISMS being targeted at?

The data from the phase 1 areas suggests that ISMS has predominantly been used for males over 15 years-old. It has also been mainly used for those young people who are a risk to others, i.e. offenders, rather than those who are a risk to themselves or a risk from others, although ISMS is still used for a number of these latter cases. In some areas, it was felt that an MRC may cause vulnerable young people unnecessary and additional distress. If the MRC is seen as the punitive part of ISMS, as it is for some cases, then punishing young people for being at risk when they are committing no offence seems very harsh to many professionals. However, our case study research covered a number of cases where an MRC had been given because young women had been absconding.

Who is receiving ISMS?

Analysis of data from a sample of referrals to the Children's Reporter also showed that previous grounds for referral to a Children's Hearing for those on ISMS or ISS tended to be for offending (92%) or because they were 'beyond the control of any relevant person' (72%).

Information drawn from the case studies presented a relatively consistent pattern of complex needs in ISMS cases. Both the young men and women placed on ISMS, generally, have histories characterised by multiple referrals to the Reporter for care and protection and offending grounds. They were also, generally, found to have been referred earlier in their lives and to have had a history of contact with the Children's Hearings System resulting in Supervision Requirements having been made several years prior to the ISMS. Issues such as substance misuse, truancy and absconding from home appear in the backgrounds of the young men and women.

The case studies did demonstrate differences in the use of ISMS for young women and young men. Concerns around young women, placed on ISMS, centre around placing themselves at risk through absconding, substance misuse and fears about personal safety (which usually translate into fears about sexual exploitation while out in the community). For young men placed on ISMS, offending behaviours appeared predominantly to have influenced the use of the disposal.

Who do ISMS work for and how?

In our survey of Children's Panel Members, ISMS was generally considered as being more appropriate than secure accommodation for 'persistent offenders' and for younger children. Secure accommodation was generally preferred for 'serious offenders' and for cases where there was a risk of self harm or absconding. ²

The case studies indicated that most of the positive outcomes attributed to ISMS were perceived by social workers and carers to be a consequence of the range of interventions and access to resources that were the IS facet of the package, rather than the MRC. Key elements were perceived to be the 'forced' engagement of young people with services and support, and the length and consistency of support. However, the MRC was believed as having a positive contribution in certain types of cases, e.g. in allowing young people to develop a set of credible reasons for not engaging with offending peers. Some of the young people interviewed appeared to reduce risky behaviours because of the potential consequences of not meeting the ISMS conditions.

Based on the interviews (with the young people, their carers and key workers) reviewed in the case studies, the ISMS disposal facilitates access to opportunities to build protective factors in the community, such as engaging with college placements and building relationships with family members and friends. In terms of accessing community resources and maximising social inclusion, the IS packages appear to have increased young people's use of education, employment agencies and health agencies. ISMS, therefore, appears to work by allowing young people access to resources that could act to build protective factors (which, in turn, reduce risk), and to receive structured interventions that also serve to reduce risk.

What are the factors contributing to programme success?

There appear to be a number of major factors that need to be in place in order to implement a successful ISMS programme. Difficulties in achieving all of these requirements have led to delays in implementation of ISMS and a lack of throughput of ISMS cases in certain areas.

The major factors are:

1. The programme must be driven forward from an early stage by a suitably skilled individual who takes responsibility for the organisation and implementation of the key elements that need to be in place to provide a service to meet local need. S/he should be appropriately supported to do this.

2. There needs to be high-level management support for the programme within the local authority.

3. Effective partnership arrangements need to be built early on, especially involving Education, Social Work, external providers and others as appropriate. The different agencies and workers need to be aware of their and other's roles and responsibilities.

4. There needs to be a supply of suitable accommodation for young people in an area.

5. The programme needs to be promoted, especially to the police, the Courts and the Children's Hearings System so that the disposal will actually be considered in cases where it is appropriate. Its profile also needs raised in the wider community so that people are aware of it and what it is intended to do.

6. There needs to be an effective link-up with secure providers to ensure that those young people in secure are assessed for ISMS and that work (such as assessment work) is not being duplicated.

7. The multi-agency case conference approach to screening and assessment seems to be the best way in which to gather all of the opinions of the professional staff involved in a case and discuss and debate the issues surrounding a young person and what they need.

8. There needs to be flexibility in programme construction and delivery so that the needs of different types of young people are met.

In our survey of stakeholders, we asked them what they considered to be the most effective aspects of the ISMS programme. The crisis response mechanism had the highest rating (considered by 84% of respondents to be either effective or very effective). This was closely followed by the external intensive support providers. Our survey of Panel Members produced similar results.

The services delivered by Includem were singled out by the large majority of interviewees in the case studies as being the most positive and effective aspect of the intensive service. Such commendation was also found in many case files and noted in some Children's Hearing reports. There appear to be a number of similarities between case study findings and the wider evidence base on good practice in the supervision of people with complex needs and risk in the community. The Pro-social Modelling approach, contains at its core the key elements of engaging with the client's motivation while simultaneously building on the relationship between the worker and the young person. The quality of the relationships that developed appeared to involve themes of trust, empathy, a non-blaming approach, holding optimistic views, and being able to articulate the young person's and family members' feelings and problems.

What are the barriers to programme success?

There is some confusion over the guidance on who ISMS should be targeted and this has also affected throughput, with disputes between the different agencies as to whether ISMS is suitable for particular cases. This sometimes leads to ISMS not being considered by professionals. The Scottish Government states that ISMS is intended for those who meet Section 70(10) conditions and, therefore, act as a disposal for a relatively small number of young people ³. It appears doubtful whether this guidance has been followed consistently in all areas throughout the evaluation period, especially with the relatively high number of ISS cases in relation to ISMS cases. However, there does seem to be a case for using ISMS (or ISS) as a preventative measure for young people who may meet this criteria in the future.

In some of our case studies, it appears that some intensive support interventions, such as substance misuse services, did not adequately address areas of need that are specifically related to the risk of further offending.

Lack of training for stakeholders is also a barrier in terms of lack of awareness of the programme and understanding of what it is intended to do among key professional groups. In our survey, just over half of the stakeholders received training, meaning that a large proportion of stakeholders had received no ISMS training. It is likely that training has been targeted at those most core to service delivery, however, it appears that some keys groups of professional staff have still to receive training, notably those working in secure accommodation and children's units.

Education is a key element of ISMS service provision as many of the young people on the programme will have been out of mainstream education for some time. Achieving levels of educational attainment and/or qualifications are key to future employability as well as in improving self esteem. Direct employment of teaching staff has been seen to be the most effective way of providing this, otherwise a key need is not addressed and it is more difficult to keep a young person occupied on a programme. Most areas now have the level of education provision that they require, but it has been a challenge to provide it. Other areas are still struggling to provide education at anticipated levels because they have found it difficult to fit education provision around the high needs and risks of some of the young people and pull together a package of 20-25 hours per week for young people who have been excluded from education for some time.

Access to suitable housing and accommodation appears to be a major problem across the phase 1 local authorities and means that young people who meet the Section 70(10) criteria and may be suitable for ISMS do not receive ISMS. This is because there is nowhere suitable to tag them to, or because it is difficult to maintain them on the programme. The case study research highlighted that being subject to curfew conditions also appeared directly to have a negative effect on some people, as it was reported that some residential options were taken away because the young person was being or had been electronically monitored.

What is the main barrier to successful implementation of ISMS?

Relationships between the core services delivering ISMS (including Education, Social Work and IS providers) appear strong, but more work has to be done in integrating other services, notably Housing, Mental Health Services, Employment Services and Addictions Services. In our survey of stakeholders, accommodation services were viewed by 28% of respondents to be either ineffective or very ineffective and a large proportion of respondents (38%) viewed it as neither effective nor ineffective. Employment services, mental health services and addiction services were the other parts of the programme that were considered effective by less than 50% of respondents.

What are the phase 1 areas spending their ISMS money on?

Table 2 outlines the funding that each of the phase 1 local authorities were provided by the then Scottish Executive to deliver ISMS in 2005-06. Funding for 2006-07 was maintained at approximately the same levels (minus the start-up costs).

Table 2. Funding data 2005-06⁴

Source: Local Authorities
* Funding for Edinburgh is still under discussion.
** It appears that this figure is for costs incurred in the start-up year, and it may therefore include running costs

We derived cost data for each area following consultations with each of the local authorities and Scottish Government economists.

Although local authorities have used the funding well, they have not always used it for the purpose for which it was originally intended. Since a proportion of the money allocated was subsequently spent on general capacity building, or providing intensive supervision to a wider group of young people.

On both outcomes and cost data, for most of the local authorities, we found that the information available did not provide sufficiently robust data for evaluation purposes. This is demonstrated by the lack of budget allocated to monitoring and evaluation activity in several of the local authorities' funding proposals (see Table 2). Local authorities need more stringent performance measurement arrangements to be in place to allow future evaluation activity to assess the value for money of these programmes.

In the absence of this information is not possible to provide accurate estimates of unit ISMS costs for two main reasons:

1. It is difficult to separate ISMS expenditure from ISS expenditure. Both programmes are not mutually exclusive and are being run from the same funding stream, but our evaluation focused only on ISMS.

2. ISMS funding was made available to provide resources that the local authorities needed to run the programme in their respective areas. Each of the areas had different needs and requirements, therefore a fair cost comparison is very difficult.

However, there is evidence from some areas that ISMS appears to have generated cost savings against the use of secure accommodation. For instance, Dundee reported a reduction in the number of secure admissions, in the average length of stay and in the occupancy rates within secure accommodation for 2006-07 (the number of secure admissions dropped by 13 cases) and the Dundee ISMS Team believes that ISMS has played a role in achieving this.

Conclusion

Evidence from the case studies and local evaluation and monitoring work indicates that the ISMS and ISS programmes have been effective for a significant number of young people. There is particularly wide support for the IS provision.

There is some evidence that the MRC works for young people whose problem behaviour is related to their peer group and for those who require rapid response to problems. It would also seem to have a positive impact in helping carers to assert rules.

The different models of delivery in each of the phase 1 local authorities has made it very difficult to assess the impact of ISMS as a whole, exacerbated by the small number of cases in certain areas. However, there seem to be a number of conditions for success in implementation, particularly having an individual with the right skills in place quickly to drive the programme forward, building effective partnership arrangements early on, having a supply of suitable accommodation for young people in an area, and flexibility in programme construction and delivery.

« Previous | Contents | Next »