« Previous | Contents | Next »
Listen
Chapter 7: Conclusions and Implications
In this chapter we summarise the main findings of the process and outcome evaluations and discuss their implications for policy and practice. Some of the implications are specific to particular projects or types of project. However, as far as possible, we have endeavoured to identify lessons that are common to the interventions and which may have wider applicability.
Summary of client outcomes
The outcome evaluation suggests that all four projects enjoyed a considerable measure of success in achieving their objectives.
Reiver and Perth Connect Projects
The young people's accounts of their experiences of Reiver and Perth Connect suggest that their participation in them had led to a number of positive changes in their lives. For those attending the Reiver project, reductions in drug and alcohol consumption, reduced involvement in problem activities and behaviours, improved relations with family members and positive changes in their attitudes towards school were all attributed to their participation in the programme. The five clients of Perth Connect all reported substantially reduced use of drugs and alcohol. Several also claimed that relations with other members of their family had improved as a result of this. All of them said that they had altered the company they kept or the amount of time they spent with their former friends and that their involvement in anti social activities had declined. The clients of Perth Connect also claimed enhanced confidence and self-esteem as a result of their participation in the project.
A high proportion of the clients, in these two projects, also referred to the ways in which they believed they had benefited from the emotional and psychological support they received from project workers. Having somebody to talk to in confidence about things that were concerning them was greatly appreciated by many of the young people. This support assumed added significance in a context in which individuals frequently had problems of a fairly profound nature and where they often did not have anyone else in whom they could confide and seek advice.
It could be argued that, in evaluating these agencies' success, only the complete cessation of drug and alcohol use should be regarded as the appropriate outcome measure. However, the projects' aims and achievements need to be assessed in the context of the young people's lives and their starting points in terms of behaviour. All of the young clients with whom the two projects worked had been involved in serious substance misuse at the time of referral. Most were also heavily involved in other problem behaviours. The progress that was made therefore, far from being seen as some sort of failure could, alternatively, be regarded as a substantial success. As we saw from the accounts of the collateral interviewees and the clients themselves, while the young people's consumption of drugs and alcohol may not have been eliminated entirely, the place of these activities in their lives had been reduced to such an extent that both the young people's own lives and those of the people close to them had been transformed. Indeed, given the relatively brief amount of time each project is able to spend with those clients, together with the latter's continuing exposure to other more negative influences, the progress they achieved could be considered fairly remarkable. Finally, it can be argued that a gradual approach which pursues realistic and attainable goals may stand a better chance of success than one which attempts to enforce complete abstinence from drugs and alcohol and, in the process, risks alienating the young people from the project.
East Ayrshire Project
The East Ayrshire Substance Misuse Family Support Project also appeared to have played a very positive role in the lives of the young carers who attended it. According to the clients' accounts of their experiences of this project, the intervention addressed a wide spectrum of needs and did so in ways that led to significant improvements in their lives. The young carers claimed to have derived considerable benefit from the respite from the caring role that their attendance at the project provided. Importantly, nearly all of the young people said that their participation in the project had led to improvements in their home circumstances. The project also gave the young carers the opportunity to socialise with other young people. This was important given the frequent isolation the young carers experienced. An additional benefit was that it brought them into contact with other young carers with similar problems to their own. The project also created opportunities for the young carers to engage in a range of activities they would otherwise not have had a chance to participate in. For some of the young people their involvement with the project had had a positive influence upon their behaviour and the company they kept. Some of them also claimed to have benefited from the emotional and psychological support they received from the project worker and nearly all of them volunteered that their confidence and self-esteem had been enhanced through their attendance at the project. A particular strength of the East Ayrshire project appeared to be the way in which, although essentially a group intervention, it was also able to provide support for the young carers at an individual level. This meant that their emotional needs were addressed more fully than might have been the case in a project which focused exclusively on activities of a group nature.
Aberlour Outreach Project
Based on the accounts provided by clients and collateral interviewees, the Aberlour Edinburgh Outreach project also achieved considerable success. According to the parents whom we interviewed, their participation in the home-based outreach service had 1) improved their parenting skills and capacities and 2) led to a reduction in their children's exposure to drug-related risks. In addition, the parents reported having been helped to stabilise or reduce their drug use and having received valuable emotional and psychological support. These latter benefits - support with their drug use and emotional wellbeing - were important in helping to create and sustain a context that helped to promote improved parenting and create a safer environment for the children.
Client satisfaction with the projects
Few criticisms were expressed in relation to any of the interventions. For the three young people's projects the clients' main concerns related to the appropriateness of the help received and a claimed tendency on the part staff to enquire about matters which the young person would rather have kept private. The same criticisms were the most prominent ones as far as the Aberlour Outreach clients were concerned.
Clients' unhappiness with perceived invasions of privacy suggest that project workers may need to be especially sensitive to this aspect of their work, particularly in view of the high priority which clients give to confidentiality. The apparent disagreement over objectives might simply reflect clients' resistance to the advice and guidance offered by the project. This would not be surprising given that the main objective of three of the projects is to challenge established behaviour and get clients to change what they do in pretty fundamental ways. Alternatively, it may suggest that projects need to make greater efforts to explain and negotiate the service they provide. Unfortunately, our data do not enable us to determine the relative weight of these possibilities.
Lessons from the evaluation
The process and outcome evaluations identified a number of factors that had implications for the success of the projects.
Philosophy and approach
The philosophy and approach adopted by the projects was central to their success. There were two aspects to this. First, a non-directive and empowering approach on the part of the project staff enabled the client to feel valued and encouraged their participation in the project. Second, all four projects adopted and maintained a person-centred and holistic approach. The person-centred approach meant that individual goals were tailored to the needs of the clients. While plans would be developed to meet those goals, workers were prepared to respond to situations as they arose. According to project staff, the person-centred approach was critical to the success of the projects, enabling workers to build the type of trusting relationships necessary to establish underlying need, negotiate personal goals and respond flexibly to help the client achieve these goals.
While the flexible, person-centred approach appears to be central to the projects' success in engaging their client groups, it can also create problems. First, it makes the approach potentially difficult to replicate and impedes the transfer of learning to other comparable areas of activity. Second, it would appear to be responsible for creating tensions with other stakeholder groups. For example, at the delivery level the imprecise nature of an intervention can affect joint working by raising demarcation issues over areas of professional responsibility. At the funding level it can create difficulties in assessing a project's effectiveness.
Confidentiality
The establishment of confidentiality and trust was essential to the development of a productive relationship with clients. This meant that the parameters for informing other agencies about a situation needed to be addressed up front, especially where concerns about child safety might need to be shared with statutory bodies. Apart from child protection concerns, it was felt that clients needed to be confident that information would not be repeated to other professionals or family members without permission. The project workers' experience was that it could take several months for trust and an open relationship to develop with clients. In the Aberlour and East Ayrshire projects in particular, workers needed to recognise and address the fear that children might be removed into care. If these fears were not dealt with effectively they could generate considerable secrecy and suspicion and impede the working relationship.
Personal qualities of project workers
The personal qualities of project workers is an important factor in encouraging clients to engage with a project
For all three of the young people's projects, the ability of staff to establish a positive relationship with the young people was extremely important as far as the success of the projects was concerned. It greatly facilitated clients' engagement with a project if staff were regarded as being friendly and accessible. It was especially important that the project was perceived as being welcoming and that participation in it was seen as being enjoyable. In this regard, our study suggests that the quality of the personal relationships which the young people had with the project workers lay at the heart of the projects' success. It appeared that how the young people felt about the individual workers - whether they liked them, whether they felt relaxed in their company, and whether they trusted them - was a central element in determining whether or not they would continue to participate in the project. If this is true, it implies that the personal qualities of project workers may be at least as important as their professional backgrounds as far as the effective delivery of young peoples' projects is concerned. This conclusion certainly corresponds with the views of the managers of these and similar community projects for young people (McIntosh et al 2004). It also echoes the view expressed by the Effective Interventions Unit that, 'In many instances the need to relate well to young people will be of greater significance than specific knowledge about substance misuse.' ( EIU 2003, p 21)
Retention of experienced front line staff
The retention of experienced front line staff was important in providing continuity for clients. Given the importance of clients' relationships with project workers, continuity of staffing and the retention of experienced staff have major implications for the overall effectiveness of an intervention.
A number of factors were important to retaining frontline staff, including a supportive and rewarding work environment and management style. Whilst clinical supervision was regarded as an important feature of support, making adequate provision for staff to meet on a regular basis and to freely discuss case management issues was similarly important. This latter aspect of support can prove challenging when workers are required to cover different and often wide geographic areas.
Projects for young people should be enjoyable
For the projects working with young people it was important for the latter's engagement with them that, as far as possible, they should be experienced as being fun and enjoyable. The staff in all three of the young peoples' projects were well aware of this and devoted considerable time and energy to ensuring that participation in the project was as appealing as possible.
Client engagement with the Aberlour project
For the Aberlour Outreach clients, it was important that they perceive the service as being relevant to their needs. While this perception might not have been in place when clients made their initial contact with the project, the subsequent development of this recognition helped to promote their constructive engagement with it. Indeed, as we saw, one of the objectives of the project staff was the encouragement of an acceptance by their clients that their parenting was deficient in certain respects, precisely because this was regarded as being a precondition for parents' successful engagement with the intervention.
Target groups
Very few self-referrals were generated by the projects. This meant that identifying appropriate referral routes capable of supplying the right kind of clients in appropriate numbers was critical to the projects' success. All of the projects invested considerable time and energy in exploring routes by which clients, who were often hidden and initially reluctant, could be accessed. Strategies for accessing clients included: identifying key agency staff responsible for making referral decisions; developing supportive relationships with referring agencies; and employing project staff who were familiar with local service networks. The distribution of leaflets and information packs also helped to promote enhanced awareness of the project among potential referral agents. Staff changes in referring agencies meant that keeping potential referrers informed about the project was a continuing process.
Referrals
It is important for projects to work at ensuring that they receive referrals that are appropriate to them. Misunderstandings on the part of referring agencies regarding the work of the project could lead to clients being referred inappropriately. In consequence, projects not only had to continually clarify the work they undertook and the nature of their clientele, they also had to make clear the type of work they did not do and correct any misapprehensions.
Relationship with host agency
The nature of a project's relationship with its host agency was a significant factor in its success. Establishing a sense of ownership of the service by the parent organisation is essential to ensuring effective leadership, management and administrative support. Key to this is ensuring that senior staff from the host agency have an active input into all funding applications. It is also important that provision is made from the outset for those staff to meet with the project on a regular basis.
Close integration with the host organisation was felt to offer considerable benefits such as accommodating management changes and covering staff absences or unfilled posts, as well as providing resources, mutual support and training opportunities. However, too close an integration with the host agency might risk undermining a project's achievement of its goals, for example if project workers are used to cover aspects of the host agency's work.
Funding arrangements
Funding arrangements had significant implications for project planning and sustainability. The projects generally felt that they had a good relationship with the Partnership Drugs Initiative both as a funding body and as a support agency, and welcomed their supportive and flexible approach to project development. However, while projects appreciated the renewal of financial support by the funding body, the reduced rate of this support could cause considerable difficulty. In addition funding uncertainties made planning difficult, particularly where staff had left and it was unclear if posts would continue beyond the first end-point.
Promoting client involvement
The projects attached a high priority to the principle of client involvement in project development issues. As we saw, establishing appropriate mechanisms to actively engage clients in this process proved difficult. Some projects employed feedback forms for specific events and occasionally sought a direct response from clients, for example at closure reviews. Whilst these mechanisms can provide useful feedback on a service, an on-going dialogue is needed if clients are to actively guide project development decisions. The Reiver and Perth Connect projects attempted to establish client advisory groups for this purpose but in both instances these failed to secure sufficient commitment from the young people. The East Ayrshire project felt that its clients appreciated their involvement in planning the programme for each session of group activities and felt that the introduction of a comments box had been valuable. Despite these efforts, however, client involvement is an area where further experimentation and innovation is required.
Assessment and monitoring procedures
The projects' assessment and monitoring procedures could be improved. The person-centred approach and the inherent flexibility of the projects helps to explain the use made of assessment tools by project workers. These were normally used as part of the engagement process to help to identify areas of concern, to generate constructive dialogue and to establish personal goals with the client. However, it was uncommon for standard assessments to be routinely repeated at later stages to measure progress. The project workers' explanation for this was that client goals were often too sensitive to measure objectively. Overall, the non-use of standard assessment tools within some projects would appear to undermine their ability to monitor client progress, assess outcomes and measure the overall performance of the service.
Conclusion
The main question that the present research addresses is whether the projects that were evaluated are worthwhile. On the basis of the evidence we have collected, the answer to that question must be an emphatic 'yes'. The central conclusion of the evaluation is that each of the four programmes achieved a substantial measure of success in achieving their objectives.
In the case of the three young people's projects this success appeared to be accompanied by an apparent vagueness and lack of specification in relation to the nature of the intervention that was undertaken. However, this did not appear to prevent the projects from achieving considerable levels of success. Indeed, there is some evidence that these projects may have been successful in part because of their flexibility rather than in spite of it. What this implies is that interventions with young people may not need to be standardised or tightly specified - or even highly sophisticated - to be successful. What may be required is that they have a clear sense of what they want to achieve and use whatever means or opportunities are available to achieve those ends. This means that a variety of different projects with varied approaches are likely to be effective as far as meeting the needs of young people are concerned. In fact, some of the young people's lives are so barren and misdirected that almost any intervention which provides respite from the negative aspects of their lives and gives them a sensitive but positive steer is likely to be beneficial. Having said that, as we saw, there are certain things that make it more or less likely that projects such as these will succeed.
The findings of this report would appear to be consistent with the views of service providers regarding the requirements for effective initiatives with young people. For example, in a recent guide to the assessment of young people who are involved in problematic substance misuse, based on consultations with service providers, the Scottish Executive's Effective Interventions Unit emphasises the importance of effective engagement with this group as a prerequisite for successful intervention with them ( EIU 2004). The Guide maintains that the young person needs to feel comfortable with the project workers and with the processes involved in an intervention and to be committed to the project's objectives. Without such constructive engagement, the young person will either refuse to attend or, if compelled, will do so in such a way that is unproductive; for example, by being disruptive or not being prepared to enter into a dialogue with the project workers. Barriers to engagement are especially likely when the young person has been referred to a service involuntarily. In these circumstances, the first task of the intervention is to somehow secure their engagement with it. According to the Guide, other barriers to engagement with a service typically include; a lack of awareness on the part of the young person of the possible consequences of their substance use; a distrust of official agencies and their staff; and difficulties in envisaging or committing to a positive course of action to change their behaviour. Successful engagement is promoted when workers are friendly, honest, non-patronising, non-judgemental, able to see things from the young person's point of view, and able to demonstrate that they have the ability to listen to what the young person has to say. The establishment of trust is central to the development of a successful relationship between the worker and the young person.
Intervention with substance misusing parents is clearly something that requires a great deal of skill and sensitivity on the part of the workers involved. However, when these are present, our research suggests that it is an approach that is capable of achieving a considerable measure of success. This study has confirmed that there is much that can be achieved with substance misusing parents short of removing their children into care. There is a great deal in what the Aberlour Outreach parents had to say that is potentially reassuring to those service providers who feel they should be working with drug using parents in this way but may be anxious about how this might be received by them. Our study suggests that substance-misusing parents may be more amenable to this approach than is sometimes assumed. It is well known that, in delivering home based interventions to substance misusing parents, a difficult balance has to be struck between intervening in ways that are effective as far as the child is concerned while, at the same time, not alienating the parent (Taylor and Kroll 2004, Kearney et al. 2000, Banwell et al. 2002). It would appear that, according to the clients in our study, the workers at Aberlour Outreach were able to achieve this with a considerable degree of success.
The present evaluation of the four projects, while wide-ranging, was not able to examine the interventions in terms of the effectiveness of individual aspects of the therapeutic process. It is likely that specific elements of an intervention's methods and approach will have particular and varying effects upon outcomes. An evaluation of projects' performance at this level of specificity would require a research design which involved the comparison of a number of interventions with the same or similar objectives. This is something for the future. However, we do not believe that it is necessary to wait for evaluations of this degree of sophistication for us to proceed to back the sort of intervention represented in the present research. The need now is for the sorts of projects we have described in this report to receive mainstream support so that they can be made more widely available to those who require them.
« Previous | Contents | Next »