The Scottish Government

« Previous | Contents | Next »

Listen

External Evaluation of Healthy Respect:
A NATIONAL HEALTH DEMONSTRATION PROJECT: FINAL SUMMARY REPORT

III. Implementation and process of component projects

G Implementation, innovation and evidence of best practice

Details of how the diverse component projects of Healthy Respect were delivered on the ground are given in this process evaluation. From close study of the way they operated (November 2000 to May 2004), some key lessons are highlighted for the demonstration project and others working in the field of health improvement and sexual health work with young people.

There was wide variety amongst the 19 component projects included in Healthy Respect. Some were existing or re-launched services or interventions; others were entirely new. Some had a clinical focus; others were community-based, or based in local authority or health board structures. Some were focused around small and discrete tasks; others took a remit across the whole demonstration project.

19 component projects ofHealthy Respect with the ten selected case studies* (in bold)with the ten selected case studies* (in bold)

The diversity of the component projects resulted in the adoption of a case study approach. A final purposive sample of 10 selected component projects for case study (bold above) was undertaken, ensuring that it represented balance across the intervention and took account of the differences in the orientation and capacity of the component projects.

• large-scale, resource_intensive projects vs smaller-scale low-cost interventions
• interventions nested within statutory settings (health services, local authority) vs voluntary settings
• interventions in rural settings vs urban settings
• interventions focusing on vulnerable or 'hard to reach' groups vs those focused on mainstream provision
• interventions which built on a base of existing work in an organisation vs those with a 'standing start'
• interventions which already possessed links with sophisticated networks and partnerships vs others with fewer connections.

Data collection included all project documents, observation at meetings, quarterly project audit returns, project reviews, interviews with project leaders, project workers, user and non-user groups, stakeholders and linked organisations. In the analysis of the work of Healthy Respect's component projects , success was judged around specified criteria from the literature used to define best practice, and also used the component projects' and demonstration project's own aims and objectives. Thus, projects showing best practice might demonstrate:

• training or capacity building activities
• outreach or inclusion activities to draw in marginalised or hard-to-reach fractions of the target group
• work with other agencies to add value to their own effort and to enable the provision of more seamless service for the client group
• encouraging a range of levels and styles of participation by the client group (here characterised as young people)
• sustainability beyond project funding
• developing procedures for monitoring and reflection, allowing them to check for client satisfaction and quality of service provision
• dissemination of their activities

Evidence was gathered across the whole of Healthy Respect, though more intensively analysed for case studies. Case study outlines are included in the final technical report (Part One), and full reports for each of the ten case studies in Part Two. The case studies present the evidence that underpins the following summarised findings of the process evaluation.

Training/capacity building

• Training offers only an indirect way of making impact on the headline targets for sexual health improvement but considerable effort was put into this type of activity. Seven of the original 12 components (and 7 of our 10 case studies) opted to develop this work
• Most training was with professionals who in turn worked with young people. Only one project (Project I _ Developing and supporting the role of parents) trained parents. None offered to train young people directly.
• Training was often primarily a form of awareness-raising (e.g. projects based at LGBT Youth, at HOT and at Edinburgh Rape Crisis). Much training was undertaken with small groups, involved contact with different agencies and aimed to convince them of the need for training. Although in some cases, agencies were actually 'preaching to the converted' (self-selecting groups with a pre-existing interest in LGBT issues, young men's issues or combating violence against women.)
• Some training was offered for those working with very challenging young people with behavioural problems or additional support needs (Projects B(a) on Looked after and accommodated young people, B(b) on Getting the message across at Caledonia Youth, and J on Young men's sexual health).
• Though training from many components was well received, there is little or no evidence for it cascading or rippling out. Fewer one-off sessions and more sustained work would have allowed trainees to develop deeper confidence and stronger skills. Evaluations of training were often superficial records of how well people had enjoyed the sessions. There is only one instance in the case studies where longer-term impact was monitored.
• SHARE training was undertaken on a multi-disciplinary basis, but did not lead to a partnership approach in the classroom. Other staff did contribute to school SHARE delivery, but always on an unequal footing. SHARE training was well received but many felt the need for continued support after training. We know little about the impact of SHARE-training on teachers' practice.
• The capacity of staff to deal with young people's sexual health issues may have been increased via these interventions, but the wide geographical coverage of the demonstration project will dilute observable impact. Capacity building has to be an ongoing process. The rate of staff turnover means that training has to be mainstreamed and constant in order to affect working cultures and practices.

Outreach or inclusion activities

• Projects working routinely with vulnerable or difficult-to-reach groups were Project B(a) ( Looked after and accommodated young people), Project B(b) ( Getting the message across at Caledonia Youth) and Project F ( Lesbian, Gay, Bisexual and Transgender (LGBT) work). There were also projects which aimed to work directly with youth populations which did not make good use of traditional services (e.g. Project J on Young men's sexual health, Project C on Chlamydia testing).
• Mainstream services were taken out to new areas or redesigned specifically for young people in project G(d) ( SHARE Inreach/Outreach work).
• The development of the drop-in services throughout Lothian will be one of the most useful legacies of the demonstration project. They clearly need time to develop a local reputation amongst young people for confidentiality and reliability, and will need to prove their effectiveness and worth to other stakeholders. Their success will depend on continuing articulation between the educational delivery and dedicated service delivery. In addition, emphasis at managerial level on continuing the development of partnership work is vital.
• Reluctant or vulnerable groups require sustained long-term intervention rather than hit-and-run services. Good work was done in this respect by Projects B(a), B(b) and J.
• The demonstration project had a universal approach, not one particularly targeted at vulnerable groups. Its impact on the latter will therefore be more limited. Ethnic minority issues were largely ignored across the demonstration project. There is no evidence of work with a community development focus that challenges the conditions within which people make poor health choices. Most projects embraced traditional health promotion messages of individual choice and self improvement.

Interagency/partnership working

• On the ground, interagency working was well developed within the boundaries of the demonstration project, and especially amongst the projects based in voluntary organisations.
• Where agencies did work together there was evidence of real benefit for young people e.g. Projects B(a) ( Looked after and accommodated young people), Project B(b) ( Getting the message across at Caledonia Youth), Project J ( Young men's sexual health).
• Not all partners felt or were treated as equals. Larger agencies appeared more powerful whilst the status of others rarely shifted from seeming peripheral contributors.
• The alliances between health and education were weak, undermining progress on unresolved contentious issues such as child protection or protocols for dispensing contraception. This probably resulted from the lack of partners' agreements, which should have been made at the start
• There is only scanty evidence of partnership with local people. The SHARE service delivery work in Midlothian would seem to be an exception, offering valuable lessons for use elsewhere. The strong branding of Healthy Respect and its location across all the Lothians was occasionally seen as undermining, rather than building on existing partnerships and ground-level developments.

Levels and styles of participation by young people

• Developing youth participation was one of the strongest themes in early project documents but was poorly developed and managed throughout. A decision was made at an early stage to package youth involvement as a cross-cutting component, but management responsibility for the work shifted continuously. There was little evidence of project staff challenging themselves or each other on this issue.
• Most of the work undertaken at demonstration project level and in component projects was consultative or needs assessment work. There is relatively little sign of young people being encouraged to become more involved in project design or management. SHARE service delivery work in Midlothian is an exception. Few of the components really got much further than the first rung up the ladder of participation.
• Some of the staff in component projects (especially in the voluntary organisations) had experience of youth work and community development skills that could have been put to good use in demonstrating different ways of working with young people. None were really resourced to do this and there was no encouragement from the centre to develop these sorts of approaches.
• Much of the designated work in young people's participation was out-sourced to an independent agency and was consequently not integral to Healthy Respect work. It always looked like a 'bolt on' or an afterthought.

Sustainability

• Some components could rightly claim that the sustainable outcome of their component project is a changed awareness of certain issues. They did make a real change in terms of problem or issue definition. Voluntary organisations in particular worked hard both on the ground and by positioning themselves at the heart of policy debates.
• Some components saw sustainability in terms of products. Some exciting and good quality videos, training packages and written materials were produced, but all these have limited shelf life and will soon date.
• Much of the work done in the component projects was experimental (albeit mostly in a rather random and unsystematic way). It is thus quite hard to discern what was being demonstrated; certainly not the feasibility of delivering service in an economic but effective way. As such, the work would not expect to be sustained in its current form, though it opens possibilities for further work where more structured and better audited service delivery experiments could take place
• Where good work did demonstrate realistic possibilities for changing mainstream practice, there is a question mark over whether component projects were given strong enough support from the demonstration project management. Individual projects did not have the leverage or the resources to forge new partnerships and alliances on their own.

Monitoring/evaluation

• The demonstration project started with little idea about how it would monitor or evaluate progress and achievements. Proper internal evaluation procedures were put in place from September 2001.
• The internal evaluator and management team were unfailingly helpful in supplying information to the external evaluation team.
• Though efforts were made to co-ordinate and minimise the demands placed on component projects, it was evident that the time needed to undertake such tasks had not been budgeted into component project resources from the start. Small projects were particularly disadvantaged. This could be seen as a failure, not just of early planning, but also of commissioning in a demonstration project.
• Because it had not been part of the early discipline of the demonstration project, components had rarely thought through how to audit both their activity or the impact of their work, and in many this had still not been addressed at the end. Most evaluations were short-term and did not attempt to assess impact.
• Some rather poor early attempts at needs assessments and surveys are evident across a number of projects. Staff need better central support in developing research instruments and interpreting data in the context of a demonstration project. They would not normally be expected to possess these skills themselves.
• Building profile was one of the goals of Healthy Respect, and management also had to work hard at maintaining alliances between partner organisations and external interests. The need to monitor and give accurate early assessments of what is working and what is not was often in conflict with the need to keep morale high in such a contentious demonstration project.

Dissemination

• Team meetings and partners meetings (as well as publicity leaflets and project newsletters) allowed a good flow of information about the work of the different components to flow around within Healthy Respect and undoubtedly encouraged many of the interagency links that did develop.
• Smaller projects with very little staff time were disadvantaged in this respect by having to eke meeting time out of their hours however; again, this does not seem to have been anticipated at the planning and commissioning stage. A number of projects were not resourced to disseminate and/or remitted this task to the centre.
• Given delayed starts, few component projects had results as such to disseminate much before the end of the demonstration phase, and some felt that they had not really completed their work even as they reached the finishing tape.
• Much of what was reported as dissemination by the components was awareness-raising about issues, or was a way of publicising projects to get access to settings where component staff wanted work to take place.
• Overall, those working on Healthy Respect have made presentations at a number of conferences, taken part in planning and seminars with national agencies like NHS Health Scotland and so on. There are already some published, peer-reviewed papers emerging from the projects. TV and radio coverage has been extensive within the local area in particular.
• All this has undoubtedly contributed to raising the volume of discussion on young people's sexual health issues. Inevitably, it has also meant that the project drew fire from those antagonistic to certain ways of working.
• Any new phase of the work will have to tackle this issue head on by more proactive development of partnership working to build the strategic alliances and take the majority of the population along with changes in service delivery. To do this, Healthy Respect needs to learn to listen as well as talk.
• Sharing learning across the Lothian area will be important in any subsequent phase. There are more lessons to be learned about relative effectiveness and impact across a whole range of issues from the mundane (e.g. issues of siting and presentation, timing of clinics) to the more fundamental issues about service philosophy (e.g. should sexual health advice/services be embedded in much more holistic health services for young people?)
• The rapid turnover of short contract staff may mean that much of the learning is lost if it cannot be captured quickly by the associated Learning Network.

« Previous | Contents | Next »