The Scottish Government

« Previous | Contents | Next »

Listen

4. Conclusions and recommendations

It is widely acknowledged (Macintyre et al., 2001; Wanless 2004) that evidence for effective interventions to improve health, and in particular to reduce health inequalities, is scarce. It is crucial that innovative interventions are accompanied by good quality evaluation (Macintyre, 2003). We start (4.1) by making some general recommendations for future commissioning and evaluation of complex community interventions such as HaHP, SW and HR, then (4.2) make some more specific recommendations for evaluation of phase 2 of the present demonstration projects, and finally (4.3) make some general points based on what we have gleaned from documents and interviews about the experience of the evaluation so far.

4.1 General recommendations

4.1.1 Commissioning projects

Alternative methods of commissioning should be considered, that place a greater weight on the existing evidence base, and take account of questions of 'evaluability'. For example, more resources should be devoted to the formal appraisal of options prior to commissioning. As well as informing project commissioning, this should also provide a clearer picture of expected impacts that can be used to inform the design of evaluations.

Future bids for DPs should be assessed by at least two independent expert reviewers who are not part of the commissioning group, and who can be asked specifically to comment on the strength of the evidence base.

A clear distinction should be drawn between phases of project development and implementation, and time allowed in the project's lifespan for the interventions to develop and settle down, before they are regarded as mature interventions amenable to a formal process and outcome evaluation. It is difficult however to let interventions mature when they are only funded on a short-term basis, in this case three years.

4.1.2 The evidence base

Allowing for the general weakness of the evidence base, the evidence for each of the three DPs was less convincing than presented in the project bids, and in one case was virtually ignored. Though some problems, such as transferability, were acknowledged, much better use should be made of existing evidence. We recommend:

• That the evidence base for the proposed interventions be made explicit at the commissioning stage;
• That evidence about the effectiveness of the proposed intervention be distinguished from background information about prevalence of risk factors, morbidity and mortality rates, etc;
• That the evidence base for the intervention be kept up to date during implementation, with particular attention given to any emerging evidence, both by the evaluators and by the projects;
• That consideration be given to the question of the transferability of the evidence base to a Scottish context, with particular attention to the extent to which evidence of intervention effects might be context dependent and therefore inapplicable - or not directly applicable - to the target population;
• That the differences between the design of the interventions represented in the evidence base and the design for the proposed intervention be clearly identified and justified. The key is explicit clarity over what is evidence-based, what is new and what is an application to a new context.

4.1.3 Evaluation models

While there are advantages and disadvantages with both traditional and newer approaches we recommend:

• That the 'theory of change' be employed in the formative stages of future DPs where appropriate, not just in reactive mode but in shaping DPs and formulating a design that is 'evaluable'. This implies much earlier involvement of the evaluation team than was the case in the three current projects and a greater influence on their content and direction (e.g. recommending dropping particular components);
• That once objectives have been clarified and agreed, and the project is 'mature', a more traditional model of evaluation is appropriate, combining quantitative (e.g. quasi-experimental) methods to measure outcomes, continually informed by qualitative methods to focus on processes;
• That ways of retaining independence, while enabling researchers to get involved in project planning and development, should be considered, for example project steering groups on which non-grantholders are represented.

4.1.4 Commissioning evaluations

While the success or otherwise of the current evaluation studies cannot be adequately assessed until they are complete, the evaluators have clearly had to tackle a range of difficulties, some of which could and should have been anticipated at the commissioning stage. For future commissioning exercises we recommend that:

• The commissioning process should incorporate a strong element of independent peer review, as well as assessment by an internal tender panel;
• Planning for evaluation should begin at the same time as planning for the DP itself, so that evaluation considerations can be fed into the design and commissioning of the DP;
• Measures should be taken to ensure an adequate number of bidders. For complex or large scale evaluations, this may involve approaching potential bidders from throughout the UK as well as within Scotland, and bidders should be encouraged to form consortia to ensure adequate capacity, range of skills and expertise, etc;
• A clear distinction should be drawn between formative and outcome evaluation (see 4.1.3 above);
• Formal mechanisms for involving project stakeholders, including the evaluation such as involvement in project steering committees, should be considered;
• Project steering committees should also include independent scientific experts (i.e. non-grantholders);
• To keep evaluation tasks manageable, it is worth distinguishing questions about overall success and failure, which are difficult to answer precisely and involve a strong element of judgement, from questions about specific changes or impacts;
• Final reports should also be independently peer-reviewed, ideally by the same reviewers as the evaluation proposals.

4.1.5 Internal and external evaluation

We recommend that the respective roles of internal and external evaluation and monitoring should be clearly defined and agreed and a joint plan produced at the outset. This would help avoid the gaps and overlaps and would help critical interdependencies (e.g. data sharing) to be identified, and resource requirements incorporated in project bids. The interactive nature of ToC (see above) should help to clarify roles and responsibilities in relation to evaluation. This will reduce the need of DPs themselves to engage in internal evaluation which may be of uneven quality and as a consequence of poor communication produce conflicting evidence. We suggest the following division of labour:

Internal role - monitoring

The focus of the internal evaluation function should be on:

• Setting up internal management information systems to enable the monitoring of project inputs (e.g. budgets & staff), outputs and participation/uptake/use;
• Assessing reach - surveying the project's target population to assess awareness, understanding, exposure and uptake/use of programme services/activities;
• Developmental research - research with users, non-users and local agencies/professionals to assess how the project is perceived and how marketing, communications and services/activities can be improved.

External role - evaluation

The focus of the external evaluation function should be on:

• Evaluating programme results, impacts and intermediate outcomes: what changes have resulted as a consequence of programme activities/services?
• Understanding how programme implementation has contributed to achieving the outcomes intended (process evaluation).

4.1.6 Capacity building

The Task Group's recommendations above suggest that there is a need to develop skills and capacity in a number of key areas if we are to realise improvements in the future. The main areas that need to be addressed are:

1. At national level, there is a need to develop a more sophisticated approach to commissioning policy-related initiatives from local partnerships that would enable a developmental stage to be built in. During this stage, the most promising initial proposals can be worked up into full detailed bids in collaboration with funders, local partners, relevant academic experts and evaluators. At this early stage of intervention development, the evidence base for the initiative could be considered and an evidence 'template' drawn up, together with the programme theory (or ToC) and plans for evaluation.
2. At national and local level, there is also a need to strengthen skills in developing health improvement plans that are informed by the existing evidence, and which are evaluable and contribute to a learning agenda. This may require a combination of written guidance, training and ongoing advice and support. The development of a new inter-sectoral evaluation service for Scotland may be able to contribute towards this development agenda.

If the commissioning process and the evidence-basis and evaluability of projects were thus improved, this may also help to address the evaluation capacity problems experienced in relation to the academic research community. The reluctance of academic researchers to engage in the evaluation of complex community initiatives is largely related to their lack of involvement at an early stage in the intervention development process, the poor evidence basis of the initiatives, and their low evaluability.

4.2 Recommendations for Phase 2

The external evaluations of the 3 projects should not continue in their current format. Specific recommendations for the three projects are as follows:

4.2.1 Healthy Respect

There could be some justification for continuing evaluation of the schools element of HR. While the evidence from the SHARE trial to date does not suggest an effect on behaviour, further data on longer term outcomes (termination of pregnancy) is expected soon. In Phase 2, HR plans to introduce SHARE as part of its overall programme of supporting education and services in schools across a whole authority. This could be rolled out in 2 stages over 2 school years, with half the schools implementing SHARE in year 1, the other half in year 2, thus enabling the latter to serve as control schools in year 1. Because all schools in the local authority will be involved, this would avoid the problems of selection experienced in the evaluation to date, especially if schools were randomised to early and late intervention.

4.2.2 Starting Well

There is no justification for continuing the external evaluation of either SW on anything like the present scale, a view shared by the external evaluation team. The intervention is still changing so rapidly that it is impossible to know what is being evaluated. There may be a case for continuing to follow up the SW cohorts, but this can only be properly assessed once response to the 18 month follow-up is known.

4.2.3 Have a Heart Paisley

Likewise, continued large-scale external evaluation of HaHP is unjustified, given problems with non-response and the consequent risk of bias in the baseline surveys. While there seems little point continuing with the original attempt to measure population level outcomes, a focus on a small number of agreed priority areas, where sustainable approaches have or are being developed may be useful.

4.2.4 Internal evaluation and monitoring

This should continue, as part of good project management, but Phase 2 bids should include costed proposals, specifying questions, data sources and deliverables.

4.3 General conclusions

Although the Task Group's remit was evaluation, some more general issues emerged in our discussions with the evaluation and DP teams which are worth briefly mentioning. We regard these as potentially important for future projects of this kind, both to ensure that scarce skilled staff are used appropriately and to provide good value for the substantial sums of public funds invested.

The concept of a 'demonstration project' is obscure and problematic and despite attempts to clarify definitions, it still remains a fundamental issue and is likely to be a matter of continuing debate.

The commissioning process should be organised in such a way that the evidence base for the proposals (and any significant gaps), is clear and explicit before contracts or grants are awarded. Where evidence is weak, a formative stage should be explicitly built into the project's timetable, and evaluation commissioned accordingly.

The timescales for the DPs were unrealistic and uncoordinated, leading to real problems for both the project teams and the evaluators and ourselves. There seems to have been very little consideration until late in the day of an exit strategy, incorporating time to review and assimilate the results of the evaluations. It is vital that this is considered early and more thoroughly in Phase 2. Likewise, a realistic, fully coordinated timetable for the implementation and funding of any future DPs and their evaluations is essential, to ensure that the appropriate evidence is available for informing decisions about the projects and wider policy within Scotland.

This report is not conclusive, since the evaluations themselves are still in progress, but we hope it will trigger further constructive debate. Our final recommendation is that a series of workshops are held, for projects, external teams, learning networks, Co-ordinating Group and Evaluation Task Group members, and HISD colleagues to discuss further the issues we have identified.

« Previous | Contents | Next »