On this page:

National Health Demonstration Projects - Evaluation Task Group Review October - December 2003: Final Report

« Previous | Contents | Next »

Listen

Section 2 The Evidence Basis of the Demonstration Projects

Each of the DPs was developed within the context of an evidence base relating both to general issues, notably the effect of deprivation on health, and to the effectiveness of community-based interventions proposed. In this section we consider how strong the evidence - including new evidence emerging during the projects' lifespan - was in relation to each DP and how closely the DPs matched the evidence-base. Any assessment of the effectiveness of DPs should take account of these issues since a failure to demonstrate might reflect the weakness of the initial evidence, rather than a failure of implementation.

2.1 Starting Well ( SW)

2.1.1 The demonstration project

SW aims to demonstrate that child and family health can be improved via an intervention comprising two integrated components: a programme of intensive home-visiting combining parental support and parenting education, and a strengthened network of community-based support services for children and parents. The parenting education programme was not specified in the original bid, but following a review of several possibilities, the 'Positive Parenting Programme' (Triple P), developed by Sanders et al. (2000) in Australia, was chosen by SW and has been delivered both on an individual and group basis. The dual approach, combining family and community perspectives, reflected a belief that by adopting a broader ecological model of health, the impact of the home visiting programme is likely to be enhanced.

The project is located in two disadvantaged communities in Glasgow. To date, 1555 families with a child born from December 2001 to December 2003 have participated (98% of the total number of births). 19% of participants were recruited (and home visited) in the ante-natal period, the remainder (81%) having their first home visit soon after the birth of the child. The intervention consisted of regular visits from health visitors supplemented by lay 'health support workers' up to the child's third birthday. One of the areas (Glasgow South) contains a relatively high proportion (18%) of families of South Asian origin, reflecting one of the demonstration project's aims; that is, to address the needs of black and ethnic minority groups.

2.1.2 The evidence base

The original proposal for SW was accompanied by a literature review of the evidence relating to the role of poverty for child and maternal health and implications for parenting together with an assessment of the evidence on the effectiveness of family interventions in ameliorating these effects. Almost all the available evidence on interventions was derived from randomised controlled trials ( RCTs), involving allocation of families into an intervention (nurse visiting) and control arm (no nurse visiting). Two systematic reviews of those studies were also referred to. While the weight of evidence was, and remains, supportive of the SW rationale, a number of points, some of which were mentioned in the original SW literature review, are worth noting here.

  • Both systematic reviews (MacMillan et al., 1993; Hodnett & Roberts, 1999) commented on a range of methodological problems associated with the evaluation studies, including small sample sizes, inadequate randomisation, lack of blinding in evaluation (Johnson et al., 1993) and problems of attribution (intervention or extraneous factors). Even in the Elmira trial (Olds et al., 1986), methodologically judged the best (MacMillan et al., 1993), the investigators acknowledged that an intervention effect could be attributable to pre-existing differences between groups and/or reporting bias.
  • Both systematic reviews offered only cautious support, each noting variability in results between studies and between outcomes in the same study. Some studies, including one in England (Lealman et al., 1983), demonstrated no intervention effect, or effects not in the expected direction. Many of the reported differences did not reach conventional levels of statistical significance, and some were misinterpreted. For example, the SW literature review reported the results of a meta-analysis as showing 'a significant preventive effect of home visiting on the occurrence of childhood injury' (p.9) while the original review drew attention to the fact that the pooled odds ratio of 6/8 trials reporting lower injury encompassed 1.0 (Hodnett & Roberts, 1999). The error in the SW review was corrected in the later evaluation proposal, the finding being described as a 'non-significant trend' (Gray et al., 2000, p.4).
  • Almost all studies referred to were restricted to first born children, and most focussed on more deprived populations. Support for intervention effects is often found in specific sub-groups of study participants, typically the most vulnerable. Thus, in the Elmira trial, at age two, significant differences between nurse visited and control groups were found only among young (<19) unmarried mothers of low socioeconomic status ( SES) (Olds et al, 1986). In a long-term follow-up (age 15), an intervention effect on delinquency and substance use was similarly restricted to this group (Olds et al., 1997). In another trial of a 'therapeutic programme' to promote maternal competence and parenting, the effect of the intervention was observed only among low IQ mothers, higher IQ mothers doing better with a traditional programme (Barnard et al., 1988). These results generally underpin recommendations to target interventions either at vulnerable families in the community (Gutelius, 1977) or (less usually) to 'communities with high concentrations of low-income married women' (Olds et al., 1997, p.642). All available evidence, however, refers to RCTs of families rather than communities.
  • The best evidence for intervention effects are also associated with particular characteristics of the intervention: namely early (ante-natal) contact, frequent home visiting by nurses in both the ante-natal and post-natal period and long term (up to 5 years) contact (MacMillan et al., 1993). There was also some evidence that programmes with a 'therapeutic' ethos are more effective (Barnard et al., 1988; Gutelius, 1977). In addition, the SW review made much of the view expressed by Olds and others that effectiveness is more likely when proximal (e.g. home support) and distal (e.g. health and community services) influences work synergistically, a view closely related to an ecological model of health promotion.
  • Evidence in relation to Triple P (not provided in the initial SW document) supports the developers' contention that it 'has the strongest empirical support of any intervention with children' (Sanders, 1999, p.72). However, its evaluation has almost entirely been based on RCTs involving children and families with identified problems, such as maternal depression or conduct problems in the child, either receiving or in need of treatment. The extent to which the effects are generalisable to all children, and identifiable at the community level, is much less certain.
  • While there is some evidence that effects diminish with time (Gutelius, 1977; Kendrick et al., 2000), other studies demonstrate mixed results, and some point to effects that persist or emerge later. In the Elmira trial, although no significant differences in HOME scores were found between intervention and control groups at 34 and 46 months (Olds et al., (1986; 1994), mothers in the intervention groups had fewer subsequent pregnancies and at age 15 their children had lower rates of substance use, arrests and convictions (Olds et al., 1997). These findings raise the possibility of 'sleeper' effects (only emerging over time) though none of the RCTs, nor the systematic reviews, mention this.
  • Almost all studies were and remain US-based, raising the question of transferability to other national/cultural contexts. This issue was mentioned in the SW review, which noted that an exact replication in the UK was unlikely because existing health visiting services routinely do some of the work of (intervention) nurses. The implication of this (which was not commented on) is that any intervention effect of enhanced health visiting is likely to be less in the UK than in the US.
  • While several of the intervention studies included minority ethnic groups, none specifically addressed the question of the extent to which programmes were either applicable or effective in those groups. In the SW review, which gave prominence to the needs of Asian families, no studies of this ethnic minority were referred to.

In summary, the evidence on the effectiveness of home intervention programmes which was available to the SW team at the start of the project was less conclusive than often supposed. Many of the studies had methodological flaws, effect sizes were generally small and often confined to the most vulnerable of sub-groups. The best evidence for an intervention effect came from the most intensive, therapeutically oriented programmes of long duration.

Since then, further systematic reviews (Kendrick et al., 2000; Elkan et al., 2000) have highlighted methodological problems and urged caution in extrapolating beyond the US. Nevertheless, the results have been moderately encouraging. For example, in Kendrick et al's (2000) meta-analysis of 12/17 trials, a significant effect of home visiting on the quality of the home environment (as measured by the HOME score) was found, and some significant effects on other dimensions of parenting were also demonstrated in 20/27 studies. The authors noted, however, that their conclusion rested on an assessment of statistical significance rather than (as preferred) effect size. They also noted that since only 4 of 27 studies were from the UK 'caution must be exercised in extrapolating the results to current UK health visiting practice' (p. 443). Very recently, the Health Development Agency ( HDA) (2004), have conducted a review of reviews of home-visiting programmes (including Kendrick et al). Their conclusion is equally cautious, the evidence being inconclusive in relation to several child outcomes (e.g. child abuse), though more positive for others (e.g. childhood injury). There is also some good evidence of positive effects on various dimensions of parenting or mother-child interaction.

2.1.3 Differences in design

There are a number of ways in which SW differed from interventions of proven effectiveness: these differences might be expected to affect its capacity to demonstrate an effect in Scotland. The most important of these are:

  • The setting - Recognised throughout, the most obvious difference resides in the different health care systems in the US and UK, the latter having a system of statutory health visiting which routinely provides some of the features of intensive home visiting. Intervention effects in Glasgow are, therefore, likely to be smaller than those observed in the US trials. There are in addition cultural differences relating to the acceptability of such interventions which might impact on uptake, motivation and compliance, and hence effect.
  • The target population - The SW target population differs from the trials. Rather than randomising families to intervention and control arms, it targets whole communities, so that intervention effects must be detected through comparison with a control community. This reflects the underlying philosophy of the project, particularly the adoption of an ecological model, but there is no direct evidence of such a community-wide effect. Given that several ( US) studies have demonstrated effects only on particularly vulnerable groups, it could be that the comparison of communities will obscure important effects in particular sub-groups. In principle this issue can be addressed by sub-group analysis, but only if the analyses are specified in advance and, where necessary, built into the design.
  • Ethnic minorities - One of the SW areas (Glasgow South) includes a relatively high proportion of families of South-Asian origin, reflecting the prominence accorded to ethnic minorities in the proposal. While this may be desirable for a variety of reasons, there is no evidence supporting the effectiveness of home-visiting in this group, nor is it known how appropriate Triple P is in this context. Given the cultural differences in family structure and parenting, it is also unclear how applicable the key HOME measure is for this group. It is possible, therefore, that ethnic minority families will be a separate case, complicating between community comparisons.
  • The target child - SW targets all children born to mothers in the two intervention areas while almost all the trials focus on first born children. This difference is not commented on in any document and it is neither clear whether this is likely to have any effect, nor what direction this is likely to be in. It is, however, possible that an effect would be more pronounced in first-born than subsequent children.
  • The intervention - SW involves a dual intervention strategy combining home visiting with strengthened community support. In respect of both, the SW design is a departure from that of the interventions providing the evidence base.
  1. Home Visiting - while SW contains many features of the effective interventions (frequent visiting with a specified educational and therapeutic content) it is clear that only a small proportion (19%) of families were visited in the ante-natal period. The best available evidence suggests that contact before the baby's birth is important. Reduced contact in this period might weaken an intervention effect.
  2. Triple P - although regarded as the most evidence-based of all home intervention programmes, most of the evidence about effectiveness refers to specific sub-groups, notably children with conduct problems. As widely acknowledged, it is also not validated in a Scottish population, nor with respect to ethnic minorities such as those of South Asian origins.
  3. Community support - although recognised as likely to enhance the effectiveness of home visiting, there is no direct evidence from RCTs on this issue.

The SW demonstration programme, therefore, had several characteristics which distinguished it from previous effective interventions which meant that there were departures from the evidence-base template. Although there were good reasons for this, it highlights the innovative nature of SW and further underscores the importance of its evaluation. As outlined in the introduction, the DPs were required to both utilise the evidence-base and to be creative. The key, therefore, is explicit clarity over what is evidence-based, what is new and what is an application to a new context.

2.2 Healthy Respect ( HR)

2.2.1 The demonstration project

HR is a multi-strand programme (made up of 12 projects) to promote positive sexual health and relationships among young people. It has two overarching aims (1) to reduce teenage pregnancies and prevent the spread of Sexually Transmitted Infections ( STIs) (2) to communicate learning about 'how and why outcomes emerge' (original bid, p.2) to promote transferability. As originally specified, within these 2 aims, a number of specific objectives were outlined, including (Aim 1) promoting openness in discussion about sexual health, nurturance of self-esteem and responsibility and discouragement of coercive sex, and (Aim 2) the development of an HR website. HR, though predominantly incorporating existing organisations in Lothian, also developed innovative projects. Project B, Looked After and Accommodated Young People ( LAAYP) was the only project that existed before the establishment of HR. Following the original bid (November 2000), which did not identify specific projects, a revised submission (project plan, July 2001) identified 12 projects (Projects A-L) and 19 reporting components. These projects referred to a range of existing and projected initiatives located in diverse settings, including improvements in the uptake of contraception post termination (A) and Chlamydia screening among women presenting for emergency contraception (D) in family planning clinics, provision of postal testing kits for Chlamydia (C) via several outlets, sexual health promotion in Further Education colleges (E) and schools (G), contact and support for specific groups such as LAAYP, (B) and Lesbian, Gay, Bisexual and Transgender Youth (F) and parents (I) and a project focussing on media campaigns (L). In combination, these projects comprise HR.

2.2.2 The evidence base

In the HR proposal, there was no separate review of the evidence on interventions to improve sexual health amongst young people. The only forms of 'evidence' presented were descriptive, with references to official reports highlighting the importance of sexual health education, self-esteem, multi-agency working and the role of poverty and a series of tables/figures detailing the sexual health profile of young people in Scotland, and where possible Lothian. For example, data from the Health Behaviours of Scottish School Children ( HBSC) Scotland survey (Todd et al., 1999) was used to demonstrate the increasing proportion of 15 year-olds with sexual experience from 1990 to 1998, and the proportion in 1998 not using contraception at last intercourse. The implicit assumption throughout the HR proposal was that these Lothian data 'speak for themselves' and were simply further testimony to the accepted wisdom that the sexual health of young people needs to be improved. The following points are noteworthy:

  • Except in the most general terms, the HR bid was characterised by the absence of an explicit evidence-base. The evidence adduced in support of HR was general, patchy and possibly selective. Much is made of the quality of relationships, and particularly the importance of self-esteem, but no supporting references were provided. While criticism of the self-esteem thesis in relation to risk behaviours is recent, some evidence on this issue was available at the time of the proposal (West and Sweeting, 1997).
  • Most importantly, despite the fact that HR is an intervention (albeit multi-faceted), no reference was made to any intervention study in sexual health which might provide a rationale for the effectiveness either of the whole programme or any of its components. For example, there was no mention of the extensive literature on sex education in schools, (e.g. Wellings et al., 1995), nor was there any reference to a number of systematic reviews on (mainly US) sexual health education programmes. Although these systematic reviews identify several studies showing no intervention effects, on balance they provide evidence for the effectiveness of certain programmes in reducing certain outcomes such as underage sex and/or increased condom use (Kirby et al., 1004; Kirby, 1999; NHS Centre for Reviews, 1997). It can only be assumed that this literature was either known to HR but not made explicit, or that it was entirely taken for granted that sexual health education was a good thing regardless of its effectiveness.
  • HR has as one of its goals the promotion of openness in sexual health matters. In support of this, reference was made to the low teenage pregnancy rate in the Netherlands, together with the view that this results from a positive and open sexual health culture. Although this is an admirable goal, there was no explicit acknowledgement of any problem of transferability to the Scottish context.

For all the DPs, the evidence base is constantly developing, and needs to be updated as new studies and reviews are published. In the case of HR, the evidence base was unacknowledged and possibly unknown to the bidders, although at a later stage in the bidding process (some time between January and April 2001), at the request of the Scottish Executive, an outline of the evidence base was prepared which outlined the Healthy Respect component project, its rationale and the evidence base for the intervention. Since its inception important new evidence has emerged which challenges parts of its underlying rationale. The most important new study is the MRCSHARE trial which provided the model for the sexual health intervention in the HR schools project (G) and the model for its evaluation. Disappointingly, the results of the trial (published in 2002, but made available to HR in July 2000), did not show an effect of the intervention on reported age of first intercourse or condom use, two of HR's key targets (Wight et al., 2002). However, consistent with other trials (Mellianby et al., 1995), improvements in knowledge were demonstrated, and it remains possible that differences in behavioural outcomes will be found in later follow-ups and by reference to termination rates. While differences between the HR project and the MRC intervention ( HR has established links with health services and developed multi-disciplinary working) might be expected to increase an intervention effect, the evidence from the MRC study clearly runs counter to the expectations of HR and challenges the rationale. This raises the question of how a DP, or its sponsor, should react when contrary evidence emerges after the project is underway. In this case, the new evidence appears to have made no difference since the results of the SHARE trial were known to HR when the project bids were being developed.

2.2.3 Differences in design

Unlike SW, without an explicit evidence base for HR, there is no effective intervention template against which to assess the design of HR. HR proceeded as if it were an entirely new project, again highlighting the need for evaluation.

2.3 Have a Heart Paisley ( HAHP)

2.3.1 The demonstration project

HaHP is an area-based, multi-component, multi-agency project with a strong community focus that aims to prevent coronary heart disease ( CHD), to promote good health and to reduce health inequalities in Paisley, a town with high levels of CHD and of deprivation.

The original proposal set out a framework for a wide range of health promoting activities targeting the main risk factors for CHD and encompassing both primary and secondary prevention. The NHS, local authorities, voluntary groups and community groups are all partners in the development and implementation of the project. The HAHP framework encompassed both primary and secondary prevention of CHD. Four geographical locality networks provided the framework within which risk factor modifying activities are supported, co-ordinated and targeted. A fifth network co-ordinated the secondary prevention work aimed at people in high risk groups such as those with established CHD. Marketing activities aimed to raise awareness of the HaHP activities and brand and to endorse positive lifestyle messages.

A few defined initiatives were included in the initial proposal, such as the development of CHD registers as a basis for improved secondary prevention. However most of the component parts of the project were not defined in detail. The partner agencies were invited to submit specific sub-projects for funding from HaHP - focussing on the three main risk factors: smoking, diet and physical activity. Training needs were also seen as a priority issue.

2.3.2 The evidence base

Since the 1970s, many community based or population based intervention studies have been carried out for the prevention of CHD (Puska et al., 1995; Ebrahim and Davey Smith, 2001; Farquhar et al., 1990; Luepker et al., 1994). Although the original proposal for HaHP included an appendix on the evidence base, this was mainly devoted to describing the North Karelia study, with some other examples of prevention initiatives. The proposal did not include a comprehensive and critical appraisal of the admittedly large and complex literature on community based CHD prevention studies or on risk factor based interventions. This would have been impossible within the time available to the bidders. However, a good summary was included as part of the successful bid to evaluate HaHP (Hanlon et al., 2000) and later on a detailed review was conducted as part of the evaluation (Blamey, A., 2002).

  • North Karelia Project

This was set up in Finland in the 1970s (Puska et al., 1985; Vartiainen et al., 1994) and is the best known of the many community based CHD prevention initiatives. It has been extensively evaluated, through a series of population surveys and other methods (Puska, 1995). The project built on the support of local community leaders and the general public and was intended to provide a unified and comprehensive approach to CHD prevention. It involved the media, workplaces, primary care, hospitals, schools and local communities. A very wide range of initiatives were implemented, including training programmes, mobilisation of public support through local leaders, formation of housewives groups and targeting of grocery shops and the food industry.

Over the duration of the study, there has been a substantial fall in CHD mortality rates for CHD in North Karelia and in levels of risk factors. However, a similar downward trend is seen throughout all parts of Finland, raising the hotly debated question of how far the mortality decline is attributable to the specific interventions in the programme (Ebrahim and Davey Smith, 2001).

  • Other community based interventions

In the 1980s several other influential community based studies were conducted in the USA, such as the Stanford Heart Disease Prevention Programme, Stanford Five Cities Report, Minnesota Heart Health Programme and Pawtucket Heart Health Programme (Farquhar et al., 1990; Luepker et al., 1994; Carleton et al., 1995). In each case, the differences between the intervention and comparison areas in risk factor modification were small or non-significant and the attributable effect on mortality was also disappointing. For example, a report from the Stanford Five Cities project found a similar decline in disease rates in intervention and control cities, suggesting that influences outwith the programme accounted for the observed changes.

Within the UK, a community based demonstration programme in Wales, aimed at reducing risk factors for CHD, was set up in the 1980s. Heartbeat Wales drew on the experiences of the studies in Finland and the United States. While a reduction in reported smoking prevalence and improvements in dietary choice were observed, there was no net intervention effect for the programme areas over and above the observed change in the control area of NE England (Tudor-Smith et al., 1998). The investigators concluded that 'with hindsight, the difficulties of evaluating such a complex multifaceted intervention were underestimated. Further debate on the most appropriate methods for assessing the effectiveness of community based health promotion programmes is called for.'

  • Multiple risk factor intervention studies

In parallel with these community based studies, there have been several large RCTs of multiple risk factor interventions in reducing CHD mortality. Ebrahim and Davey Smith reported a systematic review and meta-analysis of RCTs in workforces and in primary care in which individuals were randomly allocated to more than one of six interventions (stopping smoking, exercise, dietary advice, weight control, antihypertensive drugs and cholesterol lowering drugs). The changes in risk factors were modest and related to the amount of drug treatment used. Interventions using education, with or without drugs, were more effective in people with hypertension and in other high-risk groups. (Ebrahim S., Davey Smith G., 1997).

  • Secondary prevention

In contrast to the equivocal evidence on community based programmes for primary intervention, there is a wealth of good evidence from RCTs and observational studies about the effectiveness of secondary prevention of CHD - in people who already have signs or symptoms of the disease. In addition to demonstrated evidence about the effectiveness of individual interventions there is also evidence from a systematic review that multidisciplinary disease management programmes improve the processes of care, reduce admissions and enhance quality of life in patients with CHD, although the impact on survival and recurrent heart attacks remains uncertain (McAlister et al., 2001). Much of this evidence was available to the HaHP team at the start of the programme and developing a systematic approach to implementing evidence-based guidelines was a major component of the secondary prevention project.

In summary, although some components of HaHP, notably the secondary prevention initiatives have an evidence base, evidence for the effectiveness of community based prevention programmes is, at best, equivocal. It certainly does not consistently demonstrate additional benefit in accelerating the background secular trends in health behaviour and in CHD mortality reduction which are apparent in Western countries.

2.3.3 Differences in design

Given the lack of a good evidence base for HaHP as a whole, the design of the demonstration project only compounds the difficulties. The problems in designing and implementing an effective evaluation of population based projects like this have previously been highlighted (Tudor-Smith et al., 1998; Puska, 2000).

The HaHP project has also taken place against a background of falling CHD mortality rates in Scotland and there is little realistic chance of demonstrating additional benefits in accelerating these trends. The secondary prevention elements had more potential to demonstrate effectiveness and to highlight useful lessons for other parts of Scotland. In addition to this lack of a strong evidence base for community-wide CHD interventions as a whole, there are a number of other ways in which the design of an effective evaluation of the HaHP project was problematic. The most important of these are:

  • Interventions

Both the intensity and the quality of the interventions used in most community based intervention studies are difficult to measure and this was particularly true of HaHP.

Many small projects were funded with variable levels of activity, both over time and throughout the geographical area. This made it difficult to identify which parts of the project might have an effect. Many of these specific interventions were not evidence based.

There was anecdotal evidence that, prior to HaHP, levels of some health promoting interventions such as smoking cessation support, were lower in Paisley than elsewhere in Scotland. This could tend to exaggerate any benefit for HaHP.

In addition, HaHP took place against a background of many national initiatives on improving health in Scotland. These would have an impact on the population of Paisley and would make it very difficult to isolate any specific benefits from HaHP.

  • Setting

Although mortality from CHD in Scotland remains amongst the highest in developed countries, death rates have fallen substantially over the past 25 years and this trend is continuing throughout the whole country. The population of Paisley has high levels of deprivation and this is associated with high levels of CHD. There are significant cultural differences between the intervention area and North Karelia which are likely to impact on uptake, motivation compliance and hence effectiveness.

2.3.4 Summary

A major problem for HaHP is that community based CHD prevention studies have failed to show a clearly attributable effect, possibly even in the case of North Karelia. HaHP took place against a background of falling CHD mortality rates in Scotland. It had little realistic chance of demonstrating additional benefits in accelerating these trends.

The secondary prevention elements had more potential to demonstrate effectiveness and to highlight useful lessons for other parts of Scotland.

« Previous | Contents | Next »

Page updated: Thursday, April 7, 2005