« Previous | Contents | Next »
Listen
Section 1: Introduction
1.1 The National Health Demonstration
Projects
In 1999, the public health White Paper
Towards a Healthier Scotland pledged £15 million
over 3 years to support four National Health Demonstration
Projects (
DPs) in priority areas of child health,
sexual health of young people, coronary heart disease and
cancer. The purpose of the
DPs was to act as a testing ground for
national action and to provide a learning resource for the
rest of Scotland. They were intended to combine the best
existing evidence with innovative practice, a concept that
has been a matter of ongoing debate throughout the life of
the projects. In 2003,
Improving Health in Scotland - The Challenge
announced the intention to support a second phase of the
demonstration project programme, for up to a further 3
years, subject to a review of existing interventions and
investments.
This report considers the evaluation of the first three
years (Phase 1) of the three health demonstration projects,
Healthy Respect (
HR), Have a Heart Paisley (
HaHP) and Starting Well (
SW), with the aim of making
recommendations for the design and commissioning of future
evaluation work. We do not consider the Cancer Challenge
which was commissioned and evaluated separately but a
summary of the pilot, evaluation models, methods and
results is included at Annex A.
1.2 Purpose of this Report
The National Steering Group for the Health
DPs (now the Co-ordinating Group) set up
four task groups, one of which was the Evaluation Task
Group. Its remit is:
- to make a methodological assessment of the Phase 1
evaluations (outcomes and process) in order to develop
an account of the strength of available evidence;
- to make recommendations on the basis of available
evidence for further evaluation and policy
recommendations;
- to report to the National Health Demonstration
Projects Co-ordinating Group.
Members of the group have reviewed published research
relating to community-based sexual health, heart health and
early years interventions, and documents relating to the
process of commissioning both the
DPs themselves and the independent
evaluations. The group has met the evaluation teams and
members of the management teams for each of the three
projects, as well as the learning network co-ordinators.
This report describes our findings and makes
recommendations for the evaluation of Phase 2 of the
demonstration projects and for the evaluation of future
initiatives of this kind.
Our remit required us 'to develop an account of the
strength of available evidence'. That is to say, our
assessment depended not just on how well the evaluations
were designed and conducted but also on a) the strength of
the existing evidence base, and b) how closely the
DPs (and therefore the evaluations)
matched the evidence base. We have therefore considered the
prior evidence for the effectiveness of community based
heart disease, sexual health and early years interventions
in some detail, rather than restricting our review to
materials originating from the demonstration project
programme.
1.3 Structure of Report
The remainder of this report is divided into four
sections:
- The commissioning process
- The evidence base of the demonstration
projects
- Conclusions and recommendations
1.4 The Commissioning Process
1.4.1 The demonstration projects
As stated above, the Government's intention to set up
four national health demonstration projects (
DPs) was announced in the 1999 public
health White Paper,
Towards a Healthier Scotland. Bids had already
been invited for 'The Cancer Challenge', a screening
programme for the early detection of colorectal cancer.
Expressions of interest in bidding for the three remaining
projects, which were to focus on health promotion and
protection in the period leading up to birth and the first
five years of life, promotion of responsible sexual
behaviour among young people, and the prevention of heart
disease, were invited in May 1999. Two hundred and
fifty-eight notes of interest were received and six
organisations were invited to develop full proposals. These
were assessed by a panel chaired by the Chief Medical
Officer comprising senior officials from the Scottish
Executive Health Department and Health Education Board of
Scotland (
HEBS), and three independent experts.
The three successful bids were announced in February 2000
after the formulation of more substantive proposals and
post-tender negotiations - Have a Heart Paisley (
HaHP), Healthy Respect (
HR) and Starting Well (
SW). The
DPs are described in detail in Section
Two. Bids were not expected to set out detailed plans for
evaluation, and this was not a criterion by which they were
assessed.
1.4.2 The external evaluations
External evaluations of the three
DPs were commissioned through a
selective tender process in which six organisations were
invited to bid for one or more of the evaluation studies.
This 'selective tendering' method is the usual procedure
for procuring social research; it is less often used,
though far from unknown, in health research. Tenders were
invited
after the demonstration project contracts had been
awarded, and the successful proposals formed part of the
documentation sent to prospective bidders. A number of
those invited to tender declined on the grounds of lack of
capacity, or because they had concerns about the
'evaluability' of the projects. This reluctance to tender
may be indicative of the limited research capacity in
Scotland, but could also be because evaluating
community-based interventions presents many evaluation
problems, and does not suit traditional methods or produce
tidy publications. The fact that many academic units in
Scotland and elsewhere refuse to engage in evaluating such
interventions means in turn that there is a limited
national evidence base available for such practical
interventions.
One bid was received for the evaluation of each
demonstration project. Each bid was evaluated by a tender
panel comprising representatives of the Chief Scientist
Office (
CSO), the Central Research Unit and
Public Health Policy Division of the Scottish Executive
Health Department (
SEHD) and
HEBS (now
NHS Health Scotland)
The invitation to submit notes of interest told
demonstration project bidders that they could not have
access to research funds to develop their applications, and
also that they should concentrate on disseminating good
practice because there would be separate, external
evaluations. This, coupled with the fact that the
evaluations were commissioned separately, after the
demonstration project contracts had been awarded, helped to
cement a distinction between the implementation and the
evaluation of the
DPs which has been problematic for both
sides. This is discussed further in Section 3.
The objectives of the evaluations were extremely broad.
They required the evaluators to develop methods which would
provide an overall estimate of the success or failure of
each demonstration project:
The aim of the evaluation is to determine how far the
projects meet their own aims and objectives, and to use
this information to judge their contribution towards the
overall goals of the demonstration project initiative.
...
The overall range of objectives for each project is
broad. Some of the proposed health outcomes will respond
slowly, even if the interventions are effective. Some may
be straightforward to measure, but many raise difficult
methodological issues associated with identifying and
attributing effects. Each set of aims and objectives
contains outcome as well as process elements, and the
evaluation must cover both, in order to describe and
explain the projects' impact. In addition, each project is
meant to improve health within its local target population,
and to demonstrate methods that could be adopted elsewhere.
Both aims are relevant to the evaluation. (Invitation To
Tender, April 2000)
The original brief was very open with regard to research
methods:
Bidders are invited to propose methods for the
evaluation. These will vary between demonstration projects,
and between objectives, but it is envisaged that a
combination of quantitative and qualitative methods will be
required, using primary data gathered via surveys and
interviews, and secondary analysis of monitoring data.
Methods should be rigorous, precise and appropriate to the
hypotheses being tested, taking account of the particular
problems associated with evaluation of area-based
interventions. They should be efficient in terms of the
sampling and other requirements placed on the demonstration
projects. They should be geared towards estimating the
effectiveness of the projects in meeting their objectives,
and understanding the reasons for the success or otherwise
of the projects. The overall conceptual and methodological
framework should be made explicit. Tenders should indicate
clearly which outcomes they propose to measure, and give
reasons for the choice. If surrogate outcome measures are
proposed, evidence of their validity should be offered.
(Invitation To Tender, April 2000)
Nevertheless, the fact that the commissioning process
was managed by the
CSO, on behalf of the Public Health
Policy (now Health Improvement Strategy) Division of the
SEHD may have given rise to certain
expectations about how the evaluations should be conducted.
It was suggested to us that the
CSO's involvement ultimately encouraged
a view that a traditional approach relying on controlled
comparisons should be adopted.
« Previous | Contents | Next »