The Scottish Government

« Previous | Contents | Next »

Listen

The Independent Evaluation of 'Starting Well' Final Report

Part 1. Background to Starting Well and its Evaluation

1.1 Introduction

1.1.1 Commissioning Starting Well and its Evaluation

In 1999 the Scottish Public Health White Paper, (Towards a Healthier Scotland, Scottish Office, 1999), sought bids from health partnerships in Scotland to develop good practice in the areas of child health, coronary heart disease, sexual health and colorectal cancer. Through the Glasgow Healthy City Partnership, a multi-agency child health bid (Glasgow Healthy City Partnership Proposal, 1999) was developed and awarded funding. The project, 'Starting Well' was granted 3 million over a three-year period and was launched in November 2000. Following the decision to award funding to the Glasgow Healthy City Partnership, the Scottish Executive commissioned a multi-method independent evaluation led by the Department of Public Health at the University of Glasgow (see Appendix I for details of the evaluation team).

The initial phase of funding for Starting Well would have ended in December 2003, however, it has been granted an additional year of funding until September 2004; subject to the Scottish Executive's agreement of an further programme plan, the project will be provided with funding for a Phase II from October 2004.

1.1.2 The Starting Well Demonstration Project

Starting Well is one of a family of 'early years' initiatives being developed across the UK, which are part of a concerted attempt to break the cycle of poverty and inequality by providing support to children in Britain's most deprived areas. The overall aim of the project is to 'demonstrate that child health can be improved by a programme of activities to support families, coupled with access to enhanced community-based resources for parents and their children' (Glasgow Healthy City Partnership Proposal, 1999). As described in the proposal document, the project drew extensively on the US literature on home visiting and, in particular, was shaped by the work of Olds and Kitzman (1993, 1997, 1998). The essence of this evidence-base is that, compared with standard health care provision, intensive home visiting had significant impacts on a range of child and family health related outcomes. The key elements of the US programmes on which Starting Well focused were: intensive visiting of families within the home; the development of supportive relationships between families and their visitors; and, an emphasis on health promotion approaches. However, the complex nature of home and health visiting (Gomby et al, 1993; Elkan et al, 2000) makes this an evidence base that is not straightforward to implement and there were a number of ways in which Starting Well departed from the Olds model. These included:

• The targeting of deprived communities rather than vulnerable individuals;
• The inclusion of all new babies as opposed to only first babies;
• A lesser focus on the antenatal period than recommended by Olds due to the availability of Community Midwifery services in Scotland and specific caseload issues within the project;
• The use of paraprofessionals as part of the home-visiting delivery mechanism in addition to professional health visitors;
• The vastly different primary care context within which the evidence was derived (for example, the absence of a universal health visiting and community midwifery service; and related to this,
• The requirement to integrate aspects of project delivery with existing professional and organisational structures as opposed to an entirely standalone intervention.

Within Starting Well the overall project aim was addressed in three principal ways:

• the introduction of an augmented programme of home visiting to all families of new babies born within two geographical areas within the City of Glasgow, selected due to their relative socio-economic disadvantage (Greater Easterhouse ⁶ and Gorbals/ Govanhill/ North Toryglen, known respectively as the East and the South). Engagement with the project has been very high and at the time of writing, 1271 families have received input from the project.
• the development of enhanced local community supports and structures within these areas; and,
• the development of integrated organisational services that respond to the needs of children and their families both within the local areas and across Glasgow as a whole.

There are significant overlaps between these three strands of the intervention but the main mechanism and structures that underlie them are as follows.

1.1.2.1 The Model Of Home Visiting

The home visiting model aimed to provide intensive support to families during the first five years of the child's life. It incorporated a number of recommendations arising from Nursing for Health (Scottish Executive, 2001), including the implementation of best practice around health promoting activities and child health surveillance and an augmented public health role within nursing. A project team was initially established in each intervention area with a health visitor coordinator, Starting Well health visitors and health support workers (the latter are employed through One Plus, a voluntary organisation) and a bilingual worker in the South. As the project has developed, these have been augmented with the employment of two community nursery nurses and a community support facilitator per team.

Health visitors used a number of standardised tools to structure their visits with families. These include:

• a core visiting schedule that provided guidance on the number of visits and age-related health topics that are appropriate at different points in the child's development;
• a family health plan that stimulated the discussion of, and recorded, jointly agreed health needs; and,
• a family support scale that required staff to make judgements about the vulnerability of families at different stages.

Project team members received intensive training on a wide range of issues including child development and protection, domestic violence, speech and language, and accreditation on a Triple P Programme (an Australian parenting programme that has been adopted by the project). They were encouraged to engage in reflective practice and provide each other with peer support. The project managed to engage almost all eligible families in this overall home visiting approach

1.1.2.2 Enhanced Community Supports

To develop a mechanism by which the needs of children and families could be supported at a local community level, local implementation groups were established. It was anticipated that these groups would include representation from the statutory and voluntary sectors and from the local community. Their remit included the identification and addressing of community level issues pertaining to child and family health. They each had a development fund, with an annual budget of 20,000, which they used to support the activities of local organisations that have joined a Starting Well Affiliation Scheme. An initial ceiling of 500 was placed on individual grants.

Part of the role of the community support facilitator was to act as a bridge between the home visiting teams and the local implementation groups and to manage the use of the development fund.

1.1.2.3 Organisational Responsiveness

The mechanisms that were put in place to encourage statutory organisations to work together to develop and deliver more responsive and strategic services for children and families included the local implementation groups within the two intervention areas and the project steering group which operated with representation from senior agency staff who had a role in strategy development at a Glasgow-wide level.

1.1.3 Aims of the final report

The independent evaluation of Starting Well was designed to be both formative and summative (specific methods employed are detailed in Part 2 of the report). A number of interim reports have been produced by the research team and two journal articles are currently in press. Full details of presentations and publications are provided in Appendix II. This final project report provides an assessment of Starting Well's impact on a range of outcomes and processes and is structured in the following way:

• Part 2 focuses on the health related outcomes for children and families as assessed through a quasi-experimental survey (Part 2)
• Part 3 describes the rationale lying behind the Starting Well demonstration project as expressed by key stakeholders in the first year of the project before going on to focus on a number of key processes underpinning its implementation. These are -
  • the nature of relationships developed between families and their health visitors
  • the development of an augmented model of home visiting
  • the development of mechanisms to support strategic change

This part of the report concludes with a discussion of strategic stakeholders' retrospective reflections on the initial project rationale.

Key findings, policy implications and recommendations are drawn together in the conclusions.1.2 Methods

1.2.1 Introduction

In this section we provide an overview of the main aims of the Starting Well Independent Evaluation as commissioned in September 2000. We then outline the methods used to address them ⁷. In addition we highlight the ways in which the evaluation developed over time in response to the project itself. Ethical approval for the original research proposal and its subsequent amendments was sought and obtained from the Greater Glasgow Primary Care Trust Local Research Ethics Committee in 2001.

1.2.2 Overview Of The Independent Evaluation Of Starting Well: Aims And Methods

The main aims of the independent evaluation were as follows:

1. To measure the impact of the project on children and families;

2. To understand the theory, processes and context of the Starting Well intervention; and,

3. To analyse the policy implications of the project.

1.2.2.1 Assessing Impact

The most resource intensive component of the evaluation involved a quasi-experimental comparison of the two intervention areas with a socio-demographically similar area in the North of the city. This cohort study compared the health and development of intervention children over the first 18 months of life with a group of families receiving statutory health visiting in a demographically similar part of the city ⁸. All families with newborn were approached for consent by their health visitor between 01/06/01 and 31/06/02, yielding a total of 627 participants, or around 50% of all births ^9,10. The characteristics of participants are described more fully in section 2.2.3. Participating children were assessed on a maximum of three occasions (immediately after birth, then at six and 18 months) using a combination of mother-report questionnaires, observation in the home and structured interviews with the mother. Questionnaires covered: background maternal, household and area characteristics; maternal mental health and health behaviour; and attitudes towards parenting and current health-visiting service. Each participant that could be contacted at six and 18 months received a home visit from a trained research nurse who administered the HOME Inventory (see http://www.ualr.edu/~crtldept/home4.htm; Bradley and Caldwell, 1979), a standardised interview-and-observation tool that assesses the quantity and quality of stimulation available to a child in its home environment. Interpreters were made available to assist participants with no or limited English. Finally, individual-level data on e.g., number of home visits were collected from routine sources including health visitor records.

Analysis of outcomes at six and eighteen months concentrated on the 'rich' datasets composed of those families that completed all assessments up to that point (n=346 and 294 respectively) ¹¹. Key outcomes included: the total HOME score; extent of maternal depressive symptoms ¹² and dichotomised survey measures of child dental registration, and maternal satisfaction with the service. After checking for opt-in bias, multivariate analysis was carried out in order to test for intervention effects whilst controlling for a range of other predictor variables. A detailed description of methods for this component can be found in Shute and Judge (forthcoming) and in Part 2 of this report.

1.2.2.2 Theory, Processes And Context

Aim 2.1 To understand stakeholders' theoretical rationale for the Starting Well intervention

A theory of change approach (Connell and Kubisch, 1998) was used to map stakeholders' views of how and why the intervention was being implemented and to capture expectations of change within a 3 year programme of activity. This was undertaken through interviews and focus groups with key strategic players, observation at steering group meetings and documentary review.

Data were collected in the first year of the project to produce a summary of the project's underlying theory of change that fed into the process of sharpening project planning (Mackenzie, 2002) ¹³; stakeholders' perceptions of the robustness of this initial theory were reassessed in 2003.

Aim 2.2 To understand the key processes predicted to impact on Starting Well's success

Three separate studies were undertaken to address this aim:

• A primary aim of this evaluation component was to examine the formation and operation of the relationship between the child's key care-giver and their health. A second broader aim was to elicit the developing views of both sets of participants (mothers and their health visitors) on key aspects of the service. A total of 20 women (mean age 27 yrs, range 20-40 yrs) were recruited and interviewed when their child was around four months old with follow-up interviews conducted with 13 of those women at around eleven months ¹⁴. Respondents were purposively sampled across both project areas to provide instances of the following categories: first-time and experienced mothers; black and ethnic minority mothers; families with a range of emotional, physical and material needs. Each family's current health visitor was interviewed separately in advance of the mother interview ¹⁵. In order to capture ongoing service development, respondents were recruited in two cohorts and received their first interview approximately eight months apart. A total of 59 interviews were held, 56 of them tape-recorded.
• Through the initial theory of change approach two additional processes underlying stakeholders' beliefs about the mechanisms through which Starting Well might positively impact on child and family health were identified. The independent evaluation approach was amended to incorporate the exploration of these within its approach. The first of these was the process of developing and implementing an augmented model of home visiting. Qualitative interviews with both strategic and operational staff at two time points were used to explore this process. This part of the evaluation aimed to be both formative and summative (Weiss, 1998). That is, it aimed to provide feedback to the project (and the wider policy audience) of lessons learned within the early days of implementation and, to provide an assessment of how the model was perceived to have worked as the initial period of funding came to a close.

Strategic level respondents were selected to provide a range of experience from the project's senior management team, the steering group, the Scottish Executive (group interview), the wider Greater Glasgow Primary Care Trust and the employing bodies for members of the project team.

Project team respondents were selected to give coverage of the following staff groupings: community support facilitators (both facilitators participated); bilingual worker (of which there is only one); community nursery nurses (three of the four participated); health visitors (12 out of a total 21); health support workers (seven out of 16). Further detail of the methods used is provided in the interim evaluation of the home visiting model (Mackenzie, 2003).

• The second adaptation to the independent evaluation approach was to include a focus on the process by which individual family health needs were aggregated to a community level and responded to at a strategic level. This process was studied through a mapping of child and family health needs emerging through the project, an assessment of the extent to which the identification of local need impacted on the agenda and decision-making processes of relevant groups, and, of the degree to which Starting Well had impacted on strategic planning. A range of techniques including a local community survey, semi-structured interviews and focus groups were used to investigate these issues (Starting Well Evaluation Proposal, amended 2002). More detail on the local community survey can be found in Starting Well and Community Support: Exploring Relationships with Community Agencies in the Demonstration Project Areas, Berzins et al (draft, 2004).

Aim 2.3 To describe the broader material and social contexts of study areas

This evaluation component aims to describe dimensions of area context that might be hypothesised to influence health-related outcomes over-and-above the important individual-level characteristics identified in the quasi-experimental study. Defining study areas as aggregates of whole postcode sectors, the dimensions considered (derived from 2001 Census, routine medical and cohort survey data) include: basic demography; material and built environment; health; and social context. By making descriptive contextual comparison between intervention and comparison-areas, we aim to complement the impact study findings and to explore the potential for separating out individual- and area-level effects more formally using multi-level analyses.

Given that findings do not comment directly on the operation or effectiveness of the intervention, we have opted not to include them in the main body of the report but include them as Appendix IV.

1.2.2.3 The Policy Implications

The aim of analysing the policy implications of Starting Well cut across the other aims of the independent evaluation and signalled intent on the part of the evaluation team to contribute to the wider policy debate that surrounds the Health Demonstration Projects. In this respect it did not have a methodology in its own right.

« Previous | Contents | Next »