Scottish Government

The Independent Evaluation of 'Starting Well' Final Report

Part 4. Conclusions

Conclusions and Policy Implications

4.1 Introduction

In this final section of the report we discuss the main conclusions from the various elements of the independent evaluation and summarise the policy implications arising from these. In addition, we consider the more general learning associated with designing and implementing a Demonstration Project.

Before doing this we summarise some of the key assumptions lying behind the Starting Well model that emerged from the process of articulating the strategic stakeholders' implicit Theory of Change.

4.2 Impact on child and family health related outcomes

The original Starting Well proposal was based largely on an attempt to replicate a model of home visiting tested within the US (Olds et al, 1998). Despite doubts as to the transferability of the North American evidence-base to the British context and a number of limitations within the quantitative evaluation of Starting Well, findings relating to maternal depressive symptoms and HOME score are supportive of shorter-term psychological benefits for study mothers and potentially longer-term cognitive and emotional developmental benefits for study children.

More specifically, multivariate regression analysis of the quasi-experimental survey data that compared families in the intervention areas with those in a broadly similar control area in Glasgow found:

• lower rates of depressive symptoms amongst intervention mothers at 6 but not 18-months;
• no significant improvement in the quality of the home environment at 6-months but a small positive intervention effect at 18-months (p=0.088);
• higher levels of client-satisfaction with levels of health visitor support within intervention families;
• and, higher levels of dental registration at both assessments for such families.

These modest findings provide some evidence of a positive Starting Well effect although the weight of importance that might be carried by an outcome such as dental registration is open to question. More child-focused and longitudinal analysis is necessary to determine the longer-term clinical and social significance of these intermediate outcomes and to assess the degree to which a 'step-change' in child health has been effected. In retrospect it is unfortunate that the evaluation was not designed to address questions concerning the economic impact of the project; it may be that a future extension to the quasi-experimental study could usefully consider weighing the costs and benefits of this approach to child and family health promotion.

Simple comparisons of area-level context (described in appendix IV) suggested a basic similarity between the intervention and control areas that did not help interpret the above findings. Lower-level comparisons, however, (for example at the level of postcode sector) revealed the potential for more sophisticated multi-level analyses that may help tease out the relative contribution of individual and area-level factors to these outcomes. More extended individual-level regression analyses remain our immediate priority, however, if future opportunities can be found to explore these possibilities, we may not only explain more of the variance in outcomes but also gain a more informed sense of the kinds of emergent community-level factors that constrain and facilitate the operation and effectiveness of Starting Well.

4.2.1 The case study families

The impact of Starting Well was also assessed qualitatively by exploring the views of a purposive sample of case-study families and their health visitors. Through this fieldwork a process model was identified that linked intensive health visiting input to a diffuse set of benefits that might be summarised as 'enhanced support'. These included: increased confidence; reduced anxiety and isolation; the opportunity to confide; and, experience of advocacy.

The process model describes how intensive visiting equated to more time and direct contact with mothers during a period of universal need which encouraged the rapid formation of a trusting relationship, an individualised care package and the provision of more and better quality information on needs and life circumstances. This in turn, was associated with the identification of a broad range of problems and problem-solving activity and an enduring two-way (functional) dialogue between mother and health visitor. In sum, these processes promoted perceptions of enhanced support. Lack of maternal receptivity to the service and health visitor caseload pressures explained variation in process and outcomes.

Intensive visiting was found to be an effective way of delivering a more patient-centred, 'holistic' model of care. Precipitating factors include: the convenience of the home setting; the shift in power relations inherent in the mother's control of access to the setting; and a concomitant need for the health visitor to maintain access by a) providing a flexible service and b) establishing a positive, non-directive relationship.

Project health visitors praised teamwork, training and aspects of the approach (intensive support, skill mix) as strengths but had experienced resistance, scrutiny, and larger, more demanding caseloads than initially anticipated. These latter factors may, at times, have impeded their capacity to deliver the service as intended.

Support was voiced for a universal intensive service in the first postnatal months, but that had the capacity to target sub-groups of women with higher levels of identified need.

The project demonstrated that an individual family level, more intensive home visiting could provide a more supportive and family-centred approach within which health promotion could be effectively embedded. The challenges of maintaining this approach within the constraints imposed by growing caseloads and within a mainstreaming agenda are significant.

4.3 Implementing a new model of home visiting

Starting Well set itself the challenge of not only delivering an intensive home-visiting service but of implementing this through delivery mechanisms that provided a potential challenge to existing professional remits and structures. In a relatively short space of time Starting Well developed two project teams that incorporated a new type of worker (the health support worker) alongside a professional group of long-standing (health visitors).

This delivery mechanism threw up a series of challenges and variable progress was made in response to these. Here we draw out the key findings in relation to two main issues: the degree to which Starting Well was able to change health-visiting practice; and, the extent to which the project's health support worker model has worked.

4.3.1 Changing health visiting practice

The health visitors within the teams were encouraged to adopt an evidence-based and standardised approach to their practice. In reality, whilst many developed a broader understanding of the health and social issues affecting the families that they were dealing with, the extent to which a consistent approach to the quantity and quality of visiting that was undertaken is less clear. Health visitors continued to argue for the need to practice in a less standardised and more intuitive manner that cannot be entirely reconciled with a more managerial perspective (Robinson, 1995) and many demonstrated a degree of hostility to the Starting Well approach that they were being asked to implement. This has wider implications for the use of standardised family health plans as recommended by Hall (2003) and more generally for the implementation of Nursing for Health (Scottish Executive, 1999).

Over time the project acted as an advance demonstration of the kind of targeted intensive support within the context of universal provision that is recommended by Hall (2003). Assumptions about the ease of identifying the most vulnerable families were challenged over the course of the project and a need for flexible and ongoing approaches to identify vulnerability eventually recognised. A high level of interaction with all families over the first six months of a baby's life was viewed as key in developing the kind of trusting relationship that would facilitate future disclosure of vulnerability - this is a major challenge within generic caseloads.

In considering the success of the two project teams in influencing their wider LHCCs, context is everything. The projects developed in very different ways as a result of their individual team members, the dynamics between them and the organisational culture within which they were located. Those charged with mainstreaming Starting Well practice should to be alert to the potency of context. In addition, the degree to which the Starting Well model will be 'diluted' through integration with GP practices will be dependent on the project team members, the GP practices and the wider LHCC. These same factors will impact on whether a more discrete team of skilled workers manages to broaden the practice of other staff within an LHCC.

In relation to skill mix with nursery nurses and community support facilitators, the project illustrated some of the possible ways in which different professionals can work together to tackle need. However, a lack of role clarity remained within the project teams by the final year of Phase I. More explicit support mechanisms for nursery nurses were required. The addition of a professional group in small numbers to existing primary care staff requires time, training and support at a local as well as at a more strategic level. Without these individuals can flounder and integration stall.

By the end of the project, health visitors believed that they had done far less of a community support nature than they had originally anticipated but many were not concerned by this and had come to the view that this was not an area of practice that they had a particular wish to develop. This has implications for ensuring that community development approaches to child and family health become part of routine primary care (Hall, 2003). Community development is unlikely to happen if left to chance and should be planned at the strategic level rather than left to the personal preferences of individual members of staff.

The process of implementing the project's home visiting model raises some important questions about a number of assumptions with Nursing for Health and Hall Four. At the very least it highlights the hugely resource intensive effort required to change practice effectively. Particularly where members of a professional group are not wholly supportive of change, then introducing the tools of standardisation and skill mix approaches, much less changing approaches to interacting with families and communities, cannot be tackled lightly.

4.3.2 The Health Support Worker Model

A key element of the Starting Well home visiting model was the employment of health support workers from the local communities. These workers developed very diffuse and flexible roles that varied between the individuals; different health visitors also deployed them in different ways. The lack of role clarity did not appear to be problematic for the support workers themselves although some health visitors found these variations difficult to manage. A specific concern arose about the extent to which some health support workers were able to maintain professional boundaries in their relationships with service users as a result of their access to the real life complexity of families' lives. Reflective practice and supervision should be an explicit and continuous part of the support that such workers receive in this and similar interventions.

In general health visitors were positive about the role played by these additional workers and, in particular, the part played by those support workers that they worked most closely with. Nonetheless, stories of negative experiences with health support workers were told. It is important that, whilst lessons are learned from critical analysis of incidents that go wrong, individual stories are not generalised to a wider group of workers.

A further layer of innovation in the health support worker model was that the staff were employed, trained and managed by One Plus, a voluntary sector organisation. This offered the workers support from an organisation with expertise in employing those re/entering the labour market and was part of the project's strategy for tackling social inclusion.

Perhaps inevitably this dual management structure led to some difficulties at an operational level with staff perceiving there to be a distinct lack of clarity over the day-to-day management of caseloads. This became a major stressor for many of the health visitors and senior managers attempted throughout the project to resolve this issue. However, those at a strategic level acknowledged that the close involvement of a voluntary sector organisation allowed a supportive work culture for the local workers that would have been inconceivable within the NHS. The problem remained however that health visitors had viewed the support workers as a means of alleviating their caseloads and instead found themselves with a significant supervisory role without always seeing their own work-load reduce. More work at the beginning and throughout the project in ensuring health visitor ownership of the social inclusion aims of the support worker model may have been valuable. This issue will become increasingly salient for public sector organisations as they devise ways of expanding their workforces to those outside the current labour market. Health care organisations that plan to complement their existing professional workforces in the future will need to be clear about the purpose, management and support of paraprofessional staff. Existing staff who are expected to share in operational management roles would benefit from early and ongoing consultation and more realistic assessments of the impact of change on existing workload.

4.4 Delivering community and strategic change

To help conceptualise the progress made in delivering both community level and wider strategic change we represented the direction of travel from identifying to tackling community health needs in 5 stages. These were:

Step 1 - that the process of intervening intensively with individual families would result in an understanding within project teams of key community health needs;

Step 2 - that a shared understanding of these needs would feed into local implementation groups; which

Step 3 - would develop and support local, community solutions; that,

Step 4 - if unable to be resolved locally would be referred to the project steering group; which

Step 5 - would act as an advocate for more strategic, Glasgow-wide solutions.

Our findings in relation to these stages are summarised below.

Step 1

Despite the limited degree to which health visitors were able to engage with the community support aims of the project, more intensive contact with families appeared to help them understand health needs at a community level. This understanding was also shaped by communication within the project teams, working with other agencies and pre-existing knowledge of the local communities. A wide range of needs was identified but none were believed to be 'new' issues.

Step 2

The process of sharing perceptions of community level need was rather haphazard within the two project teams and its success appeared to be a function of the level of collaborative working, with one team in particular demonstrating a significant lack of cohesion.

The increasing burden on health visitor caseloads and the early lack of clarity in the role of the community support facilitator (as perceived by other members of the project teams) led to a lesser emphasis on advocating for community change within the health visitor role. Once again, this questions the assumption that health visitors and the changing systems with which they work, are ready for the challenges posed by key policy documents such as Nursing for Health.

Step 3

The local implementation groups were perceived to have been successful in disbursing monies from their development funds to local organisations but less effective in securing representation from both key statutory agencies and local parents.

The role of the community support facilitators and the bilingual worker became key in bringing about more sustainable changes at a local level and in liasing with other relevant child health fora. Whilst there is evidence of much good practice at a local level, the implementation groups suffered from the poor community and statutory representation that plagues health and social care projects more generally. More strategic influence is required to build the capacity of local planning groups.

Step 4

During the course of Phase I of Starting Well, only a small number of issues were passed from the local implementation groups to the project steering group. None of these resulted in significant change at a strategic level and the members of the local implementation groups showed little knowledge of the role of the steering group. As with many complex interventions a lack of connection between the city-wide and the local; and between the strategic and the operational, served to limit the effectiveness of organisational structures that were established. These connections should be planned, supported and resourced more explicitly.

Step 5

The project steering group did not succeed as a mechanism for strong partnership working around the child and family health problems experienced in poor communities. A lack of ownership of the project beyond the health partners, and a lack of commitment to tackling broader strategic questions were evident. On the other hand, there was some evidence of more constructive strategic work occurring 'behind the scenes'. The evaluation of a range of policy interventions across the UK, and beyond, supports the view that partnership working is not the natural modus operandi but is a long hard process requiring commitment at a number of levels. Given that partnership working lies at the heart of current public policy (Barnes et al, in press), this requires to be taken seriously. More time at the beginning of the project in establishing aims around partnership working and of developing specific partnership approaches to issues arising through the project might have been useful. In addition, the steering group suffered from a lack of opportunity to work together to resolve practical issues and this fostered a serious dip in enthusiasm for the project. Partnership is not simply established through the convening of a group but requires ongoing engagement, training and cultivation.

Whether or not the initial aims were appropriate it is clear that Starting Well did not, for example, demonstrate the step change that it aspired to as measured by the most important health related outcome that we investigated, and nor did the process of service development and delivery run as smoothly as predicted. Despite this, however, the complexity of the Starting Well experience should be recognised. It was highly valued by many of the staff and individuals involved, and there are valuable lessons to be learnt from it about the implementation of future initiatives. It may even be that further analysis of the quasi-experimental data will show stronger intervention effects. There is also a possibility that such effects only emerge as the Starting Well children get a little older, provided that attempts are made to look for them. The case for doing so will be made in due course. For now we conclude with a consideration of some of the reasons why Starting Well has not met all of the expectations of those who commissioned and designed it.

4.5 Demonstration Projects - Final Considerations

In this concluding section we draw together a number of conclusions and reflections on the process of designing, implementing and learning from a Demonstration Project such as Starting Well. In particular we look at the following issues: project planning; the meaning of 'demonstration'; the use of the existing evidence-base; and, achieving professional and organisational change.

4.5.1 Project Planning

As with almost all projects operating in a complex setting, Starting Well struggled to develop detailed programme plans that were wholly testable and many of the targets that they set were felt, by some stakeholders, to be unhelpful in terms of unleashing creative approaches to tackling inequalities in child health. On the other hand, unlike many such initiatives, the project placed a relatively strong focus on monitoring systems that would allow their plans to be verified. Project planning requires to strike a balance between constriction and chaos. Planning and monitoring change in complex systems require time and capacity development; these were in short supply at the point of commissioning the Demonstration Projects. It is positive that this has been recognised at a national level in commissioning Phase II although the meshing of independent evaluation findings with the development of this second phase has been rather clumsy with very little time allowed for evaluation findings (both external and internal) to feed into the planning process. The precise model of project planning utilised within a future phase of the project is probably much less important than that planning is both systematic and flexible so that learning continues to be captured about what is working well. Careful planning should therefore be viewed as a help rather than a hindrance to strategic management.

4.5.2 What is Meant by 'Demonstration'

Project commissioners, planners, implementers and evaluators have grappled with the meaning of the term 'demonstration' and the tension that it contains between the application of an existing evidence base and the encouragement of innovative approaches to intractable problems. Once again, this phenomenon is not new to the Scottish Health Demonstration Projects (Judge et al, 2004) and nor is it a purely academic point. Demonstration Projects find themselves criticised for responding to their policy circumstances by those who believe that an intervention should be evidence-based and static; and, for implementing irrelevant interventions by those who believe that projects have a primary duty in providing lessons about the here and now. Far greater clarity and consensus is required for future projects and for Phase II of Starting Well if the project is to avoid being pulled in two opposing directions. This is particularly salient in relation to the debate around the most appropriate 'model' of Starting Well where stakeholder expressed concerns that current policy thinking about 'integration' as a primary goal, will force the project down a particular road regardless of its initial aims and underlying principles.

4.5.3 The Use of the Evidence-Base

Related to the discussion of what is meant by 'demonstration' is the question of the robustness and contextual applicability of the existing evidence base. As described earlier, Starting Well drew extensively on the US literature on home visiting and, in particular, was shaped by the work of Olds and Kitzman (1993, 1997, 1998). The essence of this evidence-base is that, compared with standard health care provision, intensive home visiting had significant impacts on a range of child and family health related outcomes. However, recent reviews (Elkan et al, 1999; Bull et al, 2004) suggest that this evidence base is far from conclusive and that much further work is required before our knowledge of the impact of home-visiting can be described as robust. In addition, the complex nature of home and health visiting (Gomby date; Elkan et al, 1999) makes this an evidence base that is not straightforward to implement and there were a number of ways in which Starting Well departed from the Olds model. These included:

• The targeting of deprived communities rather than vulnerable individuals;
• The inclusion of all new babies as opposed to only first babies;
• A lesser focus on the antenatal period due to caseload issues than recommended by Olds;
• The use of paraprofessionals as part of the home-visiting delivery mechanism in addition to professional health visitors;
• The vastly different primary care context within which the evidence was derived (for example, the absence of a universal health visiting service; and related to this,
• The requirement to integrate aspects of project delivery with existing professional and organisational structures as opposed to an entirely standalone intervention.

In assessing the degree to which demonstration projects have or have not applied evidence-based practice, the complexity of the application of evidence needs to be considered. For a range of contextual, methodological, practical and philosophical reasons it may not be appropriate to transpose evidence from one setting to another.

4.5.4 Achieving Professional and Organisational Change

A final reflection emerging from our evaluation and supported by experience across a range of similar projects is that there is a chronic lack of realism expressed in the commissioning and planning of projects when it comes to goals around professional and organisational change. Implementing new ways of working within and across professional boundaries; and, establishing meaningful community and partnership approaches should not be viewed as straightforward, uniformly supported or inevitable outcomes of delivering a project. Greater realism will be required to turn around well-established ways of working.

Notwithstanding the very real issues of design and implementation highlighted above, there is much to learn from the Starting Well experience. Although the commitment to improving the early years experience of the poorest children is not in doubt, the evidence base to guide effective action is less secure than once was thought (Elkan et al, 1999; Bull et al, 2004). This is particularly true of home visiting programmes in the UK. In these circumstances, the renewed emphasis on promoting social justice by reducing child poverty in all its forms, and the growing recognition of the importance of evaluating promising public health interventions exemplified by the second Wanless report (Wanless, 2004), suggest that the lessons to be learnt from Starting Well are important ones that should not be neglected.