The Scottish Government

Listen

Have A Heart Paisley Executive Summary
Independent Evaluation, March 2005

A NATIONAL HEALTH DEMONSTRATION PROJECT
June 2004

ISBN: 0-7559-4574-3

This document is also available in pdf format (336k)

Contents

Introduction
Findings
Conclusions

Avril Blamey,
Mulu Ayana,
Louise Lawson,
Jane Mackinnon,
Iain Paterson,
and Ken Judge.
Public Health and Health Policy
University of Glasgow
1 Lilybank Gardens
Glasgow G12 8RZ

Introduction

HaHP was established in October 2000 with a £6 million grant from the Scottish Executive. As one of four national demonstration projects it was to be a 'test bed and hot bed' - an attempt to make an unprecedented impact. The long-term aim of HaHP was to reduce the total burden and levels of inequality of Coronary Heart Disease (CHD) in the town of Paisley through an integrated programme of secondary and primary prevention. HaHP consists of fifteen linked work strands. The intention was to deliver interventions in partnership, engaging the community at all levels of the programme.

HaHP is a strategic partnership between NHS Argyll and Clyde, Renfrewshire Council and the local community and voluntary organisations. At the time that HaHP was commissioned NHS Argyll and Clyde was represented by the separate entities of the Acute Trust, Renver Primary Care Trust, Paisley LHCC and the Public Health Department from the Health Board.

THE EVALUATION

The independent evaluation covered the period March 2001 to March 2004 and was conducted by a team of researchers based at the University of Glasgow, supported by grantholders from a range of institutions. ¹ The evaluation had a range of aims and objectives. The key aims were to:

• describe the HaHP intervention (e.g. its components, rationales and processes) and evaluate its delivery;
• evaluate the impact of HaHP (in terms of its intermediate outcomes and potential for delivering on long-term outcomes); and
• establish the policy lessons that have arisen from the evaluation.

The evaluation consisted of four separate but linked approaches:

• the mapping of the social context within which the project took place;
• a quasi-experimental population survey and follow-up (using Inverclyde as a comparator area);
• integrated case studies of key settings (primary care, the community and the local authority) using qualitative and quantitative methods; and
• a theory-based /process evaluation to capture programme plans and the extent of their delivery.

The independent evaluation, due to limited time and funding, was necessarily focussed on certain aspects of the HaHP intervention. Other aspects of the demonstration project have been evaluated by internal evaluation officers and by staff in the various work strands. The reports from these other aspects of the evaluation are currently, or will be, available through the CHD Learning Network [ http://www.phis.org.uk/projects/default.asp?p=fb].

Context

Eight of Paisley's eleven postcode areas have deprivation levels that are higher than the Scottish average and Paisley's most deprived locality is the tenth most deprived postcode sector in Scotland. Some of the more deprived areas have experienced substantial population decreases between 1981 and 2001. Many community facilities are in need of upgrading and in some areas access to shops and healthy foods are limited. Similarly crime and drugs are major problems for some localities.

There are substantial health inequalities within Paisley and CHD mortality is strongly associated with increasing deprivation. However, CHD mortality is decreasing in Paisley and Scotland. Paisley has been the focus of many previous attempts at regeneration and a designated Social Inclusion Partnership currently operates within the Paisley boundary.

There have been a variety of other local and national CHD/health related activities that have been delivered during the lifespan of HaHP. These include activities funded by both government and non-government organisations (NGOs), for example, the New Opportunities Funding for sport /PE and CHD, Better Neighbourhood Services Funding, Quality of Life Funding, Hungry for Success, Integrated Schools Funding, the development of Managed Clinical Networks, the General Medical Services contract and the establishment of Community Health Partnerships.

These many influences are likely to have impacted on health generally, and CHD related health specifically, within Paisley making it difficult to attribute any changes in heart health outcomes to the HaHP intervention per se.

Findings

POPULATION SURVEY

The main aim of the population survey was to detect any changes in CHD related disease status, key risk factors or behaviours at a population level. Despite the best efforts of evaluators the cross-sectional baseline survey attracted a low response rate (28%) and resulted in a sample that failed to reach targeted participant numbers for deprivation categories or age-groups. The baseline sample was, therefore, biased in relation to age and deprivation and consisted of greater numbers of older and more affluent individuals. Although biased, the samples from Paisley and Inverclyde were very similar. Initial plans to select both cross sectional and cohort samples were changed and the survey compared only cohorts from Paisley and Inverclyde residents at baseline and follow-up.

The follow-up postal survey of those who responded to the baseline questionnaire was completed in November 2003 (omitting clinical measurements due to the limited response rate) and obtained a seventy eight percent response rate. This sample was further biased in favour of older and more affluent individuals and non-smokers. These problems further limited the usefulness of the survey.

An analysis of paired data for those individuals who had responded to both surveys (n=556) illustrating the extent and direction of changes in key variables between the Paisley and Inverclyde samples was conducted and only one variable was identified as showing a significant association. This showed a significant change in knowledge of the number of portions of fruit and vegetables that should be eaten each day to stay healthy. Although knowledge has increased in both areas, a greater positive change in knowledge was found in the Inverclyde sample compared to the Paisley sample. No other significant associations were found.

A further comparison of the direction and magnitude of changes found between those in
the Paisley sample who had engaged with HaHP compared to those who had not was conducted. Again only one significant association was found. This indicated that those engaged with HaHP reported a greater improvement in the numbers of portions of vegetables eaten per day than those not engaged with HaHP. No other associations were found to be significant.

It is important to note that the evaluators cannot say with confidence that there were only the two above associations. Other associations may have occurred that have not been measurable as a result of the various limitations of the survey.

INTEGRATED CASE STUDIES

The integrated case studies focused on two settings (primary care and community) and one organisation (local authority). The focus of this part of the evaluation was largely on the extent of service development and the impact of HaHP on professionals and/or agenda change, at both strategic and operational levels.

PRIMARY CARE

HaHP was slow to get started on the ground in primary care but the changes that it facilitated have accelerated both secondary CHD prevention and CHD recording systems. These developments have put the Paisley LHCC in a position to respond quickly to the requirements that are now being promoted across Scotland by the new GMS contract.

HaHP has managed to bring about some useful practical changes within the primary care setting. These included the establishment of CHD registers in all practices, the facilitation of secondary prevention clinics in all practices and the development and implementation of the CHD patient pathway. Community nurses have developed a defined health-promoting role and pharmacists have become more involved in prevention activity (such as the promotion and provision of smoking cessation services). Partnerships with secondary care have been greatly improved and in the latter stages of the project more links were being made with the local authority education department and the community.

In many ways, however, HaHP has remained on the periphery of the primary care agenda. Whilst there have been individuals in primary care very committed to the HaHP agenda, both at strategic and operational levels, the project has not been adopted across the LHCC in a consistent way. For example organisational dynamics, such as limited financial incentives and GPs being independent contractors, militated against greater success and penetration of the project amongst GPs. Opportunities for involvement, through multi-disciplinary training, primary prevention and developing links with the wider community, were not fully embraced by GPs. Similarly strategic areas of work such as the implementation throughout the LHCC of the Health Promoting Health Service Framework have had limited impact to date.

COMMUNITY

HaHP has funded 143 community projects and has furthered community capacity at an operational level within the locality areas. Whilst many of these projects existed in some form prior to HaHP (30% thought to be new groups) the money has been used to extend or develop community activities. Many individuals have gained personally from their involvement as participants or community volunteers within HaHP. The locality team faced a number of barriers at the outset such as community reticence due to previous negative experiences of community-targeted projects. They have worked hard at building capacity in the community and there are many positive reports of the initial and ongoing support they provided.

Monitoring information and demographic details were not systematically collected by the internal team, which prevents conclusions from being drawn on whether the appropriate target groups attended or adhered to the community activity. Little can, therefore, be stated about the quality, intensity and reach of the community interventions.

Community groups viewed HaHP primarily as a funding source. Long-term sustainability of community projects is questionable due to budgetary constraints in the statutory agencies. If projects are not self sufficient, they may end up tailoring their activities to the aims of future funding initiatives rather than maintaining their heart health focus.

The project has been less successful at engaging and sustaining the involvement of community representatives within the more strategic HaHP groups and mechanism. The findings suggest, perhaps not surprisingly, that community representatives worked best in the strategy group that dealt with the direct funding of community run projects rather than in more clinical, planning and topic development roles.

LOCAL AUTHORITY

There were four local authority projects; Healthy Eating Active Living (HEAL), which provided healthy eating and menu advice to catering managers in community care establishments; Health at Work, a CHD screening programme for a small number of employees; Healthercise, a gym-based exercise programme and a community-based guided walking initiative; and the Health Promoting School Programme.

The local authority (LA) was slow to respond to the opportunities afforded by HaHP. In the context of mainstream and other short-term funds being targeted at the LA, HaHP funding was relatively small. As a result HaHP was mainly used at a project and operational level, rather than to encourage structural changes to core services or the development of new services and policies.

Many LA projects remained isolated within the service that they were placed or within the wider local authority. There is, however, evidence that some aspects of the projects have the potential to influence their wider service area and the development of policy.

• There is potential to stimulate the development of a council wide nutrition policy from the HEAL project.
• The Health Promoting Schools programme was the largest investment of all four LA projects. This funding was used in different ways by the various educational establishments, mainly for project-based activity rather than strategic or long-term policy change. It is likely, however, that the HaHP school-based work, backed by the national activity in relation to this area, has helped to influence the development of a new strategic post within the Local Authority Education and Leisure Directorate to integrate the management of a range of health and lifestyle funded programmes.
• Healthercise has also contributed to the need for this latter post and to the development of a health focussed access officer working across Planning and Leisure.

On the whole HaHP has remained on the periphery of LA work and made little contribution to the wider community planning agenda. Again the impact of the four projects on individuals cannot be established due to a lack of, or appropriately presented, internal monitoring data.

THEORY-BASED EVALUATION

The theory-based aspects of the evaluation focussed on the extent to which HaHP has successfully delivered its overall plans, in particular the cross-cutting mechanisms that HaHP thought were crucial to success. The mechanisms related to the extent to which HaHP:

• applied both evidence-based and innovative practice;
• improved partnership working to jointly deliver synergistic programmes;
• fully engaged the community at all levels of the programme;
• achieved agenda change in the key agencies responsible for service delivery;
• saturated Paisley with new and expanded opportunities to motivate behavioural and cultural change that reduced CHD; and,
• addressed health inequalities in relation to CHD.

Initial plans for HaHP were overly ambitious and as a consequence timescales for delivery were lengthened and expectations with regard to outputs and outcomes reduced. In most instances, therefore, the projects did not fully achieve their initial goals.

HaHP was to lead CHD prevention development and create lessons for the rest of Scotland. This task, however, was hampered because, at the start of HaHP, developments on the ground were less well advanced in Paisley than in other NHS Argyll and Clyde areas, or elsewhere in Scotland. For example Argyll and Clyde NHS staff indicated that smoking cessation services were more advanced in Inverclyde than Paisley and 'Phase Four' cardiac rehabilitation services and pre participation screening for exercise amongst high-risk clients was already in place within Greater Glasgow.

Tension existed in HaHP between developing innovative practice and applying current evidence. It was not clear to the project whether they were to demonstrate the efficacy of new interventions, to apply evidenced-based interventions in a wider arena (i.e. to test effectiveness), or to test the effectiveness of the combined and integrated approach.

The concept of evidence-based practice was applied at the conceptual level across most of HaHP; however, there were areas of operational practice where the use of evidence or 'best practice' was limited. In some instances projects targeted inappropriate target groups or were not expressed in a manner that allowed them to be evaluated. An example of this was the focus of the HEAL project on promoting fruit and vegetable consumption and physical activity amongst a frail elderly population. Whilst this is an important target group in general terms it is not a key focus for CHD prevention. Despite these issues being highlighted by formative evaluation, few plans were adapted to tackle these problems.

There have been substantial improvements in partnership working and relationships across HaHP. In particular strong relationships have been developed between primary and secondary care and are beginning to emerge in relation to the community. Similarly some of the local authority projects have forged partnerships with key HaHP and NHS staff, however, there is still scope for greater joint working and actual joint delivery between the local authority and the NHS at both the strategic and operational levels. Leadership and personality issues have exercised both negative and positive influences on aspects of HaHP's development.

Whilst brand awareness of HaHP is high, and the project has engaged substantial number of community volunteers, engagement at the participant levels is harder to determine due to poor monitoring. HaHP has had only limited success at engaging the community at the strategic level.

HaHP has not yet succeeded in influencing the agendas and policies of statutory agencies, in creating public advocacy for change in health related policies or in adapting Paisley's environment to become more health enhancing.

There has been a lack of clarity over the types of inequalities that are being reduced through HaHP and the contributions of different project strands to this objective. This is partly as a result of limited baseline data being available about service access, but is also due to a lack of agreed criteria for detailed targeting or approved measures of change.

The internal evaluation of HaHP experienced a range of difficulties. Some of these problems related to the delay in commissioning the independent evaluation and the consequent delay in agreement and publication of the key focus of the inter-related studies along with the limitations of the survey. However, the internal evaluation process lacked a clear management structure and there were many problems in addition to these factors. These included: problems in recruiting and retaining staff; the lack of relevant data that can be disaggregated to a local level for use as a baseline; and delays in prioritising the key focus of several aspects of the internal evaluation work (even after there was agreement over the areas to be covered by the independent evaluation). HaHP struggled to differentiate between the concepts of internal monitoring and evaluation and were slow to develop appropriate monitoring procedures for key aspects of their programmes. As a result little can be evidenced about the reach and impact of individual or combined projects and the saturation of Paisley by HaHP.

The exception to this is the secondary care aspects of the project that will in time have data on reach and impact. Existing and future internal evaluation reports can be accessed through the CHD Learning Network http://www.phis.org.uk/projects/default.asp?p=fb.

Conclusions

There is much to learn from the implementation of HaHP. Those involved in HaHP have committed enormous time and effort (many on top of their already busy day jobs) over the last three years to deliver a range of new projects and services within the town of Paisley. This activity has expanded existing services to reach new target groups (e.g. the smoking cessation services) and/or have remodelled services (e.g. the care pathway), delivered refurbished facilities (e.g. the rehabilitation service) and increased community capacity for preventing CHD (e.g. the community programmes). Secondary and primary care have overcome substantial barriers such as professional reticence, technological difficulties and data protection problems to devise mechanisms that have substantial potential to improve care and treatment for those at high risk of CHD. In addition a range of new partnerships and jointly delivered projects have been developed and implemented.

The evidence base for the secondary care activities is well established and these programmes are likely to lead to measurable change amongst those targeted. The CDR register has substantial potential to influence treatment patterns in both secondary and primary care and to identify where individuals at risk of CHD are going untreated. Early technology problems have now been overcome and the register could soon be used in a proactive fashion to identify, recall and encourage improved treatment.

The many testimonies from enthusiastic participants in the HaHP services demonstrate that those who participated have received enormous support and encouragement in their attempts to tackle their health issues. However, despite the establishment of this range of activities and the undoubted power of personal testimonies, there is limited evidence that indicates that HaHP has managed to achieve a shift in total CHD risk or in key risk factors or behaviours at a population level, or amongst key targeted sub-groups.

There is also much to learn from areas that remain problematic or have been less successful.

Whilst it cannot be totally ruled out that change in CHD risk status or factors has occurred and gone unmeasured, the detailed process information that has been gathered by the independent evaluation suggests that HaHP has suffered from many of the difficulties and repeated many of the mistakes of previous CHD prevention programmes. Within the timescales of this evaluation they have been unable to substantially influence mainstream policies and agendas and have tended towards more individually focussed interventions. Further examples of these problems are explained below.

In its initial plans HaHP had hoped to implement policies (such as smoking and nutrition policies in workplaces) and influence wider agendas and services of key agencies through advocacy and partnership working. However, the process evidence shows that stakeholders themselves believe that this has not been achieved to any substantial degree. In a similar way to other previous CHD projects, it seems that HaHP has tended towards more individually focussed interventions. This is likely to have resulted from a lack of support for local policy change resulting from such issues not being addressed at national levels, or as a result of the substantial barriers to achieving local policy development (limited influence over senior stakeholders and statutory agencies or lack of influence over local retailers and workplaces).

The available process and outcome evaluation, and limited monitoring information, would suggest that HaHP has not been able to saturate the town of Paisley. Whilst brand awareness is high, the limited available data suggest that this has not yet been translated into large-scale regular participation in intense enough activities to achieve population or sub-group change. Coupled with these problems of reach and intensity, the actual effectiveness of some programmes was also limited or questionable.

Even in the relatively successful secondary prevention sphere there are still barriers to be overcome. The numbers targeted by the rehabilitation programme are limited due to the capacity of existing staffing levels. If successful, this service may be difficult to roll out elsewhere due to the substantial staffing and refurbishment costs. Similarly, the capacity of the CDR register to identify unmet need and improve the treatment of patients is substantial but will rely heavily on the cooperation of GPs and primary care staff and on the continued availability of NHS pharmacological budgets to support the prescriptions of the most effective drugs (as identified by NHS QIS standards).

Many of the problems around the use of evidence, intervention intensity, and scope for saturation, had their roots in the early planning phases of the project and resulted from a lack of early scrutiny from both the SE and HaHP of the range of plans submitted by different agencies. Again many of these potential problems were highlighted early in the evaluation process but were not fully tackled.

REFLECTIONS

The above findings and conclusions, combined with other recent literature from the evaluation of complex community initiatives, indicate that there is a need for government and NGOs to review the role of short-term funding for pilot initiatives and demonstration projects.

Such projects are based on the assumption that ring-fenced monies given to partnerships, with specific restrictions and accountabilities attached, can lead to desired change being delivered in specific areas of government policy. It is presumed that the agencies involved in these partnerships have both the will and the capacity to utilise these funds to create innovative projects that will deliver dramatic results in limited timescales for issues that have been long-term intractable problems for central government.

The feasibility and appropriateness of such expectations need to be addressed. Each project and commission sets out with the best of intentions. However, rarely is time taken to consider the range of other tasks and projects (with clashing, overlapping or additive demands) undertaken by the same range of beleaguered partners. Neither commissioners nor implementers stop to consider the full range of evidence that should inform these projects if they are to succeed and in many instances existing evidence of what works is limited or is not in a format that is easy to interpret and apply to local contexts.

There are many lessons that can be taken from HaHP that should influence any future short-term funded initiatives.

• Short-term population interventions are unlikely to reach enough members of the public, to deliver intense enough interventions or to fully engage enough community members in their design delivery or evaluation, particularly if they fail to allow adequate time for consultation of evidence and effective planning.
• Projects should have fewer, more realistic and more focussed aims and should select interventions that allow saturation in specific topics or target groups within the set timescale. More work needs to be done on how to present existing evidence in a manner that is useful to such community-based initiatives and on further developing evidence about 'how to intervene' rather than 'on what'.
• Change in the agendas of the key agencies are unlikely to be achieved through such interventions unless the project has the capacity and political backing to capture the attention of very senior agency staff and local politicians. To achieve this, projects require high quality leaders with relevant domain knowledge and a track record in that local area or a strong reputation from similar work elsewhere. Such posts are likely to require greater authority, formal influence and remuneration than they tend to receive. Even then such projects will require substantial time commitment from senior agency staff which is unlikely to be feasible unless their existing demanding roles are adjusted.
• Consideration of a range of complex community-based project documentation and evaluations would indicate that there are major capacity issues amongst public sector agencies with regards to integrating effective planning, monitoring and evaluation. In a world where increasing funding is being released to pilot and test interventions, further support for, and development of, these skills amongst key managerial and operational staff is vital.
• It is crucial that when pilot initiatives are launched that all parties are clear about expected lessons. Terms such as 'demonstration', 'innovation', and 'evidence-base' require rigorous consideration and clear articulation from the outset. When projects are implementing a range of different initiatives to achieve higher-level outcomes, clarity is needed over whether projects are required to roll out existing effective practice or to develop new practice, and whether work strands or the combined intervention is what is being tested. If combinations of these things are required the evaluation resources and expertise need to reflect such demands.
• If dramatic change is expected in short timescales, then it is likely that partners and agencies will have to take risks and attempt innovative service changes. This strategy is highly unlikely to be successful unless central government suspends normal accountabilities, or potential restrictions (such as temporarily underwriting or suspending commercial contracts for areas such as school nutrition) and is prepared to countenance failure. The increasing focus on local accountability and performance management is likely to be counterproductive to such risk taking unless expectations are re-negotiated between partners and commissioners. Similarly there are expectations that local agencies can deliver on agendas that central government will not address itself, such as major areas like nutrition retail policy and smoking policies. The solutions to these issues are more likely to lie within national than local policy.
• It appears increasingly evident that dramatic change will not be achieved by such interventions unless they are given feasible timescales and genuine national support to deliver and to fully engage local communities. The types of changes achieved in the more successful of the CHD or chronic disease interventions projects resulted from intense activity with small high-risk sub-groups or from long-term community advocacy to tackle upstream policy issues with the direct support of central government.
• It is vital that evaluation is considered from the outset in any future pilot initiatives and that priorities for key areas of evaluation are agreed and monitoring processes subsequently focused towards these priorities. It is also important that internal and external evaluation roles are clearly defined and that monitoring is seen as the responsibility of those running programmes.

In conclusion, HaHP may be more successful in its subsequent phases if it redesigns its programme to focus on fewer headline objectives, and attempts to address these through interventions that have a strong evidence-base. During its transition phase, the project has been attempting to address these issues and intends to utilise the criteria in the RE-AIM framework (i.e. reach, efficacy, adoption, rigorous implementation and maintenance) to select appropriate interventions for its integrated programme [ http://www.re-aim.org]. The project is also considering means by which it can address more upstream interventions. Such a programme needs to be 'aspirational' but without being over ambitious or unrealistic with regard to resources and timescales.

Footnote

1 The Grantholders for this evaluation are based at the University of Glasgow, NHS Greater Glasgow and the University of Paisley.