The Scottish Government

« Previous | Contents | Next »

Listen

Final Report :The Independent Evaluation of Have a Heart Paisley

5: Methods

The independent evaluation consisted of four separate but linked approaches. A balanced perspective of the overall processes, impacts and outcomes of the demonstration project has been achieved through triangulating data uncovered using a variety of methods and from focussing on different levels (strategic, operational and participant) of HaHP across different timescales. The four approaches were:

the mapping of the social context within which the project takes place;

a quasi-experimental survey;

a theory-based approach; and

a range of integrated case studies using qualitative and quantitative methods.

Table 5 summarises the methods used and illustrates the stakeholders and levels that were the focus of each method. Each of these approaches/methods is then detailed. Fuller explanations of each of the methodologies and their respective findings can be found in the individual reports for these areas of work [Blamey, 2001/2003, Ayana, Blamey and Reid 2002, Ayana and Blamey 2003, Lawson, Paterson and Blamey 2002, Paterson, Blamey and Judge 2002. Paterson and Ayana 2003].

Ethical approval was received from NHS Argyll and Clyde's local medical ethics committee for the overall project and then subsequently for each of the separate strands of activity.

Table 5: Representation of the mixed methodologies applied

5.1: Mapping the social context

The findings from the literature review (Blamey 2002) highlighted the importance of understanding the context within which interventions are delivered. Context has a part to play in: determining the relevance of the interventions to specific population subgroups and their needs; establishing baseline measures for follow-up; and identifying secular trends and competing or overlapping initiatives that may confound or reinforce the interventions. These latter issues are vital to attributing correctly the effects of the interventions.

In order to understand the context within which HaHP was delivered, the evaluation team mapped a picture of the Paisley area (and the chosen comparison site of Inverclyde). This was done using a mix of existing secondary quantitative data and qualitative data sources, and conducting some additional primary data collection and secondary analysis for the early period of the HaHP project. It was not possible to do this prior to, or at the start of, HaHP as the evaluators were not commissioned until six months after HaHP was launched.

The range of data used in this process includes CHD prevalence data for Scotland and for relevant local areas where this was available, documentary review of existing research conducted in the relevant areas, additional primary data collection via the baseline survey (detailed below) and early interviews (n=16) carried out with key community informants within Paisley. The key findings from this contextual work are detailed in section 6.

5.2: The quasi experimental survey

A quasi-experimental survey was designed to detect changes that might result from HaHP at the population level. The methods used are detailed below.

A quasi-experimental (pre and post intervention) survey was conducted using a postal questionnaire with additional medical examinations. The survey was conducted on the intervention site of Paisley and a comparator site. The comparator site of Inverclyde was chosen as both it and Paisley were part of Argyll and Clyde NHS Board and had similar levels of CHD and socio-economic deprivation as Paisley. Information about programmes and services in the areas was accessible to the evaluators as Board employees were knowledgeable about activities in both Inverclyde and Paisley local authority areas. However, whilst these factors helped in terms of matching the two sites, their close proximity meant there was potential for contamination. The problem of contamination, however, was likely to be an issue for anywhere in Scotland and so the proximity of the site was thought to have more benefits than limitations.

The postal survey focussed on a range of demographic measures as well as general health, CHD related health, CHD related behaviours and risk factors, and other relevant information such as mental health and socio-economic influences on health. The baseline survey was also accompanied by a health examination that recorded objective health measures including blood pressure, total cholesterol and BMI (Paterson, Blamey and Judge 2002). Trained nurses completed the examinations according to recommended guidelines and following strict protocols.

As illustrated in Table 6, the original evaluation proposal aimed to randomly recruit a total sample of 1620 participants split between Paisley (n=810) and Inverclyde (n=810) from the Community Health Index (CHI). The aim was to draw this sample to provide three equal grouping (n=270 x 3) from neighbourhood areas based on deprivation categories (using the Carstairs and Morris DEPCAT scores). These groupings were group one -neighbourhoods with DEPCATS scores 1, 2,and 3; group two -neighbourhoods with DEPCATS 4 and 5 and group three -neighbourhoods with DEPCATS scores of 6 and 7. The samples drawn from within each of these groups were to provide equal numbers of males and females across a range of age bands (20-29, 30-39, 40-49, 50-59 and 60-69).

Table 6: Intended initial sample for the population survey

From the above cross sectional sample it was proposed that two cohort samples would be identified (n=210 x2) providing equal numbers of males and females across the stated age groups and neighbourhoods and that these individuals would be followed over time whilst a new cross sectional sample (n=600) would be drawn using the same process as at baseline for the rest of the follow-up (See Table 7).

Table 7: Intended target numbers for the cross sectional and cohort and follow-up surveys

A substantial amount of time during the first year of the independent evaluation was spent gaining ethical permission, recruiting appropriate staff, designing and piloting questionnaires and implementing the quasi-experimental survey. Confidentiality issues, which were particularly sensitive at that time due to relatively new arrangements with local Calidcott guardians, meant that those selected randomly from the Community Health Index (CHI) had to be invited through the Director of Public Health (DPH) to "opt in" to the survey before they could be directly contacted via the research team. This meant that the DPH sent a letter to the randomly selected sample from the CHI along with the study questionnaire. Subjects then completed and returned a card to the NHS Board indicating their willingness or otherwise to take part in any aspect of the study (e.g. completing the questionnaire only or the questionnaire and a medical). Those indicating a willingness to take part could only then be approached directly by the research team to arrange an appointment to attend a nurse led examination, or to return their questionnaire directly if they were not participating in the medicals. The examinations (where objective health measures were gathered) were conducted on alternate months in the control and intervention sites during the period from March to September 2001. Those attending a medical were asked to bring their questionnaires with them so they could have support to complete them, if needed, and so they could be checked over by the nurses for completeness. Those indicating a 'no' response received no other communication and those not responding received a second mail out of the invitation letter several weeks after the initial invite.

Because of expected problems in recruitment resulting from the 'opt in' procedure and the likely existence of survey fatigue in the Renfrewshire area due to similar studies having been conducted (the Boards own lifestyle survey and the Renfrew/Paisley study) it was agreed with Argyle and Clyde NHS Board that it would be necessary to over-sample from the CHI to achieve our intended sample size. In total 2720 individuals were randomly identified from the CHI corresponding to the age groups, neighbourhoods and genders that were required and were sent invitations to participate in the survey and medicals.

Flow chart A indicates the response rates to the invitation letters after the second mailing, and shows how the final participant numbers were achieved.

Flow Chart A: Response rates from the CHI after second mailing

The above flow chart indicates that the major drop-off in response rates was at the point of the initial invitation. Once individuals had agreed to participate (n=830) the response rate remained relatively high with 747 (90%) participating in some fashion.

Table 8 shows more information with regard to numbers of refusals, non-responders and issues with mailing that further illuminates the response rate after the second mailing. The 830 participants detailed above who indicated their willingness to participate are accounted for below in the rows marked with an *.

Table 8: Breakdown of decisions/responses received after second mailing

Table 8 confirms that the greatest loss of subjects was at the point of the initial invitation and enclosure of the questionnaire with 44% failing to respond after the second mailing, 18% stating they were unwilling to participate in any part of the study and 7.5% of addresses being wrong

In summary, despite substantial efforts, including over-sampling, the issuing of second invitations to non-respondents and repeated follow-up of those failing to attend appointments, the questionnaire survey only achieved a response rate of 28% and 27% respectively for the intervention and control sites. This gave a total of 743 respondents.

Whilst this is disappointing, response rates to other surveys have been relatively low. Argyll and Clyde NHS Board's own lifestyle survey conducted in 2001 which covered a wider range of health issues had a 39% response rate and did not require a medical examination.

Due to limited resources the original cross sectional sample in each of the three groups would allow a difference in the mean change between any two groups of 0.3 of a standard deviation to be detected with 80% power at the 5% significance level. The cohort sample was expected to provide greater power (e.g. to detect up to 0.2 of a standard deviation). Any loss to follow up would reduce this further. Therefore, even with a good response rate, the evaluation team were aware that small changes in many of the wide range of CHD related behaviours and risk factors that might result from the HaHP interventions might be missed. This subsequent poor response rate meant that our targets for each of the DEPCAT and age/ sex sub samples were not reached. It also meant that the evaluators could not have confidence that the survey was fully representative of the people, risk factor status and behaviours that they wanted to measure. As might have been expected the respondents are over represented in relation to older groups and lower DEPCATS than the Paisley and Inverclyde populations. On the more positive side the samples achieved in the intervention and control site were very similar. Both samples were similar in terms of age range, gender, levels of affluence and key measures of CHD risk factors and behaviours. (Paterson, Blamey and Judge 2002)

The Scottish Executive (SE) were kept closely informed of these issues and a series of discussions took place to consider various options (including boosting the Paisley and Inverclyde samples for the national Scottish Health Survey [SHS]) to supplement the baseline or to make maximum use of the samples achieved. Due to the similarities of the intervention and control samples it was agreed that they would be followed-up on a range of specifically chosen indicators at the end of the evaluation period. It was agreed, however, that follow-up would use all of the limited number of respondents as a cohort and that medical examinations and objective measures would not be repeated at the follow-up. The questionnaire was, therefore, shortened and the indicators selected focussed on areas where change was most likely to occur based on the formative evaluation findings. All key risk factors were still included but fewer questions were included. The Follow-up survey was conducted as late in the evaluation period as feasible. Even this timescale, however, meant that the cohorts were followed-up during the period of October-November 2003, leaving a gap of only about two to two and a half years from baseline. This is a very short period within which to expect substantial changes in entrenched behaviour and risk factors to occur.

Flow chart B, on the subsequent page, indicates the survey numbers and response rates to the follow-up survey.

Flow Chart B: Survey numbers and repose rates to the follow-up survey

In terms of the follow-up sample, twelve respondents who were not contactable when sent findings from stage one were removed from the study between 2001 and 2003. A further eleven individuals died between baseline and follow-up and another eleven had moved out of the area or were not contactable. A total of 709 individuals were, therefore, included in the follow-up survey. The follow-up survey achieved a response rate of 79 per cent and 78 per cent respectively for the intervention and control sites. In total 556 respondents were recruited for the follow-up study.

The above flow chart illustrates that the response rate for the follow-up survey was high (78.4%). The characteristics of the respondents and non-respondents were explored and the impact on the survey results is considered in section 7.1.2. The summary key findings from the quasi-experimental survey are detailed in section 7. A more detailed report is available for the baseline survey (Paterson, Blamey and Judge 2002).

5.3: Theory-based approach

The theory-based aspect of this evaluation has utilised a developing evaluation approach called the 'Theories of Change' that is promoted by the U.S. based Aspen Institute. Similar to other theory-based methods this approach stresses the importance of understanding and encouraging implementers to articulate the intervention plans and the underlying theories, rationales and logical links between activities, outputs and outcomes that support the plans. The implementers' explanations of the plans are then referred as their 'theory of change'. Several detailed reports/papers relating to the 'theory of change' process and learning from this aspect of the evaluation are available (Blamey 2001, 2002, Mackenzie and Blamey In press). Once articulated these theories can be investigated to establish the extent to which they are plausible (e.g. based on evidence and logic), do-able within resources and timescales, are testable (e.g. capable of being evaluated) and are meaningful (e.g. will result in relevant and useful outcomes). By aiding programme implementers and commissioners to articulate their theories about the intervention it is hoped the resultant 'theory of change' can help to refine and improve implementation plans, and can be used as framework to help prioritise evaluation issues and appropriate methods. By revisiting this framework over the period of the project learning can be derived about the extent to which plans have, or have not, been delivered as intended and the reasons for this.

This approach has involved the use of documentary review and initial interviews with the management group to articulate the programme plans and theories. These plans were represented as programme logic models for the overall and individual strands of the interventions. These logic models were discussed and further developed with operational staff via in-depth interviews. These interviews with 33 stakeholders were repeated on three subsequent occasions (i.e. 3 x 33 interviews over the course of the programme) and findings were supplemented by evaluator observation at Management Group meetings and a focus group with commissioners, as well as a range of ongoing group discussions with key stakeholders. This aspect of the evaluation (with agreement from implementers and commissioners) focussed predominantly on the extent to which HaHP delivered on cross-cutting issues such as; the use of evidence based practice, the achievement of community saturation, partnership working, community engagement, achieving agenda change and addressing inequalities. Summary findings from this aspect of the evaluation are detailed in section 8.

5.4: The integrated case studies of settings/organisations

The integrated case studies focussed on two settings (community and primary care) and on one organisation (the local authority) and utilised both qualitative and quantitative methods. They also involved stakeholders and participants at different levels within these areas (e.g. those involved at a strategic, operational and participant levels). The settings and methods used were further detailed in Table 5.

The original independent evaluation plans were revised in discussion with the commissioners and implementation team in response to the theory articulation process and the limited response rate to the baseline survey. Nevertheless, activity undertaken by the internal evaluation team overlapped with these plans. Decisions were made jointly in relation to what aspects of the independent evaluation would, therefore, not be fully delivered (see section 7.2). The focus of much of the evaluation activity in these settings was on identifying the extent of service development and the impact of that on professionals and/or agenda changes within these specific organisations. The greatest focus was, as a result, on the strategic and operational levels in these settings.

5.5: Analysis

The qualitative interviews and focus groups were all tape-recorded and later transcribed and analysed using a qualitative data computer analysis package, Atlas-ti. Notes were taken in cases (n=3) where interviewees did not wish to have their interviews recorded. The data were analysed according to predefined themes from the relevant literature (which had informed schedules) and emerging themes from the data itself.

The data from the surveys were entered onto databases and were analysed using the SPSS statistical package and a variety of descriptive and analytical statistical techniques (basic descriptive techniques, Chi Square test of association and paired T-tests).

« Previous | Contents | Next »