« Previous | Contents | Next »
Listen
CHAPTER TWO: METHODS
Introduction
2.1 The WMS initiative was a complex (multi-faceted), community-based, health promotion intervention (Hawe et al. 2004), of the kind where causation and attribution are often difficult, if not impossible, to establish (Wimbush & Watson 2000). The evaluation of such initiatives often has two major problems: unrealistic expectations of the evaluation and of the intervention itself (Nutbeam 1998). Therefore, such initiatives require mixed methods of enquiry which take account of the contextual and mediating variables likely to have an impact or effect on the outcome(s). Hence the project used qualitative methods to help to understand each project's structure and process, whilst at the same time collecting quantitative, standardised data for comparative analysis as outlined below:
- systematic literature review;
- secondary analysis of routinely collected men's health and health service use data in all pilot project health board areas;
- development and analysis of WMS pilot project monitoring framework ( MF);
- interviews with key informants associated with all pilot projects;
- case studies; and
- economic evaluation.
Systematic literature review
2.2 A scoping search identified existing systematic reviews of effectiveness of interventions aimed at improving men's health. A systematic search was conducted to identify studies in the English language since 1999. Further details can be found in a separate report (Robertson et al. 2006).
2.3 A systematic review of primary studies was also conducted in April 2006. The search strategy was designed to be sensitive enough to allow for diversity in the types of possible studies. Key journals were hand searched: Journal of Men's Health Gender, Men's Health Journal and International Journal of Men's Health. Reference lists of included studies were scanned. Relevant citations were downloaded into Refworks (2006). Primary studies focusing on men's health had to target men by intention. Studies were excluded if they were (a) part of the systematic reviews; (b) cost-effectiveness only studies; or (c) used historical controls.
Secondary analysis of routinely collected data
2.4 A secondary analysis of men's health and health service use data from all pilot areas was conducted to establish a baseline to assess whether projects were reaching their target population. Data were extracted from the Registrar General for Scotland, NHS Scotland Information Services Division ( ISD), Census 2001 and The Scottish Health Survey 2003. Information included, for example, age, educational attainment, healthy life expectancy, marital status, smoking status, GP and dentist attendance.
Development of WMS pilot project monitoring framework ( MF)
2.5 The WMS research brief required the development of a consistent framework for the collection of monitoring information on the operation and outcomes of the pilots. However, it became clear that a framework on its own would not support staff working in the pilots to collect useful or useable data for evaluation purposes, internal or external. Therefore, two data collection tools were developed to support the MF across all pilots. The first collected data from the projects on awareness raising and outreach activities and the second collected data from individual men who underwent a comprehensive health check.
2.6 The MF and the data collection tools were developed through extensive consultation with the various stakeholder groups as described by Douglas et al. (2006b). This process involved:
- formal and informal consultation with all individual project leads;
- various Research Advisory Group ( RAG) meetings;
- consultation with Scottish Executive Health Department ( SEHD) and Health Improvement Strategy Division ( HISD); and
- discussion with pilot project staff.
Data analysis
2.7 Quantitative data analysis was carried out on the WMS pilot project monitoring data. T-test & Chi-squared tests were used to describe details of service users.
Interviews with key service provider informants - all pilot projects
2.8 Semi-structured interviews took place with key service provider informants in two phases. Both phases were carried out in all 18 projects with interviews lasting between 20-60 minutes. Specifically designed interview schedules were used. However, the researchers explored any lines of enquiry thought to be of interest and relevant during the interview.
2.9 The first stage included 30 project staff and 14 partner agency staff. For the second phase, not all individuals were contactable, and one declined to take part again. The first phase interviews were predominantly face-to-face with the researchers visiting all project areas. All second phase interviews were conducted by phone. A thematic data analysis approach was employed (Gomm et al. 2000), supported by QSRNViVo (Gibbs 2002). The full evaluation report (Douglas et al. 2006b) presents verbatim quotes to illustrate the findings.
Case studies
2.10 Three individual projects were selected for case study investigation. The aim of the case studies was to capture and explore in more detail the operation and outcomes of a range of pilots that were generally representative of the types and locations of the services being piloted within the WMS project overall. Projects were selected on the basis of:
a. the type of service delivered by the pilot ( service delivery):
b. whether the service was new or a development of an existing service ( service development) and
c. whether the project area could be considered urban, rural or remote ( geographical location).
The case studies used additional interviews with, for example, project staff and (non-) users; mapping of supporting services; analysis of project documentation; and observation. The case study were projects in Highland, Fife and East Glasgow (Douglas et al. 2006a).
2.11 Documents studied included meeting minutes, bid documents and any reports. These were scrutinized for relevant data and to determine if they were consistent with other findings. Service mapping information, e.g. supporting structures and partnerships essential for service delivery, came from project bid documents, interviews and project documentation. Draft service maps were produced and checked for accuracy with project co-ordinators. Researchers observed projects, especially clinics, although it was not possible to directly observe a client consultation. Extensive notes were made during these observation periods and these data were used alongside interview data.
Economic evaluation methods
2.12 The economic evaluation considered the cost-effectiveness of various approaches taken in the pilot projects. Project findings have also been integrated with information drawn from the literature and other sources to provide a more complete model of costs and consequences.
2.13 Intervention costs came from projects' accounts and other documentary evidence supplemented by a questionnaire. The range of outcomes or consequences from pilots was potentially wide and diverse. The relatively short timescale of the interventions made it impossible to measure changes in final health outcomes, and information from the literature was used to model health effects. Data on lifestyle factors amongst WMS users, their willingness to change behaviour and the rate of referrals for interventions, were combined with information about the effectiveness of interventions to estimate health outcomes. This framework was also used to estimate the added value of the WMS based on assumptions about alternative provision, such as routine GP visits.
2.14 The value which men placed on different attributes of WMS was elicited directly by the use of a discrete choice experiment ( DCE) approach. A postal survey of a large population-based sample was carried out in an area where no WMS had been established.
« Previous | Contents | Next »