« Previous | Contents | Next »
Listen
WELL WHAT DO YOU THINK (2004): THE SECOND NATIONAL SCOTTISH SURVEY OF PUBLIC ATTITUDES TO MENTAL HEALTH, MENTAL WELL-BEING AND MENTAL HEALTH PROBLEMS
CHAPTER TWO: RESEARCH METHODOLOGY AND ANALYSIS
Questionnaire design and piloting
2.1 The survey was designed to collect data to inform policy development in all key areas of the National Programme's work and link with the broader health improvement and social justice policy agendas. The questionnaire was developed by a multidisciplinary advisory group comprising representatives from Scottish Executive Health Department and Finance and Central Service Department, NHS Health Scotland, The Royal College of Psychiatrists, Edinburgh University, The Scottish Development Centre for Mental Health and the 'see me' Campaign.
2.2 Specific topics covered in the questionnaire were as follows:
general health
length of residency in, and familiarity with, the local neighbourhood
social engagement and informal support networks
a measure of possible psychiatric morbidity
perceived positive and negative influences on mental health
control over factors affecting mental health
personal experience of mental health problems
the social impact of mental ill-health
recovery from mental ill-health
sources of information about mental health problems
perceptions of the portrayal of mental health problems in the media
awareness of adverts/promotions about mental health problems
familiarity with key campaigns and initiatives, including 'Choose Life', 'see me', 'The Breathing Space' telephone advice line and referral service, Mental Health First Aid and The Scottish Recovery Network
attitudes towards mental ill-health
the perceived prevalence of mental ill-health
attitudes towards specific symptoms of mental ill health
2.3 With regard to the latter topic area, scenarios depicting people with symptoms associated with depression, schizophrenia or stress were adapted from a study by Link
et al (1999) on public recognition of mental illness. Each scenario was constructed to meet the criteria of the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition, for the disorder in question and related to a man (called Robert) or a woman (Shona). Respondents were presented with one of the six scenarios and asked questions about the person in the scenario and the symptoms he/she was displaying. At the end of the section, respondents where asked to say what condition they believed the symptoms described.
2.4 In designing the questionnaire, it was important to ensure comparability with the 2002 survey where possible but to reflect advances in policy development (e.g. the focus on supporting recovery). The main changes, additions and refinements made to the questionnaire for the 2004 survey were as follows:
existing items on psychological well-being were replaced with the GHQ12 - a well-established survey instrument designed to detect possible psychiatric morbidity in the general population;
existing social engagement items were replaced with standard items used in previous research by MORI, the General Household Survey (2000/1) and the Health Education Monitoring Survey (1998);
response categories in the items focusing on positive and negative influences on mental ill-health were refined in light of findings from the 2002 survey and suggestions from the advisory group;
the diagnosis question in the scenarios section was revised. In 2002, respondents were asked how likely they thought it was that Robert/Shona was experiencing depression, schizophrenia and stress. In 2004, it was decided that respondents should be presented with a full list of mental health problems and asked to pick from this the most likely diagnosed mental health problem relating to the symptoms described;
a new question was added to measure the perceived prevalence of mental ill-health in Scotland. This was adapted from an item included in the Department of Health Survey,
Attitudes to Mental Illness in Great Britain (Taylor Nelson Sofres 2003);
three new questions were included to gauge respondents views on, and experience of, recovery from mental ill-health; and
another new set of questions was added to assess awareness of specific mental health campaigns and policy initiatives, including the 'see me' campaign, 'Breathing Space', 'Choose Life', Mental Health First Aid and The Scottish Recovery Network.
2.5 While it was important to make changes to the questionnaire, as detailed, it was also important that the length of the interview was kept to 30 minutes. In addition, certain questions worked better than others in 2002 so, after discussion, it was decided to exclude questions which could not be adequately explored in the context of this survey. The items removed focused on:
contact with GP;
drug use and alcohol consumption;
caring responsibilities;
factors seen to constitute good and bad mental health; and
priorities for government spending in the area of mental health.
2.6 The questionnaire was piloted among 21 members of the general public in two areas of Scotland, one urban and one rural. The full questionnaire for the 2004 survey is reproduced as Annex E.
Methodology
2.7 The survey was undertaken with a representative sample of the Scottish adult population (including the Highlands and Islands) and conducted face-to-face in respondents' homes between 24 May and 12 August 2004.
2.8 Households were sampled as follows:
Addresses were sorted into five groups by amalgamating NHS Health Board areas. These were:
Central Belt/West (Greater Glasgow, Lanarkshire and Forth Valley);
Lothian & Fife;
Borders & South (Borders, Ayrshire & Arran, Dumfries and Galloway);
Highlands and Islands (Argyll & Clyde, Highland, Western Isles, Orkney & Shetland); and
North East (Tayside, Grampian).
2.9 The target number of interviews was allocated to each area in proportion to the adult population of the area.
Within each area, Census Enumeration Districts (EDs)
3 were selected with probability proportionate to population to provide clusters of addresses within which interviewers would work. A total of 105 EDs was selected.
Within each of the sampled EDs, 25 addresses were drawn at random. This assumed that an average of 10% of addresses would be vacant or non-residential and at the 90% of valid addresses around 20% of households would refuse to participate and around 15% would not be contacted. Thus, we anticipated a response rate of 65% and a total of around 1,500 achieved interviews.
In advance of the survey fieldwork, a letter from the Scottish Executive was sent to each sampled address, providing essential information about the survey.
Interviewers made at least six calls at each sampled address including calls at the weekend and in the evening.
Interviews were conducted with an adult member of the household (aged 16 and over) selected at random.
2.10 All fieldwork was conducted using Computer Assisted Personal Interviewing (CAPI). where data is collected on lap top computers. The main strength of CAPI is that interviewers do not need to check questionnaire routing, which leads to improved data quality and avoids the need for a separate data entry process. This also allowed the GHQ12 section to be set up for self-completion, mimicking the pen-and-paper self-completion method traditionally used to administer the GHQ questions. Almost all respondents completed the GHQ12 module (93%, the same as MORI achieved in the 2002 Scottish House Condition Survey, which also incorporated the GHQ12 module, but using pen and paper self-completion).
2.11 The target number of interviews for the survey was 1,500 and the total number of addresses allocated was 2,625. In the event a total of 1,401 interviewers were achieved, a shortfall of 99 interviews (or 6.6%). This was in spite of two waves of sample re-issue, in which addresses at which no contact had been made and any 'soft' refusals were re-allocated to interviewers to work again. While this process delivered a conversion rate of around 20%, this was not sufficient to make up the shortfall.
2.12 The 2002 survey also fell short of the number of interviews targeted. In that survey, a total of 1,381 interviews were achieved from an allocated sample of 2,880 addresses, in other words, 20 fewer interviews from 255 more issued addresses.
2.13 In 2002 the shortfall appeared to be due to fieldwork being conducted between July and September -a relatively brief period for a substantial survey. Consequently, the non-contact rate was relatively high, at 19%. This compares with 12% in the 2003 Scottish Household Survey and 11% in the 2003 Scottish Crime Survey, both conducted by MORI. The refusal rate was also relatively high in the 2002 survey, at 23% (in the 2003 Scottish Household Survey and 2003 Scottish Crime Survey the comparable figures were 17% and 14% respectively).
2.14 In the 2004 survey, fieldwork was conducted over a longer period, from mid-May to early August, so there should have been little problem in re-issuing addresses. Additionally, the refusal rate in 2004 was considerably lower than for 2002 and more in line with what we would expect for this type of study, at 16%. However, as in 2002, the level of non contact at sampled addresses was fairly high, at 19%, and this was the main factor underpinning the shortfall in the number of interviews achieved. This was in spite of multiple calls to the sampled addresses at evenings and weekends. Table 2.1 shows the number of calls that were required to achieve an interview.
Table 2.1: Call at which an interview was achieved
| Call 1 | Call 2 | Call 3 | Call 4 | Call 5 | Call 6 | Call 7 | Call 8 | Call 9 | Call 10+ |
Interview at call | 30% | 24% | 18% | 11% | 7% | 5% | 1% | 1% | 1% | 2% |
2.15 This shows that although 95% of interviews were achieved within the required six calls, 5% of addresses required more calls than this to achieve an interview.
2.16 To assess the extent of bias in the sample, it is important to consider whether people who did not take part in the survey differed in socio-demographic terms from those who did. At addresses where respondents refused to participate in the survey, interviewers recorded reasons for refusals as well as a limited amount of estimated socio-demographic characteristics of 'refusers'.
2.17 This data shows that 28% of those who refused said that they were too busy or they were always busy while 15% were put off by the subject matter and 4% said they were just not interested. In terms of the profile of refusers, 15% were from elderly adult households, while 11% were from families with children and 12% were classified as having some 'other' background. For the remaining refusers, no profile information was coded by interviewers. In large-scale surveys of this kind, MORI consistently finds that elderly households are among those most likely to decline to participate, usually because they feel that surveys are not relevant to them or that they are not well enough to take part.
2.18 The tendency for refusals to be higher among some groups of respondents than others (and we can only assume that this is the case based on the limited amount of profile data recorded by interviewers) does not appear to have introduced any notable bias into the sample. There are two main points to note in this respect. First, in spite of the relatively high level of refusals among elderly households, the survey in fact over represents elderly age groups and under-represents younger groups. The data were weighted to reflect this and to bring the achieved sample profile more into line with that of the general population. Second, and as the table below illustrates, the weighted profile of the sample is in line with the unweighted profiles of other large scale national Scottish surveys conducted by MORI over recent years, including the 2003 Scottish Household Survey.
Table 2.2: Sample profile
Sample Profile
| Mental Health 2004 (weighted) | Scottish Household Survey
2003 |
Base: All respondents | (1,401) | (13,968) |
| % | % |
Male | 47 | 45 |
Female | 53 | 55 |
16 to 24 | 14 | 11 |
25 to 34 | 16 | 14 |
35 to 44 | 21 | 19 |
45 to 59 | 25 | 26 |
60 to 74 | 18 | 21 |
75 + | 8 | 9 |
Higher managerial and professional occupations | 11 | 11 |
Lower managerial and professional occupations | 24 | 27 |
Intermediate occupations | 11 | 14 |
Small employers and own account workers | * | 4 |
Lower supervisory and technical occupations | 9 | 12 |
Semi-routine occupations | 19 | 17 |
Routine occupations | 19 | 15 |
Source: MORI
2.19 Further information on the survey administration, including response rates etc. is given at Annex A.
2.20 In 2002, a booster sample of 100 Black and Minority Ethnic (BME) people was included in the study design. In the event, only 51 BME interviews were actually achieved, meaning the amount of analysis that could be conducted on the data was limited. In the run-up to the 2004 survey, consideration was given to the possibility of repeating the booster exercise using a larger sample. However, the Executive, in consultation with MORI, decided against this approach, as the primary focus of this survey was on a nationally representative sample and considerable effort would have been needed to achieve a booster sample that could be used for robust analysis of the non-white population. Quite apart from this, it was felt unreasonable to assume that issues pertaining to the non-white population with regard to mental health are necessarily different from those among the white population. In the event that such differences
do exist, it may be equally unreasonable to assume that the non-white population is homogenous in this respect.
2.21 In our sampling strategy we ensured that BME groups were represented proportionate to their number in the population (According to the 2001 census, minority ethnic groups currently comprise around 2.1% of the Scottish population). In addition to this, we ensured that our interviewers were fully briefed on the importance of ensuring high response rates and the significance of the non-white population. MORI surveyed 26 respondents from the non-white population. As the ethnicity profile in table 3.2 shows, the proportion of non-white respondents surveyed in this study is in line with the 2001 census.
2.22 Eighty-five per cent of respondents said that they would be willing to be re-contacted for further research by the Scottish Executive. This compares with 75% in 2002.
Analysis
2.23 Prior to reporting, the survey data needed to be weighted to account for the fact that only one person was interviewed in each household. This meant that adults in multi-adult households had a lower chance of participating in the survey than adults in single adult households. Without weighting, the data would be biased.
2.24 Analysis of the data showed that the interviews were not proportionately spread across NHS Boards and there was some evidence of differences between the age-sex profile achieved sample and what was expected. Population distributions and age-sex profiles from the 2001 Census data were used to weight the data to correct these imbalances.
2.25 Computer tables were prepared to a specification agreed with the Scottish Executive. In the tables, responses to each question were analysed against a number of key variables, namely:
Sex
Age (7 groups - 16-24 through to 75+)
Age interlocked with sex (6 groups - 16-34, 35-54 and 55+)
Ethnicity (2 groups - White and non-White)
Social capital scale (4 groups - most engaged to least engaged)
Social Class - based on National Statistics Socio-Economic Classification
4 (NS-SEC) (8 groups)
Working status (2 groups - Full time and Part time)
Household income (5 groups - ranging from less than 5,200 per annum to 36,400 or more)
Ease of managing on income (3 groups - easy, manageable, difficult)
Affluence of area (5 bands - most to least affluent based on the Index of Multiple Deprivation (IMD)
5 which uses 2001 census data)
Qualifications (4 groups ranging from no qualifications to professional qualifications)
Urban/rural split (6 groups ranging from most urban to most rural)
NHS Board Area (5 groups - Borders and South, Central Belt West, Highlands and Islands, Lothian and Fife, North East)
Long-standing illness/disability/infirmity (3 groups - Limiting condition, non-limiting, none)
Experienced a mental health problem (3 groups - A problem of their own, someone close with a problem, no contact at all)
General Health (2 groups - good and poor)
2.26 In addition to these basic 'cross-breaks', multivariate analysis was conducted to explore the strength of relationships between variables of particular attitudes and experiences. The analysis included Segmentation and CHAID (Chi-squared Automatic Interaction Detector) analysis.
2.27 Segmentation analysis is a way of simplifying survey questions into a smaller number of themes or 'factors', by grouping together items that are answered in similar ways. The process involves
factor analysis to look at common themes, followed by
cluster analysis to segment the sample into groups based on these themes. The demographic composition of each of the typologies can then be analysed.
2.28 The CHAID analysis was used to test the strength of relationships between attitudes towards, and experience of, mental ill-health and a range of socio-demographic, behavioural and attitudinal variables. CHAID analysis segments a population into two or more groups, according to their tendency to exhibit a particular characteristic. It then continues to split these groups until no more statistically significant differences are found. CHAID uses statistical techniques (based on chi-square analysis to select the key drivers (such as gender, age or particular attitudes) of the characteristic and, for these key drivers, identifies which groups of respondents are the most likely to exhibit the characteristic (A full discussion of the secondary analysis techniques adopted can be found in Annex G).
2.29 As noted in the technical appendices to the report (see Annex B) survey respondents represent only a sample of the total population. All survey results are subject to sampling variability which means that observed differences between sub-groups may not always be statistically significant i.e. they may have occurred by chance. Throughout the report, differences between sub-groups are commented upon
only where these are statistically significant - where we can be confident that such a difference is unlikely to have occurred by chance. Additionally, where data is presented in table format, significant differences between results for different sub-group categories, e.g. for men and women or for those aged 18-24 and those aged 75+, are emboldened. To avoid confusion, significant differences between comparable findings for 2002 and 2004 are not highlighted in this way, but rather are discussed where applicable in the text of the report. The formula used for calculating significant differences and a guide to statistical reliability is appended in Annex B.
2.30 Where percentages do not sum to 100%, this may be due to computer rounding, the exclusion of 'don't know' categories or multiple answers. Throughout the volume, an asterisk (*) denotes any value of less that half a percent.
2.31 It is important to note that the findings presented throughout this report are based on what people
say about their attitudes towards, and experiences of, mental health problems and related issues. It may be that some respondents have chosen not to reveal particular information, for example, that they have experienced a specific condition or that they hold negative attitudes towards mental ill health. This point should be borne in mind at all times when interpreting the data.
« Previous | Contents | Next »