The Scottish Government

« Previous | Contents | Next »

Listen

4. Evidence of bias

In this chapter we develop the previous chapter's conclusion about the scope for bias in the surveys by examining each for evidence of bias. We compare the sample characteristics of the telephone survey with the corresponding characteristics from the face-to-face survey. As we noted earlier, both surveys have the potential for bias so we also compare the surveys with corresponding data from the 2001 Census and the Scottish Household Survey. ²⁶

4.1.1 Comparison of survey respondents and the Scottish population

Table 9 shows the extent to which the samples for both surveys reflect the structure of the Scottish population as measured by the 2001 Census. There is evidence that the survey samples under-represent some age groups and over-represent others. For example, the face-to-face survey shows a pattern typical of face-to-face surveys of adults, with under-representation of young adults and over-representation of older adults.

The telephone survey shows a slightly different profile. The earlier discussion of the characteristics of mobile-only households might be expected to have resulted in under-representation of young adults in comparison with the Census population. In spite of this there is very little difference. Males aged 20-24 years and 45-54 years are under-represented in the telephone survey as are all adults aged 65 years or more. Females aged 35-64 years are over-represented.

Table 9 - Comparison of respondent age/sex profiles and 2001 Census

Shows percentage of all respondents in each age/sex group

* indicates that difference is significant at 95% confidence limits

What is also apparent is that differences of similar magnitude in both surveys (such as for males aged 45-54 years) are significant for the telephone survey but not for the face-to-face survey. This is a reflection of the telephone survey's larger sample, which should lead to greater precision. Differences that are likely to occur by chance in the relatively small face-to-face survey should not be expected in the telephone survey.

The gender balance of each sample is shown in Table 10. This compares the proportion of men within each age group for both surveys in comparison with the corresponding figures in the 2001 Census. This shows evidence of systematic bias against men in almost all age groups in the telephone survey (except men aged 75 years and over), although among men aged under 25 years the difference is not significant. The proportion of men in the samples of respondents aged between 25 and 64 years is significantly lower than in the Census.

In only one of the age groups in the face-to-face sample is the proportion of men significantly different from the proportion in the corresponding group in the Census. This shows bias against men aged 45-54 years.

Table 10 - Gender balance of survey samples

Difference between proportions of men in each age group compared with 2001 Census

* indicates that difference is outside 95% confidence limits for the survey.

The discussion of respondent selection above (section 2.3.1) indicated that there was potential for the 'last birthday' selection method used in the telephone survey to lead to over-representation of women. This would tend to be supported by the evidence here. However, the respondent selection method was changed in January 2005 to use the method recommended by Rizzo et al ²⁷ with no significant improvement in the gender balance of the sample, suggesting differential participation by men rather than problems with respondent selection linked to the use of the 'birthday' method.

Table 11 - Percentage of men each age group in the telephone samples from Jun-Sept 2004 and Jan 2005

4.1.2 Comparing key demographic variables

Table 12 shows the results of comparing the telephone survey with three sources of comparator data: the face-to-face survey, the 2003 results from the Scottish Household Survey and the 2001 Census. Although this highlights differences between the surveys, there are a number of factors that need to be remembered in assessing the results.

• apart from the face-to-face and telephone surveys, the other data relate to different time periods and some variables, such as tenure, might be expected to change (albeit slowly) over time
• while the intention of bringing in other data sources is to avoid over-reliance on the face-to-face survey (there is no reason to assume that all differences between the face-to-face and telephone surveys is caused by bias in the telephone survey), these surveys - including the census - will themselves have some degree of error.
• the data sources are different sizes and the significance or otherwise of differences will be affected by sample sizes.

Nevertheless, the comparisons are useful because they give us an indication of whether differences between the telephone survey and the face-to-face survey are also observed with comparison to the other sources and in the same direction. For example, in comparison with the face-to-face survey, the telephone survey appears to under-represent households in social rented accommodation. This is also seen in comparison with the SHS and with the 2001 Census. On this basis, we would conclude that the telephone survey is biased against social rented households. This is unlikely to be caused by missing data in the telephone survey - 4.8% of telephone survey respondents refused tenure information - since analysis of other variables (such as MOSAIC classification, SIMD classification) suggests that the households with missing tenure are more broadly spread across social groups.

Where the comparisons across the three sources are not consistent (for example, the proportion of respondents in detached houses) we could not confidently conclude that the telephone survey was biased and if the results from the telephone survey were consistent with the other sources, it would seem more likely that the face-to-face survey is biased on this measure.

We need to bear in mind that the sample being compared in Table 12 excludes TPS subscribers. As we showed in Table 2, TPS subscribers have particular characteristics, being more likely to be owner-occupiers, living in detached or semi-detached houses, with two or more cars, in two adult households. The comparisons here do not take account of how the inclusion of TPS subscribers might have affected the profile of the telephone sample although we would expect it to have exacerbated bias against social rented households, against single adult households and against no car households.

Finally, we also need to bear in mind that the face-to-face survey, the SHS and the Census will include mobile-only households and as set out in section 3.2.2, the absence of these households from the telephone survey will have an impact on the demographic profile. For example, the proportion of social rented households in the face-to-face survey would drop from 26% to 23% if mobile-only households were filtered out. This gives some measure of the impact of the telephone survey being unable to include mobile-only households.

Table 12 - Demographic comparisons - telephone compared with face-to-face, SHS and 2001 Census

* indicates significant differences

These comparisons show considerable consistency in there being significant differences between the telephone survey and each of the other sources indicating bias in terms of:

• Tenure - the telephone survey appears to be biased against social rented tenures and we would expect the inclusion of TPS subscribers to exacerbate this. Also, over three-quarters of no phone or mobile-only households are in social rented tenures.
• Property type - terraced houses are under-represented and the inclusion of TPS subscribers would tend to make this worse. TPS subscribers would also increase the proportion of households in semi-detached houses and substantially reduce the proportion of households in flats/maisonettes.
• Number of adults in households - single adult households are under-represented compared with the face-to-face survey and the SHS. TPS subscribers are less likely to be in single adult households suggesting that this difference would be wider if the sample had included TPS households. This is one of the key characteristics of mobile-only households.
• Total household size - one person households are under-represented and households with three or more people are over-represented. TPS subscribers are less likely to be in single person households and less likely to be in households of three or more persons so including them in the sample would exacerbate the under-representation of single person households and reduce the over-representation of households with three or more people.
• Number of cars - the telephone survey under-represents households with no car and over-represents households with two or more cars. TPS subscribers are more likely to have two ore more cars and much less likely to have no car suggesting that these biases would be worse had the TPS subscribers not been screened out.
• Employment status - among respondents aged 16-74 years, the telephone survey over-represents adults in full-time and part-time employment and under-represents retired people and people who are permanently sick or disabled. Including TPS subscribers would make very little difference to the proportion of respondents in employment but it would increase the proportion of retired respondents so would reduce bias against retired people.
• Marital status - TPS subscribers are more likely to be married so had they been included, the profile of the telephone sample would be more like the face-to-face survey but less like the Census.

It appears that the main impact of excluding over 20% of households by screening out TPS has been to make the achieved telephone sample less biased than it might otherwise have been. In most cases, the characteristics of the TPS subscribers would be expected to exacerbate the biases apparent in the sample. It is difficult to be more precise about this because as table 2 also shows, there is some instability in the profiles of the July-September sample and the January 2005 sample which seems to result from increasing missing data - up from around 3% of respondents to 5% - and from the growth of TPS subscription between June 2004 and January 2005, which alters the characteristics of the non- TPS sample in January compared with July-September.

4.2 Attitudinal differences in the surveys

This section makes comparisons between the telephone survey and the face-to-face survey on various attitudinal variables. Differences between the two surveys on these variables are less clearly evidence of bias than are provided by consistent differences in demographic characteristics and any differences that are seen are likely to be influenced by differences in the profiles of the two samples.

There were two sets of attitude questions in each survey and both were asked at the same points in the interview. For both sets of respondents the first substantive question of the survey was the battery of questions about "problems in Scotland" followed by questions on crime in the local area, perceptions of safety and worries about various types of crime.

The second block of questions on how common different types of crime are in the respondent's area comes in one of the follow-up modules asked of half the sample in the face-to-face survey and a quarter of the sample in the telephone survey.

Table 13 shows that there are significant differences between the two samples on a number of the measures of the extent of problems in Scotland. Telephone respondents are more likely to say that standards of education, alcohol abuse, standards of housing and standards of public transport are a 'big problem' in Scotland. They are less likely to say that unemployment is a 'big problem'.

Table 13 - Attitudes about social problems in Scotland

Table 14 shows some slightly contradictory views in the two surveys to the extent that although telephone respondents are more likely to have lived in their local area for 10 years or more and significantly less likely to feel that crime in their area has increased in the past two years, they are less likely to be concerned about personal safety (Q4, 5 and 6) in their local area but more likely to worry about every type of crime asked about (Q7). One partial resolution of the apparent contradiction might lie in the perception that rather than having got worse or better, telephone survey respondents report that crime in their local area has stayed the same - perhaps as safe or unsafe as it ever was.

The findings on fear of crime in their local area are consistent with the findings from section 4 that the telephone survey substantially under-represents older people of both sexes and older women in particular. Fear of crime is generally associated with older people and older women in particular. ²⁸

The other possibility is that the lower fear of crime is a reflection of tenure imbalance - the other main demographic bias identified at section 4.1.2. If owner-occupiers are more likely to feel safe etc compared with social rented tenants the over-representation of owner-occupiers in the telephone survey would then lead to the difference between the two surveys. However, while this might offer some explanation, both the owner-occupiers and social rented tenants in the telephone survey are more likely than their face-to-face counterparts to 'feel safe'.

Table 14 - Worries about crime and personal safety in local area

* indicates significant differences

Even if we could explain the differences in the 'safety' findings, there remain the contradiction between 'safety' and 'worry' - telephone respondents feel safer in their local area and at home than do face-to-face respondents but they are more likely to worry about each type of crime. Possibly telephone survey respondents make a clearer distinction between safety in their local area and worry about crime in general.

4.3 Conclusion

Compared with the face-to-face survey, the telephone survey shows significant differences on a number of demographic variables. We acknowledged from the outset that there was scope for bias in the face-to-face survey but comparison of the telephone survey with the 2003 Scottish Household Survey and 2001 Census underlines the differences and, in some cases, indicates that the difference is greater than the face-to-face survey suggests. While some of the demographic bias might be a reflection of the exclusion of mobile-only households, the demographic characteristics of TPS subscribers suggests that had these households been included, the demographic bias in the telephone survey would, in general, have been greater.

The differences in the attitudinal variables are more complex and simply indicate that the two surveys have significantly different responses on these measures. We cannot say on the basis of this that one survey is more biased than the other in what they record. While some of the differences between the two surveys are what might be expected given the age, sex and tenure biases in the telephone survey sample, this does not explain the apparent contradiction between perception of 'safety' and the greater reporting of 'worry' about crime among telephone survey respondents.

« Previous | Contents | Next »