« Previous | Contents | Next »
Listen
SCOTTISH HOUSEHOLD SURVEY: Methodology 2003/2004
4. Limitations of the data
There are a number of important methodological and data issues that users need to be aware of when using the SHS data.
Like all sample surveys, the SHS can only produce estimates and these estimates are limited by a number of factors.
- Sample coverage - although there are no geographical exclusions to the survey, the sampling frame does not cover the whole population because of a combination of inherent limitations and administrative errors and delays.
- Sampling variability - all samples can differ from the population by chance. This is often referred to as sampling error.
- The number of cases that analysis is based on - estimates based on large samples are more accurate than those based on small samples.
- Bias in the achieved sample - if a sample under-represents sections of the population or if a large proportion of people do not answer some questions, the estimates may differ substantially from the population for reasons that are not a result of chance. For example, in 2003, the unweighted sample of adults is 57% female and even after weighting 55% of the sample is female, but the true figure in the population is only 51%. This is an example of bias caused by young males, in particular, being difficult to contact or refusing to take part in the survey.
The SHS is also limited in the amount of detail it can collect about some topics. For example, it was not designed to provide reliable "economic" statistics (e.g. unemployment rates and average earnings).
The SHS's information about the economic status of members of the household reflects the view of the respondent to the "household" part of the interview, and so may not conform to official definitions of employment and unemployment, for example. As a result, the SHS cannot provide estimates of unemployment that are comparable to official statistics of unemployment.
There are several reasons why the SHS data on income may not be particularly reliable.
- The SHS asks only for the income of the highest income householder and any spouse/partner. No information is obtained about the income of any other members of the household. Therefore, the SHS may under-state greatly the income of households which have several working adults, for example.
- The interviewee is asked to provide information about his/her income, and that of any spouse/partner, "off the top of the head", as part of an interview on many other topics. There is no requirement to refer to (e.g.) pay slips or bank statements to check the figures.
- Some people may not know the correct figure (particularly in the case of the income of a spouse/partner), and may just provide a guess, perhaps based on a level that they remember from some time ago.
- Other interviewees may choose to under-state their income because they do not want to reveal how much they really earn.
- Because many people are unwilling or unable to provide information about their incomes, the values of some or all of the main components of income have to be imputed for about a third of the households in the sample.
As a multi-purpose survey of households, the SHS is not designed to provide the kinds of information about economic activity and household income that can be obtained from more specialised surveys such as the Labour Force Survey and the Family Resources Survey, which have questions and procedures which are designed to obtain much more reliable information on those matters than the SHS can collect. The SHS has questions on such topics only for selecting the data for particular groups of people (such as the unemployed or the low-paid) for further analysis, or for use as "background" variables when analysing other topics (such as the means of travel or the frequency of driving).
Although the SHS has a large sample that covers the whole of Scotland, it has some geographical limitations because of the sample sizes in small local authorities and because it is designed to be representative only at national and local authority level. This means:
- users need to be mindful of the sampling errors for analysis but especially when this is based on breakdowns within a single local authority
- it is not appropriate to undertake geographical analysis below local authority level since the sampling techniques used in some local authorities cannot guarantee representativeness in smaller areas.
« Previous | Contents | Next »