The Scottish Government

« Previous | Contents | Next »

Listen

4. Limitations of the data

There are a number of important methodological and data issues that users need to be aware of when using the SHS data.

Like all sample surveys, the SHS can only produce estimates and these estimates are limited by a number of factors.

• Sample coverage - although there are no geographical exclusions to the survey, the sampling frame does not cover the whole population because of a combination of inherent limitations and administrative errors and delays.
• Sampling variability - all samples can differ from the population by chance. This is often referred to as sampling error.
• The number of cases that analysis is based on - estimates based on large samples are more accurate than those based on small samples.
• Bias in the achieved sample - if a sample under-represents sections of the population or if a large proportion of people do not answer some questions, the estimates may differ substantially from the population for reasons that are not a result of chance. For example, in 2005, the unweighted sample of adults is 58% female and even after weighting 54% of the sample is female, but the true figure in the population is only 52%. ⁶ This is an example of bias caused by young males, in particular, being difficult to contact or refusing to take part in the survey.

The SHS is also limited in the amount of detail it can collect about some topics. For example, it was not designed to provide reliable "economic" statistics ( e.g. employment/unemployment rates and average earnings).

The SHS's information about the economic status of members of the household reflects the view of the respondent to the "household" part of the interview, and so may not conform to official definitions of employment and unemployment, for example. As a result, the SHS cannot provide estimates of unemployment that are comparable to official statistics of unemployment. ⁷

There are several reasons why the SHS data on income may not be reliable.

• The SHS only collects information from, or about, the Highest Income Householder and, if there is one, their spouse or partner.
• Information is provided "off the top of the head" as part of an interview on many other topics. There is no requirement to refer to pay slips or bank statements to check the figures.
• Some people may not know the correct figure (particularly in the case of the income of a spouse/partner), and may just provide a guess, perhaps based on a level that they remember from some time ago.
• Other interviewees may under-state their income because they do not want to reveal how much they really earn.
• Because about a third of the households in the sample are unwilling or unable to provide income information, values for some or all of the main components of income have to be imputed. ⁸

In 2004, researchers commissioned by the Scottish Executive and Communities Scotland compared the income data collected by the SHS and the Scottish House Conditions Survey ( SHCS) with the income statistics produced from the Family Resources Survey. ⁹ Their main conclusions were:

• the SHS (and SHCS) under-estimate total household income, due to collecting only the income of the highest income householder and any spouse/partner
• when households with one adult or two adults who are spouses/partners are compared, there is good agreement between the SHS/ SHCS and FRS income distributions for such households
• SHS (and SHCS) greatly under-estimate investment income and interest payments compared to FRS
• uncorrected bias in the SHS (and SHCS) age and sex distributions affects income distributions, particularly for one person households
• overall income from benefits agrees well between the surveys, but the individual benefits may be less accurately classified in the SHS (and SHCS).

As a multi-purpose survey of households, the SHS is not designed to provide the kinds of information about economic activity and household income that can be obtained from more specialised surveys such as the Labour Force Survey and the Family Resources Survey, which have questions and procedures which are designed to obtain much more reliable information on those matters than the SHS can collect. The SHS has questions on such topics only for selecting the data for particular groups of people (such as the unemployed or the low-paid) for further analysis, or for use as "background" variables when analysing other topics (such as the means of travel or the frequency of driving).

Although the SHS has a large sample that covers the whole of Scotland, it has some geographical limitations because of the sample sizes in small local authorities and because it is designed to be representative only at national and local authority level. This means:

• users need to be mindful of the sampling errors for analysis but especially when this is based on breakdowns within a single local authority
• it is not appropriate to undertake geographical analysis below local authority level since the sampling techniques used in some local authorities cannot guarantee representativeness in smaller areas.

4.1 Quarterly data for Scotland as a whole

The SHS was designed to provide results which are representative for Scotland as a whole for each quarter of the year. Although based on a large sample (nearly 4,000 households per quarter), they are still subject to sampling errors, so may well fluctuate from one quarter to the next. Therefore, apparent quarter-to-quarter changes should be interpreted cautiously, as they may well be due to sampling variability rather than representing genuine change.

This can be seen if one looks at the apparent quarter-to-quarter changes in some figures which one would expect to change only gradually from one quarter to the next - especially figures which show trends that one would not expect to be subject to short-term reversals. The SHS's quarterly Statistical Press Notices ¹⁰ provide a set of quarterly tables and charts. Examples of two of these appear on the following pages. The first example shows, quarter-by-quarter since the survey started, the (weighted) percentages of households in the sample with various numbers of cars available for private use; the second shows quarter-by-quarter figures for household tenure.

In both cases, the quarterly charts and tables show the kinds of long-term trends that one would expect ( e.g. a gradual increase in two-car households) - but with some apparent "wobbliness" in the lines. Given the nature of car ownership and household tenure, one would not expect sudden short-term departures from the long-term trend (such as a sharp fall in the percentage of homes which are owned outright). However, the survey results sometimes suggest very surprising quarter-to-quarter changes. For example, the table below the first chart shows that the (weighted) percentage of households with 3+ cars appears to vary from quarter to quarter. The cause cannot be any real change in car ownership across Scotland: it must just be sampling variability (the "luck of the draw" regarding which households were included in the sample in each quarter, and which of them agreed to take part in the survey). In the Annual Report's Appendix on confidence intervals and statistical significance, Table A3.1 indicates that the 95% confidence limits for an estimate of 5% based on a sample of 4,000 cases are about +/- 0.8%. The apparent fluctuation in the percentage of households with 3+ cars in the sample in the first three quarters of 2004 is a good illustration of such sampling variability.

The quarterly charts and tables also cover the following topics:

• rating of the neighbourhood as a place to live
• people who hold a full driving licence
• employed adults who work at or from home
• usual method of travel to work
• adults who make personal use of the internet
• adults who have given up their time to help as an organiser or a volunteer
• whether the household respondent/partner/spouse has a bank/building society account
• households with individuals who need regular help or care.

They can all be found on the SHS Web site: www.scotland.gov.uk/shs under "Publications".

Examples of charts and tables showing quarterly figures

« Previous | Contents | Next »