The Scottish Government

« Previous | Contents | Next »

Listen

7 Bias and data quality

The issue of bias arises in every survey of the population. There are a number of sources of bias, some of which reflect aspects of the survey design (such as the sampling frame or who is deemed eligible for interview). However, bias is also a reflection of those aspects of fieldwork outcomes mentioned above:

• the quality of survey administration procedures
• whether potential respondents can be found at home at times when interviewers call
• whether they are able to participate in the interview i.e. not restricted by ill health, disability or communication barriers
• the willingness of members of the public to participate in the survey.

A high response rate is generally viewed as one of the key measures of data quality and, all other things being equal, a high response rate and a large sample should ensure accurate estimates. However, to the extent that non-response to the survey is not spread evenly, either geographically or between sub-groups of the population, the resulting bias will limit the accuracy of the survey's estimates. The question of bias is considered by comparing key results from the SHS with comparator data. Since the publication of the 2001 Census, this source is the most accurate comparator for population data and in spite of being a few years behind the current SHS, population measures such as age distribution and household types change little from year-to-year.

7.1.1 Household type, property type, tenure and number of bedrooms

Single adult and large adult households are under-represented, and single pensioner and older smaller households over-represented, when household types in the 2005/2006 SHS are compared with the Census (Table 7-1).

Table 7-1: Comparison of household types in the 2001 Census and the 2007 SHS

* SHS data full sample, weighted by local authority size only

As Table 7-2 shows, the sample appears robust in terms of the variables associated with accommodation/property characteristics. Compared with the 2001 (which is six years older than the data in the SHS) there a higher proportion of houses and a lower proportion of flats. Outright ownership appears to be over-represented compared with owning with a mortgage and in total, owner-occupation is over-represented in the SHS compared with rented tenures.

Table 7-2: Comparison of housing variables in the 2001 Census and the 2007 SHS

* SHS main sample, data weighted by local authority size only
‡ includes households in shared dwellings
† Pays part rent and mortgage (shared ownership) included in 'Own with mortgage'

7.1.2 Age and sex profile of the 'random adult' sample

When a single adult is randomly selected within households, the unweighted sample of adults always under-represents those living in multi-adult households, since they have a smaller chance of selection for interview. As Table 7-3 shows, weighting to equalise probabilities of selection generally has the effect of bringing the profile of the 'random adult' sample closer to that of the adult population. The SHS data shown have been weighted both by the number of adults resident in the household and by the local authority weight described in the previous section. These two weights tend to act in the same direction, since those larger local authority areas which are 'weighted up' also tend to be ones with a higher average household size.

Table 7-3: Comparison of weighted and unweighted age and sex profile of 2007 SHS data with 2001 Census estimates

* SHS full sample. Weighted by number of adults and local authority size
** Weighted by local authority size

However, even after this design weighting has been applied, the weighted random adult sample for 2007 still does not match the profile of the adult population suggested by the Census estimates with, as expected, under-representation of younger people in general and 16-24 year olds in particular. Consequently, older people are over-represented in the random adult sample.

7.1.3 Driving and transport

In relation to driving and transport, the survey results also look broadly in line with what one might expect from other sources such as the National Travel Survey and the differences which exist are, again, comfortably within the confidence intervals associated with the two surveys. Mode of travel comparisons with other sources are less conclusive, though methodological or classification differences may be playing a part here.

Table 7-4: Comparison of key variables relating to driving and transport

* SHS full sample, weighted by number of adults and local authority size
** SHS main sample, weighted by local authority size and number of school children in household
*** SHS full sample, weighted by local authority size only
† Census figures are for method of travel to place of study, age 5-17
†† the National Travel Survey figures relate to 2004 alone, and were produced from the combined Scottish results of the NTS, the General Household Survey and the Expenditure and Food Survey. The Census figures relate to cars and vans available for private use.

7.1.4 Urban/rural classification

Analysis of the Scottish Household Survey makes extensive use of the Scottish Government's classification of areas into different degrees of urbanity and rurality. This classifies settlements according to their size and for settlements with a population of less than 10,000, their proximity to a settlement with a population of 10,000 or more. ¹⁴

Table 7-5 compares the urban/rural classification of the SHS sample for 2007 with the profile of all addresses sampled for the survey, the profile of eligible addresses and participating households. This shows that the addresses sampled in 2007 (column 2) under-represent urban areas and over-represent rural areas but when disproportionate sampling is taken into account by weighting, the profile matches the population.

Table 7-5: Comparison of all Scottish households, all sampled households, all eligible households and participating households by urban rural classification

* Weighted by number households within each unit postcode
** Weighted to reflect disproportionate sampling across local authorities
*** Weighted to reflect disproportionate sampling and non-response across local authorities

Comparison of the households at which SHS interviews were achieved and the classification of all households sampled at a local authority level shows that there is a good match between the two within local authorities. Table 7-6 compares the proportion of households in each local authority in each type of area.

Table 7-6: Urban rural classification of eligible addresses and participating households

Row percentages, all eligible addresses shown in bold, participating households in plain text

Rows may not always add to 100% because of rounding.

7.1.5 Economic activity

One area where the results of the SHS indicate the potential for significant differences from other sources is in relation to indicators of economic activity. As the following table shows, the most recent results from the Labour Force Survey ( LFS) in Scotland suggest that the SHS may be under-representing people in employment, and over-representing the economically inactive. It should be emphasised, however, that the information from the SHS shown here is based on the respondent's own classification of their economic activity (collected at the start of the interview) ¹⁵, rather than on the full International Labour Organisation definition, which is not classified by the respondent and is the basis for official estimates of unemployment. The SHS is not an official source of statistics on employment (see aboveon limitations of the data).

Table 7-7: Comparison of economic activity variables among adults of working age

* weighted by number of adults and local authority size

Figures in this table have been calculated using all working age people as the denominator, headline unemployment statistics are not calculated on this basis. Annual Population Survey data are sourced from quarterly Labour Force Survey data and the annual Labour Force Survey boost data.

« Previous | Contents | Next »