The Scottish Government

« Previous | Contents | Next »

Listen

4. Data quality

The issue of bias arises in every survey of the population. There are a number of sources of bias, some of which reflect aspects of the survey design (such as the sampling frame or who is deemed eligible for interview). However, bias is also a reflection of those aspects of fieldwork outcomes mentioned above:

• the quality of survey administration procedures
• whether potential respondents can be found at home at times when interviewers call
• whether they are able to participate in the interview i.e. not restricted by ill health, disability or communication barriers
• the willingness of members of the public to participate in the survey.

A high response rate is generally viewed as one of the key measures of data quality and, all other things being equal, a high response rate and a large sample should ensure accurate estimates. However, to the extent that non-response to the survey is not spread evenly, either geographically or between sub-groups of the population, the resulting bias will limit the accuracy of the survey's estimates. The question of bias is considered by comparing key results from the SHS with comparator data. Since the publication of the 2001 Census, this source is the most accurate comparator for population data and in spite of being a few years behind the current SHS, population measures such as age distribution and household types change little from year-to-year.

Household type, property type, tenure and number of bedrooms<

Single adult and large adult households are under-represented, and single pensioner and older smaller households over-represented, when household types in the 2005 SHS are compared with the Census (Table 4 1).

Table 4 1: Comparison of household types in the 2001 Census and the 2005 SHS

* SHS data weighted by local authority size only

As Table 4 2 shows, the sample appears robust in terms of the variables associated with accommodation/property characteristics. Compared with the 2001 (which is four years older than the data in the SHS) there is a slight over-representation of houses and under-representation of flats and, reflecting this, over-representation of owners who own their property outright relative to the Census and under-representation rented and 'other' tenures.

Table 4 2: Comparison of key variables in the 2001 Census and the 2005 SHS

* SHS data weighted by local authority size only

‡ includes households in shared dwellings

† Pays part rent and mortgage (shared ownership) included in 'Own with mortgage'
†† Although Scottish Homes no longer exists and had largely disposed of its rented housing stock the reference is retained in the questionnaire in case some tenants continue to think Scottish Homes is their landlord.

Age and sex profile of the 'random adult' sample

When a single adult is randomly selected within households, the unweighted sample of adults always under-represents those living in multi-adult households, since they have a smaller chance of selection for interview. As Table 4 3 shows, weighting to equalise probabilities of selection generally has the effect of bringing the profile of the 'random adult' sample closer to that of the adult population. The SHS data shown have been weighted both by the number of adults resident in the household and by the local authority weight described in the previous section. These two weights tend to act in the same direction, since those larger local authority areas which are 'weighted up' also tend to be ones with a higher average household size.

Table 4 3: Comparison of weighted and unweighted age and sex profile of 2005 SHS data with 2001 Census estimates

* Weighted by number of adults and local authority size

** Weighted by local authority size

However, even after this design weighting has been applied, the weighted random adult sample for 2005 still does not match the profile of the adult population suggested by the Census estimates with, as expected, under-representation of younger people in general and 16-24 year olds in particular. Consequently, older people are over-represented in the random adult sample.

Driving and transport

In relation to driving and transport, the survey results also look broadly in line with what one might expect from other sources such as the National Travel Survey and the differences which exist are, again, comfortably within the confidence intervals associated with the two surveys. Mode of travel comparisons with other sources are less conclusive, though methodological or classification differences may be playing a part here.

Table 4 4:Comparison of key variables relating to driving and transport

* SHS weighted by number of adults and local authority size
** SHS weighted by local authority size and number of school children in household
*** SHS weighted by local authority size only
† Census figures are for method of travel to place of study, age 5-17

†† the National Travel Survey figures relate to 2004 alone, and were produced from the combined Scottish results of the NTS, the General Household Survey and the Expenditure and Food Survey. The Census figures relate to cars and vans available for private use.

Ethnicity

When comparing the ethnic composition of all household members with that of the population as a whole (as recorded in the 2001 Census), there is good agreement between the Census and the 2005 SHS. For example, in the Census, 98.0% of the population is recorded as White. In the 2005 SHS 97.5% of all household members are recorded as White. Within the detailed non-White categories the differences between the SHS and the Census suggest that Black and Asian groups represent a higher proportion of household members. The largest difference between the Census and the SHS is in the proportions recorded as White Scottish and White Other British.

Table 4 5: Comparison of ethnicity in Census 2001 and 2005 SHS

Urban/rural classification

Analysis of the Scottish Household Survey makes extensive use of the Scottish Executive's classification of areas into different degrees of urbanity and rurality. This classifies settlements according to their size and for settlements with a population of less than 10,000, their proximity to a settlement with a population of 10,000 or more. ⁵

Table 4 6 compares the urban/rural classification of the SHS sample for 2005 with the profile of all addresses sampled for the survey, the profile of eligible addresses and participating households. This shows that the addresses sampled in 2005 (column 2) under-represent urban areas and over-represent rural areas but when disproportionate sampling is taken into account by weighting, the profile matches the population.

Table 4 6: Comparison of all Scottish households, all sampled households, all eligible households and participating households by urban/rural classification

* Weighted by number households within each unit postcode
** Weighted to reflect disproportionate sampling across local authorities
*** Weighted to reflect disproportionate sampling and non-response across local authorities

Comparison of the households at which SHS interviews were achieved and the classification of all households sampled at a local authority level shows that there is a good match between the two within local authorities although overall, large urban areas are under-represented. Table 4 7 compares the proportion of households in each local authority in each type of area.

Table 4 7:Comparison of 2005 SE urban/rural classification of eligible addresses and 2005 participating households

Row percentages, all eligible addresses shown in bold, participating households in plain text

Rows may not always add to 100% because of rounding.

Economic activity

One area where the results of the SHS indicate significant differences from other sources is in relation to indicators of economic activity. As the following table shows, the most recent results from the Labour Force Survey ( LFS) suggest that the SHS may be under-representing people in employment, and over-representing the economically inactive. It should be emphasised, however, that the information from the SHS shown here is based on the respondent's own classification of their economic activity (collected at the start of the interview), rather than on the full International Labour Organisation definition, which is not classified by the respondent and is the basis for official estimates of unemployment. The SHS is not an official source of statistics on employment (see Methodology, section 4 on limitations of the data).

Table 4 8: Comparison of economic activity variables among adults of working age

* weighted by number of adults and local authority size

Figures in this table have been calculated using all working age people as the denominator, headline unemployment statistics are not calculated on this basis

Annual Population Survey data are sourced from quarterly Labour Force Survey data and the annual Labour Force Survey boost data.

« Previous | Contents | Next »