The Scottish Government

« Previous | Contents | Next »

Listen

Appendix 3 - Confidence intervals and statistical significance

The representativeness of the Scottish Household Survey

Although the SHS sample is chosen at random, the people who take part in the survey will not necessarily be a representative cross-section of the population. Like all sample surveys the results of the SHS are estimates of the corresponding figures for the whole population and these results might vary from the true values in the population for three main reasons:

1. The sample source does not completely cover the population because accommodation in hospitals, prisons, military bases, larger student halls etc. are excluded from the sampling frame. The SHS provides a sample of private households rather than all households. The effect of this on the representativeness of the data is not known.
2. Some people refuse to take part in the survey and some cannot be contacted by interviewers. If these people are systematically different from the people who are interviewed, this represents a potential source of bias in the data. Comparison of the SHS data with other sources suggests that for the survey as a whole, any bias due to non-response is not significant ¹⁷.
3. Samples always have some natural variability because of the random selection of households and people within households. In some areas where the sample is clustered, the selection of sampling points adds to this variability.

Each of these sources of variability becomes much more important when small sub-samples of the population are examined. For example, a sub-sample with only 100 households might have had very different results if the sampling had by chance selected four or five more households with children.

Confidence intervals

The likely extent of sampling variability can be quantified by calculating the 'standard error' associated with an estimate produced from a random sample. Statistical sampling theory states that, on average:

• only about one sample in three would produce an estimate that differed from the (unknown) true value by more than one standard error;
• only about one sample in twenty would produce an estimate that differed from the true value by more than two standard errors;
• only about one sample in 400 would produce an estimate that differed from the true value by more than three standard errors.

By convention, the '95% confidence interval' is defined as the estimate plus or minus about twice the standard error because there is only a 5% chance (on average) that a sample would produce an estimate that differs from the true value of that quantity by more than this amount.

There is no simple "rule of thumb" for the size of standard errors: the standard error of the estimate of a percentage depends upon several things:

• the value of the percentage itself
• the size of the sample (or sub-sample) from which it was calculated (i.e. the number of sample cases corresponding to 100%)
• the sampling fraction (i.e. the fraction of the relevant population that is included in the sample), and
• the 'design effect' associated with the way in which the sample was selected (for example, a clustered random sample would be expected to have larger standard errors than a simple random sample of the same size).

Table A3.1 shows the 95% confidence limits for a range of estimates calculated for a range of sample sizes. To estimate the potential variability for an estimate for the survey you should read along the row with the value closest to the estimate until you reach the column for the value closest to the sub-sample. This gives a value which, when added and subtracted from the estimate, gives the range (the 95% confidence interval) within which the true value is likely to lie.

For example, the survey estimates that 16% of households (rounded to the nearest whole number) contain are Single Adult households ( Figure 3.1). This has a confidence interval of ±0.5%, which means that, if the estimate were 16.0% we could be 95% confident that the true value for the population lies between 15.5% and 16.5%.

To see the effect of smaller sample sizes, the survey estimates that in Glasgow, 22% of households are Single Adult households ( Table 3.13). However, only 3,294 households in Glasgow were interviewed so this estimate has a 95% confidence interval of approximately ±1.4%. Assuming that the estimate is 22.0%, this suggests that the true value lies between 20.6% and 23.4%. Clearly, the estimate for any single area is less reliable that the estimate for Scotland as a whole.

Statistical significance

Because the survey's estimates may be affected by sampling errors, apparent differences of a few percentage points between sub-samples may not reflect real differences in the population. It might be that the true values in the population are similar but the random selection of households for the survey has, by chance, produced a sample which gives a high estimate for one sub-sample and a low estimate for the other.

A difference between two areas is significant if it is so large that a difference of that size (or greater) is unlikely to have occurred purely by chance. Conventionally, significance is tested at the 5% level, which means that a difference is considered significant if it would only have occurred once in 20 different samples. Testing significance involves comparing the difference between the two samples with the 95% confidence limits for each of the two estimates.

For example, the survey estimates that there are 8% single adult households in East Dunbartonshire (±2.3%), 12% in Highland (±1.9%), 13% in Midlothian (±2.8%), and 22% in Edinburgh (±1.6%). Assuming that the estimates' values are 'exact' (i.e. that the figure underlying 8% is 8.0%), we can say the following:

• the difference between Midlothian and Highland is not significant because the difference between the two (1%) is smaller than either of the confidence limits. In general, if the difference is smaller than the larger of the two limits, it could have occurred by chance and is not significant.
• the difference between East Dunbartonshire and Edinburgh is significant because the difference (14%) is greater than the sum of the limits (2.3 + 1.6 = 4.9%). In general, a difference that is greater than the sum of the limits is significant.
• if the difference is greater than the larger of the two confidence limits but less than the sum of the two limits, the difference might be significant, although the test is more complex.

Statistical sampling theory suggests that the difference is significant if it is greater than the square root of the sum of the squares of the limits for the two estimates.

The difference of 4% between East Dunbartonshire and Highland is greater than the largest confidence limit but it is less than the sum of the two limits (2.3% + 1.9% = 4.2%) so it might be significant. In this case, 2.3 ² = 5.29 and 1.9 ² = 3.61 giving a total of 8.9. The square root of this is 2.98, which means that the difference of 4% is significant. Similar calculations will indicate whether or not other pairs of estimates differ significantly.

It should be noted that the estimates published in this report have been rounded, generally to the nearest whole number, and this can affect the apparent significance of some of the results. For example:

• if the estimate for East Dunbartonshire was 8.49% (rounded to 8%) and the estimate for Highland was 11.51% (rounded to 12%) the difference would be calculated as 3.02% rather than 4%. This is only just above the calculated 'significance threshold' value of 2.98. (Furthermore, as this value of 2.98 is calculated from numbers that have been rounded, in this case to one decimal place, it may not be exactly accurate).
• if, however, the estimate for East Dunbartonshire was 7.51% (rounded to 8%) and the estimate for Highland was 12.49% (rounded to 12%) the difference would be calculated as 4.98% rather than 4%. This is clearly more significant.

Statistical significance and representativeness

Calculations of confidence limits and statistical significance only take account of sampling variability. The survey's results could also be affected by non-contact/non-response bias. If the characteristics of the people who should have been in the survey but who could not be contacted, or who refused to take part, differ markedly from those of the people who were interviewed, there might be bias in the estimates. If that is the case, the SHS's results will not be representative of the whole population.

Without knowing the true values (for the population as a whole) of some quantities, we cannot be sure about the extent of any such biases in the SHS. However, comparison of SHS results with information from other sources suggests that they are broadly representative of the overall Scottish population, and therefore that any non-contact or non-response biases are not large overall. However, such biases could, of course, be more significant for some sub-groups of the population or in certain Council areas, particularly those that have the highest non-response rates.

In addition, because it is a survey of private households, the SHS does not cover some sections of the population - for example, it does not collect information about students in halls of residence. Please refer to the companion Technical Report, Scottish Household Survey: Fieldwork Outcomes 2003/2004 for a comparison of SHS results with information from other sources.

« Previous | Contents | Next »