« Previous | Contents | Next »
Listen
10 Possible sampling variability, and "95% confidence limits" for SHS estimates(Table 24)
10.1 Although the SHS's sample is chosen at random, the people who take part in the survey will not necessarily be a representative cross-section. For example, purely by chance, the sample could include disproportionate numbers of certain types of people, in which case the survey's results would be affected. In general, the smaller the sample from which an estimate is produced, the greater the likelihood that the estimate could be misleading. As an example, Table 22 provides information about travel to school. In some cases, the figure for a Council area is based on data for only a hundred or so school pupils, each of whom therefore represents about 1% of the total. So, which particular households were selected for inclusion in the sample could make a significant difference to the results, which are therefore subject to considerable potential sampling variability. For example, the estimate of the percentage of pupils in that Council area who cycle to school would have been two or three percentage points higher had the SHS sample included, purely by chance, just two or three more children who cycled to school. In a "low population density" Council area, the "clustering" of the sample increases the potential sampling variability: for example, the estimated percentage who walk to school could be over-estimated greatly if, by chance, disproportionately many of the "rural" sample clusters chosen were in villages with schools, and disproportionately few were in places far from schools. Hence, an estimate that (say) 50% walk to school, produced from a sample of 100 or so school pupil households, may only indicate that the true value for the area is likely to be between 40% and 60%. Results produced from a small sample could therefore be greatly affected by sampling variability. The larger the sample, the less likely it is that the results will be affected greatly by sampling variability.
10.2 The likely extent of sampling variability can be quantified, by calculating the "standard error" associated with the estimate of a quantity produced from a random sample. Statistical sampling theory states that, on average:
- only about one sample in three would produce an estimate that differed from the (unknown) true value of that quantity by more than one standard error;
- only about one sample in twenty would produce an estimate that differed from the true value by more than two standard errors;
- only about one sample in 400 would produce an estimate that differed from the true value by more than three standard errors.
By convention, the "95% confidence interval" for a quantity is defined as the estimate plus or minus about twice the standard error (from sampling theory, the interval is plus or minus 1.96 times the standard error), because there is only a 5% chance (on average) that a sample would produce an estimate that differs from the true value of that quantity by more than this amount.
10.3 There is no simple "rule of thumb" for the size of standard errors: the standard error of the estimate of a percentage depends upon several things:
- the value of the percentage itself;
- the size of the sample (or sub-sample) from which it was calculated ( i.e. the number of sample cases corresponding to 100%);
- the sampling fraction ( i.e. the fraction of the relevant population that is included in the sample); and
- the "design effect" associated with the way in which the sample was selected (for example, a "clustered" random sample would be expected to have larger standard errors - but lower fieldwork costs - than a simple random sample of the same size).
10.4 Table 24 shows the "95% confidence limits" for estimates of a range of percentages calculated from sub-samples of a range of sizes (NB: the confidence limits for estimates of x% and of (100-x)% are the same). The table was produced in the same way as the tables of "95% confidence limits" in the "Annual Report" volumes of Scotland's People (see section B4), but has a more detailed breakdown of the smaller sample sizes, because this bulletin provides figures for individual Council areas, some of which are based on samples of only a few hundred.
10.5 The interpretation of an entry in Table 24 is best explained by an example:
- the value in the cell at the intersection of the "45% or 55%" column and the "800" row is 4.1;
- this means that the "95% confidence limits" for an estimate of 55% which is produced from a sub-sample of 800 are +/- 4.1%-points;
- so the "95% confidence interval" for the estimate is 55% +/- 4.1%-points ( i.e. from about 50.9% to around 59.1%, assuming that the value of the estimate is 55.0%);
- or, on average, only 1 in 20 sub-samples of size 800 would produce an estimate that differs from the (unknown) true value of this quantity (if it is around 55%) by more than 4.1%-points.
10.6 As an example of the use of this table, it will be seen from the figure at the end of the last row of Table 1 that there were 885 households in West Lothian in the survey in 2003/2004. The second figure in the last row of Table 1 shows that an estimated 47% of West Lothian households had one car available for private use. Because that estimate was produced from data for only 885 households, sampling variability could (by chance) produce an error of several percentage points. The entry in the cell at the intersection of the "45 or 55%" column and the "900" row in Table 15 shows that the "95% confidence limits" for the estimate will be roughly +/- 3.9%-points. This means that there is a 1-in-20 chance that the estimate differs from the true value by more than 3.9%-points. It follows that there is roughly a 1-in-3 chance that the estimate differs from the true value by more than about 2.0%-points. Clearly, estimates based on smaller samples have wider confidence limits.
10.7 Because the survey's estimates may be affected by sampling errors, apparent differences of a few percentage points between the figures for two Council areas may not be "significant": it could be that the true values for the two areas are similar, but the random selection of households for the survey has, by chance, produced a sample which gives a high estimate for one area and a low estimate for the other. A difference between two areas is "statistically significant" at the conventional "5%" level if it is so large that fewer than one random sample in twenty would be expected to produce a difference of that size (or greater) purely by chance, if the two areas' true values were the same. One way of assessing significance at the 5% level involves comparing the difference with the 95% confidence limits for the two estimates. Suppose that these are +/- 3.0%-points and +/- 4.0%-points, respectively. Clearly:
- a difference which is less than the magnitude of the greater of the limits (which, in this case, is 4.0%-points) is not significant; and
- a difference which is greater than the sum of the magnitudes of the limits (in this case 3.0%-points + 4.0%-points = 7.0%-points) is significant.
Statistical sampling theory suggests that a difference whose magnitude is between these values is significant if it is greater than the square root of the sum of the squares of the magnitudes of the limits for the two estimates - in this case, the square root of (3.0 2 + 4.0 2) - i.e. the square root of (9 + 16) - i.e. the square root of 25, which is 5.0. So, in this case, a 5.0%-point difference would be considered statistically significant (at the conventional 5% level). Similar calculations will indicate whether or not other pairs of estimates differ significantly.
10.8 The same approach can be used to assess the statistical significance of the difference between the figures for a particular area for two two-year periods, such as 1999/2000 and 2003/2004. For example, if the estimates for the two periods had 95% confidence limits of +/ 3.0%-points and +/- 4.0%-points respectively, a 5.0 %-point difference would be considered statistically significant (at the conventional "5%" level).
10.9 However, when assessing the statistical significance of results from among the many sets of sample estimates for each of the 32 local authority areas, please remember that there may well be occasions on which sampling variability produces the kind of difference between areas (or change between periods) that would be expected to arise, purely by chance, in only one sample in twenty - or in only one sample in a hundred, or only one sample in a thousand. Therefore, one may well find some apparently "significant" results that are actually just the result of sampling variability, having arisen by chance.
10.10 The above information relates only to sampling variability. The survey's results could also be affected by non-contact / non-response bias: the characteristics of the people who should have been in the survey but who could not be contacted, or who refused to take part, could differ markedly from those of the people who were interviewed. If that is the case, the SHS's results will not be representative of the whole population. Without knowing the true values (for the population as a whole) of some quantities, one cannot be sure about the extent of any such biases in the SHS. However, comparison of SHS results with information from other sources suggests that they are broadly representative of the overall Scottish population, and therefore that any non-contact or non-response biases are not large overall. However, such biases could, of course, be more significant for some sub-groups of the population or in certain Council areas, particularly those which have the highest non-response rates. In addition, because it is a survey of private households, the SHS does not cover some sections of the population - for example, it does not collect information about many students in halls of residence (see section B.2.3). The "Fieldwork Outcomes" volume of Scotland's People (see section B.4) provide more information on these matters, and section C describes briefly the results of comparing SHS and Census results for two transport questions.
« Previous | Contents | Next »