The Scottish Government

« Previous | Contents | Next »

Listen

ANNEX 3 SAMPLING ERROR AND DESIGN EFFECTS

A4.1 Data collected in surveys always provide estimates of the true proportions in the population. The precision of these estimates - the sampling error - can be calculated for any estimate in the survey from the estimated element variance of the estimate and the number of people in the sample (or sub-sample). The sampling error can be expressed as a 'confidence interval', which can be added to and subtracted from the survey estimate to give a range within which it is fairly certain that the true value lies.

A4.2 The precision of estimates derived from a sample is normally measured using standard errors. Essentially standard errors are calculations of the standard deviation of the sampling distribution. They differ from element standard deviations, though, in that while the latter are a measure of variability derived from actual observations of a sample, standard errors refer to the variability of possible sample estimates that could be obtained from a series of samples. Usually, the formula used for calculating standard errors assumes a simple random sample ( SRS). However, as the SCVS sample was stratified and the data were weighted, the standard errors must take into account the survey methodology in order to calculate confidence intervals. This requires the calculation of 'complex' standard errors and design factors. These were calculated using the Stata statistical package which is regarded as the industry norm for this purpose.

A4.3 The design factor is the ratio between the complex and the SRS standard errors and indicates the effect of the sampling design (weighting and stratification) on the standard error and hence confidence intervals. A design factor of 1 indicates that the effect of the sampling methodology used in the survey is equivalent to that of a simple random sample. A design factor of 1.2 indicates that the sampling methodology is 20% less efficient than a simple random sample, while, conversely, a design effect of less than 1 would indicate that the sampling methodology improves the sampling efficiency.

A4.4 Table C1 shows the standard errors, the design factors, and the confidence intervals for the incidence rates. As can be seen from the table, the design factors ranged from 1 to 1.5. The overall average is 1.2, but that should not be taken as a 'typical' value, given the range of values across different variables. However, it suggests that using a value of 1.2 as a 'rule of thumb' for adjusting the standard errors of the survey data would safely account for the design factors associated with most estimates from the survey.

Table C1 Sampling error and design effects

Notes:
1) Rates for personal crimes (assault, robbery, theft from the person, other personal theft, minor assault and serious assault) are per 10,000 adults). Rates for household crimes (vandalism, housebreaking, theft of motor vehicle, bicycle theft, theft from a motor vehicle, attempted theft of / from motor vehicle, other household theft, motor vehicle vandalism and property vandalism) are per 10,000 households.
2) For the distinction between crimes which are 'comparable with police' and 'other survey crimes' see Annex 4
3) Source: SCVS 2006 (n=4,988)

« Previous | Contents | Next »