The Scottish Government

« Previous | Contents | Next »

Listen

Appendix 2 - Corrective weighting

In Chapter 3 we identified the exclusion of mobile-only households as a potential source of bias in the telephone survey. We also compared the response rates to both surveys and considered the potential for differential non-participation, particularly in terms of refusal to introduce bias. In Chapter 4 we examined a number of demographic variables and found that, in comparison with the face-to-face survey, the Scottish Household Survey and the 2001 Census, the telephone survey showed evidence of bias. We also noted that the telephone survey respondents differed significantly in terms of their response to attitudinal variables. The victimisation data outlined in Chapter 5 has borne out this view - victimisation is higher in the telephone survey and more so for personal crimes than for property crimes. Evidence of higher victimisation among respondents to the telephone survey who initially refused to participate also pointed in the direction of a bias in favour of victims of crime.

Non-response weighting is always problematic because while we have a great deal of information about people who respond to the survey, we know very little about those who do not. We can examine the characteristics of respondents and compare these with external data sources (as we do in Chapter 4) and infer the nature of non-response. While this identifies something of the nature and extent of bias on these variables, the real problem of non-response lies in the variables related to those characteristics that the survey seeks to produce estimates - in this case victimisation - about which, by definition, we know little.

We attempted two approaches to weighting. The first is traditional demographic weighting to bring the profile of the sample into line with reliable population estimates. This approach assumes that there is no difference between responders and non-responders in terms of the key survey measure of victimisation. Analysis in the main body of the report clearly concludes that this is not the case, making this approach methodologically unacceptable. The second approach is more unusual: we accepted the presence of bias in terms of victimisation and developed an approach that sought to measure this and calculate weights that would correct it.

Demographic weighting

In the preceding chapters we identified bias in the telephone survey in terms of the following variables:

Tenure - owner-occupiers are over-represented
Property type - households in detached houses are under-represented
Number of people in the household - small households are under-represented
Number of cars or vans in the household - households with no cars or vans are under-represented
Number of bicycles - households with no bicycles are under-represented
The age/sex profile of respondents - men were generally under-represented and older people of both sexes were also under-represented
Respondents' employment status - working households are over-represented.

The crime survey needs separate weights for the analysis of household variables (including household victimisation) and for individual variables, including personal offences. It is not possible, therefore, to develop one set of corrective weights. Correction relevant to the household (such as tenure, property type or the number of adults in the household) needs to precede correction of individual attributes. The sequence of weighting therefore needs to be:

• design weights to correct unequal selection probabilities for households - this addresses disproportionate sampling between PFAs and weighting for the number of landlines available for receiving incoming calls
• corrective weighting for household characteristics - this addresses bias in household variables and, when combined with household design weights, provides a single household weight to use in calculating victimisation rates for household offences.
• design weights for individual characteristics - this addresses unequal selection probabilities related to the number of adults in households.
• corrective weighting for individual characteristics - addressing bias on individual variables and, when combined with the three preceding weights, provides a single individual weight to use in calculating victimisation rates for personal offences.

This gives some indication of the complexity of the task since the effect of each stage of weighting needs to be carried into the next stage. For example, after the proportion of respondents in each PFA has been corrected, corrective weighting needs to be carried out within each PFA to stop the corrective weighting from changing the proportions. These combinations of design and corrective weights then form their own weighting classes within which each subsequent stage of weighting needs to be carried out.

In general terms, weighting needs to be carried out within weighting classes that are both comprehensive - all respondents are in a class - and mutually exclusive - each respondent is in only one class. This can combine a number of variables but this type of weighting is subject to practical constraints. First, the reference data need to exist in the same classes so that weights can be calculated. Second, the number of classes needs to be kept within manageable limits. For example, weighting involving three variables each with two categories results in eight weighting classes - the number of possible permutations. The addition of a third variable with two categories increases this to 16 classes. With many variables and many categories, the number of possible permutations quickly expands to a level that is unmanageable (at least for manual calculations in SPSS) and leads to the potential for empty cells where no weight can be calculated or small cells leading to large weights.

The telephone survey is affected by both of these issues. First, Census data, which would provide ideal reference values for many demographic variables, tends not to be available in complex combinations and probably not in combinations that will reflect the weighting requirements of the survey. Second, the combinations of design weights and corrective weights at both household and individual levels quickly leads to a very large number of weighting classes, as outlined below.

The Census has a further limitation in that it while it represents the best estimate of demographic variables around the time it is undertaken, we might expect some change in variables such as tenure and car ownership over the course of four or five years that make continuous reference back to characteristics in 2001 problematic. The need for complex weighting classes and for reference data that changes to reflect underlying change in the population suggested that using the SHS as a source of reference data would be more appropriate than the Census.

Although the SHS is not as accurate as the Census (and comparison with Census data, such as that in Table 12, shows the extent to which it differs) it is generally close to the Census on most variables and closer than the telephone survey appears to be. Using it as a source of reference statistics would be a compromise but one with some advantages. It is based on a very large sample and estimates can be produced at PFA level. Its estimates also have the capacity to move year-on-year in line with underlying change in the population.

The weighting process

To assess which variables would be best for corrective weighting we examined correlation within the group of demographic variables. The variables with the greatest number of significant correlations with other demographic variables were prioritised over those with fewer and those with strong correlations were prioritised over those with weaker correlations. Our objective was to reduce the number of weighting variables to a few that could make a significant contribution to correcting the range of biases evident in the data. For household variables, we prioritised the number of people in the household and tenure. These were collapsed to broader categories: two categories for the number of people ('one or two' and 'three or more') and three tenure categories (owner-occupied, social rented and other) to provide a manageable number of weighting classes - 48 when PFA is included.

As would be expected, this weighting is effective in correcting bias on the variables used to produce the classes but it also appears to be effective in bringing respondent employment status closer to Census estimates of the proportion of working adult aged 16-74 years (bringing the survey estimate down from 51.3% working full time to 47.8% compared with Census estimate of 46.9%). However, although the proportion of working adults has been improved by weighting, the proportion of retired people among adults aged 16-74 has been pushed higher than the Census estimate when there had previously been a close match between the telephone survey and the Census. Also, the distribution of property types is worse after weighting and this weighting does not appear to have made any significant impact on the bias against households with no car. This is shown in Table 21.

Table 21 - Survey estimates before and after weighting for tenure and number of people in household compared with Census estimates

The main individual characteristic that needs to be weighted is the age/sex profile of respondents. This would, ideally, be done within each of the weighting classes used for household correction but with 2 categories of sex and 8 age categories, the resulting 768 weighting cells (48 * 2 * 8) becomes unwieldy and there are likely to be empty cells that further complicate the weighting. The options to avoid this are:

• calculate the age/sex weights independently of the household weighting classes and apply them to the sample as a whole. This risks losing some of the correction already achieved
• reduce the number of age/sex categories, although even if only four categories of age were used there would still be 394 weighting cells.

The simplest solution was the former approach. The disadvantage of the second approach is that it relies on using the SHS as the source of reference data and we know that the SHS under-represents young adults and over-represents older adults. This perhaps makes it unsuitable for this task.

Using the first approach does indeed lose some of the correction that had previously been gained, as Table 22 shows. However, on some, the correction of the age/sex profile takes many of them further from the Census estimate than they were before (highlighted)

Table 22 - Survey estimates before and after weighting for tenure and number of people in household and after addition of age/sex weighting compared with Census estimates

This leads us to conclude that simultaneously correcting the many biases in the telephone survey will require a more sophisticated approach than simply using population-based ratios for single variables or even combinations of a number of variables. Rim weighting is able to calculate weighting factors based on a number of individual variables with know population totals. For example, rim weights based on age, sex and housing tenure would be calculated by a computer program iteratively adjusting the weighting factors applied to each case in a way that brings the sample distribution of each of these variables into line (or as close as possible) with the population distribution.

The second conclusion is that since we cannot find a weighting solution, we are unable to look at the impact of corrective weighting on victimisation.

Weighting to correct for bias in victimisation

In the main report we concluded that the telephone survey over-represents victims resulting in higher rates of victimisation than would be expected given the exclusion of mobile-only households and demographic differences between the RDD sample and comparator data. Comparison of the face-to-face survey with 2003 data and recorded crime data suggested that that survey was under-representing victims.

This approach tries to estimate the extent to which the surveys might be biased in favour of, or against, households with characteristics that are related to experiencing both household and personal victimisation.

The approach

If the telephone survey is systematically biased towards victims, corrective weighting that attempted to address this by matching survey and reference demographics would be unlikely to correct this bias. The underlying assumption that responders were like non-responders in terms of key survey estimates - experience of victimisation - was not true.

Reflecting on this, we attempted an alternative approach that might address any 'victim' bias more directly. If we could establish that the RDD sample was biased in favour of victims and estimate by how much, we would be able to use this as the basis for corrective weighting. The approach (for correction of household victimisation) was as follows:

1. respondents to the telephone survey were assigned to one of two groups based on their experience of any incidents of household victimisation (any or none)
2. SPSS Answertree was used to assign households to groups whose characteristics were related to experience of victimisation
3. the resulting combinations of characteristics were used to assign households to weighting classes
4. the same classes were created in the face-to-face survey and also in the Scottish Household Survey, which would be used as reference data for both samples
5. comparison of the proportion of households in each class in both datasets allowed us to calculate weights.

This approach relies on two assumptions. First, that the SHS is suitable for reference data i.e. that it is not substantially biased. This would be problematic if the SHS and the face-to-face survey were similarly biased on demographic characteristics and the telephone survey was not. This would lead to the paradoxical situation of the face-to-face survey requiring no corrective weighting and the telephone survey requiring substantial "correction". Table 12 shows that the SHS is broadly in line with 2001 Census estimates. We know the SHS under-represents young people in general and young men in particular so before calculating weights for personal victimisation, we corrected the age/sex profile of the SHS sample using the 2001 age/sex profile of adults in Scotland. The second key assumption is that participation in the SHS is not related to household or personal victimisation. If it were, it could not provide a basis to calculate weights related to victimisation.

Results

Table A1 shows that in the telephone survey the weighting classes that are most likely to experience household victimisation (index > 100) are being weighted down, with an average weight of 0.88. This suggests that overall, the telephone survey over-represents victims of household crime. This table also suggests that overall the face-to-face survey does not tend to over or under-represent households with these characteristics. For both surveys there are some groups that attract quite large weights.

The same exercise was run for personal victimisation using the variables included in the household correction and a range of individual respondent characteristics. The results of this are shown in Table A2. This suggests that the telephone survey also over-represents individuals with characteristics related to higher victimisation but with an average down-weighting factor of only 0.96 this weighting is unlikely to have any significant impact on victimisation rates.

Table A2 also suggests that the face-to-face survey under-represents individuals with victimisation-related characteristics and with an average weight of 1.15, this weighting should increase the victimisation rates for personal incidents recorded in the face-to-face survey.

7.1.1 Impact on sample demographic profiles

Table 4 shows that the addition of household corrective weighting in the telephone survey has some beneficial impact on characteristics such as:

• the proportion of households in social rented tenures
• the proportion of households in flats
• the number of single adult households
• the proportion of one or two person households
• the proportion of households with no car.

It has a negative effect on:

• the proportion of households in remote rural areas

However, there remain substantial unresolved gaps in the demographic profile compared with the SHS. We estimate that the combination of design and corrective weighting has removed 70% of the difference between the reference data and the unweighted telephone survey data. Most of this is done by the design weighting (66%) with corrective weighting only adding another 4% improvement. ³³

The addition of individual corrective weighting has improved the proportion of adults living in rented tenures, the proportion of adults with no car, the proportion adults in flats, the proportion in one or two adult households, the proportion of adults in one or two person households, the proportion of adults working full time and the proportion permanently retired from work. We estimate that while the individual design weight removes 51% of the differences between the telephone survey and the SHS, the addition of corrective weighting increases this to 68%.

For the face-to-face survey, as would be expected given the small average weight for household correction, this weighting has almost no impact on the demographic profile of the sample.

Individual weighting has the paradoxical effect of making a number of measures worse than before - taking them further away from the reference data - and taking very few measures closer.

However, it needs to be borne in mind that the weighting is not aimed at correcting the surveys' demographic profiles but at correcting any bias in victimisation. Since it acts on combinations of variables there might be instances where, for example, a combination of tenure and car ownership is weighted up and another combination of tenure and urban/rural classification is weighted down. The impact on the tenure profile might be neutral.

If we were confident that the weighting was adequately addressing bias in victimisation, it might be reasonable to conclude that in terms of what the survey seeks to measure, the remaining demographic anomalies could be ignored.

Impact of weighting on victimisation rates

The implications of tables A2 and A3 are that the victimisation rates from the telephone survey should fall for household incidents and be unchanged for personal incidents. The face-to-face survey should have largely unchanged household victimisation rates but higher personal victimisation rates.

Table A5 shows that the rate for all property offences recorded in the telephone survey has fallen, although only by a small amount. The rate for personal offences has also increased very slightly. For the face-to-face survey, the rate for property offences has fallen slightly and there has been an increase in the rate for personal offences.

On individual offence categories, the corrective weighting has narrowed the gap between the surveys in 10 of the 15 categories but by a substantial amount in only a few.

Compared with the 2003 SCS, the telephone survey rates remain lower for property offences (-13%) and higher for personal offence (33%) while the face-to-face survey remains lower still for property offences (-25%) and also lower for personal offences (-13%).

Conclusion

Even if demographic weighting had worked in correcting the sample profile, it would need to be rejected on the grounds that the basic assumption it relied on - of there being no difference between responders and non-responders - was patently untrue. While we think the approach focusing on correcting bias in victimisation might have had some theoretical potential, it has done little to change the victimisation rates in either survey and the 'victim' bias remains unresolved by this approach.

« Previous | Contents | Next »