« Previous | Contents | Next »
Listen
6. Conclusions
At the outset we established four key attributes of a survey which, if satisfied, would provide reassurance that estimates derived from the survey were likely to be an accurate reflection of the population. In this section we restate those attributes and, drawing on the evidence from preceding chapters offer our conclusions about the extent to which the telephone survey appears to be meeting these requirements.
1. A large sample, randomly selected
There are no issues with the sample sizes or sampling method used for the telephone survey.
2. Selection from a sampling frame that completely covers the population of interest
RDD samples, unless they include all mobile phones, automatically exclude a sizeable sub-group of the population that has no telephone or only has access to a mobile phone. This group has quite distinctive characteristics, which are important for two reasons. First, in its own right, the groups who are mobile-only have characteristics that suggest they are more likely to experience victimisation. We have not been able to demonstrate whether victimisation differs among mobile-only households but their exclusion from the survey undermines confidence in the accuracy of the key survey estimates. Secondly, the groups who are excluded by being mobile-only tend also to be groups most likely to be affected by non-participation in the survey. Although this seems to be true of both the telephone survey and the face-to-face survey, the lower response rate to the telephone survey creates more opportunity for this to affect the telephone survey. Analysis of the face-to-face survey does not show significant differences in victimisation for no-phone and mobile only households but the sample size is too small to completely rule out bias in the telephone survey.
More generally, although outwith the scope of this exercise, the emerging evidence of an increasing role for mobile phones as the primary means by which many adults make and receive calls is likely to present increasing problems for RDD surveys in the future. 31
The exclusion of TPS subscribers was an unfortunate error. While we cannot demonstrate bias on the basis of the sample from January 2005, which contained TPS subscribers, we cannot rule it out.
3. Achieving a high response rate to ensure that the randomness of the sampling is carried into the achieved sample
Based on our estimate of the eligibility of telephone lines where no contact was made, the telephone survey achieved a response rate of approximately 49% and recent figures show that this has been declining. The main reason for the lower response rate is a higher refusal rate - 40% of eligible respondents refuse making the telephone refusal rate 140% higher than the face-to-face survey. Non-contact is lower than the face-to-face survey but in spite of the optimism of the Review, our interpretation of the survey's victimisation rates is that non-victims have been more likely to refuse to take part, leading to the paradoxical situation that although the exclusion of mobile-only households and evidence of bias within the data would lead to an expectation of lower recorded victimisation, the rates recorded by the survey are higher than those recorded by the face-to-face survey and the rates for personal crimes are generally higher than those recorded by the last full sweep of the SCS in 2003. Thus, we feel the Review was wrong to suggest that higher refusal was unlikely to have an impact on the survey measures. Indeed, refusal seems to be the principal cause of a bias in favour of victims.
4. The absence of observable bias
The telephone survey appears to differ from all comparator sources of data to an extent that cannot be explained by sampling error. To that extent we can conclude that the telephone survey appears to be systematically biased. Although one element of this bias will be the type of non-response bias that affects all surveys, there appears to also be a systematic bias against people who have not experienced any form of victimisation.
On the basis of these criteria, we have not found sufficient evidence to conclude that the telephone survey is likely to be accurately measuring victimisation. We have been unable to devise a weighting approach that satisfactorily corrects the many demographic biases that are observable in the data but even if these biases could be corrected, our conclusion is that the telephone survey suffers a more fundamental and irresolvable bias resulting from what appears to be a tendency for non-victims to be more likely to refuse to participate.
« Previous | Contents | Next »