The Scottish Government

« Previous | Contents | Next »

Listen

1. Introduction

In June 2004, the Scottish Executive commissioned BMRB Social Research to undertake the Scottish Crime and Victimisation Survey ( SCVS), a new survey of victimisation in Scotland. In two distinct ways the SCVS was a radical departure from the previous sweeps of the Scottish Crime Survey ( SCS) that had been undertaken in Scotland since 1993. First, the sample size was increased from 5,000 interviews every three years to an annual sample of 27,000 with continuous interviewing. More importantly, the survey method was changed from a face-to-face survey to a telephone survey.

These changes were the outcome of a fundamental review of the SCS undertaken in 2003. ¹ This review set out the limitations of the previous sweeps of the SCS. These related principally to the relatively small sample of 5,000 households, which meant that victimisation could only be reported for Scotland as a whole; that trends were difficult to identify accurately and some of the results for individual types of victimisation were unreliable because of the small number of respondents reporting them. This particularly affected low-incidence but high-profile crimes such as crimes of violence.

The key recommendations of the Review were that the Scottish Crime Survey should:

• switch to computerised data collection
• have a significantly increased sample size, and
• have continuous fieldwork.

There was also a strong recommendation for the survey to have an element of telephone interviewing. Indeed, it was argued that "including a significant telephone component is fundamental, in that the objectives identified in the review cannot be met without doing so". ² Three issues took the review to this conclusion: the recommended increase in sample size could not be achieved at reasonable cost without switching to telephone; it would be difficult to find enough interviewers to conduct an expanded SCS using face-to-face interviewing and other methods (such as postal surveys) were inherently unreliable.

A number of risks were associated with telephone interviewing, such as:

• a reduction in the survey response rate from 70-75% to about 60% seemed likely ³
• the need for a shorter interview of around 20 minutes rather than the 30-40 minutes of the SCS
• less control over sampling than is offered by the Postcode Address File ( PAF) used for face-to-face interviewing because of the weaker link between administrative geography and the coverage of telephone exchanges, and
• poorer quality information and difficulties with sensitive topics. ⁴

However, these were not seen as deterrents to using telephone interviewing. Indeed the Review was optimistic about the transition to telephone and expected little change in the measures recorded by a telephone survey compared with face-to-face interviewing: "Our expectation is that the move to telephone will not lead to a variation in victimisation recording rates that would be significant enough to impact on policy studies to a worrying extent". ⁵

Nevertheless, the change in data collection method from face-to-face to telephone interviewing represented the potential for change in the data series established by the Scottish Crime Survey. Reflecting this, the Scottish Executive commissioned MORI Scotland and TNS Social to undertake a parallel face-to-face survey designed as a repeat of the previous waves of the SCS, although with a smaller sample of 3,000 interviews, to provide a measure of victimisation against which the telephone survey could be compared.

This report presents the findings of this comparison exercise. However, the objective is not simply to compare the two surveys as though one was a potential substitute for the other and the only question is which provides the better estimates. The McCaig Review did not question the ability of face-to-face interviewing per se to provide accurate estimates, merely whether the sample was large enough to provide the level of precision required and whether a sample of the size required could be achieved at a cost the Scottish Executive might be willing to bear. In general, there is no question that face-to-face victimisation surveys such as the British Crime Survey, based on large random samples with high response rates, are able to provide accurate estimates. The basic question underpinning the calibration exercise reflects the shift in survey method: does a crime survey conducted by telephone provide an accurate estimate of victimisation among the adult population resident in private households in Scotland?

Although the change introduced by a telephone survey is the main focus of the analysis, the question of accuracy does not apply solely to the telephone survey. All surveys are only estimates of what they seek to measure and like all surveys the face-to-face survey is subject to its own errors and biases. Indeed, there is some circularity in gauging the success of the telephone survey by using an approach whose weaknesses the telephone was designed to overcome (and a smaller and less accurate version of it). In assessing the telephone survey we cannot rely solely on the 3,000 face-to-face interviews and where appropriate we also provide comparisons of the telephone survey with the 2003 sweep of the Scottish Crime Survey (5,041 interviews), the 2003 Scottish Household Survey (approximately 15,000 interviews) ⁶ and the 2001 Census.

Also, recognising the limited ability of the face-to-face survey to give a definitive answer to the question of accuracy, we have stepped back from a simple comparison of the telephone and face-to-face results and gone back to first principles to ask what features of a survey give us confidence in its estimates, and how far does the telephone survey meet these criteria? Four key aspects of survey accuracy, which can be applied to any survey, provide a structure for our analysis. To provide clear evidence and reassurance of reliability a survey should show:

• a sufficiently large sample
• selection from a sampling frame that covers the population of interest, giving each a known, non-zero probability of selection
• a high response rate from the survey fieldwork, ensuring that the randomness of the sampling is carried into the achieved sample, and
• the absence of observable non-sampling bias in the data, implying that as well as being a reflection of the population on demographic measures, the achieved sample is likely to be a reflection of the population on those measures the survey is intended to estimate.

Thus, the task of the analyses presented here is to assess the extent to which the telephone survey exhibits features that would lead to the conclusion that its estimates of victimisation are likely to be accurate.

The report is structured as follows.

• Chapter 2 provides an overview of the surveys and the sample design used for each
• Chapter 3 looks at sample coverage and survey response rates for both the telephone and face-to-face surveys
• Chapter 4 examines the nature and extent of bias in both surveys and more generally examines the extent to which the surveys provide similar estimates for similar issues
• Chapter 5 looks at the key measures of victimisation provided by each survey as well as other measures collected by both surveys.
• Chapter 6 draws together the findings into our conclusions.

In addition, the calculation of response rates for telephone surveys is a complex and contested issue, with few standard definitions and no standard approach across practitioners. The assessment of which telephone numbers are eligible to take part is particularly problematic. Appendix 1 gives details of the method used to estimate the proportion of non-contact telephone lines that should be considered eligible in calculating the survey response rate. Finally, Appendix 2 provides details of the weighting approaches that were used to try to identify bias and correct bias in the telephone survey.

« Previous | Contents | Next »