The Scottish Government

« Previous | Contents | Next »

Listen

3. Sample coverage and response rates

3.1 Sampling frames

Survey samples are limited by the extent to which the source from which potential respondents are to be selected - the sampling frame - completely enumerates the population that the survey intends to represent. To the extent that the frame does not completely cover the population, there is scope for bias, although the extent to which frame under-coverage leads to bias depends on the extent to which the reason for non-enumeration is related to the survey measures. For example, the small user PAF does not include communal establishments such as student halls. A survey of student living expenses screened from a PAF sample is likely to be systematically biased.

Each potential respondent should, ideally, only be represented once in the sampling frame, giving each person the same probability of being selected. ¹³ However, provided the potential for multiple listing is known and information is collected to enable this to be accounted for after the survey, over-coverage is not a significant problem. For example, the fact that some households have more than one telephone line on which they can receive incoming calls increases their probability of inclusion in a telephone survey sample. Provided the number of landlines is established as part of the survey, weighting can restore each household's probability of selection to equality.

3.1.1 The limitations of PAF

Since the mid-1980s the Postcode Address File ( PAF) has become the best and most widely used sampling frame for general population surveys. Apart from the SCVS, it is the sampling frame for all major government surveys in Scotland including the Scottish House Condition Survey, the Scottish Household Survey and every recent sweep of the Scottish Crime Survey. PAF is a listing of postal delivery points and the small user subset used for survey sampling is known to include small offices, shops and other non-residential addresses. Vacant properties are also included in the sample. Together this ineligible sample typically represents about 10% of all sampled addresses, imposing a small fieldwork overhead cost to identify and screen them out. Most of the ineligible addresses are readily identified and screened out by interviewers in the field although there remains the possibility of a small proportion that, for example, might be vacant but are not obviously so. The extent of this is unquantifiable but is not considered a significant problem and these will remain part of the eligible sample with the effect that they tend to deflate the survey response rate by a small amount.

In terms of people excluded from PAF samples there are two issues: the exclusion of communal establishments such as prisons, hospitals, nurses' homes and student halls, and the exclusion of private dwellings within military bases and other sites that would be difficult to access. The 2001 Census shows that 1.7% of the Scottish population live in communal establishments.

Where there are multiple households at a single delivery point - shared flats or bedsits - these will appear on PAF once and interviewers need to randomly sample from eligible households in the field. This under-coverage can be accounted for by weighting. A potentially more serious problem with PAF in Scotland is the listing of tenement flats as a single address when there are, in fact, many individual dwellings. This problem has largely been resolved by two means: the progressive renumbering of addresses to create individual entries on PAF or by the inclusion of an additional field - the Multiple Occupancy Indicator - that allows selection probabilities to be adjusted before sampling. The experience of the Scottish Household Survey has been that the MOI is wrong in only about 2% of sampled addresses.

The general assessment is that PAF provides excellent coverage of private households and given the focus of the SCS on private households, the limitations of PAF do not present any significant problem of bias. ¹⁴ For the purposes of the face-to-face survey, there is no issue of frame under-coverage.

Although not a limitation of PAF, face-to-face samples tend to be clustered to achieve some fieldwork efficiencies. The penalty for this clustering is that the standard errors for the survey are larger than for an unclustered sample of the same size.

3.1.2 The limitations of RDD

There is no question that RDD samples represent the best option for telephone surveys. In comparison with published directories, which exclude lines and households that have opted to be ex-directory, RDD ensures that all residential lines have an equal probability of inclusion in the survey sample. However, there are two major limitations imposed by random-digit dialling samples for telephone surveys. The first is that the sample is, by definition, limited to those who can receive telephone calls. Second, mobile phones are also excluded from most RDD samples because there is no link between mobile numbers and terrestrial geography. It is not possible to sample mobile phones located in Scotland without incurring a significant cost in screening a sample of all mobile numbers anywhere in the UK. Thus, RDD samples exclude households with no telephone and those who only have access to a mobile phone. The impact of this is discussed more fully below.

In addition, because of the way RDD samples are generated, it is not known how many of the numbers generated are working residential lines. This creates a requirement to 'ping' all the numbers that have been generated and remove those that are not working lines or where faxes or modems answer the test on the line and then to remove known business numbers by matching with directories. Both 'pinging' and directory matching can be done automatically.

The sample that is then issued for interviewing is likely to still contain a significant proportion of business numbers, these mainly being unlisted direct-dial numbers on internal business exchanges. These must be screened out in the course of fieldwork.

Finally, there remains a proportion of numbers that cannot be classified. These are lines that ring out or are always engaged and while they might be eligible residential lines, they might be dedicated computer lines (so not eligible) or they might be public call boxes (ineligible) or perhaps dwellings that are temporarily vacant (also ineligible).

The treatment of these lines varies in the UK. In some studies they are treated as ineligible and so excluded from response rate calculations. ¹⁵ This assumes that if contact cannot be made after 10 attempts, the line must somehow be not working or otherwise ineligible. ¹⁶ In others, ¹⁷ some attempt is made to determine the proportion of lines that should be regarded as eligible on the assumption that there might be valid reasons why an eligible line might be difficult to contact.

In the United States, where telephone interviewing is longer established, this question has been addressed and there are clear recommendations from the American Association of Public Opinion Research ¹⁸ on the need to estimate the proportion of the sample of unknown eligibility that should be regarded as eligible. The size of the group of unknown eligibility and the proportion that is taken as eligible can have a significant impact on the survey response. Our approach to estimating this is discussed in Appendix 1.

3.2 Mobile telephone and mobile-only households

In considering the accuracy of estimates from telephone surveys, mobile-only households are of particular interest for two reasons:

• the potential for the composition of such households and their experience of victimisation to differ from that of the population as a whole, representing a source of bias in telephone surveys at any point in time
• the possibility that recent trends in the growth of mobile-only households might continue, representing a growing problem of bias and instability in time series data.

In recent years, two broad trends have been apparent. First, the proportion of households with no telephone has declined to the extent that for all practical purposes, coverage is complete. Secondly, the proportion of households with only a mobile telephone has grown quickly in recent years and, according to monitoring data from Ofcom, these households have distinctive characteristics. In this respect the McCaig Review was wrong to see mobile-only households as a problem to be addressed some time in a future that did not need to be considered in 2003. ¹⁹

3.2.1 Growth in telephone coverage

Research by ONS between 1998 and 2002 has shown that the proportion of UK households with no telephone declined from 4% in 1998 to 1% in 2002. ²⁰ The proportion with only a mobile telephone grew from less than 1% in 1998 to 4% in 2002. This research also showed increases in various telephone-related barriers to contact such as answering machines and call screening devices, which are discussed below.

Most recently, the position shown by Ofcom in its most recent report of telephony in the UK from February 2004, is that only 1% of homes in the UK have no telephone and that mobile-only households comprise 7% of all households. ²¹

Figure 1 - Telephone access in UK homes, February 2004

Percentage of homes

Note: Data at February 2004
Base: 2131 (Don't Knows are excluded)
Source: Ofcom

3.2.2 The characteristics of mobile-only households

As Figure 1 shows, the recent estimate from Ofcom is that 7% of homes in the UK are mobile-only. This is reflected in the results from the face-to-face survey, which found that while 91% of households had at least one fixed line telephone, 7% had only a mobile phone and 2% had no telephone. A more recent estimate, from Jan-March 2005 showed that 11% of households in Scotland had either no telephone (2%) or only a mobile telephone (9%). ²² Thus, the likelihood is that a telephone survey in Scotland can only represent about 90% of households and about 1 in 10 are automatically excluded by virtue of having no fixed telephone line. ²³

This is only a serious problem if the 1 in 10 differ significantly from the others and, most importantly, if it is likely that they will differ in terms of their experience of victimisation. Both the face-to-face survey and Ofcom's data show that mobile-only households do indeed have distinctive characteristics and are different in ways that might be related to victimisation. They are more likely to be:

• Young - the face-to-face survey estimates that 9-16% of respondents aged 16-34 years are in mobile-only households. Ofcom makes similar estimates.
• Unemployed - Ofcom estimates that 26% of unemployed households are mobile-only
• Low-income - Ofcom estimates that between 10% and 15% of households with an income of less than £17,500 are mobile-only
• Living in deprived areas - the face-to-face survey estimates that 16% of households in the most deprived 10% of areas (using the Scottish Index of Multiple Deprivation) had no phone (4%) or were mobile only (12%).
• In single-adult households - 16% of single adult households in the face-to-face survey had no phone (4%) or were mobile-only (12%).
• In rented tenures - 21% of households in rented tenures in the face-to-face survey are mobile-only.

This combination of characteristics suggests that regardless of the quality of the RDD sample, the scale or quality of the fieldwork effort established to translate a high proportion of eligible sample into interviews or its success in doing so, telephone surveys are significantly restricted by the inability of an RDD sample to reach a sizeable proportion of the population whose experience might be important to accurate estimates of the prevalence of victimisation in Scotland.

The extent of victimisation among mobile-only households and households with a fixed line telephone is discussed below.

3.3 Telephone Preference Service

The Telephone Preference Service ( TPS) is a central opt-out register where households can register their wish not to receive unsolicited sales and marketing telephone calls. It is a legal requirement that direct marketing companies do not make such calls to numbers registered on the TPS but TPS does not apply to market or social research. However, although it not required, some research companies routinely screen telephone samples and remove TPS subscribers, others vary their practice depending on the survey and other never do.

In this exercise, the RDD survey suffered a major error in that during the sampling process telephone numbers registered with the Telephone Preference Service were mistakenly screened out of the numbers that had been randomly generated for the telephone survey. This error was not detected until after six months of interviewing meaning that TPS subscribers were missing from the sample used for comparison with the face-to-face survey. The effect of this is to remove a group of eligible respondents from the sample for the survey.

There are three questions that need to be considered about this:

• what proportion of eligible households were removed from the sample?
• what is the impact of screening out TPS subscribers on the representativeness of the SCVS sample?
• what is the impact on the survey's estimates of victimisation?

We deal with the first two of these here and address the impact on victimisation in Chapter 5.

3.3.1 TPS registration in Scotland

Estimates of the proportion of domestic telephone lines registered with TPS are not routinely published but registration has increased substantially in recent years. In October 2003, 8% of residential GB telephone lines were registered with TPS. ²⁴ In June 2004, just before the telephone survey fieldwork started, an estimated 20% of UK lines were registered. The most recent estimate from April 2005 is that 42% of residential lines in Scotland are registered with TPS.

Table 1 shows the number and proportion of TPS subscribers in each of the standard regions of the UK and demonstrates a distinct Northern bias in favour of TPS registration - perhaps exhibiting more of a 'belt and braces' approach of being both ex-directory and registered with TPS.

Table 1 - TPS subscription rates in each standard region in the UK

Source: UK Changes. Figures refer to April 2005

By checking the TPS status of achieved interviews from the Jun-Sept sample is was possible for BMRB to estimate that had TPS registrants not been screened out, 24% of the achieved interviews would have been with TPS subscribers, suggesting that the screening removed approximately 22% of households from the telephone survey sample used between July and September 2004 - the fieldwork period being considered in this analysis.

3.3.2 Impact on sample representativeness

Using the sample from January 2005, we can compare the profiles of TPS subscribers and non-subscribers to estimate the impact of screening subscribers out of the June-September RDD sample.

Table 2 compares the main demographic variables from the January 2005 sample, of which 38% were TPS subscribers, with the sample from June-September sample (wholly non-subscribers).

Table 2 - Comparison of TPS and non- TPS subscribers from Jan 2005 with Jun-Sept 2004 sample, 2003 SHS and 2001 Census

* includes 13 cases whose TPS status is missing

This shows that TPS subscribers are more likely to be owner-occupiers (almost 90%) living in detached or semi-detached houses, with two or more cars, in two adult households. Compared with an estimate of 63-64% from the 2003 Scottish Household Survey and the 2001 Census, the June-September sample recorded 67% owner-occupation and the January 2005 sample records almost 70% owner-occupation. In January 2005 there is almost a 30% shortfall in the proportion of the sample from social rented tenures (17% compared with 27-28% in the two comparator surveys). Reflecting this, the proportion of households with no car has also deteriorated from 28% in June-September to 25% in January 2005 compared with 34-35% in the comparator surveys.

In terms of household size, the number of adults in responding households shows some improvement after the inclusion of TPS households. The number of adults in households in the January 2005 sample is very close to the estimate from the SHS.

Overall, the impact of including TPS households in the sample seems to be to exacerbate some of the demographic bias observed in the June-September sample, which, as we discuss below, already showed bias in favour of owner-occupiers and against households with no car. Adding TPS households brings the distribution of the number of adults in households closer to the SHS.

The exclusion of TPS households means that the June-September sample had a significant problem of sample under-coverage. Approximately 10% of households were excluded because they have no phone or were mobile-only and a further 22% of households were missed because TPS subscribers were screened out. Combining the two suggests that almost one-third of all Scottish households were not covered by the sample used for the telephone survey.

The exclusion of TPS households is likely to partially compensate for the exclusion of mobile-only households since the two groups seem to be very unlike each other in demographic terms. Where respondents in mobile-only households tend to be in rented tenures and in relatively deprived areas, TPS subscribers are more likely to be owner-occupiers and relatively more affluent.

3.4 Survey response rates

Survey response rates are a key measure of survey quality. The response rate indicates the scope for the achieved sample to differ from the population as a whole through differential non-response. For example, if those who are regularly not at home are at higher risk of victimisation than those who stay at home, a high level of non-contact indicates significant scope for bias since those most readily interviewed might be expected to be people least likely to be out of the house and, consequently, at lower risk of victimisation. In this scenario, estimates derived from a survey with a low response rate or a high non-contact rate will tend to underestimate victimisation. Alternatively, if people who have not experienced victimisation are more likely to refuse to participate, perhaps feeling that the survey is not relevant to them, estimates will tend to overstate victimisation. Of course, both might be true, making it a matter of chance whether the victimisation rates from a survey with a low response rate are accurate.

The importance of the response rate also helps to explain differing practice in relation to the treatment of non-contact in calculating response rates. If lines that cannot be contacted can be dismissed as ineligible, the response rate that can be claimed for the survey is (quite artificially) bolstered, making the survey appear to be more representative than it is or at least to have less scope for bias than it actually does.

3.4.1 Response to the face-to-face survey

The face-to-face survey achieved 3,034 interviews compared with an interview target of 3,038 (99.9%). This was achieved from a total of 5,018 addresses issued for long interviews, giving an unadjusted response rate of 60.5%. After removing ineligible addresses, the response rate from eligible addresses was 67.3%, 2.7% lower than the target of 70%. Thus, although the number of achieved interviews was close to the target, the response rate was 96.4% of the target. This reflects levels of deadwood varying across local authorities from that implied by the SHS experience.

The response rates to the face-to-face survey are summarised in Table 3.

Table 3 - Face-to-face survey response rate

This overall response rate disguises variation between different types of area as measured by the Scottish Executive's urban/rural classification and the Scottish Index of Multiple Deprivation, as Table 4 and Table 5 show.

Table 4 - Face-to-face survey response rate by urban/rural classification
Excludes ineligible addresses

One valid address did not match the lookup files for urban-rural classification

In terms of the urban/rural classification, response in the large urban areas - the four cities of Glasgow, Edinburgh, Aberdeen and Dundee - is substantially lower although this was anticipated in the sample design and response rate rates meaning that large urban areas are not under-represented in the achieved sample, making up 39% of the achieved sample compared with 40% of all eligible addresses.

Table 5 - Face-to-face survey response rate by SIMD deciles
Excludes ineligible addresses

Five valid addresses did not match the lookup file for SIMD

As with large urban areas, the lower response rates in deprived areas were largely anticipated in the sampling with the result that compared with 14% of eligible addresses in the most deprived areas, 12% of the achieved interviews in these areas.

Nevertheless, while the sample design might have been able to anticipate lower response and avoid under-representation - where response rates are lower, there is greater scope for bias and since both rurality and relative deprivation are linked to victimisation, this would be expected to have some impact on the estimates from the survey.

3.4.2 Response to the telephone survey

The telephone survey achieved 9,509 interviews compared with a target of 10,000 interviews (94.6%). This was achieved from a total of 43,504 telephone numbers issued to interviewers (after pinging and the removal of known business numbers), giving an unadjusted response rate of 21.9%. However, ineligible lines make up a much higher proportion of the issued sample than is generally the case in face-to-face surveys and after lines whose ineligibility has been established are removed from the total, the response rate increases to 38.4%. The full breakdown is shown in Table 6.

Table 6 - Telephone survey response rates

At first glance this looks more like the response rate of a good postal survey - a method dismissed in the McCaig Review as inherently unreliable - but this still presents only a partial picture of response to the telephone survey because of the high proportion of lines whose eligibility for the survey cannot be determined. This includes not only those lines where contact has been made but the eligibility the person or households could not be established (the fifth line of Table 6) but also the non-contact category (line 2) since without making contact with someone at the end of a telephone line, it cannot be clearly established if it is a residential number.

In the face-to-face survey, only 2% of the issued addresses were recorded as unknown eligibility and whether they are considered eligible or not can only affect the response rate by 0.3 percentage points. Counting these addresses as eligible gives a response rate of 67.5% whereas including them all as ineligible increases the response rate to 67.8%. In the telephone survey 32% of the issued sample that was not excluded as non-working or a business line was of unknown eligibility. How these are treated can have a significant impact on the survey response rate and this is discussed further below.

3.4.3 Unknown eligibility

Whereas the calculation of response rates for random face-to-face surveys is both well established, straightforward and consistently defined by survey practitioners, practice is less well established for telephone surveys.

In a face-to-face survey it is relatively simple for interviewers to decide if addresses are eligible for inclusion in the survey. First, they have to find the address and, standing outside, can observe if the address is obviously demolished, derelict, vacant or non-residential. If in doubt, the interviewer can attempt to determine if the address is occupied directly by approaching occupants or indirectly through neighbours. In no one answers, signs of occupation can be looked for - lights may be on, there may be sounds of occupation, there might be curtains at the windows, name plates on doors, furniture in rooms etc. In short, the number of instances when an interviewer on a face-to-face survey might be unsure of an address's eligibility should be small.

In contrast, if a telephone line is permanently engaged or ringing without answer it might be non-working or it might be a working residential line connected to the internet or to a call screening device. The interviewer or dialler knows nothing more about the line and cannot deduce or assume anything about its status. Once a telephone line has been dialled many times over an extended period and at different times of day and always with no clear outcome i.e. always ringing or engaged or answered by a machine, it is reasonable to question whether the line is eligible for the survey. However, there are many circumstances when an eligible line might never be answered, just as there are clearly circumstances when face-to-face interviewers cannot make contact at occupied addresses in spite of many calls over an extended period.

• The occupants are always out - this would include people who work away from home, work shifts, have long working hours or active social lives (eligible).
• If the line is primarily used for internet access and frequently engaged or if it is left unanswered for voice calls because friends and family use a mobile number, the line would still be eligible. If the line is used solely for internet access and does not even have a telephone attached, the line would be ineligible.
• The property is unoccupied but the line is still receiving incoming calls for example, it could be a rented flat between occupants (generally would be eligible but not while unoccupied).
• The line is at a second home or holiday home (ineligible).
• The line is at an occupied address but the household uses call screening (eligible).

In the telephone survey, if the entire sample whose eligibility was unknown were assumed to be eligible, the response rate would be 38.4%. However, if the entire unknown eligible sample is assumed to be ineligible, the response rate would be 54%. This would still be low by the standards of large-scale social surveys. The likelihood is that the proportion is neither zero nor 100% but apart from that, little is known about what proportion of these lines should be considered eligible. Yet, a reasonable estimate is important to the final survey response rate.

Our best estimate is that 73% of the lines that had no contact after 10 or more attempts or which were permanently engaged can be considered ineligible. The rationale for this is explained in Appendix 1 but this estimate gives a response rate of 49% for the telephone survey.

Table 7 - Telephone survey response rates

3.5 Comparing response to the telephone and face-to-face surveys

Table 8 below compares the response rate from the face-to-face survey and our best estimate of the response to the telephone survey and shows how the two differ (F-T = face-to-face rates minus telephone rates). Clearly, the fieldwork outcomes from the two surveys are substantially different. Most notably:

• The telephone response is substantially lower than the face-to-face response.
• The principal cause of the difference is a much higher refusal rate - 41% of eligible households refuse to take part in the telephone survey compared with just under 17% in the face-to-face survey.
• The non-contact rate for the telephone survey is lower.

Table 8 - Comparison of face-to-face and telephone survey response rates

This comparison suggests that one of the principal (methodological rather than cost) advantages of a telephone survey envisaged by the McCaig Review - improved contact rates - has been achieved to some extent. The Review argued that:

[while] the telephone approach will have a higher refusal rate than face-to-face it has the potential to improve on initial contact rates provided that the advantages of cheap contact are taken up. Most face-to-face surveys will have a callback requirement of three or four times but with telephone this can extend to ten or more with a higher proportion being undertaken at evenings and weekends. There can be much more controlled and effective cover of different times of day with the possibility of returning to non-contacts after a month or so to pick up households that were absent in the first contact period. It can be hypothesised that reducing non-contacts is more important than reducing refusals when estimating victimisation levels. ²⁵

It is not spelled out in the review what the hypothesised benefits of refusal over non-contact might be.

3.6 Conclusion

Population coverage is an essential feature of any survey sample and our analysis suggests that while PAF gives good coverage of the population in private households, small proportion of the adult population living in communal establishments is excluded. This represents 1.9% of the adult population. Frame under-coverage excludes an estimated 9% of households in Scotland from the telephone survey. Although this means that the telephone survey can still cover 91% of households, the fraction that is not represented has distinctive characteristics that are likely to be related to victimisation, meaning that it represents a source of bias in the telephone survey. We would expect the omission of these groups to tend to understate victimisation.

In addition to the exclusion of households that cannot be covered by an RDD sample that excludes mobile phones, the mistaken exclusion of TPS subscribers has further reduced the survey coverage by 22% of households. In total, the sample for the telephone survey excluded around 30% of households in Scotland. Our analysis of the characteristics of TPS respondents shows that they have distinctive characteristics, being 90% owner-occupiers and more likely to live in multi-adult households, in detached and semi-detached properties and to have one or more car. These characteristics will have implications for assessing the extent to which non-response (rather than sample under-coverage) has affected the profile of the telephone survey sample and the estimate of victimisation.

The telephone survey is also achieving a response rate that is substantially lower than the face-to-face survey, caused by a substantially higher refusal rate of 41% compared with 17%. While there is certainly scope for the face-to-face survey to be biased, the combination of frame under-coverage, exclusion of TPS and low response means that it is unlikely that the sample achieved by this sweep of the telephone survey would have avoided substantial bias.

« Previous | Contents | Next »