Scottish Household Survey 2019: methodology and fieldwork outcomes

Methodology of the Scottish household survey 2019 and information on fieldwork targets and outcomes.


6 Survey Response

Summary

  • The final number of social survey interviews achieved was 10,577. This represents a response rate of 63 per cent and exceeds the target of 10,450 interviews.
  • The SHS response rate has fallen in recent years, having fallen by three percentage points since 1999/2000. The response rate of 63% in 2019 was one percentage point lower than the previous four years and four percentage points lower than the 2014 response rate of 67%.
  • The long-term average response rate for 1999-2011 was 67.9%. However, it should be noted that the calculation has changed slightly for 2012 onwards as a portion of the addresses of unknown eligibility are considered to be eligible whereas previously they would all have been classed as ineligible. This calculation change would have led to a lower response rate in years prior to 2012, if it had been calculated on the same basis.
  • There was a target of at least 80 completed physical surveys for each local authority along with a target of 3,004 surveys for Scotland.

6.1 Introduction

This section presents the fieldwork outcomes for the sampled addresses.

The final number of social survey interviews achieved was 10,577. This represents a response rate of 63 per cent and exceeds the target of 10,450 interviews. Since the design changes to the SHS were introduced in 2012 the target number of interviews has been met in recent years, namely 2017 to 2019. However, the SHS response rate has fallen by three percentage points since 1999/2000 and the impact of this drop in response rate is discussed in section 6.3. The performance of the physical survey is described in section 6.5.

The SHS response rate has fallen in recent years, having fallen by three percentage points since 1999. The response rate of 63% in 2019 was one percentage point lower than the previous four years and four percentage points lower than the 2014 response rate of 67%. The maintenance of the response rate in a climate of declining response rates on other population surveys is commendable.

Previously the contractual required response rate was 67%. However, in 2017, the SHS steering group agreed to lower the target in the new contract. The group agreed on a 65% rate for the contract specification 2018-2021. The 2018-2021 contract states that "the target response rate for the survey will be at least 65%".

The SHS 2018-2021 contract states that "The household response rate assumption for each Local Authority will be updated annually and set as the mean response rate for each Local Authority over the last three sweeps of the survey for which response rate data are available. Response rate here is the successful completion of the household interview." This is subject to the following conditions: (i) the response rate assumption for any Local Authority will not be below 55%; (ii) the response rate assumption for any Local Authority will not be above 80%; and (iii) the Scotland level response rate will not be below 65%. If the third condition is at risk of being breached then each Local Authority's response rate assumption will be uniformly increased to ensure the Scotland level response is 65% or above for each survey year.

Survey response is an important indicator of survey quality as non-response can introduce bias into survey estimates. Standardised outcome codes (based on an updated version of those published in Lynn et al (2001)[20]) for survey fieldwork were applied across the SHS, SHeS and SCJS. The outcome codes paper includes guidance on the appropriate categorisation of interview outcomes. This will allow consistent reporting of fieldwork performance and effective comparison between the performance of the surveys.

6.2 Scotland-Level Summary

Table 6.1 shows a detailed breakdown of the SHS response for all sampled addresses for Scotland. The addresses of unknown eligibility have been allocated as eligible and ineligible proportional to the levels of eligibility for the remainder of the sample. This approach provides a conservative estimate of the response rate as it estimates a high proportion of eligible cases amongst the unknown eligibility addresses.

The table shows that the overall household response rate was 62.6 per cent. This is below the long-term (1999/2000 to 2011) average response rate for the SHS of 67.9 per cent, below the 2012 to 2014 rates (67%) and below the 2015 to 2018 rates (64%). The effects of the drop in response rate are discussed in section 6.3. It should be noted that from 2012 the calculation was changed slightly and a portion of the addresses of unknown eligibility are now considered to be eligible, whereas previously they would all have been classed as ineligible.

Table 6.1: Fieldwork outcomes, Scotland
Fieldwork Outcome Sample % of issued % of eligible
Responding households 10,577 57.4% 62.6%
Random adult interview complete 9,776 53.0% 57.9%
Refused
Office refusal 733 4.0% 4.3%
Refusal at introduction/before interview 3,806 20.7% 22.5%
Broken appointment - no re-contact 237 1.3% 1.4%
Total refused 4,776 25.9% 28.3%
Non-contact
No contact with anyone at the address 488 2.6% 2.9%
Contact made at address, but not with target respondent 366 2.0% 2.2%
Total non-contact 854 4.6% 5.1%
Other non-response
Ill at home during field period 31 0.2% 0.2%
Away or in hospital throughout field period 124 0.7% 0.7%
Physically or mentally unable/incompetent 209 1.1% 1.2%
Language barrier 63 0.3% 0.4%
Lost interview 0 0.0% 0.0%
Other non-response (not covered by categories above) 0 0.0% 0.0%
Total other non-response 427 2.3% 2.5%

Unknown eligibility
 
Inaccessible 0 0.0%
Unable to locate address 57 0.3%
Unknown if occupied, due to non-contact 209 1.1%
Other unknown eligibility 4 0.0%
Total unknown eligibility 270 1.5%  
Estimated eligible addresses in set of unknown eligibility addresses 253 1.4% 1.5%

Total eligible addresses
16,887 91.6% 100.0%

Not eligible
 
Not yet built / under construction 17 0.1%
Demolished/derelict 51 0.3%
Vacant/empty 887 4.8%
Non-residential 241 1.3%
Address occupied but not resident household 266 1.4%
Communal establishment / institution 26 0.1%
Other ineligible 37 0.2%
Estimated ineligible addresses in set of unknown eligibility addresses 17 0.1%
Total not eligible 1,542 8.4%  
All issued addresses 18,429 100.0%  

6.3 Drop in response rate

The response rate of 63% in 2019 was one percentage point lower than the previous four years and four percentage points lower than the 2014 response rate of 67%. Response rates are shown in the Table 6.2 below.

Table 6.2: Response rates over time
1999/00 2001/02 2003/04 2005/06 2007/08 2009/10 2011 2012 2013 2014 2015 2016 2017 2018 2019
Response rate 66% 67% 68% 69% 66% 69% 69% 67% 67% 67% 64% 64% 64% 64% 63%
Achieved sample 30,227 30,639 30,822 31,013 27,238 28,404 14,358 10,644 10,652 10,633 10,325 10,470 10,683 10,532 10,577

The long-term average response rate for 1999-2011 was 67.9%. However, it should be noted that the calculation has changed slightly for 2012 as a portion of the addresses of unknown eligibility are considered to be eligible whereas previously they would all have been classed as ineligible. This calculation change would have led to a lower response rate in years prior to 2012, if it had been calculated on the same basis.

Due to a drop in the response rate in 2015, an analysis was conducted to investigate the likely impact of this drop in response rates. This was undertaken by modeling the 2014 sample results to examine what the impact of a 3% drop in response rates would have been had a lower number of interviews been achieved, and comparing this against the 67% full sample results.

Analysis showed that the demographic composition of the sample was largely unchanged (age, gender), with only the most deprived SIMD quintile and other urban areas showing a 1 percentage point drop in their respective shares of the total sample.

Analysis of mean differences in the population estimates from the two samples for a basket of full sample questions from the SHS, and full one third sample questions from the SHCS module, were also undertaken.

This showed that the absolute mean differences for the total population estimates across the different basket of questions within the household and random adult parts of the survey, including the Scotland Performs National Indicators, were very small, at around 0.1 percentage points.

A few sub-group categories had one or two 'maximum' differences in estimates of around 1 percentage points, including age and social and private-rented sector sub-groups. However, these differences are unlikely to be statistically significant due to small subgroup sample sizes.

Sub-national analysis was not considered. It is expected that there would be a greater impact of this lower response rate for Local Authorities and other sub-national geographies.

6.4 Local authority performance

Table 6.3 shows levels of ineligible addresses, response rate and random adult conversion.

Table 6.3: Response rate and eligibility by local authority

    Ineligible addresses Responding households Random adult interviews
Local Authority Sampled addresses n % n % of eligible n % of HH ints
Aberdeen City 677 122 18% 314 55% 304 97%
Aberdeenshire 620 63 10% 363 65% 342 94%
Angus 472 39 8% 251 58% 234 93%
Argyll and Bute 457 89 19% 249 68% 243 98%
Clackmannanshire 372 24 6% 222 64% 213 96%
Dumfries and Galloway 431 43 10% 277 71% 256 92%
Dundee City 411 38 9% 244 65% 219 90%
East Ayrshire 488 31 6% 275 60% 259 94%
East Dunbartonshire 468 13 3% 276 61% 258 93%
East Lothian 431 30 7% 297 74% 273 92%
East Renfrewshire 457 17 4% 259 59% 235 91%
Edinburgh City 1,489 110 7% 871 63% 839 96%
Falkirk 442 19 4% 237 56% 223 94%
Fife 911 68 7% 526 62% 477 91%
Glasgow City 1,987 163 8% 977 54% 853 87%
Highland 648 88 14% 345 62% 329 95%
Inverclyde 435 32 7% 247 61% 236 96%
Midlothian 457 15 3% 291 66% 279 96%
Moray 431 42 10% 256 66% 225 88%
Na h-Eileanan Siar 383 49 13% 278 83% 275 99%
North Ayrshire 481 41 9% 251 57% 233 93%
North Lanarkshire 885 49 6% 522 62% 462 89%
Orkney 367 50 14% 258 81% 246 95%
Perth and Kinross 428 22 5% 237 58% 203 86%
Renfrewshire 478 14 3% 263 57% 249 95%
Scottish Borders 432 42 10% 267 68% 239 90%
Shetland 376 40 11% 244 73% 217 89%
South Ayrshire 449 43 10% 263 65% 256 97%
South Lanarkshire 862 60 7% 480 60% 423 88%
Stirling 376 27 7% 240 69% 212 88%
West Dunbartonshire 426 22 5% 243 60% 236 97%
West Lothian 402 20 5% 254 66% 228 90%
Scotland 18,429 1,525 8% 10,577 63% 9,776 92%

Argyll and Bute, Aberdeen City, Orkney, Highland, Na h-Eileanan Siar, Shetland and Aberdeenshire[21] were the areas where the highest levels of ineligible addresses were recorded. For Argyll and Bute and Na h-Eileanan Siar, high levels of ineligible addresses were expected as both areas contain a high number of holiday and second homes which are not eligible for the survey. Expected levels of ineligible addresses for Highland, Orkney and Shetland were all above 10%.

The two lowest household response rates in 2019 were in Glasgow City (54%) and Aberdeen City (55%). In addition, seven other local authorities had a response rate under 60 per cent in 2019.

The conversion from household interview to random adult completion was
92% in 2019 (this was 92% in 2016 to 2018, 91% in 2015, 92% in 2014 and 93% in 2012 and 2013).

6.5 Monitoring and reducing the respondent burden

The Code of Practice for Statistics[22] states that "Statistics producers should be transparent in their approach to monitoring and reducing the burden on those providing their information, and on those involved in collecting, recording and supplying data." Furthermore it states that "The burden imposed should be proportionate to the benefits arising from the use of the statistics."

The following steps are/have been taken to reduce respondent burden.

  • Addresses selected for any of the surveys (SHS, SHeS, SCJS) are removed from the sample frame for a minimum of 4 years so that they cannot be re-sampled for another survey.
  • Samples sizes are no greater than required to obtain robust local authority estimates.
  • The size of the SHS questionnaire was not allowed to increase following the recent consultation[23] on its contents. Where there was a strong policy need for new questions, the space/time required for these new questions was created by dropping existing questions or moving them to a biennial basis.
  • The SHS social interview is carried out using Computer Aided Personal Interviewing (CAPI). Routing is built into the CAPI script which ensures that respondents are only asked questions that are relevant to them, e.g. respondents are only asked questions on mortgage repayments if they have a mortgage.
  • Interviewer briefings are held annually. These events provide interviewers with an opportunity to feedback on improvements that could be made to the questionnaire to assist them conducting interviews and reduce the burden on respondents.
  • Interviewers were asked to choose the logo on the tote bag which was given to respondents as an incentive to complete the survey in 2019.
  • The Scottish Household Survey team engages its users in the design of new products and in the steering and evaluation of the dissemination of the survey results. The Scottish Household Survey run an annual user day and we normally have around 70-80 attendees every year. We advertise this on the Scotstat email distribution list and through twitter. In addition, we send out annual report evaluation questionnaires to our users to find out views of our publications.
  • For large-scale new developments such as the change away from our Excel tables to our new web-based data explorer, we have collected views user through quantitative questionnaires – but also through face-to-face testing and meetings with local authority staff.

6.6 Physical survey

As described in section 2, just under half of the SHS sample was assigned to the physical survey module. For completion of the physical survey, respondents had to agree to make an appointment for a surveyor to make a follow-up visit and to complete the appointment. Table 6.4 shows the number of households assigned to the physical module which responded to the main SHS and the conversion to completion of the physical survey.

There was a target of at least 80 completed physical surveys for each local authority along with a target of 3,004 surveys for Scotland. The result of continuing lower than estimated levels of conversion from household survey to physical survey, meant that there were fewer than 80 physical survey responses for ten of the local authorities in 2019[24]. However, the decision to 'open up' an additional stream to the physical survey gave a lower shortfall on the minimum target of 80 per local authority than in previous years (11 local authorities in 2018, 9 in 2017, 16 in 2016, 17 in 2015, 24 in 2014, 21 in 2013 and 20 in 2012). The 2,997 surveys achieved for Scotland in 2019 was 7 fewer than the target of 3,004.

Table 6.4: Conversion to full physical survey
Local Authority Conversion rate from household interview to physical survey Physical survey complete
Aberdeen City 63.1% 77
Aberdeenshire 59.0% 82
Angus 73.2% 90
Argyll and Bute 60.3% 70
Clackmannanshire 71.0% 71
Dumfries and Galloway 72.0% 90
Dundee City 61.6% 77
East Ayrshire 52.6% 70
East Dunbartonshire 59.4% 82
East Lothian 57.5% 77
East Renfrewshire 62.7% 84
Edinburgh City 57.3% 208
Falkirk 70.3% 90
Fife 59.1% 140
Glasgow City 55.1% 225
Highland 64.7% 86
Inverclyde 53.4% 71
Midlothian 59.2% 90
Moray 66.1% 82
Na h-Eileanan Siar 61.8% 89
North Ayrshire 71.7% 86
North Lanarkshire 51.8% 103
Orkney 82.6% 100
Perth and Kinross 72.1% 88
Renfrewshire 48.2% 54
Scottish Borders 54.2% 77
Shetland 76.6% 85
South Ayrshire 68.9% 82
South Lanarkshire 52.6% 110
Stirling 64.9% 74
West Dunbartonshire 65.1% 82
West Lothian 76.6% 105
Scotland 61.9% 2,997

Contact

Email: shs@gov.scot

Back to top