« Previous | Contents | Next »
Listen
3 Data collection methods and instruments
3.1 Use of Computer Aided Personal Interviewing ( CAPI)
In common with many other large-scale government surveys, the SHS is carried out using Computer Aided Personal Interviewing ( CAPI). This offers a number of important advantages over traditional pen-and-paper interviewing for a survey of this kind. These include the following.
- CAPI allows greater complexity in questionnaire design, since routing and 'loops' in the interview can be automated and thus effectively hidden from the interviewer. It also eliminates the need for complex selection procedures during an interview, since random selection can be built into the program.
- Overall data quality is improved because the need for a separate data entry stage is eliminated and because automatic routing and range and logic checks reduce the scope for interviewer error.
- Preliminary data are available at the end of each day's fieldwork and the lack of a separate data entry stage allows faster turnaround of results more generally.
- The CAPI system generates detailed information about the timing and duration of interviews, allowing fieldwork to be monitored more closely.
Until 2007, the SHS script had been based on In2itive software. The inclusion of the Scottish Centre for Social Research means that the interview is now also scripted using Blaise. The two versions of the script collect exactly the same information and although there are small differences in how the systems operate, these create no observable differences in the survey data.
3.2 Questionnaire development and changes
The original SHS questionnaire was developed between August and December 1998 by the contractor team, working in conjunction with the Technical Group for the survey from the Scottish Executive. This followed a broad consultation exercise in which interested parties from a range of policy areas, academia, voluntary organisations and other bodies were invited to propose topics or specific questions for inclusion in the survey.
The core of the SHS questionnaire is intended to remain constant, but there is also scope for the inclusion of different modules over time. A simplified version of the questionnaire for each survey year as well a detailed Topic List giving a broader look at past and present SHS topics can be found on the SHS website at www.Scotland.gov.uk/SHS.
This consultative approach was followed for the 2007 script. Also, the Scottish Government undertook a fundamental review of the Scottish Household Survey in 2006 to ensure that the SHS continues to meet Scottish Government needs and is appropriately focused on policy areas that reflect Government priorities. As a result of this consultation and review process, the SHS questionnaire for 2007 is substantially changed compared with 2006. The accompanying volume on the survey questionnaire notes the changes made between 2006 and 2007, during 2007 and between 2007 and 2008.
3.3 Questionnaire structure, length and content
It was noted earlier that the questionnaire falls into two parts: the first collecting information about the composition and characteristics of the household from the Highest Income Householder or their spouse/partner; the second focusing mainly on the attitudes and experiences of a random adult member of the household. The former is intended to generate data representative of Scottish households and the latter data representative of the Scottish adult population resident in private households. It should be noted, however, that for reasons of space a handful of 'household' questions are also asked of the 'random adult'. These address household events or characteristics about which any adult member of the household would be likely to know.
A simplified version of the 2007 and 2008 questionnaire can be found on the SHS website ( www.Scotland.gov.uk/SHS). The broad topic areas, however, are as follows.
In the first half of the interview (with the highest income householder or spouse/partner), respondents are asked about:
- household composition and characteristics of household members
- type of property/accommodation
- cars in household and access to public transport
- children in the household, childcare, satisfaction with schooling and travel to school
- employment status of the highest income householder
- household income from employment and other sources
- savings and household finances.
In the second half of the interview (with the 'random adult'), respondents are asked about their own:
- housing experiences, including homelessness
- education qualifications
- perceptions of the local area
- travel to work or education and use of private and public transport
- congestion
- travel on the previous day
- perceptions of services and local government
- experiences of neighbourhood disputes
- volunteering
- participation in sports and cultural activities
- health problems and caring responsibilities
- employment status
- individual income from employment and other sources.
3.4 Problems and errors in the survey scripts
Scripting the SHS for 2007 encountered a number of inter-connected problems related to the number and nature of changes to the questionnaire, the introduction of new survey modules, the time available for scripting and testing, and the sequence of scripting and testing of the In2itive and Blaise questionnaires.
This caused a number of minor routing errors in the script - mainly in the culture and sport module - and the omission of some data from the main survey script. Two major problems arose with the In2itive software and a significant error in the Blaise script, which are detailed below.
3.4.1 In2itive software
Tendering for the SHS 2007-10 contract was on the assumption that Ipsos MORI and TNS would continue to use the same CAPI system as has been (very successfully) used since the survey started in 1999.
There were substantial revisions to the survey and for simplicity, the script was developed in three distinct parts reflecting the new requirements of the survey - the main SHS script, the Culture and Sport module and the revised travel diary. However, efforts to combine these into a single script were unsuccessful. We do not know precisely why it would not compile into one script. The best explanation is that the combined script breached some inherent programming limitation in the software such as the number of forms being processed or the number of routes involved in the script.
Attempts to compile the script and to investigate the problem delayed the start of fieldwork in 2007 and in the end, the need to get a working script into the field meant that an alternative solution had to be found. This was addressed by releasing the In2itive version of the SHS as two projects - one for Culture and Sport interviews and another for the main SHS. The main project was set up as two linked sub-scripts - the travel diary sits outside the main script as a linked module.
This set-up has very little impact on interviewers. The sample is pre-allocated to the streams required for all of the sub-sampling within the script and each script has been programmed to reject addresses allocated to the other script. The possibility of addresses being interviewed on the wrong script - main sample on the culture and sport script or vice versa - has therefore been removed.
Addresses are also identified as either Main sample or Culture and Sport on the contact sheets provided to interviewers. They therefore know which project to use, which showcards to use and how to introduce the survey.
In the main interview, where addresses are allocated to the travel diary stream, at the stage in the interview for completing the travel diary the interviewer sees a screen telling them to postpone the main interview and start the travel diary script. The transition is transparent to the respondent. However, there are instances of interviewers postponing the main script to undertake the travel diary and then being unable to restart the main interview.
Over the course of 2009, In2itive will be replaced by Nipo (for TNS interviewers) and SPSS Dimensions (for Ipsos MORI interviewers).
3.4.2 Errors in the In2itive and Blaise scripts
The problems with compiling the In2itive script had knock on effects on the quality of the scripts that were issued to interviews. Specifically, the intention of having a fully checked In2itive version of the questionnaire before the Blaise version was scripted proved impossible. As a result, both scripts were being checked and edited at the same time, within a timescale that was shorter than envisaged. This led, inevitably, to errors in both scripts. Most were minor and subsequent releases of the script corrected these.
A significant error in the Blaise script meant that the script went into the field without question HA3 which records the relationship of each household member to the Highest Income Householder. As well as omitting important information at this question, the information at this question is used in subsequent routing to control the collection of the contribution to household income made by the HIH's spouse (where there is one). This error affected 631 households interviewed between the end of January and the end of March. Of these, we had telephone numbers for 561. These respondents were recontacted to collect data for HA3 and household income for some other variables where data was missing. 6 The information was successfully collected from 542 people.
3.5 Survey fieldwork
The main fieldwork for the survey has an on-going monthly cycle. Interviewers are required to make up to six calls at an address (an initial visit plus five 'call-backs'). In addition to the immediate reissue of contact sheets that have been wrongly completed or where the required number of call-backs has not been made, there is an on-going programme of reissuing 'non-contacts' in a bid to maximise the response rate. At the end of each fieldwork year a significant number of valid but 'non-contact' addresses remain 'live'.
The response rates for the SHS need to take account of the continuous nature of the survey. The data file for each year will contain a small proportion of interviews conducted on sample drawn the previous year. Similarly some of the addresses issued during any year will not be carried out until after the data file has been closed for analysis. These interviews are carried into the next data file. The response rates therefore report the outcomes for addresses sampled for a given period regardless of when the interview was carried out. Details of the most recent response rates are given at paragraph 4.2.1.
Because of the problems with the scripts, the fieldwork for the 2007 survey did not start until 25 January 2007 and over the first two months of the survey it became apparent that the main interview was substantially longer than the planned 45 minutes and this was having a detrimental effect on the interviewers' ability to meet their fieldwork targets. In June 2007, it was agreed to revise the fieldwork targets for 2007 downwards by 500 to reflect this.
A substantial number of changes were made to the script in June 2007 to reduce the average interview length to the target 45 minutes. These mainly involved introducing more streaming into the survey, reducing the proportion of the full sample that were asked questions, although some questions were deleted in June. The questions collecting the ethnicity and religious persuasion of all household members were replaced by questions asked only of the random adult.
« Previous | Contents | Next »