The Scottish Government

« Previous | Contents | Next »

Listen

CHAPTER 7 SAMPLE MAINTENANCE AND ATTRITION

7.1 This chapter considers the methods that will be required to maintain a nationally-representative sample, and to minimise attrition of sample members over the life of the new study. Attrition is a challenge faced by all longitudinal studies and problems associated with non-response were a major issue for the former SSLS. Measures to address the question of attrition must be a prominent feature of any new longitudinal study of young people.

7.2 When estimating the potential attrition rates for the new study we have attempted to use comparisons with similar surveys in the United Kingdom and elsewhere in the world. However, the proposed design of the new study is such that direct comparisons with many of these studies can be problematic. In particular, we believe it is necessary to be cautious when estimating the response rate, and therefore attrition, for the first follow-up sweep of a new study. ¹⁶

7.3 Following from the design options recommended in Chapter 6, this section highlights the particular challenges that each proposed design will face. It draws on other longitudinal studies of young people and best practice in minimising attrition and what could be applied in the new study. Finally, this chapter provides estimates for likely attrition in the study for each of the designs.

Design A

7.4 Design A features a first contact with young people within the school setting during the compulsory stages S1-S4 as described in the previous chapter.

7.5 In the proposed design for Design A, the initial survey sweep would be conducted in lesson time within school. As such non-response at the initial sweep will largely be a product of schools choosing not to take part, rather than the non-response of individuals. The overwhelming majority of sampled young people will complete the survey, with the only exceptions being those whose parents opt them out of the study and those who are consistently absent (although we have suggested follow-up procedures for the latter).

7.6 While this will mean that the response rate for pupils within schools will be very high, ¹⁷ it also means that the response rates we could expect to see at the first follow-up sweep will be lower than what has been achieved in equivalent surveys. This is because respondents who might usually be a non-responder at the initial sweep of a study will be going forward to the first follow-up sweep of research. As such, the response rate that will be achieved in the first follow-up sweep of the new study will be more akin to that of an initial sweep of a longitudinal study than that of a follow-up sweep.

Comparisons with other studies

7.7 When estimating the response rate that could be achieved at the first follow-up sweep of Design A of the new study we have looked at the most appropriate stages of other longitudinal studies of young people.

7.8 In the USA, the first follow-up sweep for the ELS:2002 among young people who have left school, or who have transferred to a new school, is conducted in a very similar way to that proposed in the new study ¹⁸. A response rate of around 80% was achieved among young people who had left school and approximately 70% among those who had moved school. The young people who had left school were a combination of drop-outs and those who had graduated early and the response rates for both of these groups were relatively similar. The response rate for young people who had moved school was significantly lower but it should be noted that this will most likely be related to difficulties in tracing these respondents. As such the 80% response rate among young people who had left school is the best guide to a response rate for Design A of the new study. It should be noted that ELS:2002 respondents were given a substantial incentive of $20.

7.9 The survey design for the 2003 cohort of the Longitudinal Surveys of Australian Youth ( LSAY) was similar to that proposed for Design A at sweep 1, but subsequently there were two main differences. The first is that LSAY solely used telephone interviews rather than employing a mixed mode approach. This was possible in part because telephone numbers were available for a very large proportion of sampled respondents. The second, and most important difference, is in the timing of the follow-up sweep. The initial sweep of the 2003 cohort and the first follow-up sweep were both conducted in the same year. This meant that there was less likelihood of sample members moving in the intervening period and that the initial sweep would still seem fresh to respondents. The cumulative response rate for the first follow-up sweep was 83%.

7.10 The LSYPE in England differs from the proposed design of the new study, and the American and Australian studies, in that no contact took place in school. Instead, all surveying was conducted face to face in home with both the sampled young person and any parents/guardians who were living with them. However, it is still worth considering the response rate achieved in the first sweep of LSYPE as this will be similar in dynamic to the first follow-up sweep of Design A of the new study.

7.11 The unweighted response rate for the initial sweep of LSYPE was 74% but there are additional factors to note which could lead to either a higher or lower response rate for the new study. Firstly, respondents in the proposed Design A will have completed a survey sweep in school and have received regular survey information and reminders in the intervening years. Secondly, there would be a smaller burden on respondents and the interview length will be shorter. Against these points, contacts in subsequent sweeps of Design A will largely be by telephone rather than face to face, which means that it will be easier for respondents to refuse. Secondly, there will not be the parental pressure to take part in the study that contributed to some of the success at the initial wave of LSYPE. Finally, the initial wave of LSYPE was conducted in 2004 while the first follow-up sweep for the new study would be conducted in 2012 and response rates have been slowly declining over time.

Measures to maintain the sample

7.12 The proposed Design A for the new study has a gap of two years between the initial contact in school and the first follow-up at the respondents' home address. If contact is not maintained with respondents during this time there is a greater chance that they will forget about the survey. This would undermine one of the elements in securing high response rates at follow-up sweeps in longitudinal studies - recognition of the survey. In addition to this, the sampled young people will be becoming more mobile around this time and maintaining regular contact will increase the likelihood of tracing those who move. As such, we would recommend that a number of steps are taken to maintain contact with participants and build a study identity between sweeps.

7.13 A minimal approach would involve setting up a study website and sending a "Research Findings" report out around a year after the initial sweep of interviewing in school was conducted. However, we would recommend a more comprehensive approach to increase the study identity and maintain interest.

7.14 In the first instance an initial "thank you" letter could be sent to respondents along with a branded gift of some sort. The gift itself would not be of a large monetary value and could be something as simple as pocket diary. This first contact would be soon after the initial sweep (4-6 weeks) so that the link with the questionnaire they had just completed in school and the survey brand can begin to be established. In addition to building the survey branding, this first contact will also serve to identify sample members with incorrect contact details at an early enough stage for the researchers to seek corrections or clarifications.

7.15 A secondary contact could then take place around a year after the initial sweep providing the study member with relevant and interesting findings from the first sweep of the research. A final contact could then take place in the run up to the first follow-up sweep of the study, this would alert them to the fact that the survey team wish to speak to them again soon and enable contact details to be checked (see Chapter 6 for the costings for a "keep in touch" exercise).

7.16 Estimating potential response rates for surveys is an inexact science. As discussed above, because the initial sweep of Design A will be conducted in school, the most crucial stage in terms of panel attrition will be the first post-school follow-up sweep. The assumptions that are made about response rates at this sweep are the ones that will impact most on projected sample sizes at later sweeps of the research. Table 7.1 shows the estimated attrition for Design A.

Table 7.1: Design A: estimated attrition over time

7.17 There are a number of assumptions that underpin these estimates, the main ones are outlined below:

• The fieldwork model is that outlined in Chapter 6
• Multiple contacts would be made with respondents between the initial sweep of school based research and the follow-up sweeps
• Full access to administrative databases to enable tracking of movers
• Two yearly intervals between sweeps

Design B

7.18 The proposed design for Design B follows that used in previous SSLS in that the initial sweep will be conducted in the first year after compulsory education. However, the data collection strategy would be changed significantly with the adoption of a mixed mode approach rather than a reliance on postal questionnaires. This change to the data collection strategy would lead to a significant increase in response rates and therefore in the reliability of survey measures. Greater detail about the proposed data collection strategy for Design B is contained in Chapter 6.

7.19 Table 7.2 gives the estimated response rates and sample sizes for Design B.

Table 7.2: Design B: estimated attrition over time

7.20 There are a number of assumptions that underpin these estimates, the main ones are outlined below:

• The fieldwork model is that outlined in Chapter 6
• Multiple contacts would be made with respondents between sweeps
• Full access to administrative databases to enable tracking of movers
• Two yearly intervals between sweeps

Differential Attrition

7.21 The SSLS has suffered in recent years not just from relatively high rates of attrition, but also from differential rates of attrition, in particular, in respect of young people with low levels of attainment and from more deprived backgrounds. This meant that estimates from the survey have become increasingly unreliable for the population and, in particular, these groups.

7.22 These problems have also been encountered in the equivalent English research, the Youth Cohort Study and consequently the methodology for cohort 13 was changed with a move to face to face interviews rather than a reliance on postal questionnaires. While fieldwork had not yet concluded, the initial results indicate that response overall is high and, even more encouragingly, there is little differential response between sub-groups of young people. This echoes the findings from the LSYPE which also uses a face to face interviewing methodology. It should be noted that both studies offer incentives to respondents and this, as well as the change to the mode of data collection, may be a factor in reducing differential non-response.

7.23 The proposed designs for a new Scottish longitudinal study focus on a mixed mode approach with initial telephone contacts followed by postal and face to face follow-ups for non-responders. While this approach may lead to slightly lower overall response rates than for the two English studies, we believe that this would be minimal. In particular, we would expect that differential rates of attrition would be minimised and as a result key groups such as low attainers and those who are not in education, employment or training would be present in the sample in sufficient numbers ¹⁹.

« Previous | Contents | Next »