The Scottish Government

« Previous | Contents | Next »

Listen

CHAPTER 3 THE EXPERIENCE OF LONGITUDINAL STUDIES INTERNATIONALLY

3.1 There are a number of longitudinal studies around the world that focus on the pathways young people take as they complete compulsory education. As part of this options appraisal we have examined a number of contemporary studies from different countries to identify their key design elements and how successful they have been in retaining study members. In particular we have focused on the current major longitudinal studies in the USA, Australia and England. In addition to this, we have also looked at a smaller-scale study of young people that has been running with great success in Edinburgh.

USA - Education Longitudinal Study of 2002 ( ELS:2002)

3.2 The Education Longitudinal Study of 2002 ( ELS:2002) is the fourth large scale cohort study tracking young people and their progress through compulsory education and on to post-secondary education and/or the workforce conducted in America. The first study started in 1972 and subsequent cohorts began in 1980 and 1988.

3.3 The first wave of research of the current study was conducted with young people in 10 ^th grade when they were 15/16 years old. Subsequent waves of research have taken place in 2004 (17/18 years old) and 2006 (19/20 years old). At present one or more follow-up studies may be conducted starting four or more years later. However, it is most likely that there will just be one follow-up in 2012 when respondents will be 25/26 years old and transcripts from their higher education courses will also be available.

Initial wave of research

3.4 ELS: 2002 adopted a two stage sample design with schools selected first and then 10 ^th grade pupils selected randomly within each school. Independent schools were selected at a slightly higher rate than other schools in order to ensure adequate sample numbers as they traditionally had a lower co-operation rate. In total, a response rate of 62% was achieved for the school sampling stage. Schools that were willing to assist in the study were required to provide a full list of enrolled pupils from which the final pupil selection took place. Approximately 26 pupils were selected from each school.

3.5 After pupils had been selected consent letters were sent to parents informing them of the study. The majority of schools were happy for this letter to require only passive or implied consent (ie unless parents actively opted their child out of the study, their consent would be assumed). However, a minority of schools were unhappy with this approach and insisted that parents had to actively opt in to the study. As might be expected the response rate for young people at these schools was significantly lower than at schools with a passive or implied consent procedure. The study directors believe that this was not because parents at these schools were less happy to take part in the study - the numbers of opt outs were in fact similar at both sets of schools - but that parents lacked the initiative to actively put their child forward for the study as the procedure required.

3.6 The survey of the sampled young people was administered in schools using paper questionnaires and comprised a number of elements including tests in maths and English as well as a self-completion questionnaire. Parents of young people who took part in the study then became eligible to be interviewed themselves. The parental interview collected background information about household characteristics and was completed using a mixed methodology of telephone and postal self-completion interviews.

3.7 As with previous studies, the first wave of research for ELS:2002 was conducted with young people on site within schools on an initial survey day. Subsequent "mop up" days were then arranged as necessary to collect information from young people who were not present at the initial survey day.

3.8 Incentives were only paid to young people when it was felt that the response rate was likely to be depressed by external factors. When schools required active consent from parents a survey day prize draw with two $20 prizes was conducted to encourage participation. When a school would only allow the survey to be conducted out of school hours (after school, weekends or holidays) each participating student was offered $20.

Response

3.9 There was a very high response rate of around 87% for young people at schools that agreed to take part in the survey. A large amount of the non-response can be directly attributable to schools that required active consent from parents. However it should be remembered that the study did encounter relatively high levels of school level non-response prior to this stage. This will largely be a result of the high levels of burden placed on schools taking part in the study. In an effort to reduce this, schools were allowed to hold their survey day whenever was the most convenient for them within a 6 month period. In addition to this, study endorsements were also obtained from a large number of relevant groups to emphasise the importance of the study.

3.10 The response rate amongst parents whose child had taken part in the study was encouragingly high at 87%.

Second wave of research

3.11 The second wave of research for ELS:2002 was conducted in 2004 when sample members were 17/18 years of age. As students in America generally stay in education until they are 18 the majority of sample members were once again interviewed in school.

3.12 There were, however, groups of respondents for whom it was not possible to conduct interviews in school. These were:

• respondents at schools who no longer wished to participate;
• respondents who had moved to a new school; ³
• respondents who had graduated early from High School;
• respondents who had "dropped out" of full-time education.

3.13 These respondents were followed up with interviews outside of school. In the first instance interviews were attempted using telephone data collection. If the telephone interviewers were unsuccessful in finding a sample member, their details were passed to a tracing unit who used various agencies to find a correct address and current telephone number. All sample members who could not be interviewed over the telephone were passed on to field interviewers who attempted to conduct interviews with respondents in home.

Incentives

3.14 At the second wave of research it was intended that all respondents who participated would receive a $20 incentive. However, this had to be confirmed with individual schools and some preferred not to allow students to receive this incentive. Some schools allowed pupils to receive $20 in gift vouchers while others asked for the equivalent sum to be donated to the school library or school council. Fewer than three per cent of schools would not permit incentives of any type.

Response

3.15 The response rate at the second wave of research was 89% although it did vary to a large degree by the type of respondent. Nearly 95% of young people who took part in the initial wave and who were still in the same school took part in the second wave of research. Just under 4 in 5 young people who had left education, either through dropout or early graduation, took part in the second wave (78% and 82% respectively). The hardest group to retain were those young people who had transferred school, but coverage here was still relatively impressive with 71% taking part at the second wave. The overall response rate for ELS: 2002 was 77%. The breakdown of the type of interviews conducted is included in Table 3.1 below.

Table 3.1: Type of interviews

Summary design elements

• First sweep conducted at 16 (2 years before the usual leaving age)
• Sweeps conducted biennially
• First two sweeps conducted in schools
• Passive consent from parents where possible
• Incentives from second sweep onwards
• Extensive procedures for absent pupils
• Parental interview at first sweep
• High response rates in school - declining when attempting to reach those outside of their original schools

Australia - Longitudinal Study of Australian Youth ( LSAY)

3.16 The Longitudinal Surveys of Australian Youth ( LSAY) is a program of surveys of young people jointly managed by Australian Council for Educational Research ( ACER) and the Australian government Department of Education, Science and Training ( DEST). The surveys focus on the education and labour market experiences of groups of young Australians, beginning from their middle years of secondary schooling. Data collected include basic demographic variables, as well as information about educational and labour force participation and experiences extending over a number of years.

3.17 The present form of the LSAY program commenced in 1995 with a national sample of Year 9 students and there have been two subsequent cohorts in 1998 and 2003. The program also brings together earlier longitudinal studies conducted by ACER under the Youth in Transition ( YIT) program, and by the Commonwealth under the Australian Longitudinal Survey ( ALS) and Australian Youth Survey ( AYS) programs. The data collected through LSAY are deposited with the Australian Social Science Data Archive ( ASSDA) for use by other researchers.

LSAY 2003

3.18 The third cohort for LSAY in 2003 was timed to coincide with the PISA study in Australia of the same year, so that it could be used to provide the basis for the longitudinal study. As a result of this the PISA sample drawn in Australia in 2003 was actually larger than that required by the OECD.

3.19 All young people who had taken part in the PISA study of 2003 then formed the sample for the 2003 cohort of the LSAY. The first interviews with the young people as part of the 2003 LSAY were actually completed in the same year as PISA in the form of telephone follow-up interviews at home. These first interviews focused on areas that were outside the focus of PISA, including follow-up information about their school and part-time work.

3.20 Subsequently this cohort has been interviewed again in 2004 and in 2005, each time by means of a telephone interview at home.

Response

3.21 The response rates achieved for the 2003 cohort of the LSAY have been relatively high.

3.22 A total of 12,551 respondents completed the PISA study in 2003 and were therefore eligible for the first follow-up interviews in the same year. Of these 932 (7%) either provided incorrect contact details or opted out of any follow-up research. A total of 10,370 interviews were conducted from the useable sample of 11,619 PISA respondents. This equates to an un-weighted fieldwork response rate of 89% and an overall response rate of 83%. ⁴

3.23 The second follow-up study was conducted in 2004 when the cohort were aged 16/17 years old, one year after the PISA and initial follow-up wave. Interviews were conducted by telephone in the respondent's home, the same approach as in the 2003 follow-up interviews. A total of 9,378 interviews were conducted from the sample of 10,370 respondents who completed the initial follow-up wave, which is an un-weighted response rate of 90% and an overall response rate of 75%.

3.24 A third follow-up study was completed in 2005 when the cohort was aged 17/18 years old. As with the previous two follow-up surveys, interviews were once again conducted in the respondent's home by telephone. A total of 8,691 interviews were conducted from the sample of 9,378 respondents who completed the previous wave, which is an un-weighted response rate of 93% and an overall response rate of 69%.

Summary design elements

• First sweep conducted at 15/16 as part of PISA
• Initial sweeps conducted annually
• First sweep conducted in school
• Follow-up sweeps conducted over the telephone
• High response rates for follow-up sweeps

England - Youth Cohort Study ( YCS)

3.25 The Youth Cohort Study ( YCS) series dates back to 1985 when Cohort 1 Sweep 1 took place. YCS is a significant and extensive study of young people's transitions and has informed key policy decisions over a number of years. The first survey (or 'sweep') for each cohort of those sampled takes place in the first year after compulsory education, with subsequent sweeps at varying time intervals; these are usually annually but occasionally take place at shorter intervals). The thirteenth YCS cohort is being conducted in Spring 2007.

3.26 Surveys at each stage provide information about the routes young people follow. The YCS is funded and managed by the Department for Children Schools and Families (formerly Department for Education and Skills ( DfES)).

3.27 The primary aim of the YCS is to track young people's transitions from compulsory education into further education, training and the labour market. The longitudinal nature of the YCS enables the Department to establish the education and employment paths that young people take and explore how things develop over time. Finding out about the different routes that young people take helps the Department plan services for different types of young people. It also enables the Department to establish appropriate points in the lives of young people where policy intervention helps to improve their educational and employment outcomes. The YCS was initially modelled on the SSLS and so the similarities between the two are unsurprising.

YCS 13

3.28 Prior to cohort 13 a mixed mode data collection approach had been established for YCS. An initial self-completion postal survey with a web completion option (first introduced at YCS12), was supplemented by a telephone interview stage to increase response rates among non-responders for whom a telephone number could be identified.

3.29 However, as with SSLS, response rates had been declining steadily over time. In YCS12, this approach resulted in an overall response rate of 46%. While this might be viewed as just about acceptable if replicated across the sample, in fact the response rate varied substantially by Year 11 attainment, ranging from 19% among those with no qualifications to 63% among those with 8+ A*-Cs. Since the initial sample for YCS had been selected to be representative by attainment, this design has led to sample sizes for those with low attainment being too small for analysis at subsequent waves of the survey.

3.30 Consequently there have been a number of changes to the traditional design of the YCS to ensure that there are sufficient numbers of young people from all levels of attainment. The main changes that have been made are:

• the issued sample is no longer broadly representative of Year 11 attainment and instead low attainers have been over sampled;
• data collection at sweep 1 will now take place via face to face interviews rather than using a mixed mode approach;
• the issued sample size has been reduced from approximately 25,000 to 10,000.

3.31 The issued sample size can be reduced since face to face data collection has significantly higher response rates than postal, telephone or web surveys. Adopting a face to face data collection strategy also changes the way in which the sample was selected for this cohort as there is a need for respondents to be clustered (for YCS 13 this was by postcode sector). At present there has been no decision made about the data collection methods that will be used for subsequent waves.

Summary design elements

• Declining response rates for traditional approach (like SSLS)
• Particular problems for low attainers and those not in education, training or employment
• New approach adopted 2007 (cohort 13) with face to face survey in home
• Reduced initial sample size but significantly higher response rates
• Sample skewed to over represent low attainers
• Initial sweep in first year after compulsory education (16/17)

England - Longitudinal Study of Young People in England ( LSYPE)

3.32 The Longitudinal Study of Young People in England ( LSYPE) was designed to meet a need for information to support government policy development around young people's transitions from compulsory education into further education, training and the labour market. While existing research (including the Youth Cohort Studies) had previously provided important information in this area there were still important gaps. In particular, it was felt that it was important to track the development of young people from their early teens, while they were still in education, rather than just studying them at the post-16 stage. In addition to this, LSYPE also collects background information about the young person's circumstances, including interviews with the parents/guardians with whom they are living.

3.33 The target population for LSYPE comprised all students in Year 9 schools in England. In practice, and largely for practical reasons, the survey population excluded pupils in small schools, boarders and foreign nationals residing in England solely for the purpose of attending school. As fieldwork for LSYPE was conducted face to face at the young person's home a two stage sample design was used with schools adopted as primary sampling units to allow for some clustering of addresses.

3.34 In the maintained sector, schools classed as 'deprived' (and hence students attending deprived schools) were sampled at higher rates than other schools (and pupils attending other schools). Furthermore, in the maintained sector the numbers of students belonging to various minority ethnic groups were also boosted. On average around 33 pupils were selected at each school.

3.35 Fieldwork for the first wave of LSYPE took place in Spring/Summer 2004. Interviews were conducted in the young person's home via Computer-Assisted Personal Interviewing ( CAPI) with the sampled young person and any parents/guardians living in the same household. Fieldwork for the second and third waves followed a similar model taking place at the same time of year in 2005 and 2006. One slight difference with the third wave of research was that interviews were not required with both parents/guardians and proxy data was collected for any partners.

3.36 The total interview length at Wave 1 was 90 minutes comprising a 35 minute interview with the sampled young person and a 55 minute interview with their parent(s)/guardian(s). At Wave 2 less information was required and the overall interview length was shorter at 70 minutes with an even split between young person and parent(s) of 35 minutes for each. At Wave 3 the questionnaire length was significantly reduced to 35 minutes with 25 minutes for the young person and 10 minutes for the parent.

Incentives

3.37 At the first wave of research all sampled young people were given a £5 gift voucher conditional on them taking part in the study. At Waves 2 and 3 the incentive remained the same but instead it was given unconditionally to all young people. Fieldwork for the fourth wave of LSYPE is about to begin and the incentive has been increased now to £8.

Response

3.38 At the first wave of research 15,770 interviews were completed at households ⁵ from an initial sample of just over 21,000 with an un-weighted response rate of 74%. At the second wave of research interviews were completed at 13,539 households to give an un-weighted response rate of 86% and an overall response rate of 64%. In the third wave of research interviews were conducted at 12,435 households for an un-weighted response rate of 92% and a cumulative response rate of 59%.

Content

3.39 Young people in LSYPE have been interviewed every year from Year 9 onwards. These interviews have focused on what happens at school, their plans for the future and various risk behaviours among many other topics. In addition, the parents/guardians of young people have also been interviewed at each wave and this has allowed for detailed background information to be built up.

Summary design elements

• First sweep conducted at age 13/14
• Sweeps conducted annually
• Interviews conducted in home
• Both parents and young people interviewed
• Comprehensive information collected about most aspects of young person's background
• Very high response rates (exceeding initial plan)
• Incentives paid to young person

Scotland - Edinburgh Study of Youth Transitions and Crime ( ESYTC)

3.40 The Edinburgh Study of Youth Transitions and Crime ( ESYTC) is a longitudinal study involving an entire year group of children, namely those eligible to start the first year of secondary education in the City of Edinburgh in 1998. The overarching aim of the study is to increase understanding of criminal behaviour among young people. The cohort comprises approximately 4,300 young people who were aged between 11_ and 12_ years at the start of the study. Annual sweeps of data collection are conducted, with the intention of tracking the cohort through their teenage years and into early adulthood.

3.41 Given the size of the cohort, the most feasible and cost-effective method of data collection was via self-completion questionnaires, on an annual basis, within Edinburgh schools. More detailed contextual information on the nature and meaning of young people's offending behaviour was achieved at sweep two by semi-structured interviews with a sample of cohort members. And at the neighbourhood level, the relationship between community characteristics, local crime rates and individual offending is being analysed by means of a Geographic Information System ( GIS).

3.42 A unique feature of the study is the number of agencies and data sources to which access was successfully negotiated. The most extensive access negotiations involved the Edinburgh schools, including mainstream secondary schools, independent schools and schools for children with additional support needs. As well as allowing the self-completion questionnaires to be administered to the cohort each year, information was also collected annually from school computer records and, at sweep two, teachers supplemented this information with a short "strengths and difficulties" questionnaire.

3.43 The fieldwork for the first sweep of the study was mainly conducted between September and December 1998. Some schools asked for the administration of the survey to be completed over one or two days to minimise disruption which was also the most efficient approach for the research team. However, other schools preferred the survey to be administered during a particular kind of lesson and this meant that fieldwork within these schools could be spread over a number of weeks (depending on timetables) which was slightly less efficient for the research team.

3.44 The next 5 sweeps of the study (2 to 6) were also conducted in schools and again every effort was made to ensure that disruption to schools was kept to a minimum with researchers working to a schedule that the schools preferred. From sweep 5 onwards the number of young people who were no longer in school began to increase as their compulsory education ended. Attempts were made to survey those young people who had left school via postal questionnaires and some telephone interviews but, as might be expected, this was significantly less productive than attempting to survey the young people in school.

3.45 The seventh sweep of the research is due to take place in 2007 when study members will be 20. As study members will no longer be in school the data collection strategy has been revised. At this sweep sample members will be interviewed using a combination of web-based questionnaires and computer-assisted interviewing.

Response

3.46 The success of the access negotiations was such that every school in the maintained sector in Edinburgh agreed to take part in the study. There was also a good level of response among schools in the independent sector which are usually among the more reluctant to participate in research.

3.47 Response among pupils in participating schools was generally very high for the first three sweeps, with around 95% of eligible pupils taking part in the study. At sweeps 4 and 5 response began to decline as more young people were no longer in school but response rates were still high at 93% and 89% of eligible cohort members respectively. Sweep 6 was conducted in 2003 when sample members were 16 and therefore a proportion had by now left compulsory education. As a result of this the response rate declined and 80% of sample members completed the survey at this sweep.

Summary design elements

• Attempted census of all S1 pupils in Edinburgh
• Sweeps initially conducted annually
• Interviews conducted in school for early sweeps
• Very high response rates in schools (all maintained schools agreed to take part)
• Later sweeps required interviewing outside of schools and response rate was more challenging

« Previous | Contents | Next »