The Scottish Government

« Previous | Contents | Next »

Listen

CHAPTER 2: LONGITUDINAL RESEARCH RESOURCES

(a) Resources Reviewed

2.1 The previous overviews of longitudinal research resources undertaken were often part of a broader review including cross-sectional surveys, but between them they identify a good range of relevant data sources for the purposes of this study. The longitudinal research resources can be classified in terms of one over-riding criterion - 'fitness for purpose'. This will vary from one project using the data to the next depending on the research question that the data are required to address. Thus the list below is only loosely linked to any idea of a relevance hierarchy though generally some studies tend to be more central to current purposes on most criteria than others. Coverage of features of Scottish policy and systems are likely to be best served by studies designed and run in Scotland (a). Comparative value within the UK is best served by UK wide studies with ideally a boost sample for Scotland (b). Multilevel enquiries, where ecological context is an important feature of the research topic, will be best served by area studies embracing variation at the level of the community and local institutions as well as the individual (c). Administrative data used to augment (a) to (c) can greatly enhance its value (d).

a) Longitudinal studies designed and implemented exclusively in Scotland to address specifically Scottish issues and contexts alongside the more general demographic information needed to cross classify them, e.g. Growing up in Scotland ( GUS)
b) UK-wide longitudinal studies containing sufficient numbers for analysis in Scotland, especially those including a Scottish boost to ensure adequate numbers for analysis, e.g. British Household Panel Survey ( BHPS)
c) Area based studies located in Scotland on topics for which findings could be reasonably generalised to the rest of the Scottish population, e.g. West of Scotland Study ( WoS 1116,2007)
d) Administrative data bases from which longitudinal datasets can be constructed, e.g. Department of Work and Pensions' Work and Pensions Longitudinal Study ( WLPS)
e) UK wide longitudinal surveys including a Scottish sample, but insufficient numbers for robust Scottish longitudinal analysis, e.g. Families and Children Study ( FACS)
f) Surveys carried out in England without a Scottish sample but much relevance for Scottish policy objectives and possibly for guiding the development of analogous Scottish surveys, e.g. English Longitudinal Study of Ageing ( ELSA)
g) Overseas and comparative studies, e.g. Study of Health an Retirement in Europe ( SHARE)

2.2 Broadly the study focussed on the first four of these types of survey (a-d) with overall 29 studies reviewed in depth (See Table A, Appendix 3). The reason for including administrative data in the table is that in the course of doing the review, especially arising from the interviews conducted with senior Analytic Services personnel in the different policy divisions, the critical role of administrative data became a major theme. It was argued that there was a pressing need to link the different sources together in health, education, employment and so on, would transform their value in policy monitoring terms. Linkage with existing longitudinal surveys, such as the British Household Panel Study, the UK Household Longitudinal Study and the British Birth Cohort Studies would also greatly enhance their value for policy research purposes.

2.3 It was pointed out at the expert group meeting that the general emphasis in the initial selection of studies was on social, educational and health policy areas. Longitudinal data sources on business activity tended to be lacking. Yet the economic goal of "Wealthier and Fairer" with a major role for business had been placed first among the Government's strategic objectives. The addition to the list of the Inter-departmental Business Register ( IDBR), the Annual Survey of Hours and Earnings ( ASHEDP) and the WPLS based on tax and benefits data helped make good the gap.

2.4 Table 2.1 lists all the studies that have been reviewed, restricting selection to those meeting the top four criteria in the list above including information about their relevance for Scottish purposes. Funding sources are also shown with the current main source in italics and sample sizes. It should be noted that other studies not included in Table 2.1 could prove valuable in offering relevant information for Scottish policy: examples are those based exclusively in another UK country, such as:

• the English Longitudinal Study of Ageing ( ELSA on health and aging in England
• the cross-national study of Health and Ageing in Europe ( SHARE), modelled on ELSA. Each study's design and coverage was analysed in terms of each of the fifteen National Outcomes to which they were most closely related (Table A1 Appendix)
• the Longitudinal Study of Young People in England ( LSYPE) - an annual follow-up survey of 19,000 young people in year 9 (aged 13-14) of secondary schools in England.

Table 2.1 Longitudinal Studies Reviewed

* = Boosted Scottish sample; ** = administrative data

2.5 As noted earlier it became clear from the work of the expert group and the interviews with officials that a classification of longitudinal resources in terms of importance in relation to meeting the objectives of the scoping study as originally formulated was not necessarily correct. The longitudinal studies spanning the whole of the UK offered the opportunity for comparative analysis between Scotland and the other UK countries, which gave added value to the data collected than that obtained from studies located in Scotland alone. This is because, as argued earlier, comparative analysis enables different policy contexts to be taken into account in evaluating the effectiveness of Scottish policies, rather in the nature of the natural experiment. Such reasoning also applies to the Scottish regions, especially the central belt as opposed to the Highlands and Islands, and other rural areas and the differences between the large conurbations such as Glasgow and the other towns and cities of Scotland.

2.6 It is also the case that the value of particular kinds of longitudinal survey located in Scotland, as opposed to other places, will depend to a certain extent on the subject matter of the survey. Thus the distinctive education and judicial systems in Scotland compared with the rest of the UK have components that demand special coverage but are easily overlooked in UK-wide surveys. The UKHLS is a notable exception with a planned Scottish supplementary module. On the other hand, some features of all policy areas, especially those such as health and environment with a bio-physical base, translate well from one UK country to another and common data therefore applies. Thus in such cases, a study carried out in one part of the UK will generalise to the others. Recognition of the ability to capitalise on work carried out elsewhere had the double advantage of saving money in the Scottish budget for undertaking longitudinal research. It also enabled Scotland to learn from the experience (and mistakes) of earlier projects.

2.7 Good examples here are the English Longitudinal Study of Ageing ( ELSA), which currently has no Scottish equivalent - though one is under consideration - and the Longitudinal Study of Young People in England ( LSYPE), which similarly has no direct counterpart in Scotland.

2.8 An issue of central importance to Scotland, identified in Chapter 1, which took on a different form and had less salience in English terms, was out-migration, now being replaced increasingly following devolution with inward-migration. The critical issues here relate to the gross migration effects in either direction as opposed to the net migration effects identified with population expansion or decline. A further complication is people who live part of the year in both countries. However, there is currently no single source that covers migration comprehensively and reliably. The National Passenger Transport survey suffers from the problem that much travel in and out of Scotland is via England. Employment ( DWP) and GP registration data ( NHSCR) is good for data on entry to Scotland but is much weaker on out-migration because individuals fail to register leaving. Such surveys as the LFS and the new UKLHS can identify migrants but the numbers are too limited for a research sample in their own right. The SLS solves the numbers problem but the ten year interval between censuses limits the coverage of changing migration patterns.

2.9 The other side of the issue of comparison across the UK regions is comparability within and between the regions of Scotland. As noted previously there is a major emphasis in the Scottish Government's strategic plans on shifting responsibility for the formulation and delivery of services to the local community - in practice, local authorities and the partnership arrangements existing below this level. One administrative data facility of particular interest is the data zones which supply small area statistics for units as small as electoral divisions (wards in England and Wales). Such geographically defined units are enormously valuable in relation to monitoring the effectiveness of local policy, again offering quasi experimental opportunities when comparing different regions. For example when Standard Grade was introduced in Scotland, its introduction was staggered over an extended period in the 1980s and 1990s across different regions; so those that did not have it could in principle be compared with those that did. The opportunity was thus offered to find out to what extent the introduction of the new qualification had made a difference, for example, to young people staying on in education and the level of achievement they were able to demonstrate. This is a good serendipitous example of what is often referred to as a 'waiting room' design.

(b) Scottish Sample Sizes

2.10 Sample sizes are a perennial problem in UK longitudinal surveys, including a Scottish sample. With under 10% of the UK population living in Scotland, longitudinal surveys based on currently participating samples of up to 10,000, such as the birth cohort studies can only expect to achieve a thousand cases in Scotland and although attrition tends to level off at older ages, numbers are still likely to continue decline to a point where separate Scottish analysis ultimately becomes no longer viable e.g. less than 500 cases.

2.11 Substantial sample sizes of 2,000 or more are generally needed to produce reliable statistics on the Scottish population alone. A study like GUS is particularly valuable in starting from higher numbers than this to ensure that representation of the Scottish population can be accompanied by much detailed analysis of special sub groups such as those in urban and remote areas, for example. The answer for the more recent UK studies has been to boost Scottish samples e.g. the Millennium Cohort Study and the BHPS. It was notable that an awareness of these boosts was not common among policy analysts, which perhaps accounts for the reason why there has been relatively little expansion of the use of the BHPS since the boost occurred.

2.12 It was pointed out by the expert group, however, that such boost samples are only strictly necessary when there is likely to be a strong interaction between the Scottish system and the longitudinal information collected on the population connected with it, i.e. the national context/person interaction will vary between England and Scotland, e.g. for post -16 transitions. If such interactions are not present then 500+ cases in the Scottish sub sample is likely to be sufficient.

2.13 Boosts are, in any event, never entirely satisfactory for ongoing studies, because of the missing data for sample members at younger ages preceding the period when the boost was introduced. In this respect the expansion of the British Household Panel Study into the replacement UK Household Longitudinal Study, eight times as large, is highly significant for Scotland (a sample of 7,500 individuals can be expected). Such a sample will certainly be large enough for national analysis and some area comparison, but even then not down to the level of local areas, such as comparing Glasgow with Edinburgh. Only specialised area-specific studies, such as the West of England Studies and "Go Well in Glasgow", can generate the data that are needed to make specific generalisations about the local population in relation to a given policy outcome.

(c) Matching the studies against the performance outcomes

2.14 We now return to the 15 National Outcomes focusing on the way all the designs and coverage of the 29 studies reviewed bear on each of them. Table 2.2 needs to be read in conjunction with Table 2.1 which supplies the acronyms for each of the studies and their names spelt out in full.

Table 2.2: National Performance Outcomes by Longitudinal Surveys

* Note: abbreviated descriptions - the full descriptions are in Appendix 3

* Note: abbreviated descriptions - the full descriptions are in Appendix 3

* Note: abbreviated descriptions - the full descriptions are in Appendix 3

2.15 The overall picture we get from Table 2.2 is high relevance of the reviewed studies for some outcomes and relatively little if any for others. Thus outcomes 3 - 8 concerned with education and employment are well covered by a large number of studies, whereas others such as 1 concerned with business environment are barely touched by them and what is there depends on administrative data. Those outcomes concerned with community and environment are similarly weakly represented in the studies. Indirect measures may in some instances be all that can be used.

2.16 Another general point arising from this first attempt at classification is the potential value of the multipurpose longitudinal studies such as the BHPS/ UKHLS and the 1958, 1970 and Millennium birth cohort studies. However the prime focus of Table 2.2 is coverage, which may not always be so important when date of data collection and other features of design are taken into account. It is then that the more specialised continually updated resources, such as the Labour Force survey and the associated DWP and business administrative data, come into their own. This makes the point that the use of multiple longitudinal data sources and triangulation across them to draw conclusions is essential strategy to make best use of them.

« Previous | Contents | Next »