The Scottish Government

« Previous | Contents | Next »

Listen

EXECUTIVE SUMMARY

Chapter 1 Background to the scoping study

1. This scoping study was commissioned by the Scottish Government ( SG) to assist in the development of an evidence base through longitudinal research, and to support evaluation of the Government's performance framework. The focus was mainly on longitudinal studies involving the follow-up of individuals over time rather than cross-sectional surveys used for monitoring performance in the population at national level, or at the level of institutions such as businesses or hospitals; though these have a role in multi-level frameworks for analysis that link National Outcomes to local goals and service delivery. In the course of the study administrative data sources emerged as highly important and were also reviewed.

2. The study was directed at:

a) Understanding the policy context in which the evaluation of research resources will add value
b) Reviewing the design features and coverage of the different longitudinal research resources available in terms of relevance to the Scottish Government's National Performance Framework focusing on the 15 National Outcomes by which the effectiveness of the framework will be judged
c) Demonstrating the ways in which different longitudinal resources might be used singly or in combination to evaluate the strategic outcomes and weighing their relative value
d) Identifying gaps in the longitudinal resource portfolio that need to be filled by enhancing particular studies or supporting new ones
e) Identifying cross-cutting issues that need to be resolved in relation to the development of the portfolio and its most effective use

3. The study drew on evidence collected from three main sources:

a) Documentation relating to the key studies identified as most directly relevant to the scoping study's requirements
b) Senior officials in the SG Analytical Services division, including directors of analytical services in the different policy domains relating to the economy, education, health, criminal justice, community, environment - to elicit their experience of, and views about, the use of longitudinal research in their own sphere of interest and the need for new studies
c) An 'Expert Group' of university users of Scottish longitudinal data to advise on the evidence base, conceptual framework and analytic approach for the scoping study and to comment on the draft report

4. The SG's strategic purpose with seven associated targets is to "to focus Government and public services on creating a more successful country with opportunities for all of Scotland to flourish through increasing sustainable economic growth". The first stage of the work was to clarify the content and operationalisation of the performance framework through which the purpose will be realised at the various levels at which it operates. These are cast in terms of the "Virginia State Model" of service delivery, which focuses on measurement of progress towards agreed goals, rather than specific targets to be achieved. Heading the framework are the five strategic objectives: wealthier and fairer, smarter, healthier, safer and stronger, greener, followed by 15 outcomes for assessing progress towards meeting the objectives and 45 indicators for detailed performance monitoring. The study focused on the 15 National Outcomes addressing the question of how longitudinal research might assist in their evaluation. Building on previous reviews of Scottish survey data sources, the 29 longitudinal studies and administrative data sources judged most relevant were reviewed (summaries are in the Appendix to the main report).

5. The studies reviewed varied in design including:

• birth cohort studies following up large samples from birth to adulthood with data collection often at extended intervals
• household panel studies with annual repeated measurement of all household members over the age of 16
• age cohorts starting at different stages of life such as adolescence or old age
• area studies following samples based in a particular locality

In reviewing the studies attention focused on their main design features including coverage, repeated measurement and sample size (for UK wide studies - the size of the Scottish sample) and methodological considerations, including the impact of data loss through sample attrition on study viability, data quality and bias.

Chapter 2 Longitudinal Research Resources

6. Six types of longitudinal data sources were identified for detailed examination of which four turned out to be the most immediately relevant to the scoping study.

a) Scottish national longitudinal studies - designed and implemented exclusively in Scotland to address specifically Scottish issues and contexts alongside the more general demographic information needed to cross classify them, Growing up in Scotland ( GUS), Scottish Longitudinal Survey ( SLS), Scottish School Leavers Study ( SSLS - being replaced), Scottish Survey of Achievement ( SSA)

b) Area based studies located in Scotland - on topics for which findings could be reasonably generalised to the rest of the Scottish population: Edinburgh Study of Youth Transitions and Crime ( ESYTC); GoWell Glasgow Study of Community Health and Well-being ( GWG); West of Scotland ( WoS 11-16 and WoS 20-07 Studies); Scottish Housing and Regeneration Project ( SHARP)

c) UK wide longitudinal studies - containing sufficient numbers for analysis in Scotland, including those with a Scottish boost, to ensure adequate numbers for robust analysis: British Household Panel Survey ( BHPS)/ UK Household Longitudinal Survey ( UKHLS); Millennium Cohort Study ( MCS), National Child Development Study ( NCDS -1958 birth cohort), 1970 Cohort Study ( BCS70 -1970 birth cohort); Labour Force Survey ( LFS); Workplace Employee Relations Panel Survey ( WERS)

d) Administrative data sources - for linkage to the longitudinal surveys: Annual Survey of Hours and Earnings ( ASHEPD); Interdepartmental Business Register ( IDBR); Scottish Continuous Recording System ( SCORE); Work and Pensions Longitudinal Study ( WPLS)

The other sources were surveys carried out in England without a Scottish sample but of relevance for Scottish policy objectives and possibly for guiding the development of analogous Scottish surveys, e.g.English Longitudinal Study of Ageing ( ELSA); Longitudinal Study of Young People in England ( LSYPE) and overseas and comparative studies, e.g.Study of Health an Retirement in Europe ( SHARE).

Chapter 3 Research Scenarios and Vignettes

7. To illustrate the kinds of research scenarios in which longitudinal data had a part to play in the evaluation of the National Outcomes, short vignettes were constructed linking relevant variables at the different levels - individual and community - at which they were measured. In constructing the vignettes, one for each outcome, we first identified the implicit proposition that lay behind each of them, e.g. investment in education will raise skills levels; smoking constitutes a risk to health. The next step was to identify the longitudinal data that would best support explanatory models linking circumstances, experiences and personal attributes to the given outcome at the appropriate level - sometimes within a single multilevel framework.

8. For convenience the outcomes were grouped in terms of the broad policy domains to which they refer - economy, education and life long learning, health (one outcome); protection (including criminal justice), community and public services, and environment. The exceptions are those that span most of the others, inequalities and identity, which are discussed separately at the end. Alongside the vignettes examples are given of longitudinal research projects to illustrate the kinds of investigations that longitudinal surveys support.

(a) Economy

9. This spans two features of localities, as well as the national picture: an attractive place in which to do business and the supply and take-up of employment opportunities. The former is measured by business start-up, growth and profitability, and the latter by the employment record of individuals in the population at large. Together they merit a multilevel data survey design structure defined by locality with employees embedded in the businesses located there and the wider social infrastructure that supports them. Relevant data sources are administrative, including IDBR, WERS, ASHE and WPLS, and from survey sources of which the UK-wide LFS and WERS, with longitudinal components, are the most important for adult employees. The research scenario would be secondary analysis of data sets linked across all sources or, in the longer term, a longitudinal study of business development and employment in a number of contrasting localities. A key issue to resolve is how to cover first entry into the labour market, where the proposed replacement for the SSLS will be the obvious data source.

(b) Education and Lifelong Learning

10. Education is the area in which longitudinal data comes into its own because of the relative ease with which individuals can be followed through primary and secondary school and later into post compulsory education and the labour market. Three outcomes are embraced; skills as reflected in: general and vocational qualifications; basic and core skills, capabilities and engagement, including entry into the labour market and its counterpart the ' More Chances, More Choices' group; cognitive and behavioural skills acquired in the early years. The research scenarios can operate at the level of individuals through modelling the formative influences on educational progress, or within a multilevel framework, with particular emphasis on school and other education system effects. The components of the 'learning careers' involved include family circumstances and parental involvement followed by school experience and achievement.

11. The main longitudinal survey sources are plentiful and include the UK-wide longitudinal cohort studies, NCDS, BCS70, MCS and GUS and the panel surveys, BHPS/ UKLHS. Linkage to existing administrative sources and the use of the under-exploited SSA, based on an annual survey of pupil attainment, needs to be explored. However it should be noted that the focus and remit of the SSA is being revised in order to reflect the aims and requirements of the Curriculum for Excellence. The precise form that the SSA will take is therefore still developing. A planned resource comprising individual data on transition from school to the labour market, based on careers guidance data, will be valuable for tapping the special features of the Scottish post 16 education system, which the UK-wide surveys do not cover in the detail required.

(c) Health

12. Health is another area where longitudinal data research has much value in contributing to the understanding of 'cause' and 'effect' especially when the effect is a health outcome, such as smoking related for example, to the incidence of disease - lung cancer. Relevant measures of the outcome are of two kinds. The first is identified with morbidity and mortality, which is encompassed by the SLS with great scope for additional administrative data linkage through national records. The second relates to psychological and physical well being and lifestyle including the health related behaviours such as smoking, drinking and diet. The key data sources are the birth cohort studies - NCDS, BCS70 - originating in perinatal mortality and child health concerns and in the longer term, MCS, and GUS. The most comprehensive evidence base, however, is to be found in the area-based West of Scotland Studies ( WoS), funded by the Medical Research Council.

13. As previously, the research scenarios follow the approach of modelling the effects of different influences on heath related behaviours, such as smoking and drinking, different disease outcomes and healthy old age. In the case of WoS, analysis can take into account features of the local community in a multilevel framework. The major issue to resolve is whether to establish the Scottish Longitudinal study of Ageing in parallel to the comparable English study ELSA.

(d) Protection

14. The emphasis here is on children and families at risk and the defeat of crime through the criminal justice system. Outcome measures relate, at the individual level, to recorded and self-reported crime and exposure to crime, and at the area level, to crime rates. Key studies are the birth cohorts and the UKHLS, which will include data on experience of, and committal, of crime. In the longer term GUS and MCS, offer much potentially for investigation of the outcomes for children and families at risk. The most relevant current study is however, the one specifically targeting crime and young people, ESYTC, which has six waves of data collected across the period 12-18 and is now moving into the adult period of life (although without funding). This dataset has very rich information about all aspects of youth transitions and crime linked comprehensively to administrative data about crime in Edinburgh and offers much scope for secondary analysis targeting this outcome. Other area-based datasets such as WoS and GWG also contain relevant data for analysing the conditions and experiences that precipitate crime and desistance to crime.

15. Research scenarios focus on first identifying the risk factors that predict later negative outcomes at school and in the community with a view to discovering what can reverse them and secondly, modelling the neighbourhood and family characteristics that predict exposure to crime and the shaping of criminal careers.

(e) Community and Public Services

16. The emphasis in the outcomes here is on sustainability, resilience and high quality services principally at community level. Measures relate to quality of community life, in relation to such features as housing and neighbourhood, participation in the community and use of services. The best longitudinal data on community comes from the various area studies such as GWG, WoS and ESYTC. All contain much information, including linked administrative data, about the different facets of community life including use of services and the role of disadvantage at individual and community level. These studies offer rich resources for research. They also bring in the topic of housing, which does not feature in the National Outcomes, but is clearly a critical aspect of effective functioning in all areas of community life.

17. What is lacking here is longitudinal research on rural communities and the best we have comes from the Scottish Longitudinal Study, which is limited in terms of its coverage. Again a multilevel modelling framework for analysing the way these different features relate to the outcome measures at the level of the individual (take responsibility) and the community (sustainability) appears the best approach.

(f) Environment

18. The outcomes here concern the impact of environment on quality of life and in return the impact of the consumption component on the global environment. Although the first is fairly straightforward to operationalise from existing longitudinal resources the second is more difficult. The built environment in the city context is well covered by the area studies such as GWG and WoS. The natural environment has to rely on administrative data and the SLS for longitudinal coverage. The annual Agricultural Census of Farms and the IDBR are relevant administrative sources, which could supply sampling frames for more intensive longitudinal study. The Scottish Continuous Recording System ( SCORE) that supplies a record of housing provided by registered social landlords is a useful source on the built environment. UK-wide studies such as BHPS, NCDS and BCS70 and GUS, and potentially UKLHS, all provide quite detailed information about housing circumstances and histories, but numbers are small for Scottish rural areas.

19. Again the case for a longitudinal study specifically targeting rural communities and incorporating the issue of inward and outward migration is a strong one.

20. Consumption patterns can be established from cohort study and household panel data to a limited extent and cross-sectional surveys such as the Family Resources survey, but their impact on the environment cannot easily be assessed. Research scenarios are thus heavily constrained in this case by the relative lack of good longitudinal data.

(g) Inequalities

21. Inequalities relate to all personal attributes encompassed by the fifteen outcomes especially the first four but also extending to the others in reflecting the variation between individuals in response to the strategic objective - 'wealthier' balanced by 'fairer'. Again the cohort and panel studies provide perhaps the best data sources for this outcome, because they can evaluate changes in the distribution of the given outcome and its structural correlates such as class and gender across time. GWG is an important example of a project that focuses on inequalities in different domains of life especially health and housing. Another potential source is the Department of Work and Pension's new Wealth and Assets Survey and the WPLS. Other studies focussing on particular sub populations in difficult circumstances will also come into the frame, such as the 'Life Opportunities Survey' (LOS) which will explore the "barriers disabled people experience" - as the study will also include non-disabled respondents the inequalities can be identified and monitored. In addition, the planned Longitudinal study of refugees ( LSR) will be an important source.

22. Modelling of inequalities follows the same standard framework operating at the individual level this time. The question to address is whether the strength of relationship between different attributes such as earnings and structural factors such as social class, gender and ethnicity is increasing or decreasing over time in response to policy initiatives and what are the variables such as qualifications that can moderate it.

(h) Identity

23. A number of the studies reviewed contain some data related to the concept of national identity, but none explore it in enough detail to allow distinct longitudinal analysis of the outcome in its own right. Furthermore, as many of these studies are UK-wide, the questions often refer to 'British' rather than 'Scottish' identity as in the case of the most recent second British Election Panel Study ( BEPS). To pursue the issue properly would require either a module devoted to identity in a study like the new UKHLS or investigation indirectly via activities such as voting.

Chapter 4 Cross Cutting Issues

24. In the course of the scoping study a number of cross-cutting issues arose that bear on the development of national strategy for longitudinal research resources in Scotland and the investment priorities to be attached to it.

(a) Longitudinal Studies Support

25. The main issue to resolve is the choice between continuing or terminating existing studies or expanding the portfolio with new studies. The evidence points to giving high priority to:

• continuing the key Scottish national studies - GUS, SLS and replacing SSLS
• continuing to support the UK-wide longitudinal studies particularly those having a Scottish boost, which provide sufficient numbers for specialised Scottish analysis including first, the MCS and the BHPS/ UKHLS, second, those without a Scottish boost but just sufficient numbers for analysis, NCDS and BCS70
• continuing support for the specialist area studies that operate effectively in Scotland WoS, GWG and ESYTC as holding potentially a key place in the portfolio.

All of these studies merit much more extensive exploitation for policy purposes.

26. Consideration should be given to establishing new studies of:

• ageing and health - the Scottish version of the English Longitudinal Study of Ageing ( ELSA) - the need for which is currently being scoped
• businesses and employees in rural and urban environments in a multilevel area-based design
• life in rural areas extending to in- and out-migration
• youth transitions c.f. the 2004 (13-25) Longitudinal Study of young people in England

(b) Longitudinal Resource Strategy

(i) Longitudinal Awareness

27. Longitudinal research enables the relationship between input of services and delivery of outcomes to be better understood. The data sources that best support this goal are the specifically Scottish studies and the multi purpose GB and UK-wide longitudinal panel and cohort studies, especially where a boost sample has been applied. The longitudinal research resource, in which there has been much Scottish investment, is underused but has much value for the appraisal and development of the national performance framework. More could be done to promote more widely the use of the resources for policy purposes.

(ii) Administrative Data Linkage

28. All the studies reviewed will be greatly enhanced by linkage to administrative data especially the Scottish Longitudinal Study, which is thin on coverage but very good on sample size (274,000). The key administrative data sources identified are SSA (in its current form) ¹, ASHEPD, IDBR, DWP and SCORE. Their linkage is best achieved within a coordinated strategy to ensure that the right ethical and data protection safeguards are in place and involving all SG policy divisions and analytical services. The plan to establish a 'secure data lab', in the Scottish Government offices for the research use of such administrative data presents positive opportunities in this respect.

(iii) Local Factors

29. Longitudinal studies for Government purposes tend to be seen in national terms. We draw the conclusion, however, that certain area studies, focussing on particular policy domains such as crime, health and disadvantage, because of the multilevel depth of enquiry, offer much of value to policy development. This also links to one of the over-riding themes of the performance framework that national purposes and objectives are ultimately delivered locally through service provision. Understanding policy delivery on the ground, through exposure to it of sample members in longitudinal studies, is therefore a very important part of the research requirement. Area-based studies are particularly valuable in this respect and more of them could usefully be established. However the balance needs to be struck between what is a valid indicator for local evaluation purposes and what is needed for national evaluation to make valid comparisons across Scotland. Local studies also offer collaborative opportunities such as between the city of Glasgow and Glasgow University's Department of Urban studies in GWG. They can also support such tactics as phased introduction of new policies creating quasi-experimental opportunities for policy evaluation.

(iv) Comparative Analysis

30. The advantage of UK-wide longitudinal studies such as BHPS/ UKLHS, the British Cohort Studies, MCS, NCDS and BCS70 and SLS is the opportunity they provide for comparative analysis between the different countries of the United Kingdom. There is therefore much value in investing in boosts, to enable separate Scottish analysis to be carried out within the comparative framework. There is a trade off between this approach and that involved in GUS, for example, which relates to, but is quite distinct from its nearest counterpart, MCS. In the wider comparative framework certain studies like the English Longitudinal Study of Ageing, are part of a cross-European collaborative project, the Study of Health and Retirement in Europe ( SHARE), for which the Scottish version of ELSA (should any such study be taken forward) could also become eligible.

(v) Capacity and capability

31. The Office of the Chief Researcher has been prominent in developing research capacity across government and promoting knowledge transfer through joint PhD studentship programmes. However, competence in universities and parts of Government in the production and use of longitudinal data still has room for improvement. Joint arrangements between local authorities and local universities on the GWG model are still rare and Scotland has yet to be awarded a node in the ESRC National Centre for Research Methods. Expansion of longitudinal research methods training must therefore be a priority.

Chapter 5 Conclusions and Recommendations

32. Longitudinal data has much to offer for policy development and evaluation in the areas of education, health and crime and potentially could offer more in the areas of economy, community and the environment. Consideration could usefully be given to developing the longitudinal resource base for policy purposes in four ways:

1. Coordinated strategy for the production and more effective use of longitudinal data in all domains of policy development and evaluation.
2. Coordinated strategy for integrating administrative data, linking to longitudinal survey research resources and formulating a comprehensive policy for research access and use.
3. Development of the performance framework in the light of longitudinal research insights and development of the longitudinal resource base to match better the performance framework's needs.
4. Enhanced capacity to match new longitudinal investment opportunities.

« Previous | Contents | Next »