The Scottish Government

« Previous | Contents | Next »

Listen

CHAPTER 1: BACKGROUND TO THE SCOPING STUDY

(a) Range of the study

1.1 This report describes the findings of a scoping study directed at informing the Scottish Government about the use of longitudinal research in the evaluation of the Government's national performance outcomes. The work builds on previous reviews undertaken by Longview for the Economic and Social Research Council ² and a series of overviews of longitudinal research resources in Scotland, including one carried out for the Scottish Government by ScotCen in 2001. ³

1.2 The report's primary concern is with the design and content of longitudinal research programmes, surveys and resources that bear on the National Performance Outcomes. It also highlights gaps in these resources where new studies are likely to be helpful. In furtherance of this aim the work focussed on six features of the research brief, which have been addressed in the order shown below:

a) Understanding the policy context in which the evaluation of research resources will add value
b) Comprehensive coverage of the different longitudinal research resources in terms of relevance to The Scottish Government's strategic plan
c) Analysing the design features and coverage of the studies in the most effective way to enable their value to be understood by matching longitudinal resources to particular policy outcomes
d) Demonstrating the ways in which different longitudinal resources may be used singly or in combination to evaluate the strategic outcomes and weighing their relative value
e) Identifying gaps in the longitudinal resources portfolio that need to be filled by enhancing particular studies or supporting new ones
f) Identifying cross-cutting issues that need to be addressed in relation to the development of the portfolio and ensuring its most effective use

1.3 This chapter sets out the main features of the study, including the Scottish government's performance framework to which the use of longitudinal data is directed. Chapter 2 following examines 29 longitudinal studies and data sources of potential value in the evaluation of the Scottish Government's National Outcomes. This is supported by Appendix 3, which supplies summaries of the studies grouped in terms of the National Outcomes to which they relate - as specified in the 2007 Spending Review. Chapter 3 contains the bulk of the text in setting out research scenarios in the form of vignettes relevant to each of the 16 National Outcomes grouped in accordance with 6 major policy areas. Chapter 4 considers investment priorities and cross-cutting issues that need to be resolved in developing national longitudinal research resources strategy for Scotland.

The report both supplies the findings of the scoping study while also serving as a resource for government and other users of longitudinal data working within the national performance framework. The National Outcome or Outcomes of interest can be tracked to the vignettes setting out the research scenarios to which they relate. Technical details of the relevant resources can then be pursued through the summaries in Appendix 3. The next step is a visit to the relevant website to examine the study's documentation. Frequencies and other descriptive data for relevant variables can then be appraised, if supplied, or run off direct from the data sets. An analysis programme can then be specified.

Longitudinal Research

1.4 Longitudinal research takes many forms so it is important to make clear what we have identified as particularly relevant to the purposes of the review. In the broadest sense longitudinal research involves the follow up of any set of entities in which changes can be observed over time. Entities may range from nation states through institutions such as hospitals, down to individuals in the population at large. Broadly our focus is on the last of these, i.e. individuals in defined populations either in this case the population of Scotland as a whole or sub-groups defined by location in the population, e.g. geographically or institutionally, to which the Scottish Government's National Outcomes relate. Thus although certain outcomes relating to the business context ('wealthier and fairer'), and the environment ('greener'), are directed more at the macro level, e have not reviewed the range of typically cross sectional surveys or studies of institutions, like schools, hospitals or businesses through which assessments of change for this purpose is achieved. This defines the remit as concerning a range of studies:

• birth cohort studies that follow up individuals from birth through to adult life
• age cohort studies that begin data collection at a particular stage in life, such as adolescence or old age, usually defined by chronological age
• household and family panel studies, where data on all eligible individuals ( e.g. 16 or older) are collected within the unit of the household

1.5 Research using these data sources has a number of foci of which the most relevant for Scottish Government purposes are:

• predicting from observed data a given outcome at a given age in later life and testing the prediction through data collected later ( e.g. "Born to Fail")
• explaining a given outcome in terms of what has been observed earlier ( e.g. "Origins of adult obesity")

1.6 Where cohort studies and panel studies extend over a period of time through repeated cohort studies, then the effects of social and policy change (cohort effects) may also be observed through the changes in relationships between explanatory and outcome variables.

1.7 Such research needs to be distinguished from studies set up specifically to evaluate the effects of Government policy, where data collection is directed at sections of the population identified with exposure or not to a particular policy. Where policy implementation is universal and a control group is clearly not possible such evaluation is limited to monitoring changes in the population typically using administrative data sources. The report refers to this work, but more in relation to its potential in enhancing the coverage of longitudinal surveys rather than as a replacement for them. The ideal evaluation model is one which controls all extraneous variation with which the policy effect could be confounded achieved by randomising the allocation of individuals to treatment and control groups. But for ethical reasons such 'Randomised Control Trials' ( RCT) are rarely applied in policy evaluation in the UK and are not considered further here.

1.8 In any event our focus is not so much on direct evaluation of this kind, than on modelling the patterns of relationships between variables to which policy refers more generally. When there is differential exposure to new policy intervention such as the banning of smoking in public places ( e.g. the Scotland - England timetables of implementation), then something can be learned from longitudinal surveys of the populations involved about the policy's effectiveness and for whom it has worked.

1.9 The report also focuses more on research resources than attempting an exhaustive review of findings. This is again in aid of identifying the features of infrastructure for evidence building on which policy development can be effectively based. We review longitudinal resources identified as relevant to the review in some detail in the next chapter.

(b) Sources of Evidence

1.10 The work involved a detailed review of the content and supporting documentation of some 29 longitudinal surveys and administrative datasets, ranging from those that were exclusively designed for Scottish purposes and based in Scotland to other UK wide projects with much relevance to Scotland. The studies were initially identified from the previous overviews and supplemented through the interviews and the work of an 'expert group'. Each of the longitudinal resources were classified in terms of a number of dimensions of which the most important initially was the extent to which the focus of design and coverage was specifically Scotland, as opposed to, for example, the whole of the United Kingdom. The latter included the UK-wide longitudinal surveys where to ensure adequate data for Scottish analysis the samples had boosts to double or treble their naturally occurring size in Scotland as determined by probability sampling. Others had just sufficient numbers for analysis (generally approaching 1000 people currently participating) as they stood. Full details are supplied in the next chapter.

1.11 As the brief for the scoping study was focussed at the level of the National Outcomes, the task for the research team was to identify longitudinal research resources that could most effectively bear on their achievement. Most resources would have to be, of course, historical in the sense that the data they comprise had been collected in advance of the strategic outcomes being specified. However, this does not diminish their relevance as they supply the basis for modelling processes that are relatively constant over long periods. Where there are changes in the relationships of the variables involved, earlier surveys in a time series of repeated longitudinal surveys supply the baselines against which the likely effects of new policies and delivery processes can be assessed. Other resources were current with new data collected or in the pipeline. A third category was that in which no longitudinal research resource was available to match the outcome, in which case the argument for new data collection had to be made.

1.12 To gain better understanding of the role these longitudinal research resources might play in the evaluation of SG's National Outcomes evidence was collected from two main sources

(i) Government Officials and External Experts

1.13 Interviews with senior analytic services personnel directing the use of data for Government purposes in different policy areas were arranged by the Chief Researcher's project liaison officer. External experts were contacted directly. Table A1 ( Appendix) lists all the people interviewed. Against the background supplied by the 2006-2009, Scottish Government Spending Review (2007) which encompasses much of current Government thinking, interviews with officials were directed at:

a. Gaining understanding of the Analytic Services role in the policy area
b. Experience of and use made of longitudinal data - to identify those resources that were:
i. Most valuable
ii. Important but not so far picked up by the study
iii. On the way
c. Perspectives on the value of longitudinal research in relation to the evaluation of the Government's National Outcomes
d. Where in the policy domain new knowledge gained from longitudinal research was most needed
e. Key stakeholders in longitudinal data

1.14 Interviews with external experts were directed at illuminating the design, coverage and future plans in key studies and the use of longitudinal data in particular specialist areas such as housing. They included the Chief Housing Officer (Policy Review and Development) Glasgow City Council, who is a leading expert on housing research in Scotland. Principal Investigators ( PIs) of key studies were contacted to clarify features of design and coverage and to learn about future plans.

(ii) Expert group

1.15 Apart from evidence collected direct from individuals and from relevant documentation, we also established an expert group of leading Scottish academics with expertise and experience of longitudinal research and its use for Scottish Government purposes (see Appendix1). The group met early on in the project to help develop the parameters of the study and identify key individuals to talk to and relevant data sources not included in our initial specification. In the first meeting, many ideas were put forward which helped to shape the course of the review, especially in working out the kinds of cross-cutting analysis of data sources that were likely to be most fruitful. Additional longitudinal resources were also identified such as the Annual Survey of Hours and Earnings Panel Data ( ASHEPD), the Individual Business Register ( IDBR) and the Department of Work and Pensions Longitudinal study ( WPLS) based on DWP data. A second meeting was held in late March with four of the group to consider initial findings and some draft material for the main report. Most valuable inputs into the work were gained on both occasions and through follow-up correspondence. Finally a meeting was held by video link to discuss the draft report.

(c) Government purpose, strategic objectives and performance outcomes

1.16 The interviews with policy analytic services personnel proved particularly helpful in enhancing our understanding of how the government's 'strategic purpose' is to be realised through the five strategic objectives and the 15 National Outcomes. We needed to find out how this aim was being worked out in each policy area and at each level from national government via local government down to the local community. The evaluation framework and the standards by which its effectiveness is to be judged has been strongly influenced by the 'Virginia model' of local service delivery base based on the joining up of services. An important feature of the model is that outcomes are not framed in terms of output targets, which have to be achieved to prove success, but define goals towards which local administrations can work to bring about improvement in specified directions.

1.17 Thus the Government's National Outcomes can be viewed as part of a hierarchy of goals and aspirations, varying in their degree of specificity and consequently utility in relation to policy evaluation. At the top of the hierarchy is the Government's purpose "to focus Government and public services on creating a more successful country with opportunities for all of Scotland to flourish through increasing sustainable economic growth". The purpose is elaborated through seven economic targets that set out, as specific benchmarks, the achievements to which policy is directed: economic growth, productivity, participation, population, solidarity, cohesion. The key to their realisation is evidence of performance improving, being maintained or worsening e.g.:

• economic growth comprises achievement of the UK level by 2011;
• productivity is for Scotland to rank in the top quartile for productivity against OECD trading partners by 2017;
• participation is to maintain the position in labour market participation as the top performing country in the UK and to close the gap with the top 5 OECD economies by 2017;
• population is to match European population growth over the period from 2007 to 2017 supported by increased healthy life expectancy;
• solidarity is to increase overall income and the proportion of income earned by the three lowest income decile groups as a group by 2017;
• cohesion is to narrow the gap in participation between Scotland's best performing regions by 2017;
• sustainability is to reduce carbon emissions over the period to 2011 by 80%.

1.18 The five strategic objectives head the plan for realising the purpose and targets providing the supporting policy strands through which the purpose will be achieved and comprise:

• Wealthier and fairer - Enable businesses and people to increase their wealth and more people to share fairly in that wealth.
• Healthier - Help people to sustain and improve their health, especially in disadvantaged communities, ensuring better, local and faster access to health care.
• Safer and stronger - Help local communities to flourish, becoming stronger, safer place to live, offering improved opportunities and a better quality of life.
• Smarter - Expand opportunities for Scots to succeed from nurture through to life long learning ensuring higher and more widely shared achievements.
• Greener - Improve Scotland's natural and built environment and the sustainable use and enjoyment of it.

1.19 The five objectives spanning the key functions of Government identify the aspirations that the Government sets for its strategic plan. Their measurement for the purposes of evaluating the extent to which the objectives have been achieved requires more specificity. This is achieved through the national performance framework developed for the current (2007) three-year spending review, which sets out fifteen National Outcomes to be achieved (approximately 3 per strategic objective) over a ten year period. The framework also specifies the indicators through which measurement of the outcomes can be achieved at national and local level, largely through the collection of relevant administrative data, 45 in all.

1.20 A key theme that emerged from the interviews was the relationship between national and local policy development and evaluation. While the national evaluation requires standardized performance appraisal across the whole of Scotland, local evaluation adheres to the principle that delivery processes and the means of evaluating their outcomes can be negotiated locally in the form of 'Single Outcome Agreements' - adding another 52 local monitoring indicators to the 45 national indicators.

1.21 It was agreed with the Commissioning Panel to focus attention more at the upper level of the framework, especially on the National Outcomes rather than the performance indicators. This is because the achievement of the outcomes can be seen more in terms of a research enterprise, i.e. gaining understanding through statistical modelling of relationships between variables how likely it is that the delivery of a given service or intervention, in a given set of circumstances, will achieve a desired outcome. This will be reflected in changes in individuals in response to experience of, or exposure to, the policy. The performance indicators can then be used to supply direct evidence of this change in the time series that can be constructed from administrative data, e.g. employment rate, air pollution, smoking prevalence and so on before and after implementation, with some local variation. Figure 1.1, derived from the Spending Review, sets out the relationships between the five strategic objectives and the National Outcomes schematically.

Figure 1.1 Strategic Plan

1.22 There are two features of this system which are distinctive and set it apart from its counterparts in other parts of the UK, especially England. First, as noted earlier, in accordance with the Virginia Model of service delivery the National Outcomes are used as goals to which local administrations can work in the delivery of services to bring about improvement in specified directions. These replace the 400 or so delivery targets that prevailed before the new performance framework was established.

1.23 The second distinctive feature is the responsibility given to local Government for both interpreting what needs to be done to achieve a given outcome and how the service is best delivered to achieve it. This may even include in some instances a specific target to be aimed at. As noted, the degree of negotiated autonomy in such an arrangement extends even to the indicators themselves that are most appropriate to identify whether the delivery has succeeded. However, notably at our expert group meeting the point was made that such a degree of devolved responsibility carries with it, in a sense, an even bigger requirement of standardising certain features of measurement. Without such standardization comparisons between one area and another become impossible, which means that relative improvement (or deterioration) of services cannot be appraised. Nor can the effects of different policy interventions be compared - i.e. if Shetland and Glasgow roll-out policies in different ways then they need some standardized local data to evaluate the effectiveness of each of them

1.24 We acknowledge that the report cannot attempt to embrace the whole range of local variation, as the Commissioning Panel, in their response to our interim report, made clear. We should focus on longitudinal research relevant to the National Outcomes. However it is difficult to divorce the issue entirely from the work of the review, not least because it draws attention to the multi-level framework, through which policy is actually arrived at and is implemented. This may be taken to support more widespread interest than hitherto by Government in local area-based longitudinal studies of which Scotland has some excellent examples.

1.25 National level enquiries tend to be given priority in establishing population parameters for the explanatory models that are evaluated, but when it comes to applying their results to particular places questions may arise as to how relevant they are. On the other hand, as we consider later, in certain parts of some policy areas where the cultural component is minimal or non-existent, such as health, results are likely to be more widely applicable, e.g. the consequences of air pollution for lung disease. In this instance the local study has the definite advantage of enabling the multi-level features of service delivery via schools and other services to individuals and families to be fully investigated, supplying insights of relevance to the whole of Scotland. It is for this reason that the Avon Longitudinal Study of Parents and Children ( ALSPAC) has been incorporated in the UK National Data Strategy funded by government and the research councils.

1.26 The approach of operating across all levels of the performance framework while focusing particularly on the top two has distinct advantages in relation to the project brief. This is because, as senior policy analysts told us, the major interest is not so much what predicts the achievement of a particular indicator, but the strength of the relationship between variables measuring policy and practice input with variables identified with policy outcomes or goals. In this scenario the immediate question that arises is how the five strategic objectives, through realisation in the fifteen National Outcomes, are best operationalised. Our Expert Group made the point that the linkage derived from the Spending Review, as shown in Figure 1.1, was subject to considerable variation depending on the precise interpretation put on the objectives. Thus, for example, implicit in almost all the objectives is the creation of not only better performing individuals in health, education, the economy and so on, but a more equal society. As improvements in life in one domain will affect improvements in others, most of the outcomes in various ways relate to this performance objective. Health inequalities, for example, though not mentioned explicitly, could be seen as embracing all 15 outcomes. Nevertheless the appeal of Figure 1.1 is that it does immediately set out the parameters of a possible research agenda. In this sense the top two levels of the performance framework turned out to be the right level of appraisal at which to work.

1.27 Another basis for classifying not so much the goals of a society, but the issues that need to be addressed in data terms, is that of the UK's "National Data Strategy". This links the scientific interests of the research councils with those of the UK Government. It is informative to compare the SG's five objectives with the five key research challenges that the national strategy identifies: ageing population, migration, globalisation and the emerging economies, child development and education, carbon economy and environmental change.

1.28 The notable difference with Scottish objectives (except implicitly) is migration. The loss of the best-educated section of the population in the past is now being reversed - as in Ireland - by a degree of in-migration much of it from the A8 European countries. Longitudinal research is a means of identifying such population movements in relation to the National Outcomes and is of particular importance in the Scottish context. The fact that longitudinal migration data is high up on the agenda for national investment in the UK will therefore provide a benefit to Scotland. This would be helped by making the topic explicit in the strategic plan.

1.29 Internationalisation of the research agenda is another important feature of the UK strategy as is the need to bridge current academic discipline boundaries and to improve data access. The research potential of administrative data held by the agencies that are central to the Scottish Government's plans is considerable, which includes pensions, social security, health, business, and vital statistics. Means need to be found for expanding access to these sources. All of these data strategy aspirations can be seen as elaborating key elements of Scottish longitudinal resource strategy. Again Scottish benefit can be derived from UK investments.

(d) Methodological considerations

1.30 In using longitudinal data to draw judgments about policy effectiveness the distinction has already been made between monitoring the effects of policies and evaluating the policy as a whole in the sense of linking (policy) cause with (outcome) effect. Our concern is with the latter for which the analytic strategy required is statistical modelling. We postulate a process through which a delivered policy will convert into a hypothesised outcome or outcomes and longitudinal data can be used to estimate the model's parameters, i.e. the relative effects of the policy intervention as opposed to that of other variables on the given outcome. There is a whole range of possibilities here from examining, very tightly, the likely returns to a particular policy intervention such as the current policy of reducing class sizes in early primary school, or community regeneration in the East of Glasgow - holding constant the effects of all variables that might be confounded with the policy shift - to 'structural' models embracing multiple causation. In applying the strategy a number of problems have to be confronted some of which are common to survey research generally and others take on particular force in longitudinal enquiries.

Selection bias

1.31 In interpreting the parameter estimates of statistical models in causal terms we have to beware of confusing improvement with self selection of families and individuals distinguished by certain attributes who took advantage of the new provision. We therefore need to control statistically the personal attribute variables lying behind such selection effects. This entails using studies that have sufficient breadth of coverage both cross-sectionally and longitudinally to make sure that all the relevant variables are included in the analysis. Longitudinal data sets are generally better able to yield such controls than cross-sectional data sets, because the temporal order of variables is established in the longitudinal design.

Ecological fallacy

1.32 Another potential hazard is the 'ecological fallacy' in which area level improvement is taken to signify improvement for all the individuals living there. Thus as officers put to us, urban regeneration as in East Glasgow may be judged successful on the basis of an indicator such as employment rate. But this masks three separate reasons for the improvement only the first of which is identified with the policy goal: increased employability of the people living there; higher employability of people moving there to take advantage of improved opportunities; lower employability of people moving out. Only individual level longitudinal data can decide between them.

Ecological validity

1.33 The final form of information that can bear on the effects derives from a different mode of research altogether. Case studies of individuals and communities can often supply evidence of changes that are happening in the light of interventions, which cannot be revealed in national statistics. Particularly the nuances of process in the translation of delivery to outcomes in particular ecological contexts can be revealed through ethnographic and socio-biographical studies generating much rich data from observation and interviews with participants on the experience of the delivery process and how to improve it. Combined approaches with statistical studies offer the best strategy, using triangulation across different data sources, to gain insights into what works and what doesn't under what circumstances.

Measurement

1.34 The measurement of the input and outcome variables raises methodological challenges as does the sample on which a given longitudinal survey is based. Notably in Figure 1.1 the outcomes for any give objective combine both highly specific operational measures together with contextual features of the environment in which the policy is being implemented or the service delivery taking place. Thus Wealthier and Fairer combines the relatively straightforward skills enhancement and increased employment opportunities with a 'nice place to do businesses and 'inclusive national identity'. Apart from the inherent difficulty in assessing the latter, over time meaning and significance of these outcomes may change as what is judged to define quality of the local context shifts or matters more for one group of people rather than another. Thus, as was pointed out by the expert group, for the newly graduated a nice business environment with good local amenities may include pubs and clubs, whereas for young families high quality schooling and a safe environment may have top priority.

Sampling

1.35 The nationally representative sample takes precedence for judging the effectiveness of national policy in terms of national policy outcomes. But for the local policy delivery - where scope to adapt lies at the heart of the Government's approach - distortions may be introduced through generalisation of national findings to all localities. It may be bad enough to generalise findings from English longitudinal research on the effects of poor housing on social exclusion to Scotland, but to go further by extending them to Glasgow, where homelessness and workless families, for example, are way above the national average, may fail to recognise that some aspects of the local policy context do not match the national one. On the delivery side there is also much variation. Community management partnerships in Glasgow, for example, include the Glasgow Local Authority as one partner among many, whereas in most other places the LA is automatically the lead partner.

1.36 Such cautions can of course be overstated. Much of policy relevance to one area is relevant to another, e.g. on matters of health and ageing. And an area study, such as the medical science-based West of Scotland longitudinal enquiry, has policy messages for the whole of Scotland, if not the whole of the UK and elsewhere. The main point to stress is that a national framework for evaluation makes most sense the more local variation can be encompassed within in it; fine tuning of its findings to local delivery needs and circumstances is then likely to be easier. The research solution is moving increasingly towards large scale multi-level longitudinal research designs to enable estimation of the separate effects of national and local variation. But such approaches are relatively rare among existing studies. We argue later for an intermediate stage where local longitudinal studies on special topics complement national ones.

Sample loss

1.37 Undoubtedly the biggest challenge to meet in longitudinal research is reduction in participation over time. Such sample loss or 'attrition' occurs for a variety of reasons ranging from untraceable after a house move to refusal to take part. If the dropout is random, then the only loss statistically is to the precision of estimates as the sampling error goes up with the decline in sample size. The stopping point comes when the sample decline renders the study unusable, With the large samples typical of the UK wide longitudinal studies this rarely happens, though the Scottish sample within them may on occasion be too small, e.g. for particular sub-groups, to rule out a particular piece of analysis. The more serious issue arises when the attributes of the drop out are related to the outcome variable of interest in which case the estimates obtained will become increasingly biased. The major studies do a lot to reduce drop-out to the minimum and generally such biases are small mainly residing in more men than women and the less educated more likely to leave the study. However, for sub-samples they may be much larger; so if the study is to be used, the bias needs to be controlled by re-weighting to restore the wave 1 distributions of key variables. Where the missing data is at the level of individual variables through non-response, refusal to answer a question and so on, statistical imputation methods may be needed to replace the missing values by estimates of what they should be based on all of the available data.

Fading relevance

1.38 Another challenge to be met in longitudinal data use resides in the datedness of the study as whole or of any given sweep. Many life course processes such as those to do with health and illness remain relatively stable over time. Others will shift as the social context changes for political or other reasons reflecting social trends. Another yardstick for judging the value of given study therefore is whether the findings from it still have relevance for current conditions. If they do not then the study may be useful in supplying baseline data for the changes that have occurred. But this does not remove the need for contemporary data and consequently the need for continual updating of the longitudinal resource with new studies.

(e) Analytic approach

1.39 The commissioning panel and the expert group endorsed the strategy of starting with a review of the design and coverage of longitudinal studies to establish their relevance to the strategic objectives and the National Outcomes. To meet this purpose a grid cross-classifying each study against each outcome would then be constructed. The next step was less obvious. A study by study analysis would not only be in danger of drowning the reader in the details of each study, but would tend to down play complementarities between them. The final strategy was shaped by discussion at the expert group meeting, which argued for a two pronged approach. The first would be to set out a series of research scenarios in the form of 'vignettes' in which studies linked together around the variables they contained could be used to address the objectives and the outcomes in each of the main policy areas. The second would go further in terms of identifying crosscutting issues that would need to be resolved in establishing where the best pay-off from investment in particular studies was likely to lie.

« Previous | Contents | Next »