The Scottish Government

« Previous | Contents | Next »

Listen

CHAPTER FIVE: PROPOSALS FOR THE SUMMARY EVALUATION OF CTOG

OVERVIEW

5.01 The objective of this chapter is to make recommendations for the evaluation of the 'higher order' levels of CtOG, i.e. the impact of CtOG's six higher-level Objectives (1.04) and the impact of CtOG as a whole (1.03).

5.02 It is recommended that evaluation of CtOG should comprise the following three elements:

• Evaluation of poverty rates and dynamics in Scotland
• Evaluation of change to CtOG Objectives
• Evaluation of CtOG as a process.

5.03 An 'evaluation of poverty rates and dynamics in Scotland' is required to evaluate the overall aim of CtOG, which is a poverty reduction strategy with three tenets, i.e. to prevent individuals or families from falling into poverty; to provide routes out of poverty for individuals and families; and to sustain individuals or families in a lifestyle free from poverty.

5.04 An 'evaluation of change to CtOG Objectives' must be an integral part of any higher order evaluation of CtOG, i.e. that above the level of CtOG Targets.

5.05 An 'evaluation of CtOG as a process' is not an essential component of any higher order level evaluation of CtOG. However, CtOG is not merely a programme of work with a set of clearly specified outcomes. It is also a blueprint for a more participative and collective way of effecting social change (improvement) in Scotland. As such, it is recommended that the whole process of 'doing CtOG' be considered as an integral part of the summary evaluation.

5.06 Each evaluation option is self-contained, which permits any combination of options across evaluation elements. For example, it would be possible to combine an elementary evaluation of poverty rates and dynamics in Scotland, with a more detailed evaluation of change to CtOG Objectives, and a comprehensive evaluation of CtOG as a process. This framework affords a high degree of flexibility to the Scottish Executive to prioritise resources set aside for evaluation.

5.07 In this chapter, elementary evaluations are presented as Option 1, more detailed evaluations as Option 2 and comprehensive evaluations as Option 3. The more detailed evaluations of Option 2 also include the work undertaken for the elementary evaluations of Option 1. The comprehensive evaluations of Option 3 also include the work undertaken for the detailed evaluations of Option 2.

DESIGN CONSIDERATIONS

5.08 Brief note should be made of three issues, which were central to the evaluation design; the extent to which CtOG should be concerned with evaluating whether the gap between more and less disadvantaged has been closed (5.09); the extent to which the impact of CtOG can be specified (5.10) and evaluation of processes (5.11). Consideration of these issues leads to a research design that comprises quantitative (extensive) and qualitative (intensive) analysis for each evaluation element (5.12).

5.09 Unlike some of the SJS milestones, which CtOG superseded, the emphasis of the CtOG Targets and Objectives is improving quality of life of the most disadvantaged, rather than explicitly 'closing the gap' between those who are more and less disadvantaged in Scotland ¹⁵. However, it would be prudent to maintain a focus on the extent to which 'gaps have been closed', i.e. to place changes to the lives of the most disadvantaged against the wider context of change in Scotland. In turn, this would facilitate an understanding of the role of CtOG in Scottish society, i.e. as a transformative programme (if gaps are being closed), as a conservative programme (if gaps are being maintained), or as a limited programme (if gaps are widening). Thus, the evaluations that are proposed are concerned to move beyond the Targets and Objectives set for improving the lives of the most disadvantaged populations and places in Scotland, to consider the extent to which the CtOG Programme has indeed effected change in reducing the gap between the most and least disadvantaged.

5.10 The ultimate goal of this evaluation is to ascertain the impact of CtOG upon the lives of the most disadvantaged in Scotland. Ideally, this would involve precise specification of the ways in which, and the extent to which, CtOG was directly responsible for effecting any changes to the lives of the most disadvantaged. However, there are two complications, which render this problematic:

• The array of factors which shape the life experiences of the most disadvantaged is so wide-ranging, multi-dimensional and inter-related that specification of the extent to which CtOG work programme actually shapes CtOG 'higher order' outcomes is akin to informed guesswork at best. It may be possible to tease apart the impact of CtOG on specific case studies (of people and places), but it would not be possible to ascertain this for Scotland as a whole.
• More fundamentally, CtOG is conceived as a collaborative effort harnessing the energies of a wide range of organisations in Scotland. It could be argued that it is against the ethos of CtOG to be overly concerned with ascertaining that the Scottish Executive's work programme is responsible for this change.

Thus, the evaluation should focus on how the CtOG work programme has contributed toward achievement of CtOG Aims, without being constrained by the necessity of determining the precise extent to whichCtOG is ultimately responsible for effecting these changes. Case studies should be used to provide an evidence-based for the positive-impact of the CtOG work programme.

5.11 It is not sufficient to merely demonstrate that CtOG is a process, i.e. a collaborative effort harnessing the energies of a wide range of organisations in Scotland. The extent to which organisations acknowledge CtOG as an overarching goal (or, at least, share the Aims of CtOG), the effectiveness with which these energies are harnessed and the experiences of those working directly to deliver CtOG outcomes should be among the 'process' issues which are subject to evaluation.

5.12 Each of the aforementioned evaluation design considerations (5.09 - 5.11) lends itself to an evaluation design that comprises both quantitative (extensive) and qualitative (intensive) analysis. There is a need, wherever possible, to present aggregate statistics to demonstrate the impact of CtOG on Scotland as a whole, and on particular sub-population and areas therein (quantitative analysis). However, equally important is the need to understand how CtOG works, which necessitates a complementary qualitative approach to analysis.

EVALUATION OF CtOG AS A POVERTY REDUCTION STRATEGY

5.13 CtOG's overarching Aims are to prevent individuals and families from falling into poverty, provide routes out of poverty and enable them to sustain poverty free lives. First and foremost, the evaluation must consider how best to evaluate the impact of CtOG as a poverty reduction strategy.

5.14 CtOG is not the only factor that will influence the levels of poverty among people and places in Scotland. Local interventions (of local government, private sector and voluntary organisations), UK level interventions (such as the government's commitment to eradicate child poverty within a generation) and global processes (e.g. the economy) will each influence levels of poverty in Scotland. An awareness of the wider context of factors that influence poverty levels is therefore required, regardless of which Option is favoured.

5.15 It should also be acknowledge that levels of poverty in Scotland are already being monitored by academics, voluntary organisations and national government. Data are readily available to undertake a Scottish-level analysis of the incidence of poverty using accepted definitions of low income and child poverty, drawn from UK-wide data. Thus, snapshot and time-series analysis of trends in the levels of poverty in Scotland are straightforward undertakings. It is proposed that the evaluation of CtOG as a poverty reduction strategy maintains an awareness of these poverty-monitoring activities, but that CtOG's poverty evaluation should extend beyond aggregate levels of poverty in Scotland.

5.16 It is important that the evaluation of CtOG as a poverty reduction strategy uses widely accepted and robust measures of poverty. Although there is no consensus of best practice, considerable progress toward an agreed measure of poverty has been made in recent years. The DWP have consulted widely among the policy community in the UK, to work toward identifying a measure of child poverty ¹⁶. More generally, an accepted definition of low income is that based on relative household income (after housing costs) ¹⁷. It is proposed that these definitions of poverty and low income are utilised in the CtOG evaluation.

5.17 Three options are presented below for consideration. These options increase in sophistication (and cost, work input) from one through three.

5.18 As with any research actual costs will clearly depend heavily upon the precise content of the research brief, particularly upon factors such as the number of datasets to be considered and the number of areas in which qualitative work should be undertaken.

5.19 Given the formative nature of these evaluation models it is also important that the proposed interview sample sizes are seen only as guides to the likely size of each evaluation option, not as precise requirements. Clearly, accurate assessments as the numbers of required interviews will not be possible until a full tender specification is complete.

'Poverty Measures' Option 1

5.20 Option 1 is to undertake a Scottish-level analysis of poverty incidence using accepted definitions of low income and poverty, drawn from UK-wide data, disaggregating the analysis wherever possible into geographical, social and socio-geographical groups.

5.21 This would involve quantitative analysis of nationwide, sub group and local level trends (of particular pertinence given the concentrations of CtOG funding in particular local authority areas, and the concentrations of deprivation in particular Scottish localities). It would provide a snapshot of poverty levels in Scotland at the end of the CtOG programme compared to a 2004 baseline. The datasets that could be included in this analysis are listed in chapter 2.

5.22 This Option would present an analysis of the current incidence of poverty in Scotland. Furthermore, it would present an analysis of changes in poverty levels in Scotland, which occurred during the initial years of the CtOG programme. Interpretation of cross-sectional aggregate statistics would provide an indication of poverty risk in Scotland (for people and places).

5.23 This Option would not allow for consideration of the extent to which any changes in the incidence of poverty were attributable to poverty having had been prevented, whether routes were being provided out of poverty, or whether people were being enabled to sustain poverty free lives, as the Option does not comprise longitudinal data, nor a qualitative component to explore processes. Furthermore, this Option would not facilitate more than a cursory account of the extent to which CtOG was responsible for these changes, as there is no means of factoring-in CtOG as an independent variable in the analysis of the data that would be used.

5.24 This Option would require the services of an analyst with an awareness of current debates on measuring poverty, awareness of the data sources available to measure poverty in Scotland, and the competence to analyse survey datasets.

'Poverty Measures' Option 2

5.25 Option 2 would supplement Option 1 by undertaking an additional study of poverty dynamics for social groups, utilising longitudinal data (from the British Household Panel Survey) and for geographical area types, using cross-sectional data (from the Scottish Household Survey).

5.26 Option 2 would comprise quantitative analysis of nationwide, sub group and local level trends. It would involve analysis of how the risks of entering poverty had changed over the operation of CtOG, for the population as a whole and for particular social groups.

5.27 This Option would provide a quantitative measure of whether poverty had been prevented (nationally, amongst particular groups and in particular areas), and of whether routes were being provided out of poverty (nationally and for particular groups) and of whether people's chances of sustaining poverty free lives had been impacted (for particular groups) during the CtOG programme.

5.28 As with Option 1, this Option would not facilitate more than a cursory account of the extent to which CtOG was responsible for any change, as there is no means of factoring-in CtOG as an independent variable in the analysis of the data that would be used. Furthermore, although change can be estimated for Scotland and for sub-groups, there is insufficient data to undertake a quantitative analysis of change for geographical areas. Finally, although utilising longitudinal data to best effect at the point of evaluation, it must be acknowledged that the impacts of CtOG may not be discernible on the incidence of poverty until a later date (for example, for those interventions with children).

5.29 In addition to those competencies listed in 5.23, this Option would require the services of an analyst with the competence in handling complex longitudinal survey datasets.

'Poverty Measures' Option 3

5.30 Option 3 would supplement Option 2 by undertaking an additional field-based investigation of poverty dynamics with population groups who have been targeted by the CtOG work programme. The focus of this research would be to consider more directly the extent to which CtOG was responsible for any change in people's lives.

5.31 Option 3 could comprise quantitative and/or quantitative analysis, quantitative analysis being better placed to present an overview and qualitative analysis being better placed to explore processes. Although both approaches are possible, the aims of this field investigation may be better served by a more in-depth qualitative investigation with a carefully selected sample population.

5.32 Option 3 would comprise in-depth interviews with key informants. For the most part, these key informants would be individuals directly engaged through the CtOG work programme. In order to contextualise the interview findings, it is further recommended that a second set of interviews are undertaken with individuals who were not directly engaged through the CtOG work programme, but who are of a similar background to CtOG participants. Although there is no necessity to interview participants from the work programme associated with each CtOG Objective (i.e. the extent of poverty reduction can be measured from quantitative data sets), there may be merit in using the breadth of CtOG Objectives to explore the range of ways in which poverty can be prevented, left and left behind. It is recommended that 50 key informant interviews are undertaken, five for each CtOG Objective and that a further 30 informant interviews are taken as 'control interviews' (with people with similar characteristics as the larger sample, but who have not received the same levels of support), three for each CtOG Objective. Unlike Options 1 and 2, Option 3 would investigate directly the contribution of CtOG to the lives of people living in, or on the margins of poverty in Scotland

5.33 As with Options 1 and 2, although utilising both longitudinal quantitative data and qualitative data to best effect at the point of evaluation, it must be acknowledged that the impacts of CtOG may not be discernible on the incidence of poverty until a later date (for example, for those interventions with children).

5.34 In addition to those competencies listed in 5.23 and 5.30, this Option would require the services of an analyst with an awareness of CtOG and the array of public policy initiatives and other factors which may influence the outcomes desired of CtOG and competence in interview based research design and analysis.

EVALUATION OF CTOG OBJECTIVES

5.35 CtOG Objectives are more challenging to evaluate than CtOG Targets, and share with the overall aim of CtOG (as a poverty reduction strategy) the difficulties of specifying causality. An awareness of the wider context of factors which each CtOG Objective is therefore required, regardless of which evaluation Option is favoured. Furthermore, and unlike evaluating the overall aim of CtOG, there is the additional challenge of specifying the indicators to be used in the evaluation of each CtOG Objective.

5.36 It will be necessary to tailor evaluation design to attend to the particularities of each Objective. The CtOG Objective evaluation is presented here as a single undertaking with the advantages this affords for a CtOG 'whole programme' evaluation. There may however be merit in drawing upon the services of subject consultants to ensure that expert knowledge is brought to bear on the evaluation of each CtOG Objective.

5.37 Three options are presented below for consideration. These options deal with the broad evaluation design principles, as opposed to the particularities of the evaluation design for each CtOG Objective. Unlike the options for CtOG as a whole (which comprised quantitative, quantitative and quantitative/qualitative options), each option comprises a quantitative component (secondary data analysis) and two qualitative components (interviews with service providers and clients).

Objective Evaluation Option 1

5.38 Option 1, part 1 is to collate and interpret a digest of Scottish-level statistics for each CtOG Objective that approximate the extent to which the outcome in question has been achieved. The datasets that could be included in this analysis are listed in chapter 2.

5.39 Option 1, part 2 is to interview a small population of key informants for each CtOG Objective. These key informants will have been directly involved in a CtOG programme of work at delivery level. The interviews would be undertaken towards the end of CtOG's operation. The interview would be undertaken in 2 local areas that had been in receipt of CtOG funded work. It is anticipated that approximately 10 interviews would be undertaken (5 in each area).

5.40 Option 1, part 3 is to interview a small population of people who have received services provided under the CtOG work programme. These interviews would be undertaken in 2 local areas that had been in receipt of CtOG funded work. Ideally, these areas would have been the subjects of investment from more than one CtOG work programme. It is anticipated that approximately 20 interviews would be undertaken towards the end of the operation of the CtOG programme.

Objective Evaluation Option 2

5.41 Option 2, part 1 is to supplement Option 1, part 1 by disaggregating the Scottish-level statistics to inform understanding of the impacts of the Objectives for geographical, social and socio-geographical groups. The datasets that could be included in this analysis are listed in chapter 2.

5.42 Option 2, part 2 is to interview a wider range of key informants for each CtOG Objective. These key informants will have been directly involved in a CtOG programme of work, either at a delivery level. The interviews would be undertaken at two points during CtOG's operation. The interviews would be undertaken in five local areas that had been in receipt of CtOG funded work. It is anticipated that approximately 20 interviews would be undertaken (four in each area).

5.43 Option 2, part 3 is to supplement Option 1, part 3 by involving a wider range of Scottish citizens in the qualitative evaluation, which would allow consideration of differential experiences among geographical, social and socio-geographical groups. These interviews would be undertaken in five local areas that had been in receipt of CtOG funded work. As for Option 1, these areas would have been the subjects of investment from more than one CtOG work programme. It is anticipated that approximately 50 interviews would be undertaken towards the end of CtOG's operation.

Objective Evaluation Option 3

5.44 Option 3, part 1 is to supplement Option 2, part 1 by undertaking time-series analysis (wherever possible) to ascertain the rate of change in conjunction with the CtOG programme. The datasets that could be included in this analysis are listed in chapter 2.

5.45 Option 3, part 2 is to supplement Option 2, part 2 by interviewing Scottish citizens who have not been directly involved in the CtOG programme, i.e. to ascertain whether wider changes are effecting change for others who are, more broadly, within the CtOG target population. Interviews would be undertaken with an additional 20 respondents (to be contrasted with the 20 interviews with key informants from Option 2, part 2). A purposive sample is recommended, to ensure representation respondents from a range of geographical areas, and with varied personal characteristics.

5.46 Option 3, part 3 is to supplement Option 2, part 3. Interviews would be undertaken in ten local areas that had been in receipt of CtOG funded work. Each local area would be selected on the basis that it had been the primary location for a different CtOG work programme. It is anticipated that approximately 100 interviews would be undertaken by the end of CtOG's operation.

CTOG AS A PROCESS

5.47 CtOG is not merely a programme of work with a set of clearly specified outcomes. It is also a blueprint for a more participative and collective way of effecting social change (improvement) in Scotland. As such, it is recommended that the whole process of 'doing CtOG' be considered as an integral part of the evaluation.

5.48 Three options are presented below for consideration.

CtOG Process Evaluation Option 1

5.49 Option 1 involves mapping core CtOG collaborative activity, both across Scottish Executive Departments and between Scottish Executive Departments and partners out with the Scottish Executive.

5.50 This would be primarily a paper exercise, followed by no more than ten telephone interviews with key informants (likely to be the CtOG Target leads) to clarify inconsistencies or gaps in the available information. This would be undertaken at two points during the operation of the CtOG programme (with the first set of interviews informing the synthesis report).

CtOG Process Evaluation Option 2

5.51 Option 2 extends Option 1 by undertaking more intensive research to explore the extent to which attitudes and working practices have changed for staff at various levels within Scottish Executive Departments and partners out with the Scottish Executive. These interviews would be undertaken towards the end of CtOG's operation.

5.52 This option would also involve a financial breakdown what each CtOG Target funded.

5.53 A possible breakdown of the interview sample is provided below:

• Key strategic policy staff within the Scottish Executive or central government bodies (10 interviews)
• Strategic policy staff from the voluntary and community sector (15 interviews)

CtOG Process Evaluation Option 3

5.54 Option 3 extends Option 2 by engaging the opinions and experiences of interested parties and key players who operate out with the formal CtOG work programme. It is anticipated that this would add an additional 20 interviews to the evaluation. These key players would be identified in conjunction with the Scottish Executive.

5.55 In addition to the financial breakdown above, this option would involve analysis of whether the funding was used to support services that would have been developed in CtOG's absence.

SUMMARY

A series of options are presented for consideration for each dimension of the CtOG programme. This framework allows the Scottish Executive a high degree of flexibility in how they prioritise resources set aside for each aspect of the evaluation.

The options consider the following three elements: a global evaluation of poverty rates and dynamics in Scotland, an evaluation of change against the CtOG Objectives, and an evaluation of the impacts of CtOG as a process.

The global evaluation options range from a high level absolute poverty analysis to an in-depth analysis of poverty trends, risks and dynamics, for the population as a whole, particular groups and particular geographical areas, over CtOG's operation.

The evaluation options for the six CtOG Objectives all include both qualitative and quantitative analysis. The levels of quantitative analysis range in complexity and coverage, as does the volume, and area of consideration, of the proposed qualitative work.

The process evaluation allows for consideration of CtOG's effectiveness as a means of effecting social change in Scotland. A qualitative method is proposed, and the volume of fieldwork varies across the three options.

« Previous | Contents | Next »