« Previous | Contents | Next »
Listen
CHAPTER TWO: CHALLENGES IN EVALUATING CLOSING THE OPPORTUNITY GAP
OVERVIEW
2.01 An evaluation of CtOG raises several issues to consider. This Chapter describes the main challenges involved in handling and interpreting the data. Some of these are specific to the nature of CtOG and the timing of this interim evaluation while others are generic to evaluation methods, although particularly pertinent in this instance.
SPECIFIC ISSUES IN THE INTERIM EVALUATION
2.02 Firstly, the evaluation has been undertaken at a very early stage in the CtOG programme. A commitment to 'close the opportunity gap' was made in the Partnership Agreement presented by the Scottish Executive on May 14 th 2003, although this ambition was not operationalised until the launch of the CtOG Objectives on July 12 th 2004 and CtOG Targets on December 9 th 2004. The operational start date for Target H is even more recent - less than one year before the start of this evaluation. The limited period in which CtOG has operated must be acknowledged in any evaluation.
2.03 Secondly, this is an interim evaluation in another sense: most CtOG Targets are working toward an end date in 2008, although an interim progress date of 2007 was also set for Target A, and alternative end-dates have been set for evaluation in 2006 (Target C), 2007 (Target G) and 2010 (Target A). For the purposes of this interim evaluation, CtOG Aims and Objectives are assumed to have an end date of 2008. Given that this interim evaluation has been completed well in advance of most CtOG end dates, it is unrealistic to expect that any CtOG Target (other than Target C), Objective or Aim will have been achieved by the end of 2006 ( i.e. at the time of writing). Consequently, this interim evaluation is concerned with the direction of travel (is progress being made?) and the pace of change (is progress sufficient for the goal is to be achieved by the end date?). This latter judgement involves a qualitative interpretation of the quantitative evidence.
GENERIC ISSUES RELEVANT TO THE INTERIM CtOG EVALUATION
2.04 Several CtOG Targets involve agencies delivering a programme of work in partnership, and collecting data to facilitate monitoring and evaluation. In some cases, the Scottish Executive does not control either the nature of the data gathered nor the means by which this is done. For example, Target D involves Community Health Partnerships developing and monitoring their own local level Targets, while Target J involves Communities Scotland evaluating Regeneration Outcome Agreements ( ROAs). In some of these cases, data was not always available when required for this evaluation.
2.05 The timing of data release is a problem for evaluating CtOG. Measuring Objectives entails drawing upon a wide range of secondary data sources, and this is not all available at the same time. Consequently, the timeliness of data used in evaluation varies across CtOG Objectives. This problem is exacerbated for this interim evaluation as it is unlikely that data published early in 2006 (and usually referring to 2005 at the latest) will record changes brought about by CtOG activities launched in 2004. This problem is exacerbated by the time-lag between data collection and publication. This is most problematic for CtOG Objectives and Aims (both of which draw on secondary data from large scale national surveys), but also affects some CtOG Targets ( e.g. the time involved in processing data returns from the 22 Rural Service Priority Areas in Target H). It is therefore important to recognise that much of the interim evaluation is based upon secondary data collected in 2005, just after CtOG became operational.
GENERIC EVALUATION ISSUES
2.06 The final set of considerations is pertinent to any evaluation of CtOG. First, although Scotland's statistical data resources have improved significantly in recent years, disaggregation of findings is uneven across social groups. For example, it is much easier to analyse across age stages than sexuality. This presents difficulties in terms of measuring the impact of CtOG across Equalities groups. Wherever possible, this evaluation has disaggregated data to identify any differential impact of CtOG on Equalities groups, but where this is insufficient, supplementary insights into this issue have been gleaned from Equalities Impact Statements (prepared by CtOG Target Owners, see A1.28 to A1.42) and interviews with key informants representing Equalities groups (A1.43 to A1.64).
2.07 Secondly, there are variations in the populations and units of analysis specified for the CtOG Targets and Objectives. For example, Target J focuses on the 'most deprived neighbourhoods' (geographical); Target B focuses on 16-19 year olds who are NEET (socio-demographic and economic); Target A focuses on the workless in seven Local Authority areas (socio-geographic and economic) and Target C focuses on an institution ( NHS Scotland). Furthermore, several of the Target populations are defined by multiple characteristics, e.g. Target H focuses on remote and disadvantaged rural areas. Care is required to focus evaluation on the specified population for each CtOG Target and Objective.
2.08 Third, all CtOG Objectives have primary and secondary goals. For example, Objective 1 seeks 'to increase the chances of sustained employment for vulnerable and disadvantaged groups - in order to lift them permanently out of poverty' (emphasis added). Evaluating the success and ascertaining the impact of these CtOG Objectives involves not only a measure of primary impact but also the secondary impact of this primary goal. In this example, if CtOG led to increased chances of sustained employment for vulnerable and disadvantaged groups which did not 'lift them permanently out of poverty', the Objective would not be fulfilled 12. This interim evaluation focuses on the primary goal of each Objective, making reference to the likely impact of this primary goal on the secondary goal, where appropriate and possible.
2.09 Fourth, CtOG Objectives 3 - 6 have two such secondary goals. For example, Objective 4 seeks 'to regenerate the most disadvantaged neighborhoods - in order that people living there can take advantage of job opportunities and improve their quality of life' (emphasis added). Thus, evaluating the success of four CtOG Objectives involves ascertaining the secondary impact in two different domains.
2.10 Fifth, evaluation of CtOG Objectives requires extensive preparatory work in defining and specifying appropriate indicators. In this interim evaluation, the evaluation of each CtOG Objective in Chapter 4 includes a section ('Operationalising CtOG Objective: Towards Measurement) which explains how the Objective is interpreted ( Interpretation) and measured ( Indicators and Data).
2.11 Sixth, it is necessary to specify what constitutes progress for each CtOG Objective and the different components of the CtOG Aims. Although CtOG Targets are tightly defined, there is still latitude in defining progress for some of them. For example, Target E will have been met if an integrated system of support is 'in place', regardless of the outcomes that such as system may have. Similarly, Target J will be considered to have been partially met if Targets A, F and D are successful, despite the fact that these Targets do not all have a remit to consider the 15% most deprived areas in Scotland. In this interim evaluation, a clear statement is provided of what is considered to constitute progress for each CtOG Target, Objective and Aim.
2.12 Seventh, as noted above (para 1.15), any evaluation of CtOG must analyse the nature of the links between different levels of the strategy; i.e. relationships between CtOG Targets and Objectives. In addition to the alignment between CtOG Targets and Objectives, each Target is likely to have an impact beyond the Objective with which it is most closely associated. Such multi-dimensional complexity is also evident when gauging the impact of delivery plans and policies devised to address a particular CtOG Target. This is acknowledged by the Scottish Executive: in addition to specifying how CtOG Targets will be achieved and what resources are committed to achieve these ends, the on-line introduction to each CtOG Target includes a section entitled 'What additional supporting activities and Targets may contribute?' For example, to achieve Target J it is acknowledged that, 'Work to achieve the targets for employment, health and education (targets A, D and F) will focus on the most deprived neighbourhoods and improving access to relevant services' 13. This interim evaluation considers the overlapping relationships between different Targets and Objectives, and the degree to which the ten CtOG Targets are reflect the six broader CtOG Objectives.
2.13 Eighth, as was also noted above (para 1.16), CtOG does not operate in a vacuum; other Scottish Executive policies and other agencies are contributing to toward CtOG Targets, Objectives and Aims. Specifying the extent to which the CtOG work programme itself shapes particular outcomes is problematic. Furthermore, CtOG has been conceived as a collaborative effort harnessing the energies of a wide range of organisations led by the Scottish Executive. It could be therefore be argued that it is against the ethos of the strategy to be overly concerned with ascertaining the Scottish Executive's 'sole' responsibility for any particular outcome. This interim evaluation tries to identify whether CtOG coincides with any changes in trends or trajectories in quantitative data, but cannot precisely determine the extent to which CtOG is ultimately responsible for these changes. This interim evaluation seeks, where possible, to attribute any changes to sources; however, care is taken to acknowledge that the Scottish Executive may be the underlying force behind a partnership that is implementing change, and in turn, that change driven by an external agency should not be understood to be a failing of CtOG, but contributing towards the strategy, and evidence of the success in working in 'partnership for a better Scotland'.
2.14 Finally, despite what its name suggests, and unlike some of the SJS milestones which it superseded, the emphasis in the CtOG strategy is on improving the lives of the most disadvantaged, rather than explicitly 'closing the gap' between those who are more and less disadvantaged in Scotland 14. Although 'closing the gap' is not the primary focus of each CtOG Target, Objective and Aim, it is useful to consider this in order to facilitate an understanding of the role of CtOG in Scottish society, i.e. as a transformative programme (if gaps are being closed), as a conservative programme (if gaps are maintained), or as a limited programme (if gaps are widening). Indeed, the brief for this interim evaluation invited comment on the extent to which 'gaps have been closed'. Consequently, this interim evaluation is concerned with both absolute and relative progress for different social groups.
SUMMARY
2.15 With the formation of a new Scottish Executive following the elections of May 2007, it is an opportune moment to take stock of CtOG progress. The challenges faced in evaluating CtOG are significant, but not insurmountable. Given an awareness of the workings of CtOG, a commitment to appraise the wider policy context within which CtOG operates, and most importantly, careful data handling and interpretation, a robust approach to this interim evaluation of CtOG has been designed, as outlined in this Chapter.
« Previous | Contents | Next »