« Previous | Contents | Next »
Listen
CHAPTER TWO: DATA SOURCES TO MONITOR AND EVALUATE CTOG TARGETS, OBJECTIVES AND AIMS
OVERVIEW
2.01 The mapping of national datasets sought to identify sources that could be utilised in the synthesis report. A digest of statistical resources available that could be deployed to address CtOG data monitoring requirements is presented in the Annex 3 of this report. In this chapter, key findings are presented to demonstrate how data availability informed the design of the synthesis report and the models for evaluating CtOG as a whole. As such, the chapter presents an evidence base for the recommendations set forth in chapters four and five.
CONTEXT
Data sources
2.02 There are a wealth of surveys that provide insight into conditions of life in Scotland. Most of these are UK national surveys, managed by the Government Statistical Service, in which the sample size is sufficient to facilitate comparisons between Scotland and other statistical regions in the UK. These have the status of national statistics (or official statistics).
2.03 Some national surveys facilitate more detailed sub-population analysis of conditions of life in Scotland. There are three ways in which this achieved. First, there are UK national surveys whose sampling frame is sufficient to facilitate such analysis, e.g. UK Census of Population. Second, there are UK national surveys whose proportionate population sample in Scotland is boosted to facilitate sub-national analysis, e.g. British Household Panel Survey. Finally, there are studies that focus on Scotland, e.g. the Scottish Household Survey.
2.04 The Scottish Parliament has assumed responsibility for research and statistics in relation to devolved matters. This involves decision-making regarding research management, and over data that are collected. Research is currently managed by Social Research at the Scottish Executive (formerly the Scottish Office). However, while the basic research infrastructure remains intact, new demands have been placed upon it.
2.05 Consequently, the Scottish Executive is a key producer and publisher of statistics on Scotland. The Scottish Executive's main vehicle for publishing statistics on Scotland is their series of statistical bulletins.
2.06 Thus, robust data to inform policy development in Scotland can be gleaned from the Scottish population of UK national statistics, Scottish national statistics (of the Scottish Executive, other government agencies and agencies outside government) and sub-national statistics.
Links with Regeneration Outcome Agreements ( ROAs)
2.07 Monitoring of progress towards CtOG Targets is intrinsically linked to a number of other policy developments in Scotland, both because of overlaps in the work programmes of CtOG and existing initiatives, and as monitoring data from existing programmes will prove useful for evaluating progress towards CtOG Targets, Objectives and overarching Aims.
2.08 The Local Government in Scotland Act, 2003 provided Community Planning with a statutory basis, and enabled the creation of Community Planning Partnerships. Community Planning partnerships bring together key participants, from each local area. They are intended to act as a 'bridge' to better link national and local priorities better 9. The Act places the following duties on:
- Local Authorities - to initiate, facilitate and maintain a Community Planning process;
- Core partners ( NHS Boards, Enterprise Networks, Police, Fire and Strathclyde Passenger Transport) - to participate in Community Planning; and
- Scottish Ministers - to promote and encourage Community Planning, including local participation by Communities Scotland in Community Planning Partnerships.
2.09 Other public bodies and voluntary organisations can be partners in Community Planning partnerships and have equal status with the statutory partners. The Act is not prescriptive about how Community Planning should be carried out, recognising that what is appropriate will depend on local circumstances. 10
2.10 In Summer 2004 the Scottish Executive announced the establishment of a new Community Regeneration Fund ( CRF) to bring improvements to Scotland's most deprived areas. The fund is particularly focussed upon the most deprived 15% of areas identified by the SIMD 2004. The primary focus of the CRF is to achieve CtOG Objective 4 (regeneration) 11. The CRF is not however the principal CtOG funding source.
2.11 The fund is to be administered, and the Objective achieved, through the actions of Community Planning Partnerships. A three year Regeneration Outcome Agreement ( ROA) provides the strategic and operational framework for their delivery. All 32 ROAs were approved by Scottish Ministers by September 2005.
2.12 As is discussed throughout this report, in practice CPPs have responsibility for delivery of multiple aspects of the CtOG programme, in addition to the central objective around which they are focussed.
MONITORING OF CTOG TARGETS
Baseline Data
2.13 This section of the report systematically reviews monitoring for CtOG Targets. All Targets owners are aware of the importance of developing effective monitoring systems, and all are progressing towards having such arrangements in place. There is however wide variation in the quality and quantity of baseline data available for each target, and some variation in the dates at which baselines were established. Variety in the available data sources also means that some monitoring arrangements provide more extensive analysis (in relation to both population groups and geography) than others. There are also disparities in the frequency at which monitoring data is collected and made publicly available.
2.14 The current CtOG monitoring data comes from a mix of administrative data, national survey data and independent monitoring.
2.15 Target delivery is dependent upon local level developments, often in a specific set of areas. Measuring change at local authority/health board level is therefore important to determine whether the overall national Targets are on progress and are eventually met.
2.16 A summary overview is provided in Table 2.1. Further information on ongoing monitoring is provided in Annex 2 of this report.
Table 2.1 Overview of Monitoring Data By CtOG Target
Target | Target end date and interim dates | Data and source | Regularity of returns | Date baseline established | Comments on quality |
|---|
A ( employment) | 2007 (December) and 2010 (December) | Quarterly claimant numbers, DWP. | Quarterly. | August 2004 for all authorities and May 2003 for Glasgow. | Strong means to measure progress against Target. |
B ( young people who are NEET) | 2008 (December) | Annual numbers of young people who are NEET, Labour Force Survey ( LFS). | Quarterly, with annual collation of results. | 2004 (1st year of Scottish boost to LFS). | Limited means to track change at a local level. Further developments involving Careers Scotland database may improve data quality. |
C ( jobs in NHS Scotland) | 2006 (31st March) | Monthly claimant numbers and job outcome rates, local Jobcentre Plus in 5 participating areas. | Monthly (final monitoring of sustained job outcomes will be available in July 2006). | N/a
Progress has been monitored in each local health authority since their respective start dates. | Strong means to measure progress against target, although further data could be useful as the initiative develops further. |
D ( cancer and CHD rates) | 2008 (31st March) | Annual mortality data supplied by GROS. Annual assessment of progress against Health Inequality targets in Health Boards Local Delivery Plans. | Annually. Data should be available from around August each year from 2006. | 2004 | All deaths from CHD and cancer are recorded by GROS. Various health surveys will record overall change as a consequence of the health improvement programme e.g. Scottish Health Survey, HEPS, SALSUS and the Scottish Household Survey. |
E ( integrated support for young people) | 2008 (31st March) | Children's services plans and joint inspection of children's services | N/a | N/a | All monitoring and baseline information is still in development. |
F ( tariff score of S4 pupils) | 2008 (31st March) | SQA attainment information (on pupils who take National Courses) and data from the school's census. | A calculation of the average tariff score of the lowest attaining 20% is made in April/May of each year. This is based upon the achievements of pupils beginning S4 in the September of two years previously. | End of September 2004 (this information was based upon the achievements of pupils in S4 in 2002/2003). | Strong means to measure progress against target. Baseline data includes both those pupils who do and do not achieve National Awards. The school census also provides additional information on pupil characteristics, which enables a more detailed analysis of the cohort. |
G ( looked after young people) | 2007 (31st March) | Annual assessment of how many young people leaving care remain in touch with their social workers, and survey data on an annual basis of how many young people are in education, employment or training. | Annually, in October of each year. | 31st March 2004. | Some limits to the available baseline, as data is not available for 23% of young people who ended their relationships with their social workers after leaving care. Pilot of individual data collection system currently underway. |
H ( rural areas) | 2008 (31st March) | Access and quality targets will be set for key services in each Rural Service Priority Area ( RSPA). These were to be agreed by January 2006. A baseline will be set in March 2006, with final analysis by March 2008. | To be decided January 2006. | As the key services are still to be agreed in consultation with rural stakeholders there is no baseline information.
Baseline data should be set by March 2006. | All monitoring and baseline information is still in development. Within a few months rural/urban analysis of the Households Below Average Income dataset will be possible, which will contribute to Target monitoring. |
J ( community regeneration) | 2008 (locally specific) | Monitoring data for Targets A, D, and F. For Targets A and F this data will relate to the 15% of most deprived areas (it is already provided in this format for Target D), not the 7 local authorities and lowest attaining 20% which are the respective focus of these Targets. There is also a slight difference in the way Target A is measured and the way employment deprivation is measured for Target J.
Locally specific monitoring of development and delivery of 32 3 year (2005-08) ROAs by Community Planning Partnerships ( CPPs), working with communities. A national framework specifies a range of expected measures.
Scottish Household Survey ( SHS) on neighbourhood satisfaction will be used to monitor environmental measures. Consideration is also being given to the use of other measures.
Consideration is being given to the use of the Ordnance Survey dataset (Points of Interest) as a means to measure access to local services. The Scottish Executive is also exploring if and how Local Environmental Audit and Management Systems ( LEAMS) could be used. | Will vary according to locality. | Baseline measures as for Targets A, D and F Locally agreed baselines for ROAs, dates locally specific.
Baseline from SHS for environmental measures. No baseline yet established for access to services. | A wide range of monitoring indicators are incorporated from other Targets, with data disaggregated to the 15% of most deprived areas with which Target J is concerned.
Communities Scotland is introducing a Performance Management Framework for ROAs and looking at how local data can be centrally collated and aggregated up. |
K ( financial inclusion) | 2008 (31st March) | Credit union membership, Financial Services Authority ( FSA). Data on numbers of money advisers, and evaluation data from money advice projects and financial education monitoring activities. | Credit union membership data is available annually. Money advice evaluation findings will report in Spring 2006 (interim) and 2007 (final report). Monitoring returns from financial education projects are available but not on a timetabled basis. | March 2003, Credit Union membership. | Comprehensive monitoring, although lack of appropriate baseline data sources means that progress against reducing debt levels in Scotland will be difficult to measure. |
DATA AVAILABILITY FOR CTOG OBJECTIVES
2.17 The six CtOG Objectives cut across the ten specific Targets. Three of the Objectives focus upon outcomes for particular groups of people, and three upon outcomes for entire populations, in particular geographical areas of Scotland. The broader nature of the Objectives means that they are best measured by monitoring national rather than local level change (amongst the specific groups and areas which are the focus of each Objective). UK, GB and Scottish survey data provide the best means to make this assessment. Below we provide a review of the best available sources for mapping progress against each Objective (in the synthesis report), and for evaluating their overall impact. We have indicated where we feel there is a need for additional survey data to be commissioned.
2.18 In this section of the report, we review the data sources that would provide contextual information to inform understanding of the CtOG Objectives, and summarise key data sources for both the synthesis report and the overall evaluation of CtOG.
2.19 A summary overview is provided in Table 2.2. Further information on data sources for evaluating the CtOG Objectives is provided in Annex 3 of this report.
Table 2.2 Overview of Data Availability By CtOG Objective
Objective | Data availability for synthesis report and evaluation | Need to expand upon existing data for synthesis report? | Need to expand upon existing data for evaluation? |
|---|
1 ( Employment) | DWP data on employment, earnings and claimant rates which can be disaggregated to a local level, including data zone level.
Increasing availability of longitudinal data to allow consideration of whether employment is sustained and the impacts upon the changes of people existing poverty. | No | No |
2 (Confidence and skills of young people) | Strong data on children and young people's achievements at school.
Detailed data on the initial destinations of most young people leaving school, but a lack of longitudinal data on future progress.
Limited data on children and young people's confidence. | No | Some question additions and a Scottish sample boost to the British Youth Panel survey may be useful. |
3 (Financial Exclusion) | There are good data to broadly assess the extent of financial exclusion in Scotland, but much poorer data with which to explore people's vulnerability to exclusion, and trends amongst particular groups. These issues will be addressed by the ONS Wealth and Indebtedness survey (currently in development). | No, but data will be limited as findings from the ONS Wealth and Indebtedness Survey will not yet be available. | The MORI financial services survey affords the opportunity of boosting the Scottish sample to provide more robust data in the shorter term. |
4 (Regeneration) | SIMD indicators and Scottish household data provide locally specific measures of quality of life.
Objective 1 and 5 data can be used to provide an assessment of employment opportunities and population health respectively. | No | No |
5 (Health) | The available data provides a strong basis for assessing changes in the health of people living in the most deprived communities in Scotland, with the SIMD and Scottish Household survey providing strong sources of relevant local level data.
High quality local data on quality of life and employment in deprived areas are also available as outlined under Objectives 1 and 4 | No | Some further exploration of the causal relationship between health outcomes and employment could be useful. Consideration could be given to placing questions related to the impact of health on employment in the Scottish Health Survey. Incapacity Benefit claim rates could also be considered. |
6 (Rural areas) | The available data provides a good means to assess access to services in rural areas, and an adequate means to assess service quality.
Measuring the outcomes of using services, specifically their impacts upon opportunity and quality of life, is more complex. Disaggregating SHS data could go some way towards this but would not provide an adequate assessment. | No | Consideration should be given to commissioning a survey of experiences of service users in rural areas, considering both their satisfaction with service provision and their views on the impacts of service use. |
DATA AVAILABILITY FOR CTOG AIMS
2.20 There are three overarching CtOG Aims: to prevent poverty, provide routes out of poverty, and to sustain poverty-free lives.
2.21 There is no single, accepted definition of poverty. Consequently, there are multiple views and continual debate on how to best measure changes in poverty rates and dynamics. In September 1999 the UK government published its first annual Opportunity for All report, setting out its strategy for tackling poverty and social exclusion and defining a number of indicators to measure poverty as a 'multi-dimensional phenomenon'. In Scotland, the Scottish Executive adopted a similar approach with the Social Justice Milestones.
2.22 In November of the same year that government's child poverty targets were announced: to eradicate child poverty by 2020, halve child poverty by 2010 and to reduce the number of children in poverty by at least a quarter by 2004/05. The interim target is being measured by looking at the number of children below 60% of median equivalised household income, as reported in the Households Below Average Income ( HBAI) analysis. This is also the measure used by the Scottish Executive to define absolute poverty.
2.23 The medium term and final targets are being measured by a 'tiered approach', with three separate measures: an income poverty measure based on a low income threshold fixed in real terms; a measure based on a relative low income threshold which rises if overall living standards increase; and a further measure combining both low income and material deprivation. Poverty will be regarded as falling when all three indicators are moving in the same direction 12. These indicators only apply to England, with the devolved administrations taking their responsibility for their own respective anti-poverty work. The Scottish Executive response to date has been the publication of the Social Justice Milestones, and more recently the Closing the Opportunity Gap strategy.
2.24 In the same way as Opportunity for All, this response acknowledges the importance of including relative poverty measures. The Scottish Executive has accepted the Townsend definition of deprivation as the starting point for all of its work towards the development of deprivation measures 13. This definition holds that:
"Poverty can be defined objectively and applied consistently only in terms of the concept of relative deprivation. [...] Individuals, families and groups in the population can be said to be in poverty when they lack the resources to obtain the type of diet, participate in the activities and have the living conditions and amenities which are customary, or at least widely encouraged or approved, in the societies to which they belong. Their resources are so seriously below those commanded by the average individual or family that they are, in effect, excluded from ordinary living patterns, customs or activities 14".
2.25 Evaluating the overarching CtOG Aims will therefore require analysis of shifts in both absolute and relative poverty rates and dynamics. Agreement of an appropriate means by which these measurements should be undertaken (likely to be in line with the agreed DWP measures outlined in 2.74) will be a vital formative stage of the CtOG evaluation.
2.26 Below we provide a review of the best available sources for mapping progress in preventing poverty, providing routes out of poverty and enabling people to sustain poverty free lives. We believe that these data can and should be included in both the synthesis report and the overall evaluation.
2.27 These three Aims cut across the CtOG Targets and Objectives. The nature of the Aims means that they are best measured by monitoring national change. However given the specific geographic and population focus of much of the CtOG work programme, ensuring that local level change and change amongst particular population groups of people, can be measured will also be key. UK, GB and Scottish survey data provide the best means to make this assessment. In addition, a number of existing analyses of poverty rates and dynamics in Scotland are already being undertaken, for example the Households Below Average Income ( HBAI) analysis and the JRF/ NPI bi-annual poverty assessment. Inclusion of these analyses will be an important means to add additional rigour to the assessment. Recent developments also mean within a few months rural/urban analysis of the HBAI dataset will be possible. The 'Scottish Executive Urban Rural Classification' also provides a means to categorise datazones as urban or rural, further enabling disaggregation of data by location type.
2.28 A summary overview is provided in Table 2.3. Further information on data sources for evaluating the CtOG Objectives is provided in Annex 4 of this report.
Table 2.3 Overview of Data Availability By CtOG Aims
Aim | Definition of success | Data | Need to expand upon existing data? |
|---|
1. Preventing poverty | By the end of CtOG, lower proportions of households considered to be 'at risk' of poverty are in poverty than was the case prior to CtOG. | Assessing whether those at risk of poverty have been prevented from entering it can be determined by longitudinal analysis of existing surveys. There are also a wide range of data sets of central relevance, including the Family Resources Survey and Millennium Cohort Survey. | No, but longitudinal analysis of existing data sets (disaggregated by population group) will be necessary.
Baseline analysis of the recent situation should also be undertaken. |
2. Providing routes out of poverty | By the end of CtOG, more households in poverty have been enabled to leave poverty than was the case prior to CtOG. | Assessing whether people in poverty have moved out of poverty can be determined by longitudinal analysis of existing surveys.
The SIMD indicators will also allow depth consideration of changes in poverty trends in local areas.
Similarly, disaggregating longitudinal cohort studies, such as the Scottish Household Survey, will allow for trends amongst people who were in poverty at the start of CtOG to be tracked over the course of the programme. | No, but disaggregating existing longitudinal datasets to map trends amongst people who were in poverty at the start of the CtOG programme will be necessary.
Baseline analysis of the recent situation should also be undertaken. |
3. Enabling people to sustain poverty free lives | By the end of CtOG, both those households which have been, and have not previously been, in poverty are less likely to enter poverty in the longer term than was the case prior to CtOG. | Assessing whether people are better able to sustain poverty free lives can be determined reference to longitudinal trends, which can be obtained via existing data sources including the Scottish Household Survey, the British Household Panel Survey and the Family Resources Survey. | No, but longitudinal analysis of existing data sets (disaggregated by previous experience of poverty and by population group) will be necessary. |
SUMMARY
A wealth of data exists in Scotland to inform life on social conditions of the Scottish people. These data have been appraised to ascertain their utility for the CtOG programme.
Strong data exists to inform monitoring for the majority of the ten CtOG Targets. There is variation in the frequency at which these data are available. Where the data are not of adequate quality action is being taken to address the deficiencies as best as is possible within the available timescales.
The work programmes for all CtOG Targets have raised a bulk of issues which are outside the scope of Target monitoring, but which should be considered in the overall CtOG evaluation. Many of these issues relate to working practices and capturing learning from variations in local level delivery.
Adequate data exists to inform evaluation of the six CtOG Objectives. For a minority of Objectives consideration should be given to boosting existing survey samples, placing specific questions in existing surveys or commissioning new survey work.
Establishing the extent to which the CtOG Objectives have achieved their specified ends (e.g. improving quality of life, lifting people out of poverty or enhancing access to opportunity) will be complex and will not always be possible without commissioning new work. Qualitative evidence (as recommended in Chapter 5 of this report) may prove to be a more effective way of exploring these possible links.
In line with current UK government and Scottish Executive policy developments, evaluation of CtOG's three overarching Aims should be done with reference to both absolute and relative measures of poverty.
Strong data exists to inform evaluation of the CtOG Aims. The available data allows for poverty rates and dynamics to be assessed at both a national, local and population level.
« Previous | Contents | Next »