« Previous | Contents | Next »
Listen
ANNEX ONE: RESEARCH DESIGN
Overview
A1.01 The research on which this Phase 1 evaluation was based comprised three stages of activity and five programmes of work, each of which is detailed and appraised in this Annex.
- Phase 1 comprised analysis of available data sources to examine the different aspects of CtOG:
- Phase 1, Part 1 comprised an evaluation of CtOG Aims
- Phase 1, Part 2 comprised an evaluation of CtOG Targets
- Phase 1, Part 3 comprised an appraisal of Equalities Group issue
- Phase 2 comprised an evaluation of CtOG Objectives.
- Phase 3 comprised a summary review to ascertain the overall impact of CtOG, with a subsidiary focus on the links between CtOG Targets, Aims and Objectives
A1.02 Throughout the evaluation, care was taken to contextualise analytical comment with reference to the wider policy arena and to the wider programme of CtOG activity. The evaluation was informed by an earlier Scoping Study1 which appraised the analytical technicalities, and subsequent consideration of the challenges faced in undertaking an interim evaluation of CtOG at an point in the strategy (late 2006 - early 2007).
A1.03 A standardised four-part analytical framework was adopted to evaluate progress towards CtOG Aims:
- Headline trend analysis - to ascertain the extent to which progress is being made toward achieving CtOG Targets, Objectives and Aims
- Trend composition analysis - to ascertain whether quantitative changes to the headline CtOG Targets, Objectives and Aims are experienced evenly across sub-populations. The approach was to undertake analysis for Equalities groups, and also at 'one level' below the headline target; e.g. for Target A, to compare change for the targeted local authorities of Glasgow, North Lanarkshire, South Lanarkshire, Renfrewshire, Inverclyde, Dundee and West Dunbartonshire
- Trend divergence - to ascertain whether changes to the headline targets in Scotland were also experienced in other parts of Great Britain
- Qualitative interpretation - to complement the quantitative data analysis, by interpreting data changes in light of the broader context of policy interventions and conditions in wider society.
A1.04 This review considers each work programme in turn.
Evaluation of CtOG aims
Objective
A1.05 The overarching Aims of CtOG are:
- To prevent individuals or families from falling into poverty
- To provide routes out of poverty for individuals and families; and
- To sustain individuals or families in a lifestyle free from poverty
Approach
A1.06 In light of the tender specification and time frame, a Scottish-level analysis of poverty incidence was undertaken, using accepted definitions of low income and poverty, drawn from UK-wide data, disaggregating the analysis wherever possible into geographical, social and socio-geographical groups.
A1.07 The evaluation of CtOG Aims is premised on being able to measure poverty. However, there is no single, accepted definition of poverty in Scotland. Nevertheless, the medium-term commitment of the UK government to eradicate child poverty within a generation, has led to the specification of a broad-based "tiered" approach to measuring child poverty using a combination of Households Below Average Income ( HBAI) data and deprivation indices. This comprises:
i. an income poverty measure based on a low income threshold fixed in real terms (below 60% of median equivalised household income)
ii. a measure based on a relative low income threshold which rises if overall living standards increase (below 60% of median equivalised household income)
iii. a further measure combining both low income (below 70% of median equivalised household income) and material deprivation
These data can be used to measure the incidence of poverty for individuals and families in Scotland.
A1.08 The approach adopted in this interim evaluation was to undertake a quantitative analysis of Households Below Average Income ( HBAI) data, informed by a 'wider' analysis of poverty (and deprivation), using a wider range of data sources.
This afforded the opportunity to measure:
- The number and proportion of people in Scotland who are experiencing income poverty at a point in time ( Headline measure)
- Aggregate trends in the number and proportion of people in Scotland experiencing income poverty ( Headline trend analysis)
- Variations in current levels and recent trends by age-stage, ethnicity and disability ( Trend composition analysis)
- Current levels and recent trends in Scotland against GORs in England, and aggregates for England and Wales ( Trend divergence)
A1.09 Primary data were drawn from the current volume of the Department for Work and Pensions' Households Below Average Income data series 2 and the Scottish Executive's Scottish-level analysis of these data 3. Supplementary data - to provide a broader analysis of poverty and deprivation in Scotland - were drawn from a wider range of sources, comprising:
- Scottish Household Survey
- Scottish Index of Multiple Deprivation
- Welfare benefits statistics
A1.10 This approach entailed certain limitations:
- HBAI comprises cross-sectional data, which unlike longitudinal data does not provide the opportunity to track changes in individuals' and families' circumstance, which is desirable for a detailed analysis of CtOG's impact on poverty dynamics
- the small sample size in Scotland from which HBAI data is drawn limits opportunities for compositional analysis
- data from the Deprivation Indices were not available when the analysis was undertaken
A1.11 It is important not to overstate the impact of CtOG on the current incidence of poverty in Scotland as the strategy has not had sufficient time to make any significant impact. Nevertheless, this approach presented an analysis of the current incidence of poverty in Scotland, and changes in poverty levels in Scotland, which occurred during the initial years of the CtOG programme. The interpretation of cross-sectional aggregate statistics provided an indication of poverty risk in Scotland (for people and places).
A1.12 In summary, the focus of the evaluation of CtOG Aims in this interim report is to interpret the broad trends in the incidence of poverty in Scotland to provide benchmarks against which the impact of CtOG might be measured when the initial stage of CtOG has been concluded in 2008.
Evaluation of CtOG Targets
Objective
A1.13 The interim evaluation was concerned to assess the extent to which progress was being to achieved for each of the ten CtOG Targets.
Approach
A1.14 A lead analyst was designated from SPIU/Inclusion to evaluate each CtOG Target. However, there was also a need to facilitate cross-cutting analysis to ensure that adequate consideration was given to the complexity of relationships between CtOG Targets and CtOG Objectives.
A1.15 The approach adopted was to interpret data routinely collected and processed by Scottish Executive analysts and Target Owners to monitor progress toward achieving their CtOG Targets.
A1.16 A standardised template was used to ensure consistent evaluation across CtOG Targets. This template is used to structure the analysis in this report.
A1.17 CtOG Target monitoring data provide a headline measure for this point in time and facilitated headline trend analysis where these data were available for earlier periods. This analysis was supplemented to provide a more detailed evaluation of CtOG Targets: variations in current levels and recent trends by sub-populations were undertaken, and SPIU/Inclusion requested further specific data from the Scottish Executive to undertake sub-population analysis, in accordance with the objective outlined in A1.04 ( Trend composition analysis)
Equalities Group Issues
Objective
A1.18 The interim evaluation was concerned to assess the extent to which CtOG was contributing toward the wider Equalities agenda in Scotland. Wherever possible, source data was disaggregated to appraise experiences among Equalities groups, and differences in rates of progress between groups, as well as rates of progress within them, were considered. There was sufficient quantitative information available to undertake disaggregated analysis by gender and age groups.
A1.19 However, not all CtOG Target monitoring data can be disaggregated by the equalities groups. It was necessary to undertake further research to compensate for the inability to disaggregate CtOG monitoring data for four of the six Equalities groups, i.e. by sexuality, ethnicity, faith group and disability status.
Overview of Approach
A1.20 The approach adopted comprised three parts:
- Desktop data analysis: where possible, CtOG Data were disaggregated by Equalities groups
- Interviews with CtOG Target Owners: the broader context of CtOG and the Equalities agenda were explored in these interviews
- Interviews with key informants from Equalities Groups: CtOG and the specific context of the Equalities population represented by the organisation from which the key informant was drawn were explored in these interviews.
Desktop Data Analysis
A1.21 Wherever CtOG data could be disaggregated by Equalities group, SPIU/Inclusion undertook a compositional analysis for Equalities groups. In practice, these analyses were largely only available by age and gender.
Interviews with CtOG Target Owners
Objective
A1.22 The primary purpose in interviewing CtOG Target Owners was to explore measurement issues pertaining to Equalities groups. Interviews with CtOG Target Owners focused specifically on the Target for which they were responsible.
Approach and Design
A1.23 SPIU/Inclusion were provided with a copy of the Scottish Executive's Equalities Impact Statement ( EIS) for each CtOG Target. EISs are systematic appraisals of the impact of Scottish Executive policy and interventions on each Equalities group. In addition to providing key background information on CtOG Targets, the EISs for CtOG Targets also provide information on information sources, consultations undertaken, evidence of differential impact on particular groups and the perceived impact of the policy. EISs were reviewed in advance of the interview with CtOG Target Owners. The topic guide for the CtOG Target Owners interviews is presented in Annex Two.
A1.24 The topics and questions for the interviews with CtOG Target Owners were informed by an appraisal of the Equalities Impact Statement and scrutiny of the arrangements for monitoring and measuring progress (information gleaned from the CtOG website).
Field Practice and Ethics
A1.25 Interviews were conducted in person and typically lasted 30 minutes (ranging from 20 minutes to 47 minutes). Interviews were conducted in accordance with the ethical guidelines of the Social Research Association. Participants were assured that direct quotations would not be used without their explicit consent and that the research team would treat their contributions respectfully. Participation was based on informed consent from each participant. Interviews were tape-recorded with the consent of the interviewee and were preceded by a brief summary of the scope of the research and the interview (Annex Two). Interviews took place in the offices of the Scottish Executive.
A1.26 At the end of each interview (and where time permitted) the interviewer presented an oral summary of the key findings to emerge over the course of the interview. Participants were given the opportunity to reflect on this summary. In practice, participants confirmed the interviewers' summary.
Analytical Approach
A1.27 All interviews were transcribed in full by professional clerical assistants; these transcripts were checked by the interviewer. Interviewers systematically analysed every transcript in accordance with a framework agreed among the SPIU/Inclusion team to provide consistency of focus across interviews.
A1.28 Interviews with CtOG Target Owners were focused on issues pertaining to CtOG Targets, as opposed to CtOG Aims and Objectives. Interviewers provided the SPIU/Inclusion analyst leading the evaluation of each CtOG Target with a digest of the key points Findings from these interviews inform the analysis reported in chapter five (Targets) of the main report.
Interviews with key informants from Equalities Groups
Objective
A1.29 The broad purpose of these interviews was to gather information to assist analysis of the outcomes of CtOG for people of different ethnicity, faith, disability status and sexuality. In particular, the interviews were intended to investigate whether progress made towards the CtOG Objectives had more impact on some groups than others, and to ascertain whether there were specific equality considerations that remained untouched by the CtOG Programme.
A1.30 The specific reasons for interviewing key informants from Equalities Groups were four-fold (see Annex 3), i.e. to investigate:
- evidence and perceptions of current circumstance in relation to CtOG Aims, Objectives and Targets.
- evidence and perceptions of particular experiences in relation to CtOG Aims, Objectives and Targets, pertaining to the Equalities Group
- experience and awareness of CtOG
- measurement issues
Selection of Method
A1.31 At the outset, it was envisaged that a combination of key informant interviews (four-eight in total, one or two from each Equalities group) and focus groups (four, one from each Equalities group) would be undertaken. However, on reflection, it was considered that focus groups would be problematic and should be replaced with one-to-one interviews with key stakeholders. Three main reasons accounted for the change in approach:
i. Initial contacts with equalities organisations indicated a low level of awareness of CtOG, which it was surmised was not likely to be any higher among the wider population of Equalities groups.
ii. It was considered that the internal diversity of equality groups potentially invalidated the focus group method
iii. For this interim evaluation, it was considered more productive to explore measurement issues and the way in which Equalities group issues are embedded in CtOG, as opposed to exploring client group's experiences of CtOG
Sample Selection
A1.32 For each of the four Equalities groups targeted, SPIU/Inclusion prepared a long-list of organisations which represented a cross-section of the population and of individuals within these organisations who could be interviewed. This long-list of organisations was discussed with the Advisory Group from the Scottish Executive and, following these discussions, a short-list of 21 Equalities groups were identified for interview.
A1.33 The agreed short-list comprised the following groups:
(i) Black and Minority Ethnic organisations:
- Commission for Racial Equality
- Glasgow Anti Racist Alliance
- Central Scotland Racial Equality Council
- LINKnet Mentoring
- Chinese Community Development Partnership
- Black and Minority Ethnic Infrastructure in Scotland ( BEMIS)
- Grampian Racial Equality Council
- Black Community Development Project
- Council of Ethnic Minority Voluntary Sector Organisations
- Scottish Council for Minorities
(ii) Faith organisations:
- The Scottish Interfaith Council
- Taleem Trust
(iii) Disability organisations:
- Inclusion Scotland
- Capability Scotland
- The Scottish Disability Equality Forum
- Enable
- The Scottish Consortium for Learning Disability
(iv) LGBT organisations:
- LGBT Youth Scotland
- Equality Network
- Stonewall Scotland
- Glasgow LGBT Youth Centre
A1.34 Representatives from 14 of these 21 organisations agreed to participate in the research.
Approach and Design
A1.35 As in the case of CtOG Target Owner interviews, the topics and questions for the interviews with key informants from Equalities Groups were informed by an appraisal of the Equalities Impact Statement and scrutiny of the arrangements for monitoring and measuring progress of CtOG Targets (information gleaned from the CtOG website). As was also the case in the CtOG Target Owner interviews, the topic guides for in interviews with equalities group representatives were used flexibly to include any key issues that emerged in the course of the interviews, and to ensure attention was given to relevant issues that were more pertinent to the Equalities population in question.
Research Practice and Ethics
A1.36 Most interviews were conducted by telephone, but several undertaken face-to-face. Typically they lasted 30 minutes (ranging from 27 minutes to 45 minutes). The same ethical and informed consent procedures were followed as in the case for interviews with CtOG Target Owners.
Analytical Approach
A1.37 As for CtOG Target Owner interviews, all interviews were transcribed in full by professional clerical assistants; these transcripts were checked by the interviewer. Interviewers systematically analysed every transcript in accordance with a framework agreed among the SPIU/Inclusion team to provide consistency of focus across interviews.
A1.38 The interviews with key informants from Equalities organisations had a broad focus, and covered CtOG Aims and Objectives more than specific CtOG Targets. Findings are primarily reported in chapters three (Aims) and four (Objectives) of this report.
Evaluation of CtOG Objectives
Objective
A1.39 The interim evaluation was concerned to assess the extent to which sufficient progress was being to achieving each of the six CtOG Objectives.
Approach
A1.40 At the outset, the research team reviewed each of the six CtOG Objectives and undertook a two-stage review:
- Clarification of the Objective: each CtOG Objective was subjected to a critical review to ascertain its precise focus.
- Identification of Indicator Set for Monitoring: having clarified the precise focus of each CtOG Objective, the SPIU/Inclusion review team proposed an indicator set which collectively would allow for monitoring.
A1.41 The suitability of datasets was determined according to the following five factors:
- Relevance to Objective
- Availability of data for inclusion in the January 2007 progress report
- Geographical coverage
- Coverage of equalities groups
- Frequency of data collection
A1.42 As anticipated at the outset, there was considerable disparity in the data which could be used to inform appraisal of the Objectives. For example, for Objective 1 there was extensive recent data available, which could be disaggregated to local area level, allowing for meaningful comparison between CtOG and non- CtOG areas. In contrast, for Objective 5 there was limited data available meaning that the assessment is more limited in its scope.
A1.43 SPIU/Inclusion afforded the Scottish Executive the opportunity to comment on the indicator set proposed for Objective 1.
Ascertaining the Overall Impact of CtOG
Objective
A1.44 The interim evaluation was concerned to assess the extent to progress in CtOG Targets, CtOG Objectives and in achieving CtOG Aims, could be directly attributed to CtOG activity. More generally, the interim evaluation was also concerned to ascertain the nature of the links between CtOG Targets, CtOG Objectives and CtOG Aims. The concern was to evaluate the extent to which trends reflect the interventions encouraged by CtOG through CtOG Targets and, more generally, by the wider body of work that is contributing to CtOG Objectives
Approach
A1.45 Care and caution are required when attempting to specify the impact of CtOG. SPIU/Inclusion reviewed the factors which would compromise any attempt to specify the impact of CtOG. SPIU/Inclusion concluded that, for this interim evaluation, it would be misguided to seek definitive conclusions specifying precisely the extent to which CtOG alone was responsible for effecting change in CtOG Targets, and more particularly, CtOG Objectives and CtOG Aims. Rather, a three-part analytical framework was proposed to offer informed commentary on the likelihood of changes in Scotland being attributable to CtOG, comprising:
- Headline trend analysis.
- Trend composition analysis
- Qualitative interpretation.
This involved a descriptive summary of non- CtOG work and extraneous factors that impinge upon poverty in Scotland, and a critical commentary on the extent to which reductions in poverty rates can be attributed to the CtOG work programme.
« Previous | Contents | Next »