« Previous | Contents | Next »
Listen
1 INTRODUCTION
Background
1.1 In 2006, the then Scottish Executive (now Scottish Government), developed the Cultural Pathfinder Programme as a way of exploring effective and practical ways to widen access to, and participation in, cultural activity. The Programme supported 13 pilot projects across Scotland, 12 of which were local authority-led, with a total budget of nearly £650,000 over two years (2006-2008).
1.2 One of the key objectives of the Programme was to extract maximum learning, knowledge and examples of good practice. In order to consolidate the learning from the Programme and gain an independent view on the performance of the Pathfinder projects, the Scottish Government commissioned EKOS to conduct an external evaluation of the Programme. This study is particularly timely, as cultural policy in Scotland continues to develop, and delivery organisations consider how best to deliver effective cultural provision at national and local levels.
Study aims
1.3 The overall aim of the study was to provide an independent evaluation of the Cultural Pathfinder Programme as a whole, identifying key lessons and making recommendations for sharing the learning arising from the evaluation process. The more detailed objectives were to:
- conduct a case study analysis of each project;
- examine the processes supporting the Programme;
- analyse the final outputs and outcomes of the Programme;
- report on the overall outcomes of the Programme, highlighting general lessons and examples of good practice that could be shared across stakeholders;
- assess the additionality of the Programme; and
- review and provide a view on legacy aspects of the Programme, including sustainability and developing the interface between cultural provision and Community Planning processes.
Study method
1.4 The evaluation was designed to focus on the strategic assessment of the Programme as a whole, with detailed project evaluation the responsibility of the individual projects. However, the requirements of the study brief and the diversity of the projects supported necessitated a case study approach.
1.5 The main elements of the process were:
- desk-based reviewand analysis of the evaluation plans, and interim and final evaluation reports provided by individual projects in order to establish project aims and objectives and examine reported performance against these aims;
- interim report summarising the Programme structure, aims and objectives, funding and the individual projects supported;
- depth interviews with Pathfinder project managers, local authority and Community Planning contacts involved in projects and national stakeholders to understand the Pathfinder projects in more detail and explore their impacts;
- case study analysis of projects drawing together the findings of the desk review and interview programme to provide a project-by-project account of the Programme and its impacts; and
- final review and analysis in which the findings were collated and fed back to Pathfinder project managers at a workshop and through the circulation of individual project case studies for comment, before production of the study report and final case studies.
Desk review and analysis
1.6 The desk review work involved the review and analysis of Programme and project documentation at a number of levels, including:
- documentation relating to the development of the Programme, including the original letters inviting applications and setting out the criteria for funding awards;
- strategic documentation at local level (eg local cultural strategies and plans) and other supporting materials provided by project managers;
- project applications and the assessment forms completed by the responsible Scottish Executive (now Scottish Government) team to score each application;
- evaluation plans submitted by each project (with the exception of the Highland Pathfinder 2) detailing the aims and objectives of the projects and the measures and indicators that would be used to assess performance; and
- interim and final project reports submitted to the Scottish Government providing an account of the performance of individual projects against the measures identified in their evaluation plans.
1.7 The desk review work was supported by informal telephone discussions with project managers, and ran throughout the evaluation period, with the final analysis focussing mainly on the final reports submitted by the projects.
Depth interviews
1.8 In addition to the desk review, the consultant team undertook an extensive programme of depth interviews with project managers and key partners and stakeholders involved in Pathfinder projects or with the Programme as a whole.
1.9 In total, 62 one-to-one interviews were conducted from an initial target of 55. Of these, 45 were conducted face-to-face, and the remainder by telephone. A number of project managers and key contacts were also interviewed more than once due either to the timing of projects (initial interview was at too early a stage in the project) or if there was a need for further clarification and input.
1.10 The interviews gathered detailed feedback on the activities undertaken in each project, the processes that supported delivery, the kinds of partnerships developed and the impacts of individual projects as well as thoughts about the value of the Programme as a whole. The interview questions can be found in Appendix 2.
1.11 The interviews were conducted over a five month period from June to November 2008.
Case study analysis
1.12 The data and feedback gathered through the desk review work and the depth interviews were collated and analysed to produce 13 case study analyses of the funded Pathfinder projects (provided in Appendix 1). While the case studies drew on the information supplied by projects in their evaluation reports, they did not seek to replicate that information.
1.13 The case studies followed a consistent structure, and provided an account of:
- the aims and objectives of each Pathfinder project;
- funding secured;
- fit with national and local strategic guidance and priorities;
- activities undertaken and processes supporting the delivery of projects;
- performance of the projects against the measures and indicators identified in their evaluation plans, in five key categories -
- activities
- participation
- satisfaction
- impacts
- value for money; and
- projects' contribution to the strategic objectives of the Programme, in terms of the extent to which they successfully -
- engaged under-represented groups in arts/cultural activities
- encouraged collaborative working within and/or across authorities
- engaged Community Planning Partnerships and other partners/stakeholders
- delivered effective consultation with communities
- delivered something new and additional
- considered and ensured the sustainability of project impacts.
1.14 Although the case studies provide an account of performance against the measures identified in project evaluation plans, detailed project evaluation was the responsibility of the funded projects. The focus of this evaluation is more strategic, and is aimed at summarising the achievements of the Programme as a whole and identifying the key lessons for future policy and delivery.
Final review and analysis
1.15 In the final stages of the evaluation process, the study team collated and reviewed the feedback from the consultations and held a learning workshop to review the emerging findings and discuss the main lessons from the evaluation process.
1.16 The workshop was held on 02 December 2008 in Perth and all Pathfinder project managers and Community Planning contacts were invited to attend. In total, six project managers and one Community Planning contact attended. The outputs of the workshop discussions have influenced the conclusions and recommendations of the report.
Study issues
1.17 The evaluation process followed a method and structure agreed with the Research Advisory Group at the outset and few major issues or difficulties were encountered. However, a couple of issues are worth brief mention.
1.18 First, the evaluation was required to work with material and data supplied by projects via the interim and final reports and in line with the format and content of the submitted evaluation plans. A number of the final reports were received by the Scottish Government later than expected (often for valid reasons), creating some delays and putting pressure on the final analysis for this evaluation.
1.19 However, more importantly, the extent to which the reports followed the structure of the submitted evaluation plans varied, and this inconsistency has created issues for the analysis process. In addition, the evaluation plans all specified quite different indicators in each of the categories identified above. As a result of these inconsistencies in approach, simple aggregation of the performance reported across the projects was not possible, and we have offered further comment on these issues later in the report.
Structure of report
1.20 The report details the findings of the evaluation process described above, and is structured as follows:
- Chapter 2 provides an overview of the Cultural Pathfinder Programme;
- Chapter 3 briefly summarises the aims and objectives of the Pathfinder projects and the broad approaches taken;
- Chapter 4 details the achievements and impacts of the Pathfinders along four key dimensions:
- participation and engagement
- cross-service and collaborative working
- interface with Community Planning
- impacts on the cultural sector.
- Chapter 5 assesses the additionality, sustainability and legacy impacts of the Programme;
- Chapter 6 reviews the use of evaluation by the projects and offers comment on the Scottish Government's support in this area;
- Chapter 7 provides feedback on the overall management of the Programme; and
- Chapter 8 presents our conclusions, together with a brief discussion of some of the wider issues and implications of the evaluation findings and key lessons for future delivery of culture.
« Previous | Contents | Next »