« Previous | Contents | Next »
Listen
6 EVALUATION AND LEARNING
Introduction
6.1 As noted in Chapter 2, the Cultural Pathfinder Programme was designed to generate useful learning on ways of engaging under-represented groups in cultural participation and about integrating culture within Community Planning. As such, the Programme placed a high priority on project evaluation and the Scottish Government provided guidance to Pathfinders on how to evaluate their projects via an Evaluation Toolkit.
6.2 All projects were required to submit an evaluation plan structured according to the guidance provided by the Evaluation Toolkit and were then required to submit interim and final reports detailing the outcomes of their projects.
6.3 As described earlier, the Toolkit suggested that projects identify a set of measures and indicators with which to measure performance, structured under five broad headings:
- activity;
- participation;
- satisfaction;
- impact; and
- value for money.
6.4 In addition, to provide a means of sharing the learning across the Pathfinder projects, the Scottish Government also supported the Pathfinder Learning Collaborative, a series of meetings bringing together the Pathfinder projects to share experiences and identify good practice.
6.5 In this section we review briefly the projects' approaches to evaluation and reflect on the effectiveness of the Evaluation Toolkit and the Learning Collaborative events.
Project evaluation
6.6 All of the projects submitted evaluation plans (with the exception of the Highland project which was not required to do so).
6.7 The evaluation plans and final reports were reviewed as part of the evaluation process, and the following issues are noted:
- there is considerable room for improvement in relation to the specification of measures and indicators in the evaluation plans. In particular, the following issues were evident in one or more plans:
- too many indicators (eg one project had 21 activity output indicators alone)
- indicators that lack meaning (eg "will discuss project plan with local and national partners");
- indicators specified without clear idea of how they would be measured (eg "cultural agencies will be more visible");
- indicators under the wrong headings (eg community empowerment is more an impact measure than one of satisfaction); and
- value for money was generally only measured by identifying the costs of different project elements.
- not all of the projects appear to have collected the required baseline data (as recommended in the Evaluation Toolkit and specified in many of the evaluation plans);
- few projects provided clear and robust data on the participation of the specific under-represented groups identified in their projects' applications;
- not all of the final reports provided an analysis of performance against the indicators identified in the evaluation plans;
- in a few cases, the evidence base for the findings of the evaluation reports was unclear, and some statements appeared to be based on anecdotal evidence; and
- a couple of the projects rejected the evaluation toolkit format altogether in favour of a more creative evaluation process.
6.8 This is not to say that all of the project evaluation work was poor - it was not, and there are some examples of good practice in this respect. The Fife report stands out as being particularly good, along with Perth and Kinross, whilst the Edinburgh project provided a clear account of performance against the evaluation plan indicators.
6.9 There was also positive feedback from most of the project managers about the usefulness of the Evaluation Toolkit. In particular, it was felt to have been helpful in providing a framework for project managers to think through the aims and objectives of their projects and how they would assess the extent to which they had been achieved.
6.10 However, the fact that some projects chose not to follow the evaluation guidance is a concern, and has made the overall programme evaluation more difficult in places.
6.11 Evaluation across the cultural sector has always been challenging, and there are well-known skills and knowledge gaps in the sector regarding the design and delivery of robust evaluation processes 15. The Evaluation Toolkit was provided, at least in part, to help address these gaps through practical advice and guidance, and for many this was effective.
6.12 Space does not permit a detailed critique of the Evaluation Toolkit here, but we would note that it is broad and in places quite ambitious, and by providing projects the freedom to identify indicators that fit their projects, the Programme has ended up without a consistent set of measures through which to evaluate its performance.
6.13 There is of course a tension here. The Programme was designed to support a range of different approaches, and the specification of a consistent set of measures for all projects may have significantly constrained the project development process. On the other hand, a consistent set of measures would have allowed not only an aggregated assessment of the performance of the projects, but also a comparative analysis across the Programme, although the specification of useful measures in this respect would be very challenging.
6.14 Unfortunately, there is no simple solution to this issue, although some consistent definitions might have helped to improve the quality of data provision across the projects eg how to measure participation in relation both to numbers of unique participants and participation by the same people over time. More consistent compliance with the evaluation processes required by the Programme is of course also an area for improvement.
6.15 Finally, while the Toolkit is strong on issues such as outputs and impacts, it is less explicit about assessing the processes through which these outputs and impacts are realised - often where the real value and learning is to be found.
Creative evaluation processes
6.16 As noted above, a couple of the projects ( SLP and Clackmannanshire) opted to undertake a creative evaluation process rather than follow the guidance provided by the Evaluation Toolkit. As such, the creative outputs of these projects constitute their evaluation.
6.17 In both cases, the creative outputs provided compelling and often moving accounts of the impacts of the projects on their participants, and are certainly very powerful tools for advocating and demonstrating the role of culture and creative processes in community consultation and well-being. In particular, these outputs can sometimes be more accessible to a wider range of audiences than lengthy reports. However, as evaluations they can lack transparency, objectivity and rigour, and often fail to provide a sufficiently clear account of how projects were managed and delivered, the processes by which impacts were achieved, and what the actual impacts were. They are also not always readily accessible (one was an exhibition in rural Clackmannanshire).
6.18 This is not to deny the value of creative evaluation. As stated, it is a powerful approach that can produce very compelling outputs capable of engaging different audiences, including policy makers. It can also be an empowering process for the participants themselves as they feel they are contributing to wider learning. However, it should be seen as additional to more formal evaluation processes rather than a replacement. Used in that context, alongside robust evidence, it can yield benefits.
Pathfinder Learning Collaborative Events
6.19 Most of the project managers interviewed felt that the Pathfinder Learning Collaborative Events had been useful, and that opportunities to share experiences and learning were always welcome.
6.20 However, a few offered some further comment, as follows:
- it was felt that the events tried to cover too much in a short space of time and, as a result, it was difficult to examine the different Pathfinder projects in depth;
- a few of the project mangers commented on the differing levels of (perceived) knowledge and experience across the Programme and felt that some of the content was either too 'basic' or too 'advanced' for parts of the audience; and
- a couple also commented that the projects were all so different that it was difficult to extract transferable lessons from the Collaborative Events.
6.21 It is also worth noting that the Scottish Government sought comment and input on the agendas and format of Collaborative events, and that project managers and partners were able to follow-up with contacts outwith the events.
6.22 There was also a view (from a few of the project managers) that the Pathfinder Programme has brought together a number of key individuals with substantial knowledge and experience of culture and its role in community well-being and capacity building. This was felt to constitute a significant opportunity to develop and share good practice. In particular, there was evidence of demand for some form of ongoing mechanisms to maintain this network and use it to inform and develop future activity and policy.
6.23 As a final comment, it is worth recognising the volume of information, data and knowledge that the Pathfinders have collected about what culture means to people across Scotland, and about the needs and aspirations of diverse communities regarding cultural provision. Many of the projects have also gathered useful evidence of the wider role and benefits of cultural participation which, when added to the already substantial literature on these issues, constitutes a valuable knowledge base and potentially useful advocacy tool.
6.24 It is beyond the scope of the current evaluation to review and compile all of that information and learning, but a number of the consultees felt that this would be a useful future exercise.
« Previous | Contents | Next »