The Scottish Government

« Previous | Contents | Next »

Listen

Section 9 Best value

9.1 Our remit

The Commission was required to:

"test the ability of existing structures to evolve and change"

"have regard to the realities of existing, and likely future, funding levels."

The Commission was also asked to:

"provide guidance and advice on initiatives which affect the wider objectives of Scottish government"

"consider how to achieve best value from existing resources and how to use public spend to lever growth in the cultural and creative industries." ¹

9.2 Our understanding

The Commission recognises the importance of best value practices and supports drives towards greater efficiency in order to release more resources for the delivery of cultural services. We are aware that much is already being done by NDPBs and local government in this area, and we recognise that there may be opportunities for further work if bodies begin to form strategic alliances or share back offices and collaborate more comprehensively than they currently do in areas of mutual interest, such as education.

It is impossible to consider better use of resources without consideration of the current impact of cultural activity. Considering the current levels of funding, the high quality and range of cultural activity in Scotland is considerable. Evaluation of the impacts of cultural activity is currently a hot topic of debate within the sector, because of the difficulties of measuring and quantifying many of the impacts. Whilst there is much research being undertaken in this area, and it is hoped that developments in quality of life and public value will strengthen the position of cultural bodies, it may be that such measures or indicators may take some time to develop. The process of rights and entitlements provides a starting point for the establishment of a framework of nationwide standards which should allow benchmarking to encourage providers to aim for above average results.

Better use of public resources has been a key factor in the development of infrastructure reforms, outlined in Section 10, and it is important that the information which follows should be considered in this context.

9.3 Consultation

Very few submissions in the first phase of our consultation and meetings raised Best Value or the areas of making better use of public investment, measuring performance and benchmarking as priority issues. In our second phase of consultation, we asked the cultural sectors and our stakeholders to provide specific ideas and suggestions on how proper standards of outcome, output and process could be established. Again, very few non public sector bodies or agencies chose to tackle this issue. From those who did respond, however, mainly local authorities, the consensus was that the subject was a challenging one.

"The cultural environment is not one which sits easily alongside targets, attainment outcomes and boxes to be ticked. While standards need to be in place, there needs to be sufficient flexibility to avoid stifling artistic expression."
Argyll and Bute Council Phase 2 submission

All respondents made reference to recent internal initiatives to develop and roll-out departmental targets and some included details of schemes that had been introduced for grant recipient organisations. Concerns were expressed by many respondents, and in particular by the cultural sectors, about the difficulty of evaluating cultural activity because of its very subjective nature.

On the issue of best value, it was clear from the consultation that the sector delivers a great deal for relatively small investments, but there was evidence, despite great improvements in this area, that the level of understanding of audience needs and requirements requires improvement. Whilst there was evidence of many initiatives to improve efficiency within organisations and some partnerships, consultation responses on the whole reflected a sector that was divided and one that had to compete internally in order to survive.

9.3.1 Current thinking in the public sector

We have examined in detail various guidance documents and research relating to efficient use of public resources. Any reform to the cultural sector needs to take into account general reforms and trends within the public sector in Scotland and in the UK. Following the Gershon Review, ² four major principles have been identified for efficiency within the public sector:

• investment in information and communications technology

• shared back-office functions

• e-enabled service provision

• the use of efficiency tools.

The applicability of these to the cultural sector is variable and, while most may help improve the efficiency of the administration of the sector, they cannot be expected to deliver savings to the cost of artistic practice. Considering the relatively small public subsidy to cultural services, and the massive impact which Scotland's artists and creative workers have, both within Scotland and internationally, it may already meet and exceed the current efficiency criteria. However, we believe that the cultural sector in Scotland should agree the importance of making savings on administrative functions and, in return, government should guarantee that any savings will be made available for the provision of core activities.

From our submissions and from wider discussion with stakeholders, there are signs within the public sector that focus is moving to a more citizen-based approach to the provision of public services, aligned in our case with the concept of cultural rights. This will require a fundamental change in work ethic and the infrastructure of the public sector and is part of medium-to-long-term priorities for government. The drive for more efficiency is very much a current topic, with job losses expected in many part of the public sector in the UK and with a virtual freeze on Civil Servant recruitment in Scotland.

Both of these trends are explicit in material emanating from the Cabinet Office, and are implicit in the Scottish Executive's Guidance to Accountable Officers on Best Value:³

• make arrangements to secure continuous improvement in performance (while maintaining an appropriate balance between quality and cost); and in making those arrangements and securing that balance

• have regard to economy, efficiency, effectiveness, the equal opportunities requirements and to contribute to the achievement of sustainable development.

9.3.2 Measuring success

Another key trend revealed in our consultation is the likelihood of an important value shift in the way in which we regard success in terms of public sector engagement. This is set to change as we move towards citizen focused policy and with the rise of the concept of public value. Public value is a system of assessing the value and the importance of public sector services in relation to that which the general public are prepared to give up - in monetary and non-monetary terms, such as time - to see a service in place. Under public value, issues such as quality of service and not merely cost will be considered. Cost efficiency of achieving success will continue to be a factor, but relative cost alone will not be the sole means of judging efficiency. In this sense, we are moving away from the most distorted versions of New Management prevalent in the 1980s and 1990s.

Our consultation confirmed our belief that in order to be able to judge the value of cultural activity, it is necessary to have a system which allows us to measure the effects of a particular project or a policy and to make comparisons with others.

9.4 Our thinking

9.4.1 Assessing cultural value

There has been a great deal of debate about the assessment of cultural value and the impact of arts. A number of organisations and key people including Demos, the Institute for Public Policy Research, the Secretary of State for Culture, Media & Sport, Tessa Jowell, and the Chair of the Arts Council of England have all recently produced statements on the public value of arts and culture. Most attempt to realign a debate that has centred around the relative merit (and therefore value of) culture in and of itself (intrinsic) or as a means by which to deliver broader social and political agendas (instrumental).

"Too often politicians have been forced to debate culture in terms only of its instrumental benefits to other agendas. . . . In political and public discourse in this country we have avoided the more difficult approach of investigating, questioning and celebrating what culture actually does in and of itself."⁴
Tessa Jowell, Minister of State for Culture Media and Sport

Both the First Minister's 2003 St Andrew's Day speech and the Cultural Policy Statement from the Scottish Executive, whilst alluding to the intrinsic benefits of cultural activity demand a cultural policy that pays "particular attention to the impact on the wider objectives of Scottish government" ⁵. It deploys both the intrinsic and the instrumental arguments, but tends to side towards the instrumental.

Instrumental arguments for the defence of culture have increased over the last ten years for three main reasons: the assault on traditional, timeless aesthetic and intrinsic values which have been challenged by the democratisation of the concept of the 'high arts'; instrumental statistics such as the number of young offenders participating in an arts project can be quantified; despite the developments in neuroscience, cognitive theory and educational psychology, a universally agreed scientific measure of the deeply subjective experiences that are at the heart of great art has yet to be declared.

"I know that arts and culture make a contribution to health, to education, to crime reduction, to strong communities, to the nation's well being, but I don't know how to evaluate it or describe it. We have to find a language and a way of describing its worth. It's the only way we'll secure the greater support we need." ⁶
Estelle Morris, Minister for the Arts

As noted in Section 2 of the report, the conflict between the intrinsic and the instrumental is one that has been overstated. The intrinsic nature of cultural activity has allowed it to be employed so successfully as an instrumental tool. The issue becomes more political, however, when we consider how to assess the impact of culture. A key question for the Commission was: should the measures used for cultural interventions be judged on the criteria of success at achieving other political agendas?

9.4.2 The importance of evaluation

The Commission's consultation revealed an awareness of the need for effective systems of evaluation:

"It is only through measurement that the value of national goals can be assessed."
Scottish Cultural Resources Access Network submission

" SCDI recommends… identifying and facilitating key outcomes of spending on cultural policy"
Scottish Council for Development and Industry submission

However, it also revealed a strong feeling among the sector that such requirements would usually be subordinate to the core functions of such bodies:

"It isn't appropriate to impose such bureaucracy that the recipient of funds spends a significant percentage of the funding in the process."
Scottish Publishers Association submission

9.4.3 Inadequacy of the current systems

Throughout our consultation, we confirmed the widely held view that there is a perceived inadequacy in the current systems of evaluation for the cultural sector in Scotland.

"Currently there are evaluation forms for arts and cultural work which could also be used for measuring the level of comfort in an airport lounge."
Royal Bank of Scotland Centre for Community Arts Research and Practice

Our work has also revealed a paucity of quality information currently available across the cultural sector in Scotland and an absence of co-ordination across various bodies collecting such data, making nationwide comparisons impossible. In drafting our report we have also struggled to make sense of inconsistencies and gaps in the information that is available. It is variable in quality, content and scope; different parts of the sector handle information differently, and different measures are used by different bodies.

In the main, the information is primarily based around inputs, (such as investment figures and those figures which can easily be logged and recorded - such as changes in library stock). These provide a poor picture of the quality, scope and magnitude of the cultural activity. A consensus exists that our present systems of evaluation, which tend to be predicated on securing continuing or future funding rather than on genuinely improving performance, do not serve the long-term interests of the cultural sector.

Proper evaluation… concentrates on improving performance rather than being used as an advocacy tool to justify further funding… What is important is the concentration on public perceptions and the use of more holistic characteristics than employed by our present evaluation systems.
Dr Stewart MacDonald, The Lighthouse submission

There was also an awareness that whilst consideration of evaluation may be a key part of securing funding that this is not always followed through:

"It is left to the recipient to do what they say and there is no obligation on those who succeed office bearers to continue to pursue those stated criteria."
Dora Betteson, personal submission

PMP's research for the Commission also provided evidence of the perceived inadequacy of the current statutory systems for sport and leisure management, museums and libraries by local authorities. Our consultation confirmed a strong feeling in the sector that the current systems of evaluation (quantitative 'bums on seats', economic 'ticket-sales' and economic impact data) should be augmented by some kind of evaluation that is sensitive to the subjective impact of culture. We need to be able to capture the transformative power of cultural activity and the concept of 'cultural value'. The Visual Arts and Galleries Association noted the need to, "create evaluation/measurement methodology appropriate to unknown and unpredictable outcomes". DEMOS held a conference in 2003, which led to the publication of Capturing Cultural Value: How culture has become a tool of government policy.

"We need a language capable of reflecting, recognising and capturing the full range of values expressed through culture. Some of those values may be covert and naturalised, they may coexist or conflict, but only with clarity about what they are can we hope to build wide public support for the collective funding of culture."⁷
Demos

9.4.4 Creating a new system of evaluation

Whilst there is almost universal agreement that there is a need for a better system of evaluation, there is little agreement on precisely what such a framework might be. There was an acute awareness that this is a complicated issue and that it will require careful consideration and further work beyond the life of the Commission:

"Detailed consideration of systems of monitoring, evaluation and setting of standards, should be out with the remit of the present Review. The Commission should take serious consideration of existing international research into 'qualitative' impacts of cultural provisions, as part of a move away from mechanistic quantitative measurements."
HI~Arts submission

It is important that work is undertaken to develop such a system with all of the people who will be required to utilise it. It is important to involve them all from the planning stage to ensure any system is appropriate to both the size and the scale of the project being assessed and to ensure that some common criteria are agreed allowing the collection of comparable data on the activities and impact for the cultural sector in Scotland.

The establishment of rights and entitlements will offer an opportunity to develop many of these criteria and should be used as an opportunity for the cultural sector to develop a system of evaluation. As we have noted, the Commission is unable at this stage to offer a comprehensive picture of such a system, but we are able to outline some of the key issues to be addressed.

A holistic approach

Research into the impact on quality of life and well-being of culture and sport is currently being undertaken by CCPR for the Scottish Executive and there is an indication that the Executive's Strategy Unit or the Office of Chief Researcher may undertake a more extensive cross-governmental study into well-being. The results of these studies will need to be considered and integrated into any new system.

The Commission is also aware of some interesting international developments and we highlight in particular the Creativity Index ⁸ being developed in Hong Kong that will provide a statistical framework for measuring the status and growth of creativity. Whilst the project is still developing the full range of indicators, many are already collected across a range of policy areas. They are not, however, currently being deployed in such a framework. We recommend further investigation of this development and, if appropriate, the adoption of the framework allowing benchmarking against Hong Kong. One of the stated long term aims for Hong Kong is to allow comparison with other Asian cities, but there is certainly the potential for an international index to develop.All of these developments are part of a general trend throughout the world that is beginning to look beyond GDP and economic growth as measures of success. The volume and effort of such developments suggest that these will become increasingly important in how we organise our societies and prioritise our development in the coming decades. Demos outlines a persuasive argument in Capturing Cultural Value ⁹ that attempts to create a language and a system built upon the discourses of anthropology, environmentalism, valuation of intangibles from business and the Cabinet Office's notions of Public Value. Demos's publication closes by suggesting an interlocking system of evidence collection and assessment, which includes many items currently collected as well as some further demands. Whilst attempting to provide practical examples, the model remains largely theoretical and some areas need further elaboration.

PMP's research for the Commission revealed the variety of systems of monitoring and benchmarking across local authorities currently in development - VOCAL's Quality Assurance Model and the English Comprehensive Performance Assessment, where DCMS is currently consulting on a 'culture' block - as well as other systems that could be amended or adapted. It is clear that, at present, no definitive performance measurement system for the cultural sector in Scotland exists.

9.5 Towards a new system of measuring culture

9.5.1 A comprehensive literature review

Our view is that one of the first major tasks is to carry out a comprehensive review of the existing literature relating to public value in the cultural context. Such a study should also consider the various assessment frameworks and toolkits that are currently available and those being developed in order to help Scotland produce its own universal assessment tailor made for the sector's needs. This is a key part of any new cultural framework for Scotland. Such a system will need to be adaptable to the needs and the scale of individual projects but will provide universal measures by which all assessments can be collated and can be compared.

9.5.2 Securing commitment from the sector

The process of establishing cultural rights and entitlements will require agreement on a system for measuring and monitoring performance against these. The most suitable framework for evaluation with the cultural sectors will need to be developed with sector councils, local authorities and all other stakeholders as it is essential that any such system has 'buy-in' from all. The resulting framework should be 'light-touch' and must be sensitive to individual projects and local variations, but should allow for key comparative data.

9.5.3 Self-assessment and an Inspectorate

We believe that the framework should also incorporate the general trend towards self-assessment from the education sector and others. In Scotland, monitoring, measuring and mentoring the cultural sector will be based upon self-assessment at organisational, local authority and institutional level. There is also a need for this information to be co-ordinated by the central cultural body, where it will be collated and reported.

In addition to self-assessment and an ongoing mentoring function, many stakeholders identified the need for robust independent evaluation, as and when required:

"Services which are not inspected are not taken seriously in the current local government context. The establishment of HMIE for Cultural Services would raise both the local and national political profile of libraries and other cultural partnerships."
Library Sector Thinking Group submission

A cultural sector inspectorate function, built upon Her Majesty's Inspectorate of Education ( HMIE) as a model, would be able to examine individual facilities on the agreed self-assessment criteria, as required, but would also be able to undertake specific studies as required, such as rural touring theatre in Scotland or educational services of core funded bodies for example.

9.5.4 Agreement on core measures

Responsibility for continual refinement and improvement will also need to be part of a co-ordinated approach. However, we believe it to be essential that the sector agrees on some core measures and ensures that these are collected consistently for at least ten years, so that data can be comparable in a way that is not currently possible.

Essentially we believe there will be a requirement in the future to:

• critically examine the aims of any project or intervention

• identify fully the costs involved

• assess the extent that aims have been met

• evaluate the investment proportionate to cost incurred or any profit realised

• assess whether the outcomes could have been achieved more effectively by other means.

"If proper objectives and milestones are agreed, written down and signed at the outset and constructive dialogue takes place throughout the partnership, then the incidences of failure to meet standards are reduced".
Diageo submission

In discussions with sector representatives, it was clear that key stakeholders realise the value of being able to explain the outcomes and effects of cultural engagement and of being able to compare and continually improve the provision of cultural services. If the First Minister's aim of placing culture at the heart of government, cross-cutting all departments, is to be achieved, cultural bodies will increasingly be required to justify itself to a wider range of clients. In such an environment, good reliable information for the sector is a priority.

9.5.5 What needs to be done?

In summary, there is a pressing need for better collection and reporting of data on the cultural sector in Scotland. Such data should include some core comparable figures that apply across the sector. We believe it is vital that the cultural sector applies effort to this project in order to deliver a tailor made system or else it may be forced to adopt a system co-opted from another sector. All assessments and reports should be made available publicly, in line with Freedom of Information Legislation. ICT solutions which could improve the system of evaluation should be explored.

It is important to note, however, that when considering further work and responsibilities in the area of assessment, care must be taken not to create systems which become overly burdensome and in which the requirement to record and assess outweighs the benefits of a more coherent understanding of the sector's impact. Recently the Scottish Parliament's Enterprise and Culture Committee concluded "that the current mechanisms for monitoring, evaluating and auditing spend on arts in the community are overly bureaucratic and burdensome, particularly to the many small and voluntary organisations active in this area. One witness told us that we ought to be 'measuring the breadth of the smile' that participation in the arts can bring." ¹⁰

It may be that we are unable to prove with a degree of statistical reliability the range of impacts of culture. A degree of faith is required from politicians to invest in the future well-being of the nation.

9.6 Making public investment work more effectively

9.6.1 Savings from central procurement and economies of scale

As we begin to implement methods of assessment that are more appropriate to the range of outcomes from cultural activity, we will improve our knowledge about the most effective approaches to cultural interventions. This will allow more targeted investment and more efficient usage of public investment. The Commission's recommendations on infrastructure outlined in Section 10 of this report will create an environment in which there will be a number of potential efficiency savings. The rationalisation of the diverse bodies and funding streams into one organisation should also present significant opportunities for savings in backroom functions such as payroll where currently each body has its own systems.

Bringing all the cultural sector bodies together into one system allows for the central procurement of both goods and services on behalf of the whole sector, in line with the Gershon recommendations. This could be undertaken on relatively straightforward areas such as stationery or cleaning, but it has the potential to be expanded into the procurement of professional services e.g. marketing services, IT and legal services on behalf of the whole sector. We would hope that any new infrastructural body for culture, established on the basis of our recommendations, and other bodies such as Scotland's National Institutions with collections, and performing companies, would follow this example by exploring the possibilities of shared services.

There are many areas where such procurement savings and economies of scale might be made, and they have been identified in Section 10.

9.6.2 Strategic alliances

During our consultation, we attended an Arts & Business seminar held in February 2005, which addressed the issue of strategic alliances between organisations and the benefits this might bring. These were identified as:

• releasing money for core functions

• improving standards and sharing best practice

• increasing impact through co-ordinating programming

• sharing staff and resources.

Participants at the seminar identified a range of barriers to such programmes, including:

• fear of losing funding

• time and energy required to build partnerships

• questions of ownership of intellectual property

• preservation of artistic identity.

A range of examples of successful alliances were presented. These included a one-off collaboration between York Theatre Royal and the York Pilot Theatre on a production that resulted in the Pilot moving in with the Royal. The Pilot accesses the marketing of the Theatre Royal whilst the Royal gains revenue from Pilot productions as well as being able to scale productions down for touring using the Pilot. In addition, the BBC and London Symphony Orchestras were reported as having pooled their administrative staff at the Barbican, whilst retaining individual directors. The Kimmel Centre in Philadelphia was also cited because of the co-location of a handful of arts organisations that shared HR, IT, PR and other 'back room' services.

We believe that strategic alliances could be explored as a useful first step towards greater collaboration between organisations in the cultural sector.

9.7 Summary and recommendations

We recommend:

1) That the cultural sector should apply its effort to the development and implementation of a tailor-made system for the performance, measuring, monitoring and benchmarking of cultural information.

2) That the lead agency should be tasked to improve the collection, co-ordination and recording of data.

3) That the key bodies in the sector should be required to adopt standard methods of recording and collating information.

4) That the lead agency adopts an agreed system of business plan evaluation.

5) That the lead agency mentors and validates a system of self assessment.

« Previous | Contents | Next »