The Scottish Government

« Previous | Contents | Next »

Listen

CHAPTER THREE METHODOLOGY

Summary

This chapter sets out the review methodology, which includes:

• the establishment of the review Steering Group
• collection of data from a range of sources
• stakeholder survey
• review of financial controls, audit and governance
• evaluation of ASH Scotland's communication strategy
• benchmarking
• identification of key issues

Introduction

3.1 The Saren Dixon Partnership's overall approach to the review had as its starting point that ASH Scotland was a mature and established voluntary sector organisation with its own strategic plan and internal performance management systems. The review team did not wish to duplicate the existing internal business and performance management systems within ASH Scotland, nor the processes for external monitoring, evaluation, scrutiny and regulation which were already in place.

3.2 The review team set out to assess ASH Scotland's activities covering the 3 E's of economy, efficiency and effectiveness, including a value for money ( VFM) assessment. The exercise included benchmarking components of ASH Scotland's performance against that of the Scottish Drugs Forum ( SDF) which was selected as a comparable organisation.

The elements of the methodology

3.3 The methodology involved a number of elements:

• the establishment of a Steering Group to support the review and provide feedback on findings at 2 stages in the process, after the initial findings and after the interim report
• the collection of data from a range of sources, including ASH Scotland, the Scottish Government, internet research and the SDF and the analysis of this
• a stakeholder survey comprising 50 interviews of key external and internal stakeholders
• review of financial controls, audit and governance
• an evaluation of ASH Scotland's communications strategy
• benchmarking
• identification of the key issues
• drilling down on the key issues, including the collection of further data and follow up with stakeholders
• the production of an interim report for comment by the Scottish Government and the Steering Group
• the production of a final report

The Steering Group

3.4 The review Steering Group comprised:

Analysis of key documentation

3.5 A range of key documentation was examined. The majority of this related directly to ASH Scotland's activities and included ASH Scotland's most recent management accounts, annual accounts, business and work plans, current and draft future strategic plans, annual reports, publications and reports, staff policies and quality systems. Other documentation examined included research findings and publications external to ASH Scotland, other evaluations and reviews of aspects of their work, Scottish Government committee papers and Scottish Drug Forum reports. For the purposes of the evaluation of the communications strategy, ASH Scotland's website was scrutinised. A list of the documents is available in Annex 1.

Stakeholder survey

3.6 Because of the difficulty of assessing ASH Scotland's particular contribution to the advances in tobacco control in the period under review, as distinct from that of other organisations with which it worked, a significant part of the evidence for the evaluation of the external impact of ASH Scotland has been gathered through the stakeholder survey. A full table with a breakdown of participant stakeholders is contained in Annex 2. It was agreed with interviewees that they would be named but that their comments would not be individually attributed.

3.7 Fifty interviews were conducted with key stakeholders in a range of categories including the Scottish Government, public health and the voluntary sector. This was designed to reflect sectors and organisations with which ASH Scotland has worked in partnership or had an important working relationship. 23 of these interviews were conducted face to face and the remaining 27 were conducted by telephone or by email. Annex 3 is the topic guide for the stakeholder interviews.

3.8 The profile of the interview sample is shown in Figure 3-1. Figure 3-2 illustrates the timeline and process of the research project.

Figure 3-1 ASH Scotland Stakeholder Survey Participants by Field

Figure 3-2 Project Flowchart

« Previous | Contents | Next »