The Scottish Government

« Previous | Contents | Next »

Listen

5 PERFORMANCE IN PROVIDING VALUE FOR MONEY 2004-2008

What is meant by value for money?

5.1 Value for Money ( VFM) analysis considers the question of whether more impact can be achieved for the amount of money spent. To examine this, VFM analyses consider:

5.2 VFM analyses, ideally, require firstly good data on costs, inputs, outputs and outcomes; and secondly 'counterfactuals' - what would have happened in the absence of the policy. Good value for money in relation to SDF would show it was better than any alternative delivery option either because it was more efficient and/or more effective in delivery of objectives for the same level of inputs with quality remaining undiminished.

5.3 The data available to this review are more limited and there are no directly comparable benchmark organisations. This means that it is not possible to compare SDF's performance with others on a strictly like-for-like basis, and that benchmarking data must be used with caution. The approach taken supplements the quantitative analysis on economy and efficiency with an analysis of delivery against targets and priorities set out in SDF's 06-09 Strategic Plan; and an analysis of stakeholder views of effectiveness. This gives a more rounded picture of performance than a purely quantitative value for money analysis would. The section is set out under the following headings:

• Inputs and economy - staffing and premises (paras 5.4 - 5.7)
• Outputs and effectiveness
• Delivery against targets set out in the 2006-09 Strategic Plan (paras 5.8 - 5.14)
• Delivery against 2006-09 Strategic Plan priorities (paras 5.15 - 5.17)
• Interviewee views of effectiveness on planned outputs from SDF's Strategic Plan 2006-09 (paras 5.18 - 5.35)
• Outputs, outcomes, efficiency and effectiveness for specific areas of work (paras 5.36 - 5.77)
• Conclusions and Recommendations (paras 5.78 - 5.99)

Inputs and economy - staffing and premises

5.4 The number of staff working at SDF during 2007/08 was 21. The total staff costs were £753k, which is over 60% of total costs. The average employment cost per employee is almost £36k. This is almost the same ⁶ as average employment costs per employee of Drugscope, although it is based in London and so might be expected to have higher costs. SDF feels that it operates in a different manner to Drugscope with more of its staff operating externally at a strategic level. Average employee costs at SDF are 64% higher than those at SAMH at £21,905. SAMH however is a much bigger organisation than SDF with 750 staff. It is a service provider and so might be expected to have more, lower-paid operational staff than SDF.

5.5 The pay and grading review during 2006/07 established new salary ranges. For the professional and manager grades, the new ranges were set in the upper quartile of the employers SDF had benchmarked against. Most (17 of 21) of SDF staff are on these grades. The salary range for administrative grades was in the upper half of the benchmark sample and the support range was in the lower half.

5.6 The 2007/08 salary (excluding pension contributions) of the Director at SDF lies in the range £51-£55k. This is substantially less than has been paid for the equivalent posts at both SAMH and Drugscope whose salaries lay between £70 and £80k. The salary of the Director at SDF is 40-44% higher than the Head and Regional Manager posts that report to him.

5.7 The second biggest area of expenditure relates to premises. SDF altered its premises in 2006/07. The appraisal carried out at that time showed moving to the Mitchell Street premises meant premises costs would rise to £45.6k per annum compared to £36.4k per annum for the Waterloo Street office. However, the new premises offered SDF more flexible space and, with some sub-letting and renting of spare capacity, the net additional cost was estimated to be £5,600 per annum. Moreover, had SDF remained in the Waterloo Street premises, the annual rent was likely to increase following the periodic review that was due. Consequently, the net additional cost of Mitchell Street was viewed as being reasonable given the greater flexibility it offered SDF.

Outputs and Effectiveness

Delivery against targets set out in the 2006-09 Strategic Plan

5.8 The 2006-09 Strategic Plan for SDF sets out four aims:

• to support an improvement in the quality of all responses to drug use in Scotland.
• to improve the general level of understanding and awareness of drug issues among practitioners, drug users, their families and the wider community.
• to act as a forum to develop and promote common responses to issues of concern and to represent SDF members.
• to develop SDF's capacity in line with existing and new demands.

5.9 Each aim leads to a number of 'outcomes' e.g. to support empowerment of communities to respond to drugs problems, which in turn have 'outputs' and targets.

5.10 Each year SDF submits a review of activity to the Scottish Government outlining progress made against each of these targets. This is not reviewed by the Board prior to submission. The reviews for 2007/08 and 2006/07 include quantified outputs. The review for 2005/06 does not, and so has not been included in the analysis below. An overview of performance covering similar information to that included in the review of activity is included in SDF's annual accounts. Delivery against targets is detailed in Annex 3.

5.11 SDF met or exceeded most (71%) of its targets during 2007/08 and almost two thirds (63%) of its targets during 2006/07 (see Figure 6). A small minority (8%) of targets were not achieved in 2007/08; this was down from around 14% in 2006/07.

Figure 6. Achievement against targets in 2006-09 Strategic Plan
SDF achieved almost three-quarters of its delivery targets in 2007/08 and almost two-thirds during 2006/07.

Sources: SDF Strategic Plan 2006-09; Reviews of Activity for 2006/07 and 2007/08, SDF submission to review team, March 2009; SDF comments on 2nd interim report. .

5.12 Targets missed or unclear in both 2006/07 and in 2007/08 were related to:

• work with people on drug treatment and testing orders ( DTTOs) and the Scottish Prison Service (2006/07, 2007/08)
• target to publish 10 bulletins per year - 8 and 9 were achieved respectively in 2006/07 and 2007/08
• support for networks dealing with young people.

5.13 There are some issues with the targets. Firstly, it is not possible to tell from the reports if some targets have been met or not (the unclear category in Figure 6 ). Secondly, a number of targets which were superseded during the course of 2006/07, were retained for 2007/08. Thirdly, most targets remained the same over the two years, including some which might have been expected to increase, to reflect that they had been exceeded by some margin during 2006/07. Fourthly, some targets conflate different elements which meant that a target could be met, without achieving the different elements within it.

5.14 These issues are not unusual but do reflect scope for the Board, staff and the funders of SDF to tighten up on performance management in future.

Delivery against 2006-09 Strategic Plan priorities

5.15 The strategic plan lists 8 'key strategic priorities':

• assisting in delivering effective community engagement and empowerment
• supporting the implementation of the Scottish Government's drugs deaths action plan
• promoting good practice and assisting improvements in service quality
• assisting the development of effective responses to the emerging problems with stimulants, particularly cocaine
• assisting the development of family focused services
• assisting the development of a closer linkage between treatment and care services and communities
• assisting improved working between criminal justice agencies and treatment and care services
• continuing to develop SDF as a membership organisation.

5.16 These priorities are listed at the start of the plan, which then goes on to set out targets in relation to each of the four aims. How the priorities will be taken forward is unclear - they are not explicitly linked to outputs or targets in the plan. This gap also means that progress on priorities is not reported back to the Scottish Government.

5.17 An analysis by the review team of which outputs and targets would appear to help to deliver the strategic priorities is shown below in

Figure 7. This analysis suggests that progress was made on six of the eight priorities; the exceptions being development of family focused services and improved working between criminal justice and services.

Figure 7. Progress against key strategic priorities set out in Strategic Plan 2006-09.
During 2007/08 progress was made on most strategic priorities.

Sources - SDF Strategic Plan 2006-09; Review of Activity 2007/08.

Interviewee views of effectiveness on planned outputs from SDF's Strategic Plan 2006-09

5.18 Interviewee views are noted below. Figures in brackets are the number of responses on particular points. No judgement is included if there were fewer than 6 responses on a particular point.

Aim 1. To support an improvement in the quality of all responses to drug use in Scotland

5.19 This aim covers most of SDF's operational areas: training, UI work, work with ADATs and community planning partnerships and so on. Interviewees were:

• Very positive on - critical incident training (8)
• Positive on - training for community groups and drug forums (13); support and develop UI groups/ UI surveys/commissioned research (31); promote the development of employability services for people with drug problems (10)
• Mixed on work with ADATs to support effective community engagement (8).

5.20 The critical incidents, UI work and AWTP are dealt with in more detail below (see Critical incidents training, Regional user involvement ( UI) work and AWTP, respectively).

5.21 The picture with ADATs appears variable, depending on the perceptions of the performance of current or previous SDF staff members in different areas.

5.22 Views on employability generally were influenced by knowledge of the AWTP work. Interviewees were very positive about that, although noting it was expensive ( ADAT stakeholder). Several stakeholders questioned SDF's locus as a direct service provider, with one ADAT stakeholder saying that other agencies were dealing with much larger numbers and at lower costs. Another said " their work has broken down barriers in terms of agencies' perceptions of former addicts' abilities". (Proactive stakeholder)

5.23 SDF's work on naloxone comes under this heading. SDF feels that it played an important role in the Lanarkshire and Glasgow pilot projects. Five interviewees and a number of user involvement volunteers commented positively on this aspect of their work. SDF's impact report prepared for the review team states that there had been 17 successful uses of naloxone in Glasgow area.

Aim 2. To improve the general level of understanding and awareness of drug issues among practitioners, drug users, their families and the wider community

5.24 This aim covers the databases of services, media contacts, and publication of bulletins. Interviewees were:

• Very positive on the bulletin (15) and information service (7)
• Mixed on the databases of services (6) and on undertaking a range of work with the media (10)

5.25 Nearly all responses regarding the bulletin were very positive. Three dissenting views mentioned lack of coverage of residential rehabilitation ( VCS drugs stakeholder), information from outwith the central belt (Police stakeholder) and that the bulletin was available only in hard copy ( VCS drugs stakeholder).

5.26 Interviewees were very positive about the information service. One interviewee from the voluntary sector said that SDF was " very good assisting them with enquiries", another from the same sector saw SDF as " an information hub for Scotland".

5.27 The Scottish Directory of Drug Services received more mixed views with some finding it very helpful and others questioning if it had been marketed enough.

5.28 Views on how effective SDF was at handling media were mixed. On the one hand, there is praise for the inputs, on the other, views from a range of sectors that SDF needed to be more proactive. However many people from all sectors mentioned that a strength of SDF was that it always put forward the users' points of view and that was essential.

Aim 3. To act as a forum to develop and promote common responses to issues of concern and to represent SDF members

5.29 This aim covers seminars, participation in government working groups, work with the Scottish Association of ADATs ( SAADAT) and consultation with members and development of member networks. Interviewees were:

• Very positive about SDF's seminars (6) and about its participation in government working groups (15)
• Positive about delivering consultation responses from the membership (13)

5.30 Seminars were very positively viewed by stakeholders from all sectors as important sources of information. A small number of interviewees ( SDF and volunteers) commented adversely on quality of organisation of the seminar on stigma.

5.31 SDF's quality of input to Scottish and UK government working groups was seen as positive and frequently praised; SDF was seen as effective in what one stakeholder referred to as a 'critical friend' role:

" They do inform those high level groups positively and usefully and bring in frontline perspectives". ( ADAT stakeholder).

5.32 Members (8) were generally positive about being consulted when SDF responses were developed whilst noting that they often put in responses from their own organisations as well:

"they ran very useful groups when the strategy [The Road to Recovery] was being developed". ( NHS stakeholder)

"I am a member - they are good at consulting" (Academic stakeholder)

5.33 However there was a minority (5) of negative views about SDF consultation responses from a variety of sources. These related to the tone and standpoint of the responses:

"I was surprised by the Welfare Reform Submission. It read like a polemic …if you are trying to engage Government, then it needs to be measured." ( VCS drugs stakeholder)

5.34 Another felt that the approach put forward in response to the consultation on 'Road to Recovery' was "never going to happen, it was a self-indulgent approach". ( SG stakeholder). The response was criticised by an ADAT interviewee who said it was "poorly researched and poorly put together." The same interviewee however commended the fact that SDF had highlighted the link between poverty and drug use again in the document.

Aim 4. To develop SDF's capacity in line with existing and new demands

5.35 This aim covers effective financial control, staff management and the need to develop funding proposals in line with the strategic plan. None of the three outputs within this aim were commented on by six or more interviewees.

Outputs, outcomes, efficiency and effectiveness for specific areas of work

Regional user involvement ( UI) work

5.36 Three of the UI projects running over 2007/08 were considered in detail: Perth & Kinross, Highland and Lanarkshire. ⁸

5.37 Income sources varied, but included Lottery funding, ADAT and SDF reserves. Objectives were to set up and maintain user involvement groups which would (1) undertake peer research work to provide feedback to services and (2) act as a stepping stone for volunteers to progress to e.g. employment, further education or training. The Perth & Kinross work included setting up a carers' group.

5.38 The numbers of volunteers involved varied widely, for example the Lanarkshire group involved a total of 16 over the year 2007/08, but this had dropped to three by early 2009, partly due to staffing changes. Highland operated with a total of 16 volunteers and Perth & Kinross with 6 during 2007/08.

5.39 The costs of each project are for a development officer and office and administrative support. Costs varied from £46k in Perth & Kinross, to £56k in Lanarkshire. Costs per volunteer involved varied from £3.3k in Highland, to £7.7k in Perth & Kinross. These bald unit costs do not however reflect that the groups also delivered a total of 5 peer research reports including one national survey during 2007/08. Neither do the costs reflect inputs by volunteers to planning groups.

5.40 What is achieved for this activity varies. A number of peer research reports were delivered to services and planning groups, and were valued. However, it is unclear how many volunteers moved on to employment/volunteering/ education and training opportunities. For example an ADAT funder of the work could not tell from monitoring information whether e.g. a small number of people had achieved multiple outcomes (e.g. secure tenancy, volunteering and part-time education) or a larger number of people had secured single outcomes. It is clear that some people do move into further education, employment and training and volunteers clearly gained in confidence and developed a range of skills.

5.41 In quantitative terms, agreed targets for this work were not met well. Of the total of 24 targets apparent for Highland, Perth & Kinross and Lanarkshire for 2007/08 (Figure 8):

• 12 were met
• 11 were not met
• it was unclear whether the remaining one was met or not

5.42 Targets not achieved included delivery of peer research reports and numbers of people progressing to employment or further training. Delivery in Lanarkshire was affected by staffing changes there during 2007/08. SDF has also expressed reservations (via comments on the second interim report) about some of the targets there, feeling in particular that the numbers of referrals and volunteers projected by the funders were aspirational. Delivery in Highland was judged more favourably by the ADAT than the monitoring reports to the New Opportunities Fund would suggest. Their view was that SDF had met or exceeded their expectations in over 80% of their work areas.

Figure 8. Achievement against target for regional UI work during 07-08
Half of the quantitative targets set for regional UI work during 2007/08 were achieved.

Sources: Lanarkshire Involvement Group Annual Report 2007/08; Highland New Opportunities Fund Annual Monitoring Report Feb 2007 - Feb 2008; Perth & Kinross Impact Report, April 2008; SDF submission to review team regarding Highland targets, April 2009.

Glasgow user involvement work

5.43 SDF's user involvement work started in Glasgow some years ago and the group there produced a number of valued pieces of work including on needle exchange, pathways to employment and nutrition and diet. The model changed in April 2008 when SDF, working with Alcohol Focus Scotland, was commissioned by the Greater Glasgow & Clyde Drug Action Team to oversee the development and implementation of the strategy for user involvement. The focus of the work is much more on supporting services to develop their own user involvement than to run a city wide user involvement group, which SDF had previously done.

5.44 SDF employs a user involvement manager and officer on this work, at a total cost of circa £91k for the period April 08 - March 09.

5.45 There are no quantitative targets set for the current work. SDF said it had tried to incorporate targets relating to inputs and outputs/outcomes but that they had not been included in the agreement.

Stakeholder views on user involvement work

5.46 Volunteers involved in SDFUI groups felt that they had benefited personally in the following ways from participating in UI work:

• Increased confidence
• Training
• A good reference attesting to reliability
• Feeling of achievement
• Something to focus on
• Respect from others

5.47 Volunteers praised one SDF development officer in particular for her commitment, knowledge and her open-mindedness in considering new approaches (a volunteer had suggested using cognitive behavioural therapy in their work and the development officer was investigating it). One said: "she has a good background - in homelessness and prescribing and she is so committed". Members of the UI group credit this worker with turning round a situation where no services in the area asked users for their feedback, to one where services, taken aback by the honesty of the peer research and the extent to which it differed from what was being said to doctors and key workers, were asking for more work to be done. This assessment was echoed by the relevant ADAT.

5.48 Current SDF volunteers thought the peer element of their research was one of the most important aspects. They felt strongly that it is easier for a drug user to speak more honestly to someone who has shared a similar experience. They also felt that anonymity of responses promoted more honest feedback. They also saw value in being able to be a positive role model to people who were still living chaotically.

5.49 The model was seen by members of one local SDF group as more powerful than an advocacy model whereby a volunteer acts as an intermediary between a service user and the provider or key worker. It was felt that the research work changed things. It showed services honest feedback - " the truth" - which was totally different to what key workers and doctors were being told.

5.50 The users consulted were generally very positive about SDF's interaction with them in both the local and national groups.

5.51 ADAT staff offered different views of the work. In one case the funder was very satisfied with the results of the research and said it was due to the sound work of the development officer. In his view, "the information I get from service users is unique and that alone is worth the investment". In another, there was concern about staffing changes and slow recruitment and a clear view that SDF had not delivered.

5.52 A repeated concern from ADAT interviewees related to perceptions of SDF's inflexibility in approaching user involvement:

• One ADAT said " SDF attitude was that as a national organisation they had a format ….something that works in the centre of Glasgow where lots of drug users congregate in one area - this approach doesn't work in what is essentially a rural area". This ADAT stopped the SDFUI work early as they felt that they were not getting value for money.
• A second said: "They seem fundamentally opposed to the model they have been engaged to work on…".

5.53 ADAT interviewees recognised that involving problematic drug users is challenging and implies constant support and contact. Nonetheless, stakeholders found uncertainty about the stability of UI groups problematic:

"Effectiveness fluctuates - they haven't had a consistent group but every now and then they've had a good, useful, productive group. They've had difficulties in keeping the group going. They've prioritised it, rightly, and done their very best.

I need to have the confidence that I can go to them and ask for views. If I don't have that confidence that they are there and working well, that will inhibit me approaching them." ( NHS stakeholder)

5.54 The role of SDF national staff in the management of local projects was commented on negatively in three instances. Issues raised were perceived lack of knowledge of the local context, and the effectiveness of providing managerial support from national HQ to locally based staff.

5.55 Views from non- ADAT interviewees were mostly positive about SDF's UI work, e.g. "One of their particular strengths is that they're quite heavily involved in UI - they provide a direct link for Scottish Government with the user and that's of huge benefit".

5.56 Some, from the voluntary and government sectors, felt that SDF's UI groups gave a partial picture in two ways. Firstly, that it (with the exception of Perth & Kinross) did not reflect carers' views and secondly, that it reflected SDF's own views. On this latter point, staff involved in the UI groups were clear that users took the lead in peer research work.

5.57 The Director of SDF feels that the organisation has learned lessons from the UI work in different areas. These concern the importance of having tangible and widespread support from services, and that SDF needs to assess feasibility of the work prior to accepting funding. He and the Head of Business, Finance and Administration confirmed that SDF has not carried out a formal performance review of its UI work.

AWTP

5.58 The Addiction Workers Training Programme ( AWTP) trains people with a history of substance misuse to be addiction workers. The project was set up in 2004 with funding from the New Opportunities Fund, the Robertson Trust and Scottish Enterprise Glasgow, supported by the Greater Glasgow & Clyde Drug Action Team. Additional funders for 2006 were the Greater Glasgow Alcohol Action Team and Glasgow Community Regeneration Fund.

5.59 The Programme won Best Practice Initiative of the Year and came runner-up in the Education Initiative of the Year categories of the inaugural Herald Society (newspaper) social care awards in 2008.

5.60 The AWTP lasts one year for a participant and there is an additional three month period of aftercare support for all trainees who begin the training, regardless of whether or not they complete the course.

5.61 The AWTP cost for 2007/08 was circa £231k. This equates to a cost per person completing the programme of circa £13.5k and a cost per person into employment of circa £15k.

5.62 Over the three years between 05 and 08, a total of 46 people started the programme. Of these, 40 (87%) completed it and 35 (75%) went into employment. Trainees' views reported in an independent evaluation were very positive, (Figure 9). People interviewed were positive about the AWTP.

Figure 9. Views of trainees about the AWTP Trainees felt that the AWTP was an excellent programme and made a big difference to their lives.

Source: Addiction Worker Training Programme Evaluation, Final Report, August 2008. FMR Research, for Greater Glasgow & Clyde Drug Action Team

Benchmarking Employability Work

5.63 The AWTP aims to help trainees find employment as addiction workers.

5.64 Cost per person moving into employment of AWTP is circa £15k, (2007/08 figures) and over three years, 75% of people starting the programme have moved into employment.

5.65 Cost per person involved in the three regional UI projects is between £3.3k and £7.7k and roughly between a total of 30-40 are involved at any one time in the three regional projects looked at.

5.66 It is hard to find direct comparisons for these programmes, particularly the UI work because of its dual purpose - their unit costs are based on volunteers involved, but they also deliver peer research reports. Some context for looking at these figures is given below.

5.67 CAIR Scotland runs the ECLIPS Moving On service for the Tayside ADATs. Most of its clients have already been involved in services to address psychosocial issues and to help stabilise them i.e. they serve clients similar to those involved in SDF's AWTP and UI groups. Year to date figures for 2008-09 as at 24 March 2009 were:

• Clients who came onto the ECLIPS programme during the year - 193
• Clients who left the ECLIPS programme during the year to move onto further education, training, employment or volunteering - 122 (63%)
• Budget is £276,731
• Unit cost per person involved is £1,433
• Unit cost per person achieving a positive outcome is £2,268

5.68 SAMH runs a variety of programmes which aim to help people with mental health problems and problem drug use to move onto employment, training or education. Their costs per person involved range from £2,388 to £6,831.

5.69 SDF has provided benchmarking figures on the success of similar programmes i.e. for former substance users wishing to work in social care. No costs are given in the document. Employment rates for the projects, two in Scotland and one in England, are between 68% and 85%.

Critical incidents training

5.70 The Critical Incidents Training programme has run since October 2004. The training aims to inform people about what to do if they suspect that a person has overdosed. Programmes are delivered to groups of drugs users, volunteers in user involvement groups and carers. The programme consistently exceeds its targets of training courses run, despite the targets being increased each year from 25 in 2004/05 to 44 in 2007/08.

5.71 The total number of people directly trained in 2007/08 was 562, an increase of 75% on the previous year's total. During 2007/08, nearly half the courses were 'train the trainers' with the intention that attendees on these courses would then cascade the information to others. SDF estimates that around 1200 people received cascaded information.

5.72 The Scottish Government funded the cost of this work: during 2007/08 it totalled circa £62k. This gives a cost per person directly trained as £110. Including those estimated to have received cascaded training, the cost per person trained comes to £52.

5.73 There is no quantitative evidence of outcomes in this work i.e. numbers of lives saved as a result of the training - such data would be impossibly hard to obtain. However, the Critical Incidents Training officer at SDF reports two known successful uses of the training and the work was viewed very positively by interviewees with a number mentioning unprompted that they were sure it will have saved lives.

Contribution to Policy

5.74 SDF contributes to policy via its involvement in a wide range of working groups and also via responses to consultations. It has also been involved in Ministerial meetings.

5.75 There is a split view from interviewees in this area. On the one hand interviewees are very positive about the effectiveness of SDF's contributions to working groups, yet on the other hand, SDF's effectiveness at influencing policy making at national level is viewed negatively.

5.76 Views from government, i.e. from the people who were responsible for development of policy were negative. SDF contributions to 'The Road to Recovery' consultation were felt by one stakeholder to be " over-simplistic"; by another as " out of touch". Speaking about a number of interactions with Ministers and senior civil servants, another interviewee felt that " they always said that that would need money, rather than e.g. take a more imaginative approach about how to negotiate to do things differently or how to change attitudes". These views were echoed by responses from other, non-governmental stakeholders in relation to the tone and content of responses to consultations ( see paragraphs 5.33 and 5.34 above.)

5.77 Views from others not directly involved in Scottish Government policy development were positive. They saw that SDF was involved in a range of working groups and so concluded that their involvement in development of policy at top level, was also positive. This is a reasonable assumption to make given that nearly all who worked with SDF staff on working groups saw their effectiveness there as very positive. So to that extent SDF is effective, but when it comes to detailed responses, both written and in bilateral Scottish Government/ SDF meetings on policy, the picture is more negative.

Conclusions and recommendations

Inputs and economy

5.78 The relatively high level of costs per employee reflects the high proportion of staff at SDF on professional and manager grades and the fact that these salary ranges appear more generous in relation to benchmarks than either the Director's or the administrative and support ranges. The relatively large gap between the salary of the Director and Heads/Regional Manager could suggest an over-reliance on the Director in the operation of SDF. The costs of premises appear reasonable.

Outputs

5.79 SDF's delivery against targets overall seems reasonable, although targets were not updated each year and could have been more stretching in some instances. SDF's delivery against targets in local UI projects is not as good.

5.80 'Key strategic priorities' are not translated into targets. This keeps them off the twin radars of the annual review of activity and the annual report, meaning that neither the Board, nor the Scottish Government, has a way of tracking progress on them. However, mapping what has been delivered on specific targets against the key strategic priorities does show that progress has been made on six of the eight.

Outcomes and effectiveness

5.81 It is possible to come to some conclusions on SDF's effectiveness by looking at both delivery against target and stakeholder views.

Aim 1 - supporting an improvement in the quality of all responses to drug use in Scotland.

5.82 Training work is well regarded here, particularly the critical incident training which is a clear strength and consistently exceeds targets.

5.83 Work to support communities to respond to drugs problems via e.g. ADATs, community based groups and drugs forums shows a more mixed picture. Performance against some targets here is unclear or not met, and no progress is evident in the performance reports relating to the key strategic priority of assisting the development of family focused services. More positively, this area of work includes a training post funded by Scottish Government to provide training and support to a range of family support and community groups throughout the country, which has exceeded targets. Interviewees involved in ADATs have mixed views about SDF's effectiveness in their localities. Annex 3 on delivery gives a flavour of the patchy geographic coverage of this work. This illustrates the almost impossible task SDF faces in achieving simultaneous wider geographical coverage given its staff numbers and budgets.

5.84 User involvement work has been a traditional strength of SDF's work, noted by a number of interviewees. SDF's delivery on local UI targets is not good however and ADAT funders of the work have mixed views of its effectiveness.

5.85 AWTP is seen as effective and its delivery hits targets. However see points below (paragraphs 5.95 and 5.96) regarding cost-effectiveness.

5.86 Work with the criminal justice system is a key strategic priority. Yet a target relating to it has been missed in each of the last two years. Nonetheless interviewees from SPS rated their work with SDF positively.

5.87 SDF's work on naloxone is viewed by interviewees as important and good.

Aim 2 - to improve the general level of understanding and awareness of drugs issues among practitioners, drug users, their families and the wider community.

5.88 Interviewees commented frequently that it was necessary to get the users' points of view heard in the media and it was generally felt that SDF had responded well to media enquiries over a number of years. Media work hits its targets, but there is concern that it is not proactive enough. This is reflected in the mixed views of stakeholders. Are the proactive briefings targets too low?

5.89 Bulletin and seminar delivery both missed targets in the last two years. Nonetheless, they are very much valued with both viewed very positively by interviewees. They are a clear strength of SDF.

Aim 3 - to act as a forum to develop and promote common responses to issues of concern and to represent SDF members.

5.90 Participation in working groups is viewed very positively and hit targets. It is a strength of SDF.

5.91 Consultation responses provoked strong views. Some liked the tone of particular SDF inputs, whilst others felt the tone and content of some were not the most effective way to try to influence government. The challenge here is to be effective in representing members' views whilst at the same time understanding and responding to the constraints on what government is likely to be able to do. There are choices here for the SDF Board to make in the tone of their responses in the future and also in how inclusive of a diversity of views they wish to make their responses.

5.92 Support for members' networks appears to meet targets in terms of numbers, but not in breadth of work: there appears to have been no support for agencies working with young people.

Aim 4 - to develop SDF's capacity in line with existing and new demands.

5.93 Targets here were all met although they were not stretching. Some views questioned the effectiveness of managerial support to regionally based workers.

Unit costs (i.e. efficiency) and cost-effectiveness

5.94 Unit costs of the UI work vary widely, reflecting the different numbers of volunteers involved in different groups. In the light of (a) benchmarking information above, (b) a record of mixed delivery on regional UI targets and (c) mixed views expressed by ADAT interviewees, there is a strong case for a full review of SDF's UI work: its purpose/s, costs, management support and flexibility of approach.

5.95 Costs per person into employment from the AWTP look high in the context of the benchmarking information available to us. Their success rate looks comparable to other projects dealing with similar clients.

5.96 It may be a case of other services benefiting from economies of scale - SDF has a very small operation in employability work compared to many other providers. The limited benchmarking information on costs ( see 4.7 -4.8 , above) suggests that SDF and its funders might want to review its methodology and costs to see what additional benefits may be generated as a consequence of apparently higher unit costs and whether similar success rates could be achieved at lower costs, by SDF or potentially by others.

Performance Management

5.97 Performance management is not as tight as it should be. Aims are linked to specific outputs and targets, but priorities are not. This breaks the 'golden thread' that links what SDF wants to achieve with how it will do it, and how it will know if it has worked or not.

5.98 Targets do not appear to be systematically reviewed each year and whether they have been achieved or not is unclear in some cases. The Board does not receive a systematic report back on performance against the strategic plan. At the time of writing (March 09), the mid year review of activity in 2008/09 required as a condition of the Section 16b funding from the Scottish Government, had not been received.

5.99 In the light of increased pressure on funding, the emphasis on outcomes in 'Road to Recovery' and the findings of the Audit Scotland report, Drug and Alcohol Services in Scotland, SDF, both staff and Board, and its major funders, should improve performance management in future.

Recommendations

R3 SDF should conduct a full review of its UI work: its purpose/s, costs, management support, risk assessment and flexibility of approach.

R4 SDF and its AWTP funders should seek further benchmarks on costs and outcomes for the AWTP; and use that information to help ensure that the AWTP is as cost- effective as possible.

R5 SDF should improve its performance management arrangements to ensure that priorities are properly reflected in the strategic plan; that performance is reviewed by the Board; that targets are appropriate and reviewed fully annually and that performance achieved against targets is set out clearly.

R6 SDF should increase the amount of proactive media work it undertakes. This need not necessarily involve mainstream media. In general, SDF may want to consider whether its use of its website and of online communication can be expanded.

R7 SDF should consider how to increase its effectiveness in contributing to policy. This should include consideration of approach in both written and meeting based opportunities.

« Previous | Contents | Next »