« Previous | Contents | Next »
Listen
1 Executive Summary
1. As part of the consultation on draft National Quality Standards for Substance Misuse Services three pilot projects were carried out across Scotland to guage the practicality of using the standards to monitor and evaluate services. All three pilots used differing methods and tools for evaluation but all three used the Quality Standards as reference against which the services were measured.
- Forth Valley Substance Action Team ( SAT) undertook to lead an evaluation of five services representing a range of sectors who had volunteered to take part
- Aberdeenshire, Moray and Perth & Kinross Alcohol and Drug Action Teams ( ADATs) ran a peer evaluation pilot which initially only covered one project.
- In Glasgow the Scottish Drugs Forum ( SDF) facilitated the Glasgow Involvement Group ( GIG) to undertake a service user evaluation of five voluntary organisations who had elected to take part.
Forth Valley Substance Action Team Pilot
2. Interviews were arranged with the five services participating in the pilot. Services were provided with a checklist to assist in gathering appropriate documentation to evidence that the service met the standards as currently drafted.
3. Service providers presented various amounts of evidence: some had recent copies of client information leaflets, assessment sheets, specific policy and procedures. While the overall standard, readability and evidence of implementation of policies and procedures was an area that could be improved, all services stated that publication of the draft standards gave them an opportunity to review materials and discuss processes for personal plans.
4. The checklist provided helped services to understand how meeting the standards could be measured. The pilot also demonstrated how meeting proposed standards could become an integral part of service review and development.
Scottish Drugs Forum Pilot
5. Members of GIG developed a questionnaire and carried out survey work with service users at five services across the Glasgow area. Despite initial exploratory work, there was lack of clarity amongst frontline staff as to why the pilot was taking place. The aim was to conduct ten interviews in each service, although only 40 were completed in total.
6. Based on those interviews, the information provided suggested that the standards are quantifiable in their current form. However, they need to be clearer to allow service users to fully understand what they can expect from a service. The service users interviewed may not have been representative of users as a whole.
Aberdeenshire, Moray and Perth and Kinross ADATs Pilot
7. This pilot explored the possibilities of conducting a 'peer led' review of services using the draft quality standards, involving review teams from outside each Action Team area, in a three way process. A template was produced to measure evidence in a quantitative way and supplemented by staff, partner agency and service user interviews to gain qualitative information.
8. Because of difficulties with timing and sickness, the reviews were not completed. However, the process did provide information for the local Action Team and services in Perth and Kinross to come together to build the foundations for a better integrated care model. Despite fears that senior management might not take recommendations forward, there was clear signs of willingness of the service(s) reviewed to change.
Main points emerging from the pilots
- All pilots found that the standards are useful and quantifiable. The collection of both quantitative and qualitative data is necessary to evidence any recommendations for change.
- Frontline staff were not fully aware of developments around quality standards at national and at local level.
- Quality standards could be more service user friendly, accessible and presented in a way that can be easily understood.
- The lack of guidance on the interpretation of the standards and a framework for measuring how services can meet them made evaluation difficult.
- Monitoring and evaluating can provide an opportunity to review material produced by services and discuss record keeping. Evaluation results can be used to support funding applications.
Actions identified from the pilots
- An evaluation framework should be developed incorporating both quantitative and qualitative methods, with guidance to assist in interpretation.
- The standards need to be promoted fully to ensure all people involved with substance misuse know about them and understand them.
- All managers, staff and users of services need to understand the purpose of the standards and their part in the evaluation process from the outset. Assistance and training should be provided.
- Measurement against the standards should form an integral part of the service review process and an independent body should carry out these inspections and make recommendations.
- Recommendations arising from the service evaluation need to be implemented by management.
« Previous | Contents | Next »