< Previous | Contents

Independent Advocacy: A Guide for Commissioners

Section 4

Measuring the Effectiveness of Advocacy

Both commissioners and advocacy groups have an investment in knowing that advocacy is effective. Public agencies have a duty to ensure that public money is being used well. Advocacy groups which seek public funds to help them do their work recognise their accountability to the public for how they use this money.

Advocacy groups know better than most that good intentions do not always lead to good outcomes. They know that this applies to their own work as well as to services, and welcome regular scrutiny.

During the consultation, several funders and advocacy agencies made the case for a system of external recognition as a way of bolstering the independent status and credibility of the advocacy scheme. Over the next year, SHS and Advocacy 2000 will be working with advocacy groups and commissioners to develop proposals and methods for a national scheme along these lines.

Commissioners and advocacy groups should be clear from the outset that as part of the grant conditions or service agreement there will be agreed processes for monitoring and evaluation.

However, the methods used for monitoring and evaluating the work of advocacy groups must be credible both to the groups themselves and to funders and should not impose a disproportionate burden on a small organisation.

4.1 DEFINING QUALITY

Different approaches to independent advocacy are needed for different people at different times and in different contexts; there is no one best model. Similarly, the evaluation method and criteria must be matched to the specific approach. The 1997 guidance states:

While these indicators are specific to these types of advocacy, there are certain general criteria for quality which apply across all types of advocacy:

• Advocacy should actively include those individuals and groups most at risk of exclusion and least able to represent and defend their own interests
• Advocacy groups should be firmly rooted in, supported by and accountable to a geographical community and/or to the constituency they serve
• Advocacy groups should be constitutionally and psychologically independent of government and of the statutory and voluntary service system
• Advocacy groups should have a robust process for management and governance which helps them stay clear about their principles and goals
• Advocacy groups should be well-informed about the wider context of policy and law.

4.2 MONITORING

Monitoring is the process of checking continuously how things are going. From the outset, advocacy projects should set up systems for gathering the routine information they need so they know how they are doing.

For example, in the first few months the project and the funders might agree to monitor 'setting up' tasks such as getting the advocate recruitment procedures in place or providing training for the management committee.

By the end of the first year, projects and funders should be able to agree what information to collect routinely. In one project this could include for example:

• the type of enquiry - what issues people want help with
• characteristics of people looking for an advocate - age, gender, ethnic origin
• the amount of time staff spend on direct advocacy
• the amount of time staff spend recruiting, training and supporting citizen advocates
• length of time people are in contact with the project/with an advocate
• feedback from people who use the project/have an advocate
• the outcomes for people who have used an advocate
• the areas where it has been difficult to engage and work with providers.

Commissioners can advise projects on what information to collect and this information will be useful in discussions between projects and commissioners. However, projects should see monitoring as primarily something which benefits them, not as a chore to please the commissioners. This means limiting the information collected to the useful minimum.

Funders should recognise that advocacy projects will develop over time. They should expect different areas of work within a project to proceed at different speeds. Numbers tend to build up gradually as the project gets established and it is more important to build capacity than to worry about numbers in the first couple of years.

As well as routine monitoring, many advocacy groups undertake periodic reviews of their work. These reviews may be purely internal, that is, undertaken by some combination of staff, management committee, advocates and people who need advocates. Alternatively, advocacy projects might invite a commissioner, another advocacy project or an independent outsider to contribute to the review.

Evaluation

Evaluation involves a planned process of gathering information, reaching conclusions and making recommendations. An evaluation of an advocacy project will seek to take into account the perspectives of all those with a stake in the work: people who need advocacy, paid and unpaid advocates, project staff and management committee members, funders, referrers and so on.

Evaluation means making a judgement of how good something is, not just whether or not it has complied with a funding specification. It means looking at outcomes as well as activities, at relevance as well as numbers, at what could have been done as well as what was done.

Evaluating advocacy is complicated. It means thinking carefully about the purpose of the project (and different stakeholders often have different accounts of this). It means listening carefully to what different people say about the difference it has made to people's lives (and sometimes the people whose lives have been affected most are not able to articulate this). It means balancing the visible stories of success with the invisible work of preventing worse from happening. It means putting a value on relationships as well as results. It means assessing how much impact advocacy has had on policies and practice in the service system, both in relation to individuals and more generally.

There is an obvious danger (apart from lack of time) in commissioners undertaking such an evaluation directly. As the representative of the service system, the commissioner - however sympathetic - cannot be seen as independent and fair in this role.

Commissioners and advocacy groups should therefore invest in regular independent evaluation of all advocacy projects. This should only start when the project is on its feet - say after the first 3 years - and then at similar intervals. The reports from these independent evaluations should be made available to funders, advocates and others.

Typically, an external evaluation will cost from £2,500 to £4,000 - and on a three year cycle this represents a good investment for the commissioners. Alternative 'in-house' approaches - for example using someone from an agency or a part of an agency not involved in commissioning advocacy to undertake the evaluation - can also work well if a person with the right skills has the time to do this. But this also costs money.

Evaluation should be a constructive but challenging process - not an ordeal, but equally not simply a mechanism for encouragement and renewal. For this reason, it is important for advocacy groups to undertake other renewal activities.

Commissioning an independent evaluation

It is useful to draw up a clear specification for any evaluation, setting out the scope of the work, who is doing it, how it will be done, why it is being done, who wants it done, who is paying for it to be done, who will get the report and what sort of actions might be taken as a result of the evaluation.

In planning one recent evaluation the following scope and focus was agreed by advocacy agencies and commissioners:

• Developmental - using a partnership approach, to highlight what is working well and where improvement and development is needed in future

• User-focused - exploring the relationship between partners and advocates, the experience of partners, the issues which have been important to partners and advocates and the roles advocates have taken up

• Scheme-focused - the work of the office and co-ordinator, the training and support given to volunteers

• Management-focused - management arrangements and the role of the management committee

• External support - what support has been provided from commissioners and others, how this helps or hinders, how it could be improved

• Relationship with providers - how the project is perceived, what impact it has had on providers.

As well as considering current outcomes, an external evaluation should pay attention to the accountability, robustness and sustainability of the agency. This might include, for example, issues such as:

• the composition and renewal of the management committee. How well does the membership of the committee reflect the agency's constituency? Are new people being recruited to the committee?

• the extent to which the agency is addressing the needs of the most marginalised people within its constituency - for example, people from ethnic minorities, people who do not use words to communicate, etc.

• the reputation of the agency within its community

• the match between the advocacy needs of the people the agency serves and the skills and resources held within the agency.

There are recognised tools designed for evaluating citizen advocacy. There are also a number of agencies and individuals in the UK with significant experience of evaluating various forms of advocacy.

Section 5

Learning from Advocacy

"Even in instances where governments don't necessarily agree with advocates they might still nonetheless invite their participation, input and presence in matters of importance. For example, governments may seek to sound out advocates on prospective policies, collaborate in getting new laws developed, invite submissions on budget priorities, authorise officials in the system to give access to advocates, encourage consumers of service to employ or use advocates, and take other measures which demonstrate that they value and welcome advocacy."

Michael Kendrick (1998) A brief description of the common ways in which governments have supported advocacy

The business of independent advocacy is to add weight to the views and interests of individuals at risk of exclusion. So advocates - whether paid or unpaid - are constantly making people's views known on specific issues. However, this is often a reactive process, dealing with issues 'downstream' as they affect individuals and groups. It is also important to pull out the broader themes from these individual issues in order to inform policy and practice 'upstream'.

5.1 THE IMPORTANCE OF LEARNING

Independent advocacy groups - both local and national - can provide valuable intelligence and feedback to policy-makers and commissioners as well as to service providers. For example, they may raise issues of professional practice which should be addressed through the clinical governance process. They may also provide useful information for Health Improvement Plans and Community Planning.

Many large provider agencies genuinely want to learn from an independent advocacy group how to improve its service. If there is no mechanism for them to do this, they may see the advocates as simply a source of complaints and aggravation rather than a source of positive ideas for change.

This process of learning from advocacy needs to be organised so that the knowledge gets to where it is needed and so that people's time is used well. Individual 'noise' has to be translated into intelligible patterns and coherent messages.

For example, an organisation of mental health service survivors is well-placed to give feedback on users' perceptions of the way they are treated by different mental health services.

A citizen advocacy agency concerned mainly with people who have a learning disability can provide valuable information on the common themes being tackled by advocates - for example, lack of daytime activity, poor access to primary health care or lack of co-ordination between different agencies as it affects individuals.

Commissioners should devise simple ways to gather this intelligence, preferably through a regular cycle of face-to-face meetings rather than relying on written reports. The onus is on health boards and local authorities to be clear about the sort of information they want, and how they plan to use it.

For their part, advocacy agencies have to spend time reviewing their work and discerning the underlying themes before they can give commissioners a clear response to the questions they ask. This takes time, and works better if it is part of a regular cycle rather than part of a one-off 'by the end of next week' consultation.

In the response to the first draft of this guidance we heard of several examples of this working in practice.

5.2 BUILDING LEARNING IN TO THE SYSTEM

The funding agreement between an advocacy group and a statutory funder should state explicitly that advocacy groups have both a right and a duty to give feedback, and that the statutory agencies have a right to be advised of problems and a duty to listen.

These meetings should be with the right people in the agencies - this may be people with a specific 'care group' or geographical brief rather than the person responsible for commissioning advocacy.

Short focused face-to-face meetings with individual agencies or with a small number of advocacy agencies doing very similar work are probably more useful than large gatherings.

In between meetings, there should be an agreed way for advocacy agencies to raise service-related issues with the commissioners.

Advocacy groups whose work brings them into frequent contact with particular service providers should be encouraged to agree protocols with those service providers on issues of access, representation and conflict resolution.

It may be helpful to clarify the ground rules for these meetings, for example:

• Information which can be traced back to individuals will be handled in confidence
• The relationship is based on mutual respect rather than confrontation
• Advocacy agencies have a duty to tell it as they see it, even if their picture is much less positive than the view of commissioners or providers
• An undertaking to raise issues through agreed channels first
• A willingness to give accurate information rather than make general allegations
• These meetings are for the advocacy agency to give feedback to commissioners and service providers, and not for reviewing the work of the advocacy agency
• There will be an agreed written note from these meetings, and commissioners should give feedback to the advocacy agency on any follow-up action.

AND FINALLY

From the 1997 guide to good practice, as a reminder:

TEN DO'S AND DON'TS FOR COMMISSIONING ADVOCACY

DO

1. Do ensure that independence is built in to the project design/service specification.

2. Do involve people who use services, people who are independent of the service system, existing advocacy and rights groups and people with experience of advocacy in deciding what sort of project is needed.

3. Do be clear what you want it to achieve, and that the project you commission will be appropriate to meet that need.

4. Do make sure that it has realistic aims and adequate resources to meet them.

5. Do acknowledge that an advocate's primary loyalty must be to the person or group they are advocating for, not a commissioner or provider of services

6. Do stress that the service provider must make sure that professional staff understand people's need to have access to, and get support from, independent advocates and advocacy groups.

7. Do remember that advocacy will, at times, lead to conflict and put pressure on services to change and adapt to meet individuals' needs.

8. Do remember that people who have been neglected, or ignored, may have difficulty expressing their needs and may be very angry. You may need to work hard to understand what they are saying, and to act appropriately.

9. Do spend time, money and effort helping agencies to develop their proposals.

10. Do remember that it is often better for advocacy groups to do one thing well than to try to meet many different aims and expectations.

DON'T

1. Don't try to control the advocacy effort. It becomes ineffective when compromised.

2. Don't set advocacy organisations up to fail by putting unreasonable expectations on them.

3. Don't develop plans for advocacy without the active involvement of people who use services or who are independent of the system.

4. Don't forget to provide continued support to advocates and self advocates.

5. Don't think that funding an advocacy organisation is all that needs to be done.

6. Don't forget that participation costs time and money. Volunteers and service users need to have their expenses paid.

7. Don't expect non-professionals to understand the jargon and to be able to complete tenders without help.

8. Don't underestimate how far a little help can go to boost morale and restore trust.

9. Don't expect an advocate or advocacy project to keep silent in the face of injustice.

10. Don't forget how you would feel if your freedom was curtailed and decisions taken for you.

Bibliography

Advocacy 2000
Key Ideas on Independent Advocacy
Advocacy 2000

Butler, K, Carr S and Sullivan F (1988)
Citizen Advocacy - a Powerful Partnership
Citizen Advocacy Information and Training

Dunning, A
Citizen Advocacy with Older People - a code of good practice
Centre for Policy on Ageing

Killeen, J (1996)
Advocacy and Dementia
Alzheimer Scotland - Action on Dementia

Lanarkshire Planning Partners (1998)
Improving Advocacy in Lanarkshire - report of a search conference
SHS

O'Brien J and Wolfensberger W (1979)
CAPE - Standards for Citizen Advocacy Programme Evaluation
Syracuse, NY
www.soeweb.syr.edu/thechp

Pearpoint J, O'Brien J, Forest, M
PATH - a workbook
Inclusion Press, Toronto
www.inclusion.com

Ward, L.
Innovations in advocacy and empowerment for people with intellectual difficulties
Lisieux Hall Publications

Wesibord, M et al. (1992)
Discovering Common Ground -How Future Search conferences bring people together
Berrett-Koehler, San Francisco
www.futuesearch.net

Wertheimer, A (1996)
Speaking Out: Citizen Advocacy and Older People
Centre for Policy on Ageing

Contacts for Further Information and Help with Evaluation

Advocacy 2000
134 Ferry Road
Edinburgh
EH6 4PQ
0131 554 7878

Scottish Human Services Trust
1a Washington Court
Edinburgh
EH11 2HA
0131 538 7717

Citizen Advocacy Information and Training
Unit 162
Lee Valley Technopark
Ashley Rd.
Tottenham Hale
London
N17 9LN
0208 880 4545

Scottish Health Feedback
5 Leamington Terrace
Edinburgh
0131 228 2167

Contacts for Further Information and Help with Evaluation

Advocacy 2000
134 Ferry Road
Edinburgh
EH6 4PQ
0131 554 7878

Scottish Human Services Trust
1a Washington Court
Edinburgh
EH11 2HA
0131 538 7717

Citizen Advocacy Information and Training
Unit 162
Lee Valley Technopark
Ashley Rd.
Tottenham Hale
London
N17 9LN
0208 880 4545

Scottish Health Feedback
5 Leamington Terrace
Edinburgh
0131 228 2167

< Previous | Contents