The Scottish Government

« Previous | Contents | Next »

Listen

CHAPTER THREE METHODOLOGY

Introduction

3.1 To meet its aims and objectives, the evaluation comprised three overlapping phases. Phase 1 focused on scoping and mapping the development and implementation of the campaign. Phase 2 was concerned with collecting data against which to assess effectiveness. The third phase was forward focused, with the aim of collecting data to inform draft recommendations for the future direction of work in Scotland to address the stigma and discrimination experienced by people with mental health problems.

3.2 The Scottish Executive research specification set out in very broad terms the range of research activities it was anticipated would be undertaken to address each of the evaluation's five objectives. These are summarised in Annex 2. The methods used included documentary analysis, face-to-face and telephone interviews, workshops, surveys and cases studies. The methods used and the range of participants involved at each phase is described below and summarised in Annex 3.

3.3 It should be noted that, although described as discrete phases, there was a degree of overlap between the different elements of the evaluation. People interviewed or participants in focus groups in phases 1 and 3, for example, were also given an opportunity to indicate their perception of the effectiveness of the campaign, although this was the primary focus of phase 2.

Phase 1: Scoping and mapping the development of 'see me'

3.4 Phase 1 was designed to address the evaluation's first aim of co-ordinating a detailed account of the development of 'see me', including examining how the campaign was established, funding allocated, its infrastructure developed and its activities decided upon. A range of different methods were used to build up a picture of the campaign's development and implementation. These included:

• Analysis of documentary material
• Interviews with key Scottish Executive decision-makers
• Interviews with other National Programme delivery agencies
• Interviews, workshops and group interviews with those directly involved in the management and operation of 'see me'
• Survey of NHS Board areas and workshops in three local areas and interviews with organisations who had implemented anti-stigma activities with support from 'see me'.

Documentary analysis

3.5 Documentary evidence covering the period from the early discussions between the alliance and the Scottish Executive in 2001 through to September 2006 was analysed. The aim was to use this contemporaneous material to develop an account of decision-making processes, rationales and outcomes, relating to the campaign as a whole and to the different campaign strands. The 'see me' campaign provided copies of around 750 documents. This was supplemented by material held by the Scottish Executive relating to the setting up of the campaign pre-2002. The material covered:

• The development of the initial proposal for the campaign
• Management meetings (including minutes and supporting documents)
• Organisational and 3-year plans
• Commissioning and management of the Communications Agency supporting the campaign
• Campaign strand development
• Funding and budgets
• Media links (including information relating to media volunteers and Stigma Stop Watch)
• Research and evaluation reports commissioned by the campaign.

3.6 To illustrate the strategic and creative decision-making processes informing the direction, design and implementation of specific campaign strands documentary material relating to the employment and workplace strand was scrutinized.

3.7 Following preliminary examination the documentary material was coded using the qualitative data analysis package NVivo to identify key themes.

Interviews with key Scottish Executive decision-makers

3.8 To obtain an understanding of the context and rationale behind the decision to fund the campaign, face-to-face interviews were undertaken with key Scottish Executive stakeholders. These included the Health Department Branch Head involved in the initial funding decision, the Director of the National Programme and the member of the National Programme team who acted as Scottish Executive 'observer' on the 'see me' management team until 2006.

Interviews with other National Programme delivery vehicles

3.9 The National Programme comprises a number of different strands, each supported by a different delivery vehicle, some of which have been established specifically as part of the National Programme, others comprise vehicles with a wider remit commissioned to undertake a specific activity on behalf of the National Programme ⁵. To examine the nature of integration and co-ordination at local and national levels between 'see me' and these other vehicles, eleven telephone and face-to-face interviews were undertaken with representatives of eight of these vehicles ⁶:

• Breathing Space
• Choose Life (two interviews)
• HeadsUpScotland
• Scottish Recovery Network
• National Resource Centre for Ethnic Minority Health (two interviews)
• NHS Health Scotland
• Scottish Centre for Healthy Working Lives (two interviews)
• Scottish Executive Health Promoting Schools Unit.

3.10 Where more than one representative of an agency was interviewed this was either because of the different operational or strategic roles of those interviewed or because different people were involved at different times over the five-year period of concern to the evaluation.

3.11 The issues covered in these interviews included:

• The nature and extent of integration and co-ordination between their own agency and 'see me' at national and local levels
• The degree of perceived influence on the activities undertaken by 'see me'
• Perceptions of the focus and approaches adopted by the campaign
• Perceived impacts of the campaign
• Views on the future of anti-stigma and discrimination work in Scotland as it affects people with mental health problems.

Individual and group interviews and a workshop with those directly involved in the management and operation of 'see me'

3.12 Interviews, a workshop and a focus group were undertaken with people directly involved in the strategic and operational development and implementation of 'see me'. The aim was to map the structures and processes of decision-making, the rationales for decisions made and the links between these and the activities undertaken.

'see me' management group interviews

3.13 A group interview was held with members of the 'see me' management group in December 2006. The workshop covered the following areas:

• The origins and early development of 'see me'
• The aims, objectives, values and principles behind 'see me'
• The purpose and functions of the 'see me' alliance, and the benefits and limitations of this model
• Funding arrangements and relationships with the Scottish Executive
• The campaign development process and 'see me' 'brand'
• The balance between national and local anti-stigma activity
• Perceptions of the impact of the 'see me' campaign to date
• The future direction of 'see me' and anti-stigma and discrimination work in Scotland as it relates to people with mental health problems.

3.14 Three of the five members of the 'see me' management group participated in the group interview. The remaining two took part in telephone interviews.

Interviews and workshop with the 'see me' staff team

3.15 A one-day workshop was held with the 'see me' staff team in November 2006. The workshop covered the following areas:

• The purpose of 'see me' and the nature of the stigma and discrimination that 'see me' aims to address
• Roles and responsibilities within the 'see me' staff team
• Agencies and organisations within the wider 'see me' network
• 'see me''s theory of change, and barriers and facilitators to achieving change
• Key learning points from 'see me' for future anti-stigma and discrimination work in Scotland.

3.16 Five members of the 'see me' staff team participated in the workshop.

3.17 In addition to this workshop a preliminary interview was undertaken with the Campaign Director and Deputy Director, followed by an in-depth interview with the Campaign Director. This covered definitions of stigma, campaign objectives, factors influencing the focus and direction of the campaign, accountability and decision-making structures, links with other National Programme delivery agencies and views on the future direction of anti-stigma and discrimination work in Scotland as it relates to people with mental health problems.

IAS Smarts group interview

3.18 A group interview was held in November 2006 with three members of the IAS Smarts team who had worked on the 'see me' campaign. The interview covered the following areas:

• The perceived purpose of 'see me' and IAS Smart's role within this
• The key messages the campaign aims to deliver and the process for developing, delivering and evaluating these messages
• Methods of working with 'see me'
• Perceptions of the impact of the campaign on the media, general public and their own organisation
• The barriers and facilitators to implementing the 'see me' campaign
• The future direction of 'see me' and anti-stigma and discrimination work in Scotland as it relates to people with mental health problems.

Surveys, workshops and interviews with organisations in the field

3.19 To obtain a snapshot of the pattern of implementation of the campaign across Scotland a survey of health board areas was undertaken followed by workshops in three localities. Interviews were also held with four organisations with whom 'see me' had worked.

NHS Board area survey and workshops

3.20 Invitations to complete a web-based survey were issued to contacts in all 15 NHS Board areas in Scotland, including the State Hospitals Board for Scotland. 'see me' provided the evaluation team with the details of relevant contacts in each of these areas.

3.21 Issues covered in the survey included:

• Examples of anti-stigma or discrimination work in the local areas in relation to people with mental health problems
• The nature of 'see me''s involvement in this local anti-stigma / discrimination work
• Structures, processes and relationships, both between 'see me' and local areas and within local partnerships
• Factors influencing local decision-making
• Target groups
• Involvement of users and carers in the design and development of local 'see me,' activities
• The extent to which local 'see me' activities have been/are being evaluated
• Future directions for anti-stigma and discrimination work in Scotland as it relates to people with mental health problems.

Responses were received from 13 Boards.

3.22 To provide more detail on local activity, follow up workshops were held in three of the NHS Board areas responding to the survey: Greater Glasgow and Clyde, Tayside and Western Isles. These three areas were selected to enable an exploration of any differences between island, rural and urban/suburban areas.

3.23 A total of 41 individuals participated in the case study workshops across the three areas; 11 in the Western Isles (over two meetings), 14 in Tayside and 16 in Greater Glasgow and Clyde. All workshops included representatives from statutory and voluntary mental health agencies, and mental health service users. Other participants included complementary therapists, college lecturers, youth group leaders, representatives from black and minority ethnic ( BME) community organisations and forensic services.

3.24 The issues covered in the workshops included:

• Local perspectives on priorities for anti-stigma and discrimination activities in relation to people with mental health problems
• Local anti-stigma and discrimination activities and approaches
• The role of 'see me' in supporting the development of local anti-stigma activities
• Obstacles and opportunities which exist when developing anti-stigma work
• Achievements to date
• Recommendations for future anti-stigma and discrimination action at local and national levels in relation to people with mental health problems.

Interviews with organisations that have developed mental health related anti-stigma activities

3.25 To supplement data on geographical patterns of implementation, interviews were held with representatives from four organisations across the public and private sectors with whom 'see me' had worked: Amey Roads, Bell College, Birchwood Highland and Standard Life. The purpose was to explore processes and structures for implementation in discrete organisational contexts. The organisations were identified in the course of the analysis of the 'see me' documentation and contact details provided by the campaign.

The interviews covered:

• The reasons and processes for making contact with 'see me'
• The position of anti-stigma campaigning as it relates to mental health problems within the organisation's wider priorities
• The anti-stigma activities developed with 'see me', including resources required
• Perceptions on the impact of these activities in terms of workforce attitudes to mental health and organisational practice
• The barriers and facilitators to implementing the 'see me' campaign within the organisation
• Plans for future anti-stigma work within their own organisation and views on the future direction of 'see me' and activities to address the stigma and discrimination experienced by people with mental health problems in Scotland.

Phase 2: Assessing the effectiveness of 'see me'

3.26 Phase 2 sought to address the second aim of the evaluation which was to determine the extent to which 'see me' has met its own strategic objectives to date and identify the factors that have facilitated or acted as barriers to effectiveness. This included assessing the effectiveness of the different campaign strands, whether and how the practice of media professionals had changed and the experiences of media volunteers.

3.27 In addition to material collected in the course of phases 1 and 3, phase 2 contributed to an understanding of the perceived effectiveness of the campaign. Phase 2 comprised:

• Collation of existing sources of quantitative data indicative of campaign awareness

• A web survey of media professionals

• A content analysis of print media
• Interviews with 'see me' media volunteers
• Focus groups with mental health service users and carers.

Collation of existing sources of quantitative data indicative of campaign awareness

3.28 Data from the national 'Well, What do you think?' survey as well as surveys commissioned by 'see me' were analysed to indicate the awareness and reach of the campaign.

Web survey of media professionals

3.29 To assess whether and how the reporting of mental health issues had changed since the launch of 'see me' a brief web-based survey of media professionals was undertaken, including those in the print, local and national radio, TV, internet media and working in different arena, for example, business, young people or the general public.

3.30 To identify a sample of media professionals, and to encourage completion of the web survey, a two-pronged approach was attempted.

• A link to the web survey was made via the Scottish Branch of the National Union of Journalists, who also forwarded the link to local branches
• From a database of 270 media professionals compiled by a member of the evaluation team from source material in the National Library of Scotland and a database provided by an SDC associate, an initial sample of 40 professionals was contacted by email and invited to participate in an interview and/or to complete the web survey. Following an initial limited response the invitation was extended to the remaining names on the database.

3.31 The web survey covered:

• Personal experience of working on a story with a mental health component
• Sources of guidance on mental health issues
• Awareness/usage of media guidelines on reporting suicide and mental health
• Awareness of the 'see me' campaign
• Awareness/experience of specific aspects of the campaign
• Impact of the campaign on own reporting.

3.32 Notwithstanding this extensive trawl the web survey only generated eight completed responses. Of this group, two indicated an initial preparedness to also be interviewed. One person subsequently took part in an in-depth telephone interview; the second person withdrew.

Print media content analysis

3.33 To obtain a sense of whether and how media reporting had changed over the period immediately prior to the campaign to early 2007, articles and headlines from three Scottish newspapers at three time periods were analysed. Researchers managed by Rethink undertook this stage of the evaluation.

3.34 The newspapers were selected to include national and regional examples, tabloid and broadsheet, covering different political perspectives and daily and Sunday publications. The newspapers selected comprised:

• The Daily Record
• Sunday Mail
• The Scotsman
• Scotland on Sunday
• The Aberdeen Press and Journal.

3.35 For the purposes of the analysis, the Daily Record and Sunday Mail were combined, as were the Scotsman and Scotland on Sunday.

3.36 The three time periods were selected to include a period immediately prior to the campaign and two subsequent periods, but to avoid campaign launches or periods of high profile activity. The three time periods were:

• 1 ^st January to 28 ^th February, 2002
• 1 ^st April to 31 ^st May, 2004
• 1 ^st January to 28 ^th February, 2007.

3.37 Articles were identified and accessed using Lexis Nexis, a professional database of newspapers. The search terms chosen were:

• 'Mental Health' and/or 'Mental Illness'
• Nutter' and
• 'Psycho'

3.38 Search terms appeared in the key word index, the headline or in the text of the article. These search terms were chosen to provide a broad coverage of articles relating to mental health.

3.39 Articles/headlines were excluded if following a preliminary analysis they did not refer to mental health or mental illness ( e.g. if the article was about learning disability). Similarly, if the term 'psycho' referred to the film title or was part of a longer word, these articles/headlines were excluded.

3.40 The headlines and content of the material identified were rated as 'positive' if they were felt to portray people with mental health problems in a balanced affirmative light; 'negative' if they appeared to refer to people with a mental health problem in a stigmatising manner, including use of pejorative terms; or 'neutral' if they felt to be impartial/factual in nature. If a rater was uncertain about a category, allocation was made following discussion with co-raters. In addition the material was analyzed to identify:

• References to mental health agencies e.g. 'see me' or SAMH etc.
• Note if any contact details for support organisations are included e.g. 'see me' or mental health services
• The use of a core set of 'discriminatory' terms including 'psycho', 'schizo', 'nutter' and 'loony' and the context of this language use e.g. whether these terms related to violence.

3.41 For 'nutter' and 'psycho' references the following were also recorded:

• Frequency of use in headline
• Frequency of use in body of text
• Context of use.

3.42 In addition to a quantitative analysis the material was subject to a broad content analysis.

Interviews with media volunteers and media trainer

3.43 An important part of 'see me''s activities has been to develop a pool of media volunteers from across Scotland comprising people with lived experience of mental health problems who are willing to talk to the media about their experiences of stigma and mental health. This approach not only enables people with first hand experience to be directly involved in the campaign, it also reflects the evidence which suggests that direct contact with someone with a mental health problem is an effective way of influencing people's attitudes (Thornicroft, 2006).

3.44 The 'see me' media volunteer programme provides both training and support for the media volunteers, and manages the requests from the press, radio and television for contributions from the volunteers. The programme was initially set up to support 'see me''s activities, but was subsequently extended to cover other elements of the National Programme. At the time of data collection in August 2007 there were approximately 55 media volunteers in total.

3.45 To explore the extent to which the media volunteers felt able to make an impact in terms of raising awareness about stigma and discrimination and changing attitudes, interviews were undertaken with a sample of the volunteers and the media volunteers' trainer.

3.46 The evaluation team provided information sheets and consent forms for distribution by 'see me' to the media volunteers who acted, or had acted specifically on behalf of the campaign. Of the media volunteers approached, eight agreed to take part in a telephone interview. The interviews covered the following areas:

• Reasons for becoming a media volunteer and perceptions about the nature of the volunteer role
• Views on the training, preparation and ongoing support received from 'see me' to equip them for the media volunteer role
• Extent of, and views on opportunities to act as a media volunteer
• Perceptions of the impact of being a media volunteer
• The most rewarding and challenging parts of being a media volunteer
• How, if at all, would they like to see the role of media volunteers in particular, and 'see me' as a whole, develop in the future.

3.47 A telephone interview was also undertaken with the media trainer engaged by 'see me' to provide media training as part of the preparation for the media volunteer role. The interview covered the perceived need for, and impact of, the 'see me' media volunteers, the process of training volunteers and the content of the training course and views on the core skills, experience and attributes required, if any, to be a 'see me' media volunteer.

Focus groups with mental health service users and with carers of people with mental health problems

3.48 Focus groups with users of mental health services and with carers of people with mental health problems were held to obtain the views of people with experience of mental health problems and carers on the campaign as well as on its future direction.

3.49 The focus group with users of mental health services was organised through the Glasgow Association for Mental Health and held in July 2007. The group comprised seven service users, one carer and two support workers. A mental health carers group was organised through Edinburgh Carers' Council and was attended by four people and one support worker.

Both focus groups covered the following issues:

• Awareness of the 'see me' campaign and views on whether it has had an impact
• The areas of people's lives that they feel are most affected by stigma and discrimination due to mental health problems
• Actions that are required to further tackle stigma and discrimination
• Priority actions for addressing the stigma and discrimination experienced by people with mental health problems.

Phase 3: Developing recommendations for the future of anti-stigma and discrimination work in Scotland as it relates to people with mental health problems

3.50 The aim of the third phase of the evaluation was to collect data to help inform the development of draft recommendations for the future of anti-stigma and discrimination work in Scotland as it relates to people with mental health problems. The focus was, therefore, not just the future direction for the 'see me' campaign, but a wider agenda with implications for a range of sectors and spheres. Phase 3 involved a two-step process.

3.51 Step 1 comprised telephone interviews with a range of agencies and organisations with a stake in anti-stigma and discrimination work either specifically in relation to people with mental health problems or in relation to the other groups at risk of stigma and discrimination (some of whom may also have mental health problems). The groups were selected following discussions with the Scottish Executive and members of the Research Advisory Group, and included:

• Alzheimer's Scotland
• Compass (Asylum Seekers and Refugee Mental Health Liaison Team)

• Depression Alliance Scotland
• Disability Rights Commission
• Dundee Young Carers
• LGBT Youth
• Mental Welfare Commission
• Mental Health Foundation

• Glasgow Mental Health Partnership
• Poverty Alliance
• Scottish Independent Advocacy Alliance.

3.52 The interviews covered the following areas:

• Views on the nature/sources of discrimination and stigma experienced by people with mental health problems and their families and carers
• Views on target groups and/or settings which should be the focus for work in the future to address the stigma and discrimination experienced by people with mental health problems
• Priority areas for future action to address the stigma and discrimination experienced by people with mental health problems
• Views on who is responsible for addressing the stigma and discrimination experienced by people with mental health problems
• Perceived scope for integrating activities targeted at addressing the stigma and discrimination experienced by people with mental health problems to other anti-stigma and discrimination activities.

3.53 Step 2 comprised interviews targeted at key sectors with a role in supporting and implementing future work in Scotland to address the stigma and discrimination experienced by people with mental health problems. The aim was to ensure that these sectors or spheres were able to inform and shape the draft recommendations at an early stage. The sectors, again identified following discussions with the Scottish Executive and Research Advisory Group, included

• Health
• Social Care and Social Work
• Regeneration
• Police and Prisons service
• Employability/Trades Unions.

3.54 In practice it was not possible, within the time remaining to the project to recruit representatives from across all these sectors, particularly in the area of employability. The agencies and organisations that were consulted included:

• Care Commission

• Communities Scotland

• Glasgow Housing Association
• Jobcentre Plus
• Trades Union
• Her Majesty's Inspectorate of Constabulary
• Her Majesty's Inspectorate of Prisons
• Greater Glasgow and Clyde Mental Health Partnership
• Social Work/ ADSW.

3.55 The interviews covered the following areas:

• Views on mental ill-health related stigma and discrimination in the interviewee's field, and steps taken, if any, to address this issue
• What are the barriers and facilitators to addressing stigma and discrimination
• Approaches to addressing stigma and discrimination which are most likely to be effective in the interviewee's field
• Perceived scope for integrating activities aimed at addressing the stigma and discrimination experienced by people with mental health problems with other anti-stigma and discrimination activities.

Analysis of material

3.56 To analyse the material collected in the course of the evaluation a two-stage approach was adopted. The first stage comprised a within-phase analysis. Material collected in the course of phase 1, for example, including the documentary material, interviews and group interviews, workshops and survey material were analysed to draw out main themes to inform the aim of mapping the development of the 'see me' campaign.

3.57 Stage 2 consisted of an across-phase analysis drawing on data collected in the course of all three phases of the evaluation. These data were coded using a standardised coding frame. The coding frame comprised higher-level codes based on the questions to be addressed and familiarity with the themes based on the stage one analyses. The data were then analysed to identify sub-themes. Using this coding frame enabled an in-depth qualitative analysis of 'effectiveness' that drew on the breadth of material collected.

Evaluation strengths and limitations

3.58 With any evaluation of this kind there are strengths but also inevitable limitations, some which stem from the nature of the programme being evaluated, some from the scope and resources available to the evaluation itself.

3.59 As indicated in Chapter 1 the aim of the evaluation was to assess the extent to which 'see me' had met its own strategic objectives. The largely qualitative nature of these objectives does however make it more difficult to draw unequivocal conclusions about the extent to which the campaign has or has not met its own objectives: there are no quantitative targets which can be used as a proxy indicator of possible impact - leaving even more open the potential for differing interpretations of what the data are saying. Further, the objectives are very broad. This means that it becomes difficult to establish the boundaries of what is properly within the remit of 'see me' and what lies outwith the scope of the campaign to address. This open-endedness means that if, for example, the campaign is perceived not to have achieved something it is unclear whether this indicates an area where the campaign has not been so effective, or in fact is not an area that the campaign is intended to address. This latter point and the possible implications for how 'see me' frames its objectives in the future will be returned to in the conclusions and recommendations.

3.60 A second potential limitation, and one common to any initiative that is part of a wider public health programme, is the difficulty of attribution. The National Programme and the different elements within it, including 'see me', will both have influenced and be influenced by the wider policy and cultural environment. This makes difficult the attribution of specific effects to 'see me'. Further, given this wider context, the extent to which some of these changes would have occurred anyway, irrespective of a campaign, is also difficult to assess with any degree of certainty.

3.61 A third limitation relates to the small numbers of participants, particularly in relation to certain groups such as the media. It does mean that some of the conclusions, both positive and negative, are based on small sample sizes. Further, although attempts have been made to include as extensive a range of stakeholders as practicable, it is recognised that claims cannot be made for the 'representativeness' of these groups or of the individuals speaking on their behalf.

3.62 These limitations are, however, counterbalanced by its strengths: in particular the wide range of different organisations and agencies it was possible to include in the process and the range of different methods employed. In addition to interviews and surveys, quantitative data from the national 'Well? What do you think?' survey as well as from surveys commissioned by 'see me', have also been analysed. This breadth of coverage in terms of participants and methods does mean that the findings and the conclusions to be drawn from these reflect not individual personal perspectives but emerge from an extensive data source, encouraging confidence in the evaluation's conclusions. Further the input received both from 'see me' and from the Scottish Executive and the Research Advisory Group throughout the course of the evaluation have provided an external source of scrutiny.

« Previous | Contents | Next »