« Previous | Contents | Next »
Listen
Chapter 2: Sample and Methodology
This chapter describes the methodologies and samples employed in the process and outcome evaluations.
The process evaluation
The process evaluation focussed on the delivery of the projects and aimed to:
- Describe and understand the nature of the projects and their evolution over the study period (covering both their design and delivery).
- Examine how project objectives have been delivered, including organisation and staffing issues, recruitment, ongoing client contacts, monitoring and evaluation, and work with other agencies.
- Identify those factors (both internal and external) that have enhanced or hindered the delivery process.
- Identify key learning points from across the four study projects.
The research methods used included: individual in-depth interviews, analysis of available project documentation and, where appropriate, observation. An initial round of site visits was carried out in each of the case study agencies in Spring 2004 and a second set of visits was conducted in Spring 2005. Information from the profiling exercise in Spring 2003 also informed the research.
In depth interviews
The core of these visits was individual in-depth interviews which were conducted with project workers, managers and administrators from the four case study projects and, in some instances, with representatives from relevant external agencies. Service providers were contacted directly or through their line managers and interviews took place in the course of the working day, usually in the workplace. Informed consent was obtained from all respondents and a project information sheet with contact details was left with them for future reference. The interviews were semi-structured and used a topic guide to ensure that key areas of interest were covered while, at the same time, enabling respondents to introduce their own priorities. The topic guides were tailored to individual projects and reflected emerging issues from previous contacts and the profiling report. The second round of interviews included similar themes to the first round, but in addition addressed topics which emerged from analysis of the first round of site visits. The majority of interviews were carried out on a one-to-one basis but a small number of paired or mini-group interviews were also conducted.
The interviews were supplemented by a shared analysis of client case notes with project workers in each project. Workers participating in the Spring 2004 round of interviews were asked to provide access to a selection of client case notes designed to reflect the range of work undertaken by the project. The case notes were used in the interviews to provide individualised illustrations of the work of each project. In some instances the researcher had an opportunity to review the case notes immediately prior to the interview while in others they were introduced during the interviews. These reviews formed part of the interview field notes which were recorded on audio-tape. Some projects provided written case material.
Between 12 and 17 interviews were undertaken per project. Interviewees represented a range of stakeholder perspectives and included project workers, project managers, workers in the host organizations and representatives of relevant external agencies. Overall, interviews were conducted with 43 respondents over the project profiling and case study periods. (See Appendix 1 for details of the process evaluation sample)
Analysis of documentation
Available documentation relating to individual projects and to the programme as a whole was analysed. This included background strategy and planning documents, project reports, project monitoring data, written protocols and other written outputs from the projects.
Observation
It was originally intended that, where possible, observation would be used to examine aspects of the projects in action to complement the accounts of project workers. However, given the sensitive nature of the projects and the need to build and maintain a relationship of trust between project workers and clients, observation tended to be confined to the nature and location of the project premises.
All of the interviews were fully transcribed and, together with the field notes, subjected to detailed thematic analysis. Researchers also drew on data collected during the profiling exercise in Spring 2003 which had included the projects ultimately chosen as case studies. Thus this report is able to broadly reflect three years in the lives of the projects. All of the projects have continued to be funded beyond the study period, although at a reduced level.
The outcome evaluation
The outcome evaluation had two components; repeat interviews with a sample of the projects' clients and interviews with people identified by the clients as being in a position to assess the impact their participation in the intervention had had on their lives.
Client interviews
Two interviews were conducted with project clients at 8 monthly intervals. The interviews, which contained a mixture of closed and open-ended questions, were conducted by trained interviewers. They were tape recorded with the interviewee's permission and the responses to the open-ended questions were subsequently transcribed. The interviews had two main purposes. First, they were intended to provide a mechanism by which any changes in clients' behaviour and circumstances following their contact with a project might be identified. Second, they were designed to obtain clients' views on the projects and their perceptions of the extent to which they had helped them.
The interviews with the young people attending the Perth Connect, East Ayrshire and Reiver projects addressed the following topics; their use of alcohol and illegal drugs; their home circumstances; their family dynamics; their self esteem; their attendance at school; their engagement in problem behaviours; whether anyone else's drug or alcohol use was giving them problems; their friends and their friends' activities; and, their perceptions of the project and any impact it had had on their life and behaviour.
The interviews with the clients of the Aberlour Outreach project explored the following; their use of alcohol and illegal drugs; their parenting skills and participation in their children's lives; their children's exposure to drug taking and other types of risk; the impact of their drug taking upon their children; and their perceptions of Aberlour Outreach and its impact on their own and their children's lives.
While the questionnaires for the second round interviews were broadly similar in content to those used in the first round, some modification was made to the open-ended questions to enable them to build upon the first round responses. The second round interviews included clients who had discontinued contact with the projects as well as those who continued to participate in them.
Steps were taken to ensure that the interview setting was such that both interviewer and interviewee could feel comfortable and secure. In the case of interviews with young people, the interviewers were instructed that these should be conducted in sight of another adult but not within earshot. In those instances in which such arrangements could not be made, someone else accompanied the interviewer. For interviews with clients in their own homes, the interviewers observed the Centre for Drug Misuse Research's safety procedures for fieldworkers. Informed consent for their participation in the outcome evaluation was obtained from all clients. This process is described in Appendix 2.
Interviews were conducted with 42 clients in the first round of interviewing; 15 from Aberlour Outreach; 11 from the Reiver project; 11 from the East Ayrshire Family Support project; and 5 from Perth Connect. The great majority of the first round interviewees (37) participated in the second round of interviews. A fuller description of the outcome evaluation sample and its recruitment is provided in Appendices 3 and 4.
Collateral interviews
There are obvious difficulties in relying upon respondents' own accounts of their progress as a measure of outcome. However, in the context of the present research, more objective measures of outcome were deemed impractical. Instead, as a way of attempting to validate the clients' accounts, collateral interviews were conducted with individuals who were in a position to assess their progress since their contact with a project. Each client was asked to identify one or two people who could talk to the research team about how their life might have changed following their contact with the project. It was explained to them that this should be somebody, a relative or a professional, who knew them well and who might be willing to talk about their progress. It was also explained to the client that the identification of such an individual was entirely voluntary. In the event, all of those who were interviewed nominated somebody who could act in that capacity.
The collateral interviews used a series of open-ended questions to explore whether, and in what respects, the client had made progress in certain key areas. For the young peoples' projects this included questions about the clients' use of drugs or alcohol, their behaviour and activities, their friendship networks, their home life, their attendance at school and any changes in how they felt about themselves. For the Aberlour Outreach clients, interviewees were asked about the latter's parenting skills and care of their children, their use of drugs or alcohol, their mental and emotional health and their children's exposure to drugs or drug taking in the home or elsewhere. All of the interviewees were asked to comment upon the role they believed the project had played in bringing about changes in the client's behaviour. No reference was made in the interviews to any claims that the clients themselves might have made concerning their progress.
Collateral interviewees were approached for the 37 clients who participated in the second round of interviews and a total of 28 collateral interviews were conducted; 11 for Aberlour Outreach, 9 for Reiver, 7 for the East Ayrshire Family Support Project and 1 for Perth Connect. Three potential interviewees declined to be interviewed (2 for Perth Connect and 1 for Reiver) while the other 6 either proved impossible to contact or did not respond to the invitation.
These interviews proved to be of enormous assistance in validating the clients' own accounts of their experiences and progress. As we will see, they also provided alternative perspectives which informed a broader appreciation of the changes that had taken place in the young people's lives and the impact these changes had had on those around them. The success of this approach suggests that this type of interview might be useful in any evaluation which has to rely on self reported outcomes as opposed to more objective outcome measures.
The value of the collateral interviewees' contributions was to some extent dependent upon their relationship to the client. As we will see, some were better placed than others to comment upon the client's behaviour and circumstances. For example, the Reiver mothers appeared to have a fairly detailed, though not comprehensive, awareness of the progress their son or daughter had made, while friends of the Aberlour parents had some difficulty in commenting on the client's parenting and child protection practices. Ideally, a collateral interviewee should be in a good position to observe the young person's progress while, at the same time, being independent of them in the sense of not being affected by it. In practice, of course, these two requirements tend to be incompatible since it is precisely those who are in the best position to comment on clients who are most likely to be affected by their behaviour.
There might be an argument for determining in advance who the collateral interviewees should be for particular groups of clients. However, for the present study we judged that the sensitivities associated with the clients' circumstances were such that this would not be acceptable ethically. Nor would it have been especially practical since it would have been likely to result in substantial numbers of clients withholding their consent for us to approach the nominated individual. Instead, we invited the clients to select someone who was both in a position to comment upon their progress and acceptable to them. However, not only was this the only tenable option in an ethical and practical sense, it also ensured a spread of interviewees with different perspectives on the clients. This, in turn, created the potential for making comparisons between different categories of collateral interviewee. While the small numbers in the sample meant that this diversity was not something we were able to exploit in a systematic way in the present study, it is something that future work using larger numbers of collateral interviewees might wish to consider.
Information from case records
In addition to the interviews, it had been intended that selected information would be obtained from the client records held by the agencies. Most clients had been in contact with the agencies for varying amounts of time before our first interview with them. This meant that it was not possible to obtain reliable data in the interview on their behaviours and circumstances at the time of their initial contact with a project. This information was clearly important as far as any attempt to monitor clients' progress over the period of their engagement with a project was concerned. In order to obtain this baseline information, it was hoped to collect data from clients' records and from other monitoring instruments held by the projects regarding their behavioural status at the time of their initial contact with the agency and to use this data to provide a baseline of information against which clients' progress over time could be measured.
Unfortunately, this proved to be unsuccessful. With the consent of the agencies and of individual clients, individuals' case records and any instruments used to monitor their progress were reviewed by a member of the research team to ascertain whether these could provide the sort of baseline information that was sought. This revealed that the information that was required was not available in a sufficiently structured or systematic form for us to be able to use it in the way in which we had intended. Where appropriate information was available, it tended to be patchy. There was little systematic or substantial recording in agency records of the progress made by clients. In addition, the monitoring instruments that we thought might also provide a way of tracing clients' progress over time were either administered inconsistently or not at all. Of course, the content of the records kept by the agencies has substantive as well as methodological implications. In particular, it raises questions about how the project workers themselves keep track of clients' progress and this is a theme that is addressed in the subsequent analysis of the process evaluation.
« Previous | Contents | Next »