« Previous | Contents | Next »
Listen
Chapter Two Research Methodology
2.1 This chapter provides an overview of the timetable for the research and the methodology used. It describes the design of the study, and addresses the question of including a quasi-experimental approach with Pathfinders and other authorities serving as experimentals and controls respectively. It summarises the main strands of evidence collected in terms of questionnaires sent to and interviews conducted with key personnel in psychological services, in Scotland's Colleges and in Careers Scotland and other agencies. In addition, it outlines the other sources from which evidence was obtained, including the staff of the university training programmes, educational psychologists in training, key personnel in local authorities and the Strategic Officers and those with whom they worked most directly. The procedures for obtaining meaningful and relevant experiences and views from young people in the second stage of the evaluation are also discussed.
Timetable
2.2 The timetable for the research is shown at Annex 1. The research programme allowed for 2 main evaluative exercises, the first being completed in June 2005, and the second in March 2006. The period between these 2 exercises, as well as incorporating data processing and analysis, was also devoted to carrying out ongoing aspects of evaluation. In particular, it provided the opportunity to undertake further evaluative work in areas where responses to the first exercise required to be supplemented with further data. It also allowed for continuous assessment of the ongoing contribution of the Strategic Officers.
2.3 In addition to the provision of periodic progress reports and evaluation summaries to the project steering group and the advisory group, 3 main reporting stages were incorporated into the research timetable - an inception report in December 2004, an interim report in December 2005 and a final report at the end of March 2006.
Design
2.4 The design adopted for this evaluation used a mixed research methodology in which different types of data, both qualitative and quantitative, were drawn from multiple sources. The 'triangulation' or cross-referencing of these data sources allowed each strand of data to strengthen the others in order to build up an accurate picture for the overall evaluation. The design incorporated a 'quasi-experimental' approach, that is, an investigation in which an existing experimental group is compared with a control group which is not subject to the process being studied. In the case of this evaluation the experimentals were the 12 Pathfinders, while the controls were the other authorities that did not have Pathfinder status.
2.5 Within the parameters of the research design account was taken of the following factors relating to the preparation of all evaluative measures (Bloom & Fischer, 1982): reliability - the extent to which the measures would provide consistent and dependable data; validity - the extent to which the data collected would be meaningful and relevant to the key research questions; utility - the extent to which the evaluative procedures would provide useful measures to inform future planning and decision-making; directness - the extent to which the measures clearly approximated to or accurately reflected the real situation under investigation; reactivity - the interaction between the evaluation and the results obtained, with a balance between low reactivity to avoid the investigation procedures distorting the comparisons, and desired levels of reactivity in which the process and closeness of the investigation would support, prompt and encourage monitoring of service development; sensitivity - the capacity of the data collection procedures to identify whether real processes of change had occurred; and feasibility - the robustness of the strategies used to accomplish the required task within the resources and timescale allocated, while maintaining suitable response levels and co-operation from participants.
Issues relevant to a quasi-experimental approach
2.6 The research specification for the evaluation included reference to how a control group could be selected for comparison with the 12 Pathfinder authorities. These were to be local authorities that were not Pathfinders, and they were to be monitored over the same period of time in relation to their provision of post-school psychological services. The purpose of the comparison was to gain a sense of the activity and added value that had taken place in the Pathfinders as a result of PSPS.
2.7 This issue of establishing a control group was explored further by the researchers in meetings with the Strategic Officers, and in discussion with the Research Advisory Group. The following key issues were identified in relation to the selection of 12 non-Pathfinder authorities to serve this purpose:
- an inadequate basis for matching against the 3 key comparisons required, namely, size of authority (including urban/rural balance), socio-economic level and type of psychological service
- the lack of co-terminous boundaries between local service providers and education authorities
- the non-random selection of Pathfinder authorities, since these were more likely to reflect services that had already shown interest and initiative in developing PSPS
- 'contamination' of the control group, given that, for example, part of the remit of the SOs was to promote PSPS in all authorities including non-Pathfinders
- a variety of starting dates for PSPS among the Pathfinder authorities
- the complication of evaluating the work of personnel not yet appointed to the project
- different models of service delivery within both psychological services and post-school providers
- the inherent difficulties in controlled assessment of outcomes in educational psychology, and most particularly the fact that psychological service delivery is provided mainly through support to other professionals and not directly between psychologist and young person.
2.8 Nevertheless, it was recognised that while a strict quasi-experimental study of this kind using traditional methodology could not be undertaken for the above reasons, comparisons with non-Pathfinder authorities would still be central to the evaluation. A quasi-experimental approach recognising the limitations outlined therefore continued to be relevant. In particular, pre-post comparisons within Pathfinders were expected to be greater than within non-Pathfinders. While many services had responded to the Beattie and Currie reports by looking towards the development of services offered to the post-school sector, extra resources were being provided only to the Pathfinders. Also, the local authority context had not changed as the direct statutory basis for psychological service provision remained the same as it did prior to the Beattie and Currie reports. This was likely to be a limiting factor in the extent to which authorities might be providing additional resources to services for provisions they did not view as mandatory. The key question therefore related to 'additionality': what added value had arisen from the investment of extra resources in the Pathfinder authorities?
2.9 Within this context the control group selected for the study was all 20 non-Pathfinder services. This overcame the matching difficulties that would have been implicit in seeking 12 other authorities for comparison with the Pathfinders, and allowed a comprehensive approach to assessing PSPS developments differentially across Scotland.
Evidence considered
2.10 Evidence was gathered from a very large number of sources including:
- the Strategic Officers
- Principal Educational Psychologists in both Pathfinder and all non-Pathfinder authorities
- other educational psychologists with a significant role in PSPS
- the Programme Directors and trainees in the educational psychology training programmes at Dundee and Strathclyde Universities, with particular reference to Dundee, where additional trainee places had been allocated to promote PSPS
- key personnel throughout the post-school sector
- other appropriate personnel in Scotland's Colleges, Careers Scotland, Scottish Enterprise and a range of other post-school service providers
- key personnel in local authorities
- young people, as the direct users and beneficiaries of post-school services.
Methods used
2.11 The main methods used for the study were: questionnaires; semi-structured interviews; focus groups (for example, with university trainees and with psychologists working in clusters); and examination of available documentation. Questionnaires provided a significant volume of quantitative and qualitative data, and these were supported by supplementary data from all other evidence sources.
Questionnaires
2.12 In consultation with the steering group, a questionnaire template was designed as the basis of all questionnaires that would be used during the evaluation. This was modified as required to suit the context of the groups that were being surveyed, guided by the principle that questions would only be adjusted to the minimum extent necessary for this purpose. This approach had the advantage of allowing the largest amount of comparative data to be collected from all groups. On this basis, 5 questionnaires were designed. Their content provided a suitable balance between closed and scalar questions that could be assigned scores, and open-ended questions that would draw out a range of qualitative information.
2.13 The questionnaires utilised the simple but highly informative strategy ('the three things') developed by Dunlop, Knott & MacKay (2002) in their research on social interaction and understanding in autism. This combines individual information generated by participants with a self-scoring method on a Likert scale. It has proved to have considerable utility to date in 2 studies, both in identifying qualitative factors of key significance to participants and in generating highly differentiable pre-post scores that can be subject to quantitative analysis.
2.14 Psychological services: the questionnaire for psychological services (Annex 2) was identical for Pathfinders and for non-Pathfinders. It was sent to all of the 32 psychological services in Scotland for the first evaluation in relation to 1 April 2005, and again for the second evaluation in relation to 1 March 2006. The response rate was 100% on both occasions. Its main content included:
- basic particulars - in this case mainly staffing ratios, including PSPS staffing information, to provide a context for the current situation in which services were operating
- whether post-school services were being delivered, and if so the nature and extent of these services
- changes and anticipated changes in service delivery in the areas of consultation, assessment and intervention, training and development, and action research
- barriers to service delivery and perceived gaps
- the support received from the Strategic Officers
- current effectiveness in relation to perception of ideal services
- overall perception of impact of PSPS initiative
- implications for educational psychology practice (this section was used in the second evaluative exercise).
2.15 University trainees: Following useful discussions with the entire group of trainees at the University of Dundee, an abbreviated questionnaire was prepared for all educational psychologists in training at the Universities of Dundee and Strathclyde (Annex 3). It extracted from the psychological service questionnaires those sections on which trainees placed in Pathfinder or non-Pathfinders would be able to comment on the service delivery and perceived effectiveness of these services in relation to PSPS. Returns were received from 27 trainees, representing a response rate of 53%.
2.16 Scotland's Colleges: The questionnaire for this sector (Annex 4) was of exactly the same structure and content. After gathering basis particulars relevant to the colleges, it covered the above areas, viewing the colleges as recipients of PSPS rather than as providing service delivery. This was sent to the principals of all 46 colleges in Scotland at the time of the first evaluative exercise, and also to other key personnel in each college, to maximise the likelihood of a completed return that would have involved someone reasonably well qualified to address the issues raised. By the time of the second evaluative exercise the colleges had been reduced to 43 through mergers, and the questionnaire was again issued to all of these. Returns were received from 38 of the 46 colleges for the first exercise and from 35 of the 43 colleges for the second. This represented responses from 82% of colleges.
2.17 Other service providers: The questionnaire for Careers Scotland, Scottish Enterprise and other training and service providers (Annex 5) was identical to the questionnaire for Scotland's Colleges except in detailed adjustments to suit the context of these providers. It was sent to a wide range of post-school providers in every area. The number of returns received was 49 from approximately 200 questionnaires issued, representing a response rate of about 25%. The difficulties associated with any attempt to have comprehensive coverage of this sector are covered in detail in Chapter 5.
2.18 Local authorities: The questionnaire for key personnel in local authorities (Annex 6) was sent to directors of education, of children's and young people's services and of community services. It covered level of PSPS delivery, the perceived value of the contribution being made, PSPS as a developing service within their authority, barriers to PSPS development, views of ideal services and examples of PSPS development. Returns were received from 16 authorities, representing a 50% response rate.
Semi-structured interviews
2.19 Follow-up interviews were held at 2 time points with all 32 psychological services, with 35 of Scotland's Colleges for the first evaluative exercise and with 37 for the second, and with 36 other service providers. These interviews allowed further exploration of the issues raised in questionnaires. In addition, 28 interviews were held with key personnel in relation to the work of the Strategic Officers.
Focus groups
2.20 Focus group meetings were held with the staff at Dundee University, the Dundee educational psychology trainees and on two occasions with the psychologists in the Tayside cluster.
Documentation
2.21 A vast volume of documentation relating to the initiative was gathered from multiple sources. This made a significant contribution to informing the evaluation. It included the detailed progress reports prepared by the 12 Pathfinders.
The Strategic Officers
2.22 Evidence relevant to the evaluation of the contribution of the Strategic Officers was obtained from all the evidence sources described. Because of the centrality of the role of the SOs, and the weakness of some of the quantitative sources of evidence in assessing this, a great deal of further qualitative information was generated for this aspect of the evaluation. This is covered in considerable detail in Chapter 8.
The views of young people
2.23 Taking account of the view and experiences of young people in relation to PSPS was viewed as an important part of the overall evaluation. The methods by which this might be done in a meaningful way were discussed with the steering group. It was felt that while focus group approaches that would be simple to arrange could provide information here, there would be a danger of deriving views from such groups that incorporated this strand in a tokenistic way, with limited real evidence to contribute to the evaluation.
2.24 It was therefore considered appropriate to work with psychological services and other agencies that might be able to co-operate in taking this strand forward, through work in relevant contexts with young people with whom they were already involved. Discussions were also held with other post-school providers, such as the national co-ordinator for the BRITE initiative, on possible avenues to approach this issue more meaningfully - for example, by asking for the experiences of young people in relation to structures that had been established by psychologists, such as new assessment frameworks. The view of service providers, however, was that at this stage this approach would not be a useful one. In relation to assessment in Scotland's Colleges, for example, part of the purpose was to move from singling out young people with additional needs and to make the process less conspicuous. Also, psychological services tended to be provided at some distance removed from young people as the ultimate beneficiaries of these services, making it more difficult to isolate specific contributions at this level.
2.25 The questionnaires sent to psychological services included a request for information on any steps they had taken to obtain views from young people on PSPS. This information was followed up on interview and was supported by further documentation provided by services. In addition, the Strategic Officers gathered and collated information about impact on young people for which they had already asked services. This supplemented the other sources of evidence available. The total body of evidence gathered on the views and experiences of young people and on the direct impact of PSPS was considerable.
Summary
2.26 This chapter has provided an overview of the evaluation timetable, research design and methods used. It has included discussion of the issues associated with a quasi-experimental design involving non-Pathfinders as a comparison group for Pathfinders. The place of questionnaires, semi-structured interviews, focus groups and documentation has been outlined. The procedures for obtaining relevant information and views from young people have also been discussed.
« Previous | Contents | Next »