« Previous | Contents | Next »
Listen
APPENDIX 1 - CONTINUOUS MONITORING SYSTEM
Below are a few observations on how the evaluation of xlerate with xl could be further developed. The suggestions discussed are quite involved and potentially go beyond xlerate with xl.
5.1. The findings from this research strongly support the notion that xlerate with xl is a very effective programme, and clearly has a substantial impact on many of the students - of that we have no doubt. However, we can say little about the longer-term implications, how it works, who are the most suitable students, how it compares to other programmes, and so on. In essence, our evaluation provides something of a snapshot taken over a year - a year during which the programme has developed and improved. These further questions could be addressed with the adoption of a continuous monitoring system, and a post-programme tracking system. Additionally, such processes could contribute in a formative way to the further development and improvement of the programme, as well as possibly contributing to the development of other similar programmes.
5.2. Without detracting from the very positive findings here, the world of research and evaluation in education (and other areas) has many examples of what appear at face value to be very effective interventions, but in the longer term turn out to be somewhat less impressive. This may well be due not to any fault in the design of the programme or those involved, but rather factors such as over-ambitious expectations, limited scalability potential, or honeymoon / Hawthorne 39 type effects.
5.3. It is worth briefly mentioning the salutary lessons from Joan McCord's (2003) 40 findings from the Cambridge-Somerville Youth Study. This was a very well-designed evaluation (with randomised controls) which looked at a programme for at risk teenage boys. By all accounts, the programme appeared to be a success - the boys and other stakeholders felt the various interventions had been beneficial. Indeed, follow-up research some 30 years later confirmed this view, with the participants being able to recount the experiences and help they received; and again in retrospect, they felt it had been beneficial.
5.4. However, when the control and treatment groups were compared in terms of outcomes such as crime rates, mental health and alcoholism, a somewhat different picture emerged. In the long term, no evidence was found to show that the programme had been of benefit. The message here is that although the evidence might appear to point one way, in the cold light of day, particularly in retrospect, a different and somewhat less impressive picture might emerge. In addition it is worth noting that McCord's study is something of a rarity in evaluations (certainly education) in that it covered such a large timescale.
5.5. There are, of course, many differences between this and xlerate with xl, so it would be very wrong to draw anything more than just the message that there is considerable value in taking a long-term view. And, of course, one of the main aims of xlerate with xl is to help children remain in education during S3 and S4 - clearly it does this; but still, how about the longer term, and more specifically, how does the programme contribute to the overarching long-term aim of A Smart Successful Scotland 41?
5.6. So how could these longer-term broader questions be addressed? We would suggest a two stage process: first an ongoing monitoring system while the students are at school; second, a regular tracking system after they leave the programme.
5.7. Monitoring: A regular monitoring system could be established in a number of schools which offer the xlerate with xl programme, as well as in some that offer other programmes. This would cover all year groups in S1 to S4 and beyond, and ideally, it would begin in the feeder primary schools. This would provide an ongoing profile of all of the children, including those doing xlerate with xl, as well as other programmes.
5.8. The system would provide academic and cognitive assessments, as well as softer behavioural and attitudinal measures - including those related to enterprise. They would tie in with the existing 5-14 and Standard Grade assessment framework, and where appropriate, extend to include specific school or programme factors.
5.9. Taking our ( CEM) current systems 42 as an example, these cover pre-school-age children through to S4 and beyond. The assessments can be done on an annual basis and typically take between 20 and 45 minutes. Increasingly, they are done on-line, and all the processing and initial analysis is carried out here, in Durham. Currently, CEM monitoring projects are used by a large number of schools around the world, with an increasing number in Scotland. From the individual schools point of view, they provide a variety of information on the performance and potential of their pupils and information about the school.
5.10. Tracking: This is more difficult. Information from the ongoing Scottish School Leavers Survey might be of some use in a comparative way, but in terms of evaluating the impact of xlerate with xl, it would be necessary to adopt a more targeted approach. A recently published report on the Princes Trust Leaving Care Initiative 43 demonstrates the potential difficulties. The research was funded by the Joseph Rowntree Foundation, and the researchers identified 148 ex-participants who were suitable interviewees. However, despite considerable efforts, they were able to contact and interview only 17 of them. 44 The key message is that it is important to build appropriate follow-up/contact strategies during, or ideally at the outset of the programme.
5.11. To conclude - We would suggest that a longer-term ongoing evaluation programme could positively contribute to the overall evaluation of xlerate with xl (and in some respects the broader DtS strategy), by:
1. considering longer-term outcomes ( i.e. beyond the end of the programme at S4)
2. making more effective comparisons with other similar or related programmes and with students not on the programme
3. relating the findings to ongoing student level assessments, both in terms of academic performance and ability, as well as softer enterprising qualities
4. providing some indication of the possible characteristics of xlerate with xl students in advance (as a result of continuous monitoring ideally from P1), and from that, begin to build up a picture of why and how the programme works
5. finally - this could be a part of Scottish eXchange of educational data 45 (scotXed - SEED) to help contextualise and distribute the findings.
« Previous | Contents | Next »