« Previous | Contents | Next »
Listen
CHAPTER TWO: RESEARCH METHODOLOGY
This chapter describes the methodology adopted by the study for both the Transitions and the Collaborative Learning/Groupwork projects. It explores sample selection, the implementation of the project intervention, the creation of measures of impact and their administration, observation and analysis. It also highlights difficulties encountered and how these were overcome.
2.1 Sampling
The Transitions project was a follow-up study involving P6 & P7 pupils from the schools in the ScotSPRinG primary school who were followed up having experienced transition to secondary school. These pupils are referred to as 'follow-up' pupils.
The Collaborative Learning/Groupwork project was an experimental intervention study. Pupils engaged in 2 science topics taught through collaborative learning techniques. These classes are referred to as the 'experimental group'. Some pupils were experimental follow-up pupils (they had been involved in the original primary project) and some were experimental non follow-up pupils (they had no involvement in the original project). Other classes were identified who did not participate in the collaborative learning activities but undertook the same range of tests as the experimental group to act as the control group.
Of course, P7 pupils from one primary school do not all attend the same secondary school. Therefore, the secondary schools receiving the majority of P7 pupils from each primary were the focus for the project. A total of 10 relevant secondary schools were identified in the West of Scotland, and a further 11 in the East. Recruitment efforts were focused on the ScotSPRinG project participants who were now attending year 1 of the Secondary school, and some of their classmates for comparison purposes. One science class from each year was targeted in each school, but this was adapted in the light of circumstance. Where possible, this class contained the largest numbers of follow-up pupils in the original sample. The project was implemented in those classes where science teachers expressed their willingness to participate.
In the Transitions project data was collected from a total of 644 pupils. Of this total, 204 pupils formed the sample of children who were followed-up after the primary school project, with 440 pupils forming a sample of comparator/control children. In the Collaborative Learning/Groupwork project data was collected from 644 pupils. Of these 259 formed the experimental group and 385 formed the control group. In terms of school participation this meant that in the East out of 11 possible schools, there were 4 experimental schools yielding 9 experimental classes, plus 4 control schools and one control class in one of the experimental schools. Three schools declined to participate. In the West, out of a total of 10 schools, 4 agreed to be experimental schools, and 4 agreed to be control schools, while 2 schools refused to take part. However, the experimental schools in the West refused to undertake the collaborative learning / group work teaching as originally planned. Instead they chose to embed the pedagogical techniques within existing curriculum teaching materials. The project thus experienced some sampling difficulties. These had been anticipated, but their scale was perhaps unexpected. In both East and West the numbers of high schools wishing to participate was not large. In the West the absence of schools willing to use the curriculum materials was problematic. The reasons for this reluctance are explored in 2.2 below.
2.2 Implementation of the project
Key participating teachers from the receiving secondary schools were identified by the senior management team in each school as leaders in the development of group work methods in their schools. In the event, most of these teachers implemented group work in one or 2 classes they already taught, and in only a small number of cases were other teachers involved in the project. The teachers attended a total of 3 CPD days. Content was developed and delivered by staff from University of Dundee and University of Strathclyde and involved:
1. general group work strategies and activities, pupil training in the skills necessary for effective group work, with emphasis on social and communication skills;
2. specific and structured group work in science in the topics of 'Materials' and 'Earth in Space'; and
3. a final debriefing and evaluation session.
The first of these CPD days was in the autumn term, the second at the beginning of the spring term, and the last at the time of the post-implementation test. At the first 2 days teachers received resource materials and consultation. The plan was that teachers would introduce pupils to generic cooperation activities in the autumn term, and at the beginning of the spring term would start them on the science-related topic activities. In the last session, teachers took part in a brief interview session and completed a questionnaire concerning their experiences and perceptions of the group work initiative or any further issues they felt were relevant.
Initially it had been anticipated that only members of the senior management team might be able to attend the CPD for logistical reasons (supply science teachers can be difficult for schools to find in Scotland). However, this fear proved unfounded. Support from colleagues and the senior management team meant that in general teachers were able to attend the days as planned.
The materials used in the CPD days varied. For CPD day 1, the ScotSPRinG generic cooperation skills materials were felt to be too large and too generic. Consequently selections of activities were made which were not so far away from the intended focus on science and which were relevant to the secondary population. The extent to which the secondary schools used these was probably variable, and observation did not occur until the spring term, but there was anecdotal reportage of these being used in at least some classrooms before Christmas. For CPD day 2 two new sets of materials were devised, akin to the ScotSPRinG specific science materials and incorporating similar principles of group working, but at a higher level and focused on new areas in science: Science topic 1 'Earth and Space' and Science topic 2 'Materials'. Whilst there was overlap between the Materials topic and the primary school topic 'States of Matter', the overlap was limited in nature. During the CPD days the teachers engaged with some of the materials and experienced group work themselves using the materials.
In the East the high schools more or less kept to the plan, although there was some demand for additional materials of modified accessibility which had not been anticipated. However, there was considerable variation in the quality of implementation of the project. In the West the high schools were more likely to want to depart from the plan, varying what they did more considerably, and this resulted not only in more customisation of materials, but also some changes in materials. The reasons for the different approaches in the East and West were as follows. Teachers were offered curriculum packs for 'Earth in Space' and 'Materials'. These curriculum packs were closely aligned to the 'Earth and Space' outcomes in 5-14 science. However, teachers in the West of Scotland refused to synthesise these curriculum packs into their planned programme of work. The stated reason for this was that the 'Earth in Space' outcomes were not outcomes that they planned to teach that year. Rather than have no participation from schools in the West, a pragmatic decision was taken to provide training and support to help teachers implement collaborative learning into their own science topics. This kept west of Scotland schools involved, but precluded them from using the curriculum packs. This project was launched upon the participating high schools without a great deal of advance notice (given the short time scale), and a longer project with more developmental time to cultivate schools would probably have resulted in greater take-up of a somewhat more orderly nature.
2.3 Developing outcome measures
Outcome measures were taken at 2 main points linked to the timing of the CPD days: the middle of Autumn Term 2006 (before the general skills training) and end of Spring Term 2007 (after the specific skills intervention). Data collection in the middle of the Autumn term 2006 provided a baseline for the post-test.
Measures included some but not all measures from the original ScotSPRinG project, or adaptations thereof, together with some new measures. Particularly important were tests of science attainment of which 4 measures were used:
1. The Forces (16 item) science test covering an area of science involved at primary school, was administered at the pre-implementation stage only, to assess enduring knowledge on this topic surviving the transition from primary to secondary school
2. Earth and Space (30 items) and Materials (30 items) specific science tests were administered pre and post the group work intervention. These tests were developed specifically for this study to measure the cognitive gains that may be attributable to the secondary school collaborative learning project.
3. A 21 item general science test based on the AAP assessment in general science derived from the full standard AAP test was also administered pre and post the group work intervention to measure transfer of learning to wider science curriculum.
Self-esteem measures (Harter) 3 together with a measure of attitudes to cooperative learning and group work ( My Feelings About Group Work - MFAGW) 4, both used in the previous ScotSPRinG project, were used to explore affective variables that may be influenced by the collaborative learning/groupwork project. Additional measures of attitudes 5/self concept in science and pupil self-assessment of transferable skills in cooperative teamwork were also used to assess the extent that gains in pupils' attitudes towards science or as effective learners were influenced by the original primary school project or the collaborative learning/groupwork project. Attitudes towards science/self concept as a learner of science are widely regarded as important indicators of future performance in school (Marsh, 2007).
Finally, as in the ScotSPRinG project, a (modified) sociometric measure was employed in order to investigate pupils' social relationships and patterns of interaction in and outside school. People in your Class was presented in the form of a matrix and asked adolescents to consider 4 key context questions (columns) regarding their relationships with all other members of their science class (rows). People in your group asked the pupils to undertake the same task, but only for the named people in their science work group. Both instruments asked the pupils to mark all those pupils in their class / group that they:
- Worked with regularly in a group
- Liked working with in science
- Liked spending time with at break time
- Liked seeing out of school
It was specifically designed to measure the 3 aspects of peer relationships that had shown substantial pre-post implementation gains in the original primary school project.
2.4 Observations
Researchers visited secondary schools at 2 intermediate points throughout the experimental period during the spring term to offer consultative support and assess the implementation quality of group work. This involved direct observation in classrooms, using an adaptation of the observation schedule utilised in the ScotSPRinG project. The nature and role of children's interactions in group and class learning contexts was recorded. Aspects of language were classified under 2 headings, Collaborative and Tutoring, as shown in Table 2.1.
Table 2.1 Codifying aspects of language from observation
Collaborative codes - learning is co-constructed
Proposition: child suggests an idea or course of action (whether low or high level), or otherwise makes some form of statement that someone else could disagree with
Disagreement: child explicitly disagrees with a suggestion or explanation offered by another
Explanation: child offers an explanation of a proposition
Reference back/continuation: child explicitly refers back to a previous suggestion or explanation, irrespective of originator ( i.e. they must refer to the content of the previous statement and point to the fact that this is something that has been said before - saying e.g. "I think the same" is not sufficient)
Resolution/compromise: child acknowledges previous statement of other and adjusts own to include content ( i.e. there must be some explicit fusion of ideas)
Tutoring codes - learning was led and managed by one member of the group
Instruction: child tells someone to say something or carry out some action
Question: child asks open-ended question (or gives other form of prompt) that directs attention to something not yet considered ( e.g. "What about keeping weight the same?" "Do you think it would make any difference if we used something solid?"); NB the key marker here is that this is a question that the asker does not want to know the answer to (they already know it)
The role of pupil explanations, questioning and responding was of particular interest. In addition to the micro-measures of implementation of group work, the broad layout, organisation structure and management context of each classroom was mapped.
2.5 Analysis approach
Analysis was undertaken on a number of bases. The Transition project explored whether experiences in the ScotSPRinG project in rural or urban schools or in single-age or composite classes advantages or disadvantages pupils on transition to secondary. The Collaborative Learning/Groupwork project led to a range of analyses. Firstly the experimental/control differences were explored. Previous data on quality of interaction in group work in the primary school was related to outcomes in the secondary school, as was new data on quality of interaction in group work in the secondary school. This was principally done via descriptive statistics and graphs using analysis of variance for the pupils as a whole, but exploration of individual classes was also undertaken to explore variation among them.
Analysis of Variance is a statistical test. It can be used to determine whether the differences between 2 groups of numbers ( e.g. the results of pupils before and after an experimental intervention) are a result of chance, or whether any differences are probably due to a pattern ( i.e. tests results improving or getting worse). Commonly the chance (or probability benchmark) used for whether differences are as a result of a pattern as opposed to just being as a result of chance is 1 in 20 (or 5% or less). This is normally expressed as a 'p' value. A p value <0.05 indicates that a result is likely to be as result of a real, rather than a random change.
« Previous | Contents | Next »