The Scottish Government

« Previous | Contents | Next »

Listen

CHAPTER THREE: RESULTS AND FINDINGS

3.1 Introduction

This chapter will first consider the results for the experimental pupils who were followed-up from primary schools. It will then consider the results for the pupils who formed the experimental group in the secondary school - first at the overall level and then at the class level - and compares these to the control pupils. Finally, it considers the relationship between the outcomes achieved and the process skills that were observed to have been used. In each section, a narrative summary of the results is provided first for readers who are less interested in the statistical detail that follows. An overall summary is provided at the end of the chapter.

In many of the detailed sections, descriptive statistics are supplemented with the inferential statistical test of analysis of variance ( ANOVA). Results of the analysis of variance are expressed as F (the base statistic), with an indication in brackets of the degrees of freedom (approximating to the numbers of cases involved in the calculation) and a probability (expressed as a number, where any number less than 0.05 indicates statistical significance ⁶). Some of the analyses of variance are one-way (a single comparison), while others are more than this (involving simultaneous comparisons between more than one variable) - this is indicated in the text.

The following reminders may be useful in the interpretation of what follows:

1. Follow-up pupils were those who had been part of the previous project in primary school and had been pursued after they had undergone transition to secondary school. Non follow-up pupils were those who were not involved in the previous project. Pupils involved in the Transition Project fall into one of these 2 categories.
2. Experimental pupils were those involved in the secondary school collaborative learning / group work project.
3. Control pupils were those not involved in the collaborative learning/group work project in secondary school.

It is possible therefore to identify 4 groups of pupils: follow-up pupils who were not involved in the secondary project (follow-up controls) contrasted with non follow-up controls and follow-up pupils who were involved in the secondary project (experimental follow-up pupils, contrasted with experimental non follow-up pupils).

3.2 Transition project

3.2.1 Summary

In summary, the ScotSPRinG follow-up pupils did significantly better than the non-follow-up pupils on a test of the specific science they had encountered in a group work context in primary school (the Forces test). The follow-up pupils also did significantly better than the non-follow-up pupils on one specific test of the science encountered in a group work context in the secondary school (the Materials test). This was evidence of the generalisation of group work skills to new curricular material.

During the secondary collaborative learning project the experimental follow-up pupils did not do better on the other specific test of science ( Earth and Space) than the experimental non follow-up pupils. Additionally, there was no evidence that the experimental follow-up pupils had any advantage in general science or in attitudes to science. Nor were there any significant findings on: feelings about group work, transferable skills in cooperative learning and self-esteem. However, there was some evidence of significant variation in the nature and type of social relationships formed by follow-up and non follow-up pupils. The follow-up pupils showed a stronger inclination to form positive social relationships with peers from the science work group rather than the class in general. These findings related to both children that they reported that they liked to play with at breaks and those whom they reported that they liked to spend time with outside of the classroom.

3.2.2 Detailed findings

Comparison of follow-ups with non-follow-ups at pre-test in secondary school identified a range of results with a mixture of significant and non-significant findings. Looking at the tests of attainment, there is evidence of superior performance in the attainment test of the Forces and in one of the specific standardised science tests developed for the secondary project, Materials as shown in Table 3.1.

Table 3.1 Significant differences in test performance

Follow-up pupils outperformed non follow-up pupils in the Forces and the Materials tests even when controlling for age/class achievement. This suggests that the primary project did indeed have a continuing effect into secondary school. Further, this effect was not only an effect in the area of follow-up, but also a generalised effect into a new area of the science curriculum covered by the Materials test which had some overlap in terms of content with one of the science topics (States of Matter) previously covered in the primary school project.

However, there was no evidence that follow-up pupils had any advantage in general science or in the Earth and Space test.

Turning to variables concerning attitudes, skills and feelings no significant difference between follow-up and non-follow up pupils at pre-test stage were noted as shown in Table 3.2.

Table 3.2 Pre-test results with no significant difference (p=ns)

Perhaps surprisingly no significant differences were noted regarding feelings about group work, transferable skills or self-esteem. In addition, no significant difference was evident on attitudes to science. However, this begs the question of whether respondents were considering science as it was in primary school or science as it was in secondary school.

A range of other relationship variables were examined which demonstrated no significant difference between follow-up and non-follow-up pupils as shown in Table 3.3.

Table 3.3 Pre-test results showing no significant variance (p=ns)

However, there was some evidence of significant variation in relation type between follow-up and non follow-up pupils. The follow-up pupils showed a stronger inclination to focus on the group rather than the class, especially in relation to the percentage of the science work group that they reported that they liked spending time with out of class. Taking the sociometric variables and aggregating them into relationship type (work vs. play) and focus (class vs. group), considering overall relationships (based on 174 follow-up vs. 306 non-follow-up cases), significant differences were found with follow-up pupils reporting:

• more positive work relationships with members of their science work group (F(2,956) = 257.63, p < .001);
• more positive personal relationships overall within the class (class vs. group) (F(1,478) = 284.69, p < .0010); and
• more positive work relationships within the class (relation type x class vs. group) (F(2,956) = 28.88, p < .001).

Other related comparisons were non-significant.

Thus in general, the follow-up pupils and the non-follow-up pupils appeared to be fairly similar at pre-test in this project (which was good for the next stage of the project), with the exception of performance in the Forces test, the Materials test, and some of the relations items, where the follow-up pupils showed a sustained advantage.

3.3 Collaborative learning/Groupwork project

3.3.1 Summary

Experimental pupils increased in terms of their science attainment scores ( AAP) during the course of the project. However, there was no evidence of a significant gain in attainment for the experimental pupils in comparison to control pupils. This was partly due to the fact that control groups also increased in attainment. There was some evidence of an experimental effect in Materials, but this was largely owing to the follow-up pupils rather than the non-follow-up pupils who were also experimental pupils in the secondary collaborative learning project, with non follow-up pupils remaining static. Thus this finding really belongs to the previous section, as the performance of these pupils was influenced strongly by the previous project that they had undertaken in primary school. In Earth and Space the experimental groups did not show significantly greater gains than the control groups. However, this was predominantly due to the fact that follow-up controls performed well in the post-test. In this topic the follow-up pupils (combined follow-up experimental and control) showed significantly greater gains than the non follow-up controls. This finding indicated that the enduring effect of the original primary school intervention was stronger than the effect of the secondary school intervention. Attitudes to science, self-esteem and transferable skills in cooperative learning showed no significant differences. For attitudes to group work, pupils in the experimental group decreased while pupils in the control group increased, but again this might have been partly due to follow-up pupils increasing while non-follow-up stayed static.

On the relational measures, the percentage of the group that pupils reported that they liked working with in science increased in the control group more than the experimental group. However, in both the other relational measures, the percentage of the group that the pupils reported that they liked spending time with at break and also the percentage of the group that the pupils reported that they liked spending time with out of school, the experimental pupils increased significantly more than the control pupils. There was some indication that the experimental follow-up pupils tended to focus upon group relations rather than relations with the whole class at pre-test. By post-test, the experimental non-follow-up pupils also tended to have shifted in the same direction. At pre-test rural pupils performed more highly than urban pupils on Forces, Earth and Space, Materials and general science, with the second and fourth reaching statistical significance. This suggests that rural pupils are not disadvantaged on entry to secondary school. There was some evidence that the groups were working, with propositions, explanations, disagreements and continuations (see exemplifications of codes in methods section) significantly more frequently in group as compared to whole class sessions. However, the groups were not working as well as at primary level. There appeared to be a lack of discussion taking place in the classroom. Discussion may be an important factor in promoting attainment as it was found to have a positive influence on attainment scores on both science topics and post-test attitudes to science.

3.3.2 Detailed findings

Turning to the interactions between experimental and control in considering the pre vs. post test scores, disappointingly there were little signs of a significant gain in science attainment for the experimental pupils in relation to control. This was partly due to the fact that control group scores also increased, from a slightly lower baseline. There was some evidence of an experimental effect in Earth and Space where standardised gains for follow-up control pupils were significantly greater than those for non follow-up control pupils (F(1,160)=4.909, p<0.05). A similar effect was observed in Materials, due to gains in test scores in the experimental group and by gains in test scores from follow-up pupils, who had undertaken a topic related Materials in the primary school project, over inflating the scores of the control group. Pupils in the control group who were not follow-up pupils showed no attainment gains in Materials. This suggests some kind of continued group work impact being evident into secondary school.

The scores obtained in General Science, Earth and Space and Materials are shown in Table 3.4.

Table 3.4 Pre-test and post-test gains in General Science, Earth and Space and Materials

Table 3.5 looks at the significant differences raising from these test scores.

Table 3.5: Significant assessment results

In general science the experimental group start with higher scores but make lower gains while the control group show greater progress. These remained significant when adjusted for age/class achievement. In other words, there is some evidence that the controls show a tendency to catch up with the experimental pupils.

On Earth and Space both the experimental follow-up pupils and the control follow-up pupils had significantly higher gains that the control non-follow up pupils This indicated that not only was there an experimental effect of the secondary school project, but there appeared to be a residual effect of the primary school project evident within the follow-up control pupils.

However, on Materials, follow-up pupils progressed, while non-follow-up were static by comparison. This was also true of the pupils' Materials scores standardised within topics - a way of accounting for schools having operated somewhat different schemes of work ⁷ (Table 3.1). Again experimental pupils gained more than control pupils, but only if they were non-follow-up. In other words, again a significant difference, with experimental pupils doing better, but only if they were non-follow-up.

There might be a differentiation here between those follow-up pupils from rural locations and those from urban locations. The statistical ⁸ results indicated that rural pupils performed more highly than urban pupils on all 3 tests. In 2 cases this difference was significant and in one case nearly so.

Turning to variables concerning attitudes, skills and feelings, there was little change in scores for attitudes, CLEFP or self-esteem. Some changes were noted in Feelings about Group Work, where there was a marginal decline in positive responses with the experimental group declining while the control group scores increased. However, follow-up pupil scores went up marginally, while non-follow-up pupil scores stayed static overall. Results are summarised in 3.6.

Table 3.6 Feelings about Group Work

On the relational measures, % of the class that pupils reported that they liked working with in science, % class liking spending time with at break and % class liking spending time with out of school were insignificant. Meanwhile, some more significant results were obtained when looking at the science work group.

• % of the science work group that the pupils reported that they liked working with in science showed control group greater than experimental group overall (experimental vs. control F (1,277) = 7.94, p = .005); but
• % of the science work group that the pupils reported that they liked spending time with at break showed experimental greater than control (experimental v control F (1,277) = 5.89, p = .016); as did
• % of the science work group that the pupils reported that they liked spending time with out of school (experimental v control F (1,276) = 6.63, p = .011).

However, it should be noted that although the scores of the experimental pupils decreased, those of the control pupils decreased significantly more.

Table 3.7 Sociometric within group scores

There was thus some indication from the sociometric data that the follow-up pupils tended to focus upon group relations rather than relations with the whole class at pre-test. By post-test, the non-follow-up pupils also tended to have shifted in the same direction.

Turning to the researcher observations that took place twice during the experimental period in the spring term, there were rarely differences between time one and time two, but there were differences between the average number of times a communication behaviour was observed per observation window during the whole class and group observations. This indicated that when the class were meant to be doing groupwork, evidence that they were actually doing so was obtained through the observations. The increase in learning productive dialogues indicated that the project was having an impact on the teaching strategies that the teachers utilised. The following types of dialogues were observed with significantly greater frequency during group work as compared to class work:

• Propositions per window (Context F (1,145) = 82.94, p < .001);
• Explanations per window (Context F (1,145) = 25.14, p < .001);
• Disagreements per window (Context F (1,145) = 11.34, p = .001); and
• Continuations of theme per window (Time F (1,145) = 15.41, p < .001; Context F (1,145) = 11.10, p = .001).

Further details regarding the meaning of the dialogue codes are available in Table 2.1.

Only the number of times a pupil was observed per window referencing back to something another pupil had said earlier in the learning experience increased significantly. The S-TOP⁹ measures (indicating a measure of the differences in suitability of learning context, activities undertaken, teacher role, and level of group skills displayed) were all insignificant.

Despite the disappointing results with regards to attainment, the whole class vs. group contrasts are much as would be expected. In the primary school project significant differences had been observed with experimental classes increasing more than control classes. One may have hypothesised that similar patterns may be observed in the secondary project as well. This is evidence that the groups were working. However, the impression is that neither context generated as much discussion as it did at primary level, and this may be the reason for the lack of experimental effects. Dialogue did help, however. Discussion, particularly at Time 2, had positive effects on post-test attitudes to science and scores on both science topics. The overall impression is that the same basic processes are at work as in the primary science project, but that there is less difference between group and non-group sessions than was previously seen.

In general, considering all the results together, there is evidence here that the project resulted in a gain in science attainment, but the control pupils advanced as much as the experimental pupils. This means that in the promotion of attainment, the alternative collaborative learning strategies in this study proved just as effective as methods already employed by the teachers in the control classes, but not more effective and the non-follow-ups advanced significantly more than the follow-ups. Thus there is little evidence of any enduring impact of the primary school project when the pupils come through into the world of secondary school.

3.4 Experimental effects - Individual analysis

3.4.1 Summary

Overall any positive findings at class level were almost equally balanced by negative findings, explaining the failure to find an overall gain in attainment when comparing intervened pupils to controls. Significant positive increases in attainment and social measure are reported for schools C and E. Otherwise it is difficult to see any consistent pattern, with every other experimental school showing at least one result in the 3 available that appears to be inconsistent with the general pattern being observed in the school as a whole.

3.4.2 Detailed findings

The analysis in the previous section considers the data as a whole, in a way which masks the variation between experimental schools. Additionally, there is a question about the control schools, since these were somewhat self-selected, and might have been schools of higher socio-economic status or offering alternative programmes of equal effectiveness.

For instance, comparing experimental teacher 1 with experimental teacher 5 (in different schools), we find a significant difference between the pupils they did group work with for both science topics despite these pupils being in more than one class. It seems that quality of implementation was variable within the experimental schools.

However, the outcomes for individual schools were very mixed. Because there was little consistency in the attainment results, attitude and observational measures were not further considered in depth. However, there were significant correlations between standardised test score gains and the frequency with which observations recorded an instance of the continuation of a theme by a science work group member for both Earth and Space¹⁰ and Materials¹¹. This observation related to the number of times that a science group member would sustain or develop a conversation about science whilst working in their group. For more detailed investigations to be worthwhile in respect of the observation schedules then a more widespread effect would have to have been evident in the experimental classes that could have been significantly related to the collaborative learning project. As such effects were either weak or not present the value of undertaking more in-depth analysis of this nature would have been limited. A summary of the effects in classroom settings is presented in Table 3.8.

Table 3.8 Summary of effects of collaborative learning in experimental classrooms

Thus overall any positive findings were almost equally balanced by negative findings. Otherwise it is difficult to see any consistent pattern, with every other school showing at least one out of the 3 results that appears to be inconsistent with the general pattern within that school.

3.5 Experimental effects - relation to quality of group work

3.5.1 Summary

There was no evidence that the quality of primary or secondary group work (as indicated by observations) consistently influenced outcomes for pupils.

3.5.2 Detailed findings

Little evidence was found that the quality of secondary group work experience as indicated by observations consistently influenced the outcomes for pupils. All the correlations between overall observed factors and results on Earth and Space and Materials were insignificant.

Gains observed in the ScotSPRinG project that were still observable after transfer to secondary school were not observed to have been influenced by the quality of primary school group work experiences. Neither were pupils from the primary school project advantaged in the secondary school collaborative learning/group work project. It appears that neither of the experiences led to substantive gains in the long run.

While it is possible to consider the relative degree of transfer from primary into secondary for particular classes and relate this to the quality of primary group work experiences in those classes, the numbers involved in these comparisons are so small that there is negligible likelihood of finding a result.

With regard to transfer being influenced by the quality of secondary school group work experiences, no consistent attainment gains were found. Therefore, it was not possible to conclude that the differences in attainment gains were as a result of the quality of the collaborative learning experience. Consequently the picture is extremely muddled for any process of aligning quality of group work with the outcomes of group work.

3.6 Overall Summary

At pre-implementation test, the follow-up pupils who had experienced group work in the primary school project showed a superior performance on the Forces test, and also on the specific science test on Materials (remembering that there was minimal overlap between this test and the topic previously covered in primary school-States of Matter). Thus at pre-implementation testing, the follow-ups showed an advantage in science attainment in Forces and Materials. In terms of social relationships ( i.e. comparing the percentage of the group and the class that pupils reported that they liked to work with and the percentages of the class that they reported the liked to play with at break time) follow-up pupils significantly showed a stronger inclination to form positive relationships with their science work groups rather than forming relationships more generally throughout the class, especially in relation to who they liked to spend time with when not in class. There were no effects on other variables.

Turning to the interactions between experimental and control groups in considering the pre vs. post implementation test scores, there was little sign of a significant gain observed for the experimental group when compared to the control group. This was partly due to the fact that attainment scores for the control groups also increased. In the AAP general science test, the control group actually performed significantly better than experimental pupils. Follow-up pupils also tended to lag behind non-follow up control pupils in the AAP test and in Earth and Space. However, in Materials, follow-up pupils scored significantly higher than non-follow-up pupils from the control group. This topic had minimal overlap with the topic 'States of Matter' covered in the original primary school project.

The peer relationships children reported on the sociometric instrument showed some significant patterns. The percentage of the class that children reported they liked working with was greater in the control group than in the experimental group. However, a different pattern was seen in the percentage of children that children reported that they liked spending time with at break and liked spending time with out of school. In these 2 measures the percentages reported by children in the experimental group were greater than the percentage reported by children in the control group. All other scores were insignificant.

There might be differences between those follow-up pupils from rural locations and those from urban locations. At pre-implementation test 2 rural pupils performed significantly better than urban pupils. In the other tests results from both groups were not significantly different from each other.

In general there is evidence that the collaborative techniques in science promote effective group work in secondary school science, with the exception of discussion which was more prominent at the primary school level. The lack of discussion may account for the lack of experimental effects. There were also differences in observations between whole class and group observations in propositions, explanations, disagreements and continuations of theme. Overall, however, we found little evidence that the quality of secondary group work experience as indicated by observations consistently influenced the outcomes for pupils.

Examining the schools individually, positive findings overall were almost equally balanced by negative findings. Every other school shows at least one set of results at odds with the general pattern for that school context.

Exploring whether gains accruing from group work transfer more or less as a function of quality of primary school group work experiences or transfer of previous knowledge/skills/attitudes did not lead to positive conclusions. There was also no evidence that the quality of secondary school group work experiences led to enhanced learning in the long run.

« Previous | Contents | Next »