Scottish Government

Evaluation of the Masterclass Initiative

CHAPTER 4 : EFFECTIVENESS OF DELIVERY OF MASTERCLASS

This chapter of the report reviews the perceived effectiveness of Masterclass amongst participants and co-ordinators. These comments are set within the context of the initial expectations held about Masterclass and other ICT based activity that was going on within the school environment.

The research also established attitudes to certain key elements within the Masterclass programme and so the chapter ends by looking at the views and comments made about the different elements which formed the foundation and on-going central support that is provided to Masterclassers. As such we look at the perceived effectiveness of specific elements such as the training provision within Masterclass, the SETT conferences and the on-line community.

4.1 EXPECTATIONS HELD AT THE OUTSET

The qualitative research indicates that Masterclass participants often do not have a clear idea what to expect when they sign up for Masterclass and there was some comment about the lack of factual detail during the initial stages. As such, someone they knew often persuaded participants to be involved, with information passed out by word of mouth means.

' There was no real feel at the beginning for what it would mean. It seemed a bit of an open ended commitment, but I thought it would offer opportunities for training and networking.'

ICT head in Secondary school

' There was only a vague vision of what Masterclass might be. It was 'here is a carrot, but the contract associated with it is open-ended.'

Local authority co-ordinator

Some thought a better outline should be in place.

' I wanted a roadmap of how it will go forward each year and to know who it's geared to, how long it will go on for and can others at the school get involved?'

ICT head in Secondary school

In the absence of any clear steer on what Masterclass is about, Masterclassers create their own pattern of assumptions as to what to expect. The extent to which the champion role was known at the outset did vary - some were aware that their involvement in Masterclass would include this element of giving back within their own school or region. However, in other cases, this was definitely not part of the package being promoted by the education authority.

'If any staff are cascading, this is because they want to, they weren't recruited to Masterclass on the necessity to cascade…. I sold it to our Masterclassers as being given an opportunity to play a key role in developing the use of ICT strategy across the region.' Local authority co-ordinator

Most taking part in the qualitative research expected a personal benefit from Masterclass in the form of improved IT skills, or to take advantage of networking opportunities and to learn from others.

Some also commented that they had felt quite scared at the outset and described the initial stages as daunting. However, their concerns had been alleviated as they began to meet other people involved with Masterclass and discovered that they were in the same position as themselves.

The extent to which all Masterclassers echo these views was measured in the survey, with participants asked to state what they had hoped to gain from Masterclass when first told about it. Initially responses were unprompted which provides an opportunity to gauge what the most front of mind expectation is. Subsequently, respondents were prompted with other possible outcomes and allowed to select those that applied. The table below shows the overall pattern of expectation that results from this analysis.

Table 4.1

Expectations of Masterclass

* other answers were given by fewer than 5% overall

At an overall level, the main expectations relate to the possibility of making contacts and becoming part of a 'learning community' with 90% or more of participants stating that :

• they hope to learn from others
• they expect to network with them and
• meet others working within the same area.

However, these are not generally the first expectations articulated (mentioned first by no more than 7-10%) and they emerge more frequently as later choices or when prompted.

In fact, a personal benefit in the form of improving the Masterclassers own ICT skills is the first expectation mentioned by the largest group of participants (32%) and it can be implied that this is a key reason for becoming involved. It is notable also that the initial perceived benefit to individuals is couched in terms of improving specific ICT skills rather than the more general improvement of ICT knowledge (14%). Relatively few (1%), initially, appear to have regarded their involvement in Masterclass as likely to improve their personal standing within their workplace and, as a possible reason for involvement, this does not appear to have been a major consideration.

In terms of other possible reasons for involvement, although Masterclassers benefited from the provision of a laptop, relatively few (5%) cite this initially as a prime expected outcome, although just under 2 in 3 mention it overall.

Few (1%) also identify initially the ICT champion role as an expectation and less than half (46%) overall appear to subscribe to this as an outcome at all. The qualitative research suggests that the terminology of ICT champion is off putting to some participants who would prefer not to lay claim to the champion title as it is seen as élitist. The extent to which participants subscribe to the champion role is evident through the proportions foreseeing a role in sharing learning with others (85% overall) and getting involved in integrating ICT within the curriculum (56% overall).

To sum up, both strands of the research reveal that for the largest group of participants a key expected outcome is improvement in their own ICT skills and this is likely to be a key motivator for any new participants. Beyond this personal benefit, however, Masterclass participants hold expectations about the nature of the community that they are joining and articulate a belief that networking and shared learning opportunities will arise.

There are a few differences in the expectations outlined by those who are both participants and co-ordinators (18 of the respondents). At the overall level, this group is more likely to regard networking as an outcome and less likely to feel that they will either improve their own ICT skills or knowledge of ICT generally or to benefit from the provision of a laptop.

4.2 ATTITUDE TO MASTERCLASS SUBSEQUENTLY

Overall views on the effectiveness of Masterclass were gauged using two attitude statements to which respondents were asked to agree or disagree, selecting a response from a 4 point scale that showed the strength of their feelings about each statement. The first statement read: 'Masterclass has been a real catalyst driving forward our use of ICT.'

Chart 4.1

Views on whether Masterclass has had a catalytic effect on use of ICT

(Source : Q13)

There is widescale support for the sentiment expressed in this statement, with over three-quarters of participants agreeing to some extent that Masterclass has had a catalytic effect on their use of ICT, albeit that most just tend to agree (51%). Those in schools are more likely to agree that Masterclass has had a catalytic effect (76%), while local authority and TEI staff especially are less convinced of this, albeit that around half or more still agree (67% and 47% respectively).

The main visible impacts associated in part with Masterclass are in terms of the provision and ability to use new ICT equipment. Beyond benefiting from their laptop, many of the schools are pursuing the use of a range of new equipment types and this will be discussed in more detail at chapter 5.

' The outcome has been a real Pandora's box. I'd never seen projectors and whiteboards before. We now have funding (Through Early Learning Forward Thinking) to get this.'

Nursery head

Views on the impact on the use of ICT in schools were also sought from a small number of co-ordinators (7 in total) who did not personally participate in the Masterclass programme. The co-ordinators are more similar in their views to TEI staff with a small majority likely to agree with the statement that Masterclass has had a catalytic effect (57% compared with 43% who disagree).

A further statement was tested to help understand views on the effectiveness of the Masterclass programme overall. Masterclass is by no means the only ICT initiative that is active within schools at present. The qualitative research found that many schools had active ICT programmes and initiatives in place - a view that was echoed by co-ordinators who often had assumed responsibility for Masterclass due to their overall involvement in local authority wide ICT initiatives. This view of Masterclass being part of an overall package was echoed by a number of the local authority co-ordinators taking part in the research.

'( Being MC co-ordinator was) a real blurring of my roles as it fits alongside a lot of other stuff that was going on anyway. Masterclass was not really an additional initiative in the area, it was integrated into what we do and has supported what we would have done. The benefit of Masterclass has been to give the subject National (Scottish) importance.'

Local authority co-ordinator

While some co-ordinators had happily subsumed Masterclass within their role, others felt an additional resource was needed to help co-ordinate Masterclass in their area and this is borne out by comments examined later about the need for local authority support to be greater in some areas.

As a further example of the inter-connections that exist, some Masterclass participants are in the process of receiving training through Early Learning Forward Thinking. This will widen their future role as ICT champions as it provides an opportunity to train others within the local authority area (training in 'how to train' is being provided and time has been ring fenced to allow then to do this). In their own view, these individuals may not have received this further training had they not been part of Masterclass. This is an important point as it makes it difficult to evaluate the effectiveness of Masterclass in isolation.

Given the above, it is not surprising that there is widespread agreement (75%) with the second statement which read 'it is hard to attribute recent developments and changes in ICT use purely to Masterclass.'

Chart 4.2

Views on whether ICT developments are due to Masterclass or other reasons

(Source : Q13)

As with the previous statement, those most likely to dispute the impact that Masterclass has had are TEI staff (88% agree with the statement compared to 75% overall).

Thus, whilst Masterclass appears to have had a catalytic effect on the use of ICT, this must be tempered with the knowledge that there are other initiatives and strategies in place which are working with Masterclass towards the same goal.

4.3 THE INITIAL 4 DAY TRAINING

For many, the official start of Masterclass was when they attended their 4 day training session in Stirling. Two thirds of those taking part in the survey say they commenced their initial Masterclass training in 2002 and 32% did so in 2003. A few (1%) said either that they had no initial training or that they did it this year.

As a group, librarians are more likely to emerge as late entrants to Masterclass with a majority of this group (62%) starting in 2003.

Some modifications were made to the initial four day training in response to feedback from participants attending the first few events. One local authority informed us that they were aware that changes had been made to the training in response to concern by those on the first programme that it was too intensive and lacked time for consolidation and reflection.

However, this is not reflected in the scores given by participants on the quality of this training. All respondents were asked to mark this out of a possible maximum score of 10 and the results overall and from those who undertook their training at different stages, reveal that a high average assessment is given. As the following table reveals, very few provide low scores on the training or indeed scores of less than 7 out of 10.

Table 4.2

Rating of quality of initial training

	10 = highest			1 = lowest
	Score of 10 %	Score of 7-9 %	Score of 4-6 %	Score of 1-3 %	Average
Overall sample	7	85	7	1	7.9
Trained in 2003	8	85	6	1	7.9
Trained in 2002	6	84	9	1	7.9
(Source Q18)

The study is also able to provide some understanding of what lies behind these assessments and reveals that there were many positive aspects. At an overall level, Masterclassers taking part in the qualitiave stage described the initial training as :

'Brilliant, motivational, eye opening.'

Nursery head

Another said

'It was enjoyable and a good opportunity to meet people - we learned as much from the people on the course, as from the speakers.'

Nursery head

Particular elements of the training that are identified by a significant proportion of respondents as especially useful are the sessions on digital technology, mentioned by 85% or more and the group sessions, mentioned by 82% specifically. However as the chart below reveals, many of the elements included within the initial training are regarded as having been useful to at least half the Masterclassers taking part in this training.

Chart 4.3

Useful aspects of initial training

(Source : Q17)

Of the three elements revealing the lowest levels of identification in terms of usefulness, there was some comment in the qualitative stage about the case study element of the initial training. Simply, some said that participants may have felt intimidated by the discussion about best practice, reflecting a lack of confidence in their abilities. Others commented that few participants seemed prepared for this element. This may explain the lower score on this element.

The talk2learn workshop was intended to inform participants on the use of the on-line community. Only half the Masterclassers subsequently participated in the on-line community, in part due to a number of problems associated with the site which are discussed later. The fact that relatively few Masterclassers went on to use the on-line community could explain the finding that the talk2learn workshop was not useful. In addition, interviews with the facilitators reveal that not a great deal of time was spent on this area, and consequently participants failed to realise the significance of their input and participation in the on-line community. On some of the programmes, the workshop was held at the end of the day and was a short session, and only a single session was held in the whole four days. Therefore, it was not emphasised as an important aspect of the overall Masterclass programme.

Despite this positivity, the qualitative research did collect suggestions from Masterclass participants on ways in which the training might be improved. These suggestions were tested in the survey element by asking participants to either agree or disagree with opinion statements about the training. A four point rating scale was used although we have merged the disagree responses in presenting the data in the following chart.

Chart 4.4

Views on initial 4 day training session

(Source : Q15)

In terms of its impact, the initial training is assessed very positively with almost all (93%) respondents stating that it has a motivating and invigorating impact and almost two thirds (65%) agree strongly that this is so. As with other aspects of Masterclass, TEI lecturers are slightly less positive than others.

A lot of the qualitative comment mentioned the quality of the speakers, with Marie Dougan and Alan November mentioned with some frequency. Again, the survey shows just how well regarded the speakers are on the whole and 77% agree that ' some of the speakers are excellent and worthy of a wider audience.' This latter suggestion arose in the qualitative research and reflects, to some extent, what is already happening with these individuals being made accessible to a wider audience through video conferencing or attendance / speaking at local events. Some Masterclassers find this especially helpful to their own cascading activity in that they will recommend colleagues to take advantage of the opportunity.

The remaining statements relate more to the structure and content of the training course. Attitudes to the statement ' they covered a wide range of issues and subjects of relevance' show broad satisfaction with course content and a vast majority (96%) agree that the coverage is relevant. However, opinion was more dichotomous when it comes to considering if the mix of participants was a good or bad thing and this reflects the range of comments made about this within the qualitative stage. On the one hand, some seem to feel that this mix is a real benefit as they have opportunities to learn from others. Others feel that the training tries to cater for too many different types of individuals and skill levels.

'I would have liked some segregation when we were doing our tasks, as some people didn't fit in well.'

Head of ICT in a Secondary school

This theme also emerges later when we consider networking and sharing opportunities and suggestions for improvement in this.

Overall 40% of the Masterclass participants agree that 'they tried to cater to too many different audiences - it would have been better to have some segregation between different types of staff at some sessions.' In contrast 58% disagree with this statement. Those who describe themselves as merely interested in ICT are more likely to agree with this statement than those who view themselves as very ICT literate.

One of the more common complaints about the course arising from the qualitative work was that it is quite intensive, with individuals talking of late nights to complete tasks. This appears to have affected a minority of respondents, with just 29% overall stating that the ' training was too intensive.' The vast majority (71%) does not appear to find the training too intensive.

However, there is a suggestion that participants need more time to be able to consolidate what they learn while on the course. The final statement pertaining to the training specifically related to the phasing and structure of the training. On the whole participants are likely to agree (61%) that 'not enough time was allowed for consolidation of knowledge,' with the ICT literate less likely to say this than their less ICT aware colleagues.

Many of these opinions and views are reiterated when Masterclass participants are asked to suggest ways of improving the initial training (Q16). The main themes emerging from the wide range of suggestions made are around :

• Making the course less time intensive; overall almost two-fifths (39%) of Masterclass participants ask for more time or less material to be covered or suggest some ways of changing the time allocation for the training session.
• Arranging the groupings of participants better is mentioned by 17% of Masterclassers. Essentially here Masterclassers are asking for segregation by sector, subject or skill level
• Altering the range of topics is suggested by 14% of Masterclassers. Some want greater choice or flexibility while others want the training to focus more on specific issues.
• Around 1 in 10 suggest that the format can be improved through better project work, practical sessions, school visits or better presentations for example.
• The timing and scheduling of different elements could be altered in the view of 7% of Masterclassers and this largely relates to a view that certain topics should have been covered at the beginning or the end of the course.
• Although only 5% mention a lack of networking opportunities during the initial training this is an important issue given expectations about Masterclass as a channel to facilitate this.
• 5% also mention that they would like more information in advance of the course.

4.4 OTHER TRAINING PARTICIPATED IN

Overall some 99% of those taking part in our survey had taken part in the initial 4 day training that is offered. Some 47% claim also to have participated in other Masterclass training sessions.

Chart 4.5

Masterclass training sessions attended since initial training

(Source : Q19a)

Reflecting the positivity expressed earlier to the digital video training elements included in the initial training, this is the topic where most (30%) have sought further training. Beyond this, just 1 in 10 have taken part in information literacy courses (especially librarians - 38%) and learning styles courses, and few have partaken of any of the other options, as yet.

This interest in digital video was evident when conducting the qualitative stage and this possibly reflects the fun element of being able to involve children in using the equipment. Within secondary schools there are a number of mentions of the children making films and entering competitions with their output, while even younger children at nursery are being encouraged to use digital cameras to take photographs of items and store them for project work.

Although relatively few Masterclassers took part in any of these further training sessions, some attempt was made in the survey to measure the quality of the session amongst those with some knowledge of this. However, base sizes are small and these are indicated in the following table by showing the numbers who provide a response in brackets. For comparability with the initial training assessments, the same scoring system was used, with a score out of 10 being allocated, where 10 is the highest score.

Using this system, the initial training is given an average score of 7.9. As the analysis shown in table 4.3 reveals, the digital video training and learning styles training are even more positively rated that the initial training with an average of 8.7 and 8.2 respectively calculated for those assessing them.

Where other training received slightly lower assessments, this reflects a situation where the majority gave scores of more than 7 out of 10. However, it should be noted that 1 in 4 of those doing the information literacy training gave it a low score though this arises particularly from teachers - librarians tend to be more positive about it.

Table 4.3

Rating of training (all who undertook each type)

	10 = highest			1 = lowest
	Score of 10 %	Score of 7-9 %	Score of 4-6 %	Score of 1-3 %	Average
Digital Video Training (152)	32	62	6	1	8.7
Information Literacy (46)	4	54	17	24	6.2
Learning Styles (46)	17	70	13	-	8.2
Additional support needs (22)	14	55	23	9	7.1
Web tools / web design (21)	10	57	24	-	7.4
Interactive whiteboard (10)	40	50	0	0	9.1
(Source : Q19b)

4.5 THE SETT CONFERENCE

For many Masterclassers, participation in Masterclass brought an opportunity for the first time to attend the SETT conference and, from the qualitative research, this appears to have been a particularly good opportunity to network and meet those Masterclassers that they had been on their initial training with. The vast majority of those surveyed (93%) had attended a SETT conference since becoming a Masterclasser. All of the co-ordinators had gone but, as a group, fewer librarians had attended (86%) ¹.

When asked to rate the conference, using the same scoring system as was used for the training elements, the overall average computed is 7.9 ². Thus the SETT conference is as well regarded amongst those attending it as the initial training is. Although less likely to go to the conference, librarians who did attend gave it the highest average score (8.5).

As suggested by the qualitative research, one of the additional benefits of SETT is the opportunity to mix with people, but the conference also serves as a very useful opportunity to learn about new equipment (and thus keep Masterclassers' knowledge up to date). One individual in the qualitative research complimented the keynote address especially and also applauded the Scottish focus of the event. Another said:

'It is a chance to get together, where you can talk and exchange ideas and use the new technology. It was all good experience.'

Teacher in Secondary school

On the whole participants are very complimentary about the SETT conference and the following chart indicates what elements of SETT are liked by Masterclassers with experience of this conference.

Chart 4.6

Likes of SETT conference

(Source : Q20c)

There is a lot of positive comment (by 73%) about the format of SETT, covering aspects such as the quality of the speakers, the seminar and workshop programmes and the layout of the event. The content also is praised by 15% of those attending and particular aspects mentioned centre on the debates and input given.

Masterclassers also enjoy the opportunity to see new hardware and software and attend demonstrations of this, and a quarter mention this as a plus of the conference. However, as Masterclassers it is perhaps not surprising that a significant proportion value the networking opportunities offered by SETT, with one third mentioning this as a positive aspect of SETT. About 1 in 10 also find the SETT conference to be a good way of identifying best practice through case studies and opportunities to share information.

A number of other themes were identified by smaller groups of Masterclassers. Just 6% mention finding the SETT conference to be motivating and inspirational and a source of new ideas, while 1% describe the conference as a useful way of broadening their perspective and making them think about the future. Some Masterclassers also mention specifically the pre-eminence given to Masterclassers at SETT and feel boosted by this.

4.6 THE ON-LINE COMMUNITY

The on-line community is one of the supporting elements provided to Masterclassers. It was originally accessed at think.com but later transferred to talk2learn.com. When used, the facility offers Masterclassers support in a number of ways. For example, each Masterclasser has the opportunity to develop their own web page within the site, on which they can upload resources and ideas. The website is also intended to offer a key networking resource to its users enabling Masterclassers to communicate with each other and discuss issues surrounding the use of ICT in education. Discussion forums have been developed for this purpose. The idea is also that Masterclassers can obtain resources and new ideas.

However, the qualitative research suggests that the facility is not used with any great frequency by a number of Masterclassers. The survey was used to gather data on levels of use of this facility and attitudes to it.

In terms of establishing levels of use of the on-line facilities, some 53% of participants claim to have used the Masterclass on-line community at all. Use is slightly higher amongst librarians (67%) than others, with minorities in a co-ordinator role making use of the facility (40% across all co-ordinators). Fewer still have had follow-up contact with any talk2learn facilitators and only 1 in 4 Masterclassers have had any contact with these facilitators. In this instance participant co-ordinators (50%) are the group most likely to have had contact with facilitators.

SEED is aware of the low usage of the website and eight "on-line facilitators" were appointed during the summer of 2003. The facilitators are Masterclass participants who are particularly committed to using the on-line community. They have more advanced ICT skills including the ability to develop websites. Their remit was to use and develop the site on a regular basis to encourage other Masterclassers to use it more regularly.

Facilitators give between 4 and 8 hours per week to website use. Their role includes activities such as developing discussion forums on the site, which aim to stimulate others to participate, keeping their personalised web pages up to date; up loading materials and resources that they have developed, and identifying useful links.

Chart 4.7

Proportion of participants making use of facilitators and on-line community

(Source : Q21a / 22a)

When asked to specify why they had contact with the on-line facilitators at talk2learn ³, a number of reasons are provided, with the principal one being to access updates and bulletins, mentioned by 19% of those with any contact. A further 12% mention newsletters and 11% were seeking help with a technical problem. A similar proportion (13%) state that they access this service just out of general interest and with no particular purpose. Beyond this a wide range of other reasons are mentioned by 10 or fewer Masterclassers.

As a measure of the value placed on this service, those who had accessed it were asked to state how useful it had been to them in their Masterclass work

Chart 4.8

Usefulness of contact with on-line facilitators

(Source : Q21c)

Opinion here is split with slightly more rating the facility as useful (56%) than not (44%). As relatively small numbers are in a position to rate the facility, it is not possible to look at differences in opinions across different types of user.

Overall 70 Masterclass participants rated contact with the talk2learn on-line facilitators as useful (Q21d). Their perception of this as useful arises largely because it provides information of general interest or gives an opportunity to catch up (29% of this group say this), it provides opportunities to share (mentioned by 20%) or it helped 11% to find the answer to a particular problem. A small proportion (7%) states also that it is a good way of networking or it has helped them in their work.

Some 54 Masterclass participants say that their contact has not been useful and the main reason behind this is a perception that the content accessed is poor, mentioned by 26%, 14 respondents. A number also have had access difficulties (affecting 15%; 8 respondents) and the other significant group of comment is on the poor design of the website (affecting 13%; 7 respondents).

As mentioned in chart 4.7, just over half (53%) of Masterclassers have made use of the on-line community, with use highest among librarians (67%) and lowest among TEI lecturers (47%) and local authority Masterclass participants (43%). However, further probing about what elements of the on-line community have been used reveals that a slightly lower proportion may have practical experience of the community as 13% of those using the community at all than say they did not use any specific parts of it.

The results of the analysis on what parts of the on-line community have been used are provided in table 4.4. This table shows the parts of the on-line community used by Masterclassers overall and also reveals to what extent they are aware of certain parts of the community. The final column shows the proportion of Masterclassers aware of a part of the community that have used it.

Table 4.4

Parts of Masterclass on-line community used

Specific points arising from this analysis are:

• levels of awareness of various parts of the community are low, with none of the elements approaching global awareness even amongst users
• the good practice section has high levels of use amongst those aware of it. Arguably increasing awareness of it is likely to lead to similar levels of increased use
• the local issues section emerges reasonably well from this analysis in terms of use, although its profile is low with only 39% of users aware it exists.

Masterclassers also identified a wide range of other parts of the community used - for example 5% mention Headstogether in response to this question and 3% refer to stickies.

A high proportion of use of the Masterclass on-line community is to browse with 83% of all users saying this (Q23a). In addition, 38% claim to have used it to seek out specific information while 17% are participating in specific activities. Librarians are the group most likely to be seeking specific information (57%), while TEI lecturers (38%) and local authority staff (28%) are more likely to be involved in specific activities.