Scottish Government

Evaluation of the Masterclass Initiative

CHAPTER 7 : BARRIERS AND SUGGESTED IMPROVEMENTS

In the final chapter presenting the results of the study, we review a number of the barriers identified over the course of the various discussions and outline some of the constructive suggestions made as to how they may be overcome.

The chapter begins with a review of the selection process for Masterclassers, presenting information on how this was managed, views on its suitability and preferences for change. Other sections in the chapter then look at the on-line facility, the programme of follow-up activity that has been implemented, the opportunities for networking and sharing good practice and the impact of resource constraints.

7.1 SELECTION OF MASTERCLASS PARTICIPANTS

All participants were asked initially how they became involved in Masterclass. Just over one third (37%) claim to have volunteered themselves with the rest being asked to take part - largely by a local authority contact (39%) or a head teacher (18%). Other routes are used less often.

Chart 7.1

Initial participation in Masterclass

(Source : Q1)

This is in keeping with the procedures described by co-ordinators and participants for identifying candidates in the qualitative interviews. Some local authorities tried to run a voluntary system by asking for staff members to be involved, but then supplementing those who came forward with some individuals who they targeted specifically to ensure that their authority had a range of candidates from different roles and parts of the educational spectrum. Many participants often did not know exactly why they had been selected to take part; some supposed it was because of being known within the local authority as interested in the use of ICT, others because they had a clearly defined role in the use of ICT.

'I'd been involved in preparing the regional response on ICT for the 5-14 curriculum as well as providing training to people in other schools.'

Head of ICT in Secondary school

'I'd been a staff tutor in ICT. I got an email from XX (named local authority staff member) saying they wanted people who were interested.'

Depute head in Primary school

'We wanted to include all grades of staff ie head teachers, curricular leaders, subjects heads, class teachers. … We selected people from across sectors (from pre-school to secondary) and one representative from each of our clusters. We didn't have room for a special education needs representative and would've liked to have one of them within the group. The people chosen were selected as having shown good learning and teaching skills and an interest in the use of ICT - they were not ICT experts. The main approach used was to ask headteachers for recommendations and to go with this.'

Local authority co-ordinator

The following chart displays some of the main reasons given by participants to account for why they became a suitable candidate for Masterclass. Only those mentioned by 30 or more participants have been included in the chart, but candidates came from very varied backgrounds covering training roles, having a strategic remit, a knowledge or an interest in ICT or simply being known as 'keen to learn.'

Chart 7.2

Reasons for being a Masterclass candidate

(Source : Q2)

One participant simply chose himself while others thought they were chosen due to their youth or because they had the time to do Masterclass. In an attempt to see to what extent Masterclass participants regarded themselves as having some expertise in ICT, the survey asked participants to classify themselves in terms of their ICT ability at the time that they started Masterclass. As reported earlier at section 5.2, most described themselves as very ICT literate (61%), with one third seeing themselves more as 'interested in ICT' - 34%. Only a few (7%) classified themselves as having done something innovative with ICT in the past.

Given the earlier comments about the mix of candidates taking part in the initial training, views were sought from Masterclass participants on which types of individuals Masterclass is most suitable for. Responses were unprompted and so a wide range of descriptions were offered. Key aspects that comprise the ideal candidate are:

• Having an interest or expertise in ICT (58% say this)
• Being enthusiastic or innovative in their thinking (50% say this)
• Being a good communicator or trainer (24% say this)
• Having a degree of seniority (12% say this)
• Having the correct mindset to succeed (7% say this)

Some 15% say that anyone could be part of Masterclass.

Masterclass participants tend to characterise the ideal candidate as someone similar to himself or herself. Hence one head of ICT described the ideal candidate as ' ICT literate' and 'senior enough to be able to push ICT on the school agenda.' Another claimed to have taken it on because she was at a point where she needed to either ' take on a new challenge or sit back and vegetate.'

We found when speaking with school based staff during the qualitative research that there were instances when individuals felt aggrieved that they or another in their school had not been given the chance to be a Masterclasser. The lack of transparency of the selection process had not helped some staff come to terms with this.

'Masterclass has been divisive within my curriculum panel as some staff members have been selected to be part of it and others haven't.'

Non participant in Secondary school

As part of these discussions we sought suggestions from candidates as to how the selection process could be improved. Various ideas had been put forward at the qualitative stage, ranging from running a competition allowing any staff member to enter, to wider circulation of details, allowing more people a chance to volunteer. However, there are clearly pros and cons to the options raised. In essence, in an ideal world, the process should make it open to all rather than running a risk of being based on 'who you know.'

The survey also generated a vast range of suggestions about how candidates should be selected (Q5), although over one third (35%) of participants said that there is no need to change the process, and 12% could not offer a suggestion to improve it. Key themes and suggestions that were identified are:

• Make Masterclass open to all staff to volunteer - 16% say this
• Promote it better thus allowing more people to have a chance to volunteer - 12% say this
• On a similar line, some 9% say more information should be given out about Masterclass
• Some 4% say a more rigorous selection system should be in place that is transparent to all

There is widespread support amongst all participants for other staff being able to take part in Masterclass. When asked to agree or disagree with the statement ' other staff in the school / our area should be given the opportunity to do Masterclass,' 83% of participants and 86% of those in a co-ordinators only role agree that this should be the case. Librarians (90%) are most keen to see others have the opportunity to take part.

Chart 7.3

Whether other staff should be given chance to do Masterclass

(Source : Q13)

Of course, one of the issues associated with allowing other staff to do Masterclass is the impact it has on staffing levels. But as one respondent pointed out:

'Getting away from school is a nightmare. But still this would be better than trying to find time with (named colleague) to learn from him.'

Non participant in Secondary school

7.2 IMPROVING THE ON-LINE FACILITY

Previous sections of this report (section 4.6 and 6.2) reveal that use of the on-line facility is both limited and infrequent. When respondents were asked to give their reasons for accessing the on-line community, its main use is to look or browse to see what is there (cited by 83% of respondents). Almost two in four (38%) access the on-line community to obtain information and under one in five (17%) accessed this to participate in activities (section 4.6 - Q37a).

Those who do make use of the facility were asked to specify if there is anything else that they think the on-line community should cover (Q37b). Large proportions of those answering this question could not specify any changes (13%) or state specifically that there is nothing else it should cover (62%). A wide range of other suggestions have been identified by respondents with the main themes being to :

• Provide information on certain / specific subjects (9% say this)
• Make sure it is kept more current with new information or details of future events (5% say this)
• Use it better for disseminating good practice (5% also say this)
• 4% made comments about its design and 3% want it to be more accessible

One suggestion from a respondent who is a co-ordinator only is that there should be better links between local authorities - in keeping with the comments made earlier about inter-local authority networking not really operating.

Although only 53% of participants have actually used the on-line facility, all participants were asked if they had experienced any problems when trying to access the Masterclass on-line community (Q24a / 38a). A very high proportion, 45% - almost half, say that they have experienced access problems and these have been exacerbated by the change from think.com to talk2learn.com.

'It's not of great use. There is too much other stuff to do and it's difficult to get on to and use.'

Teacher in Secondary school

Attempts have already been made to improve the on-line facility in order to stimulate more of the Masterclass community to use it. As mentioned earlier, Masterclass facilitators have been employed to develop the site by their own regular use of it. One area that was noted as being particularly successful was the introduction of the Great Debate. Every few months, a new issue, carefully selected for its controversial character and concerning some issue relating to the use of ICT in education, is posted on the Great Debate page and participants invited to offer comments and discussion. As indicated at section 4.6 the Great Debate is one of the more used parts of the website. The on-line facilitators have also developed their own web pages and posted resources and activities on to the site. It is hoped that through establishing a core of regular users who keep forums updated the website will start to appeal more to the wider community.

Chart 7.4

Experience of access problems to on-line community

(Source : Q24a)

Only two of the seven respondents who were interviewed as co-ordinators only make use of the on-line facility. Neither reports any problems with accessing it.

Librarians, who are one of the most frequent user groups, are also the group most likely to report problems (62%). Among all those experiencing problems (228 participants) a number (46%) admitted to operator error - in the form of forgetting their password or user name. However, access and logging on generally (37%), speed of downloading/uploading (25%) and navigation difficulties (24%) are all mentioned with some frequency also. A range of other reasons are provided by minorities of respondents (Q24b / 38b).

Chart 7.5

Access problems to on-line community

(Source : Q24b)

There is, however, support for the concept of the on-line facility and improvements in it are seen as an important channel to facilitate networking and sharing good practice. In its present configuration, it seems not to appeal.

'It's an excellent idea and an easy way to network, but you need some incentive to log on.'

Teacher in Primary school

'The website is like a seed, it can point you in ways you've not thought of before, that's part of its value.'

Teacher in Primary school

7.3 ENSURING CONTINUITY AND CONSISTENCY IN FOLLOW UP ACTIVITY

Chapter 6 outlined the amount of ongoing activity around Masterclass, looking specifically at aspects such as frequency of use of various facilities and the support offered by co-ordinators. This, in association with the qualitative findings reveals that there are issues about the continued evolution of Masterclassers learning and the extent to which they are able to fulfil a role as ICT champion. One key element of this appears to be a diminution of participants' enthusiasm and motivation from the early days, with particular comments being made that as, for example, NOF funding has ended, key support staff in local authorities have been withdrawn, with consequent loss of support through this channel. Furthermore, there is little evidence that LT Scotland have played an active role in promoting ongoing activity, other than continued development of the on-line community.

The later analysis at section 7.6 will also indicate the extent to which Masterclassers themselves face constraints on the amount of time they have to devote to Masterclass and their own development. In terms of the magnitude of the problem, participants and co-ordinators were asked to what extent they agree or disagree with the statement 'it can be difficult (for teachers) to know how to put into practice many of the ideas you get through Masterclass.' The views of participants are dichotomous, with almost as many agreeing with the statement as disagreeing.

Chart 7.6

Whether Masterclassers know how to put ideas into practice

(Source : Q13)

Librarians (53%), TEI lecturers (58%) and co-ordinators (64%) are more likely to agree with this statement than other groups.

One aid to participants would be a programme of more regular meetings - at present the frequency of these varies from area to area. However, when asked if it would be useful to have more regular meetings with other Masterclassers, a vast majority (71%) agreed that this would help.

Chart 7.7

Whether would like more meetings with Masterclassers

(Source : Q13)

TEI lecturers are unanimously in favour of this and 72% of co-ordinators favour this as well.

7.4 IMPROVING NETWORKING ACTIVITY

One of the key expectations held by participants as they start Masterclass, is that it will provide opportunities to network with others - 37% stated this spontaneously ( see section 4.1). As reported at section 5.3 this expectation was largely realised, with 90% of participants stating that Masterclass did provide opportunities for networking. However, there has been some drop off in the level of networking and only 76% of all Masterclass participants currently claim to still network with people whom they met through Masterclass.

Across the sample of participants as a whole, there is a consensus that networking opportunities can be improved.

• Overall 72% think that these opportunities could be improved
• The extent to which those in different roles agree that this is so varies, with teachers and TEI lecturers most in favour of greater opportunities to network

Chart 7.8

Whether networking opportunities with other Masterclassers could be improved

(Source : Q7e)

Co-ordinator/participants also emerge from this analysis as a group likely to favour improved opportunities, with 78% stating that they think these could be improved.

As a follow up to this question we sought information from participants on ways in which better networking could be facilitated. There were various comments made by our participants in the qualitative exercise about the difficulty they have experienced in finding time to pursue their Masterclass interests while also fulfilling the requirements of their day job. As such, it is not surprising to find a common suggestion amongst those in the survey is for cover to be provided to let them develop Masterclass ideas, or to have time set aside for this in some other way.

The other key theme emerging is a for a better system of meetings with some specifying these should be held at a local level while others would value more national opportunities to network. Again this reinforces the findings detailed in the previous section where 71% are in favour of more meetings.

It is perhaps not surprising to see comments that networking would be improved also by a better on-line service, given the comments already made about this facility.

These represent three of the main themes identified by Masterclass participants as likely to improve networking for them. The exact proportions of those identifying each theme and some other, less frequently, mentioned ideas are detailed in the following table.

Table 7.1

Ways in which networking opportunities could be improved

Minorities also mention that they would benefit from the provision of email contact lists, for example, or that conferences and training days can offer networking opportunities.

As well as the more physical dimensions of networking (covering the channels available for this and the barriers limiting these opportunities), the study also collected information on the mix of participants and the suitability of other Masterclassers as a possible peer group to network with. This focus arose from comments made during the qualitative stage about Masterclassers feeling slightly isolated by being the only individuals within their area that came from a particular educational background. It is an important issue as it impacts on the extent to which good practice can be shared.

'There are some meetings and sharing of best practice (in our area), but they are of limited value to me personally, as I am the only teacher of my subject in Masterclass.'

Teacher in Secondary school

'I'm the only librarian in my area who has been on Masterclass. Other school librarians are very spread out - it's difficult to visit each other and observe practice. We are restricted to spreading ideas to in-service days.'

Librarian in Secondary school

Masterclassers in the survey were asked if there were enough people from a similar background to themselves in terms of sector or subject area or would they have liked more opportunity to meet these types of people.

• Overall one third (32%) of Masterclassers would have liked more opportunities to meet others within their peer group
• TEI lecturers (71%) are most likely to want this
• Interest in this opportunity increases with school roll size (from 21-30% in favour where the school roll is less than 1000 to 49% in favour for staff in schools of more than 1000 pupils)

Thus certain individuals do feel slightly isolated. However for the most part, Masterclassers feel that they have benefited in terms of networking opportunities. When asked to agree or disagree with the statement 'Masterclass has been a useful way of making contact with people working on the same sorts of issues as you' a clear majority (82%) agrees that this is the case.

Chart 7.9

Masterclass as a means of making contact with others

(Source : Q13)

As might be expected given the earlier results, TEI lecturers emerge as the group least likely to have made contact with peers through Masterclass, although slightly more agree that they have (53%) than disagree (48%).

The same statement was tested among those who act as co-ordinators only. All 7 of these individuals agree that Masterclass has been a useful means for people to make contact with those working on the same areas as themselves.

7.5 ENCOURAGING SHARING OF GOOD PRACTICE

One of the aims of Masterclass is that it will facilitate the dissemination of good practice in the use of ICT across Scotland. When co-ordinators were asked to identify the ways in which their local authority provides support to Masterclassers, the most common response given was that they allowed sharing of best practice and learning with Masterclassers (88% said this - see section 6.1).

This is endorsed by the views of participants themselves, with the same proportion, 88%, of Masterclassers saying that Masterclass provides opportunities to share good practice with others ( see section 5.4). Furthermore, while the extent to which this is occurring is on the increase for a majority of those involved in sharing good practice (65%), a few (6%) report that they are less involved in sharing good practice than they were at the start of Masterclass.

Overall then, majorities appear to be involved in activity to share good practice and this is on the increase. Nevertheless, the study results indicate that this activity could be further improved. Some 60% of participants state this, with TEI lecturers most likely to say this (71%).

When suggestions are sought as to how to improve the sharing of good practice, a very wide range of ideas were put forward. Some are similar to those mentioned as likely to improve networking opportunities for example and this is indicative of the multiple roles and benefits that meetings or on-line facilities can have. Some of the key themes identified as likely to enhance the opportunities for sharing good practice are summarised in the table below.

Table 7.2

How to improve opportunities to share good practice

The main themes reflect those already identified elsewhere with majorities commenting on the need for meetings, better on-line support, time to be set aside for this or general facilitation of contacts. However, 6% mention specifically that they feel they will benefit from appropriate peer groups, with a similar proportion stating that more information or publicity would help.

7.6 THE IMPACT OF TIME AND RESOURCE CONSTRAINTS AND WHAT EFFECT THEIR REMOVAL MIGHT HAVE

In the preceding discussion on how to improve networking opportunities and the sharing of good practice, the issue of time pressure emerged as one of the constraints faced by Masterclassers when trying to take on these roles. For many Masterclassers taking part in the qualitative interviews, although they would like to keep active and access talk2learn and organise training or presentations to colleagues and develop new ideas, it is difficult to find time to do everything on top of their 'day job'.

The colleagues of our Masterclassers also commented on their being a certain lack of opportunity to learn from their Masterclasser because of time constraints. There were a number of comments about the need to ring fence time to make best advantage of Masterclass learning. However, this does not apply to all. Some Masterclassers already have time set aside within their duties for strategic work on ICT development (within the school or region) and as such, are at an advantage over others when it comes to being able to make full use of Masterclass learning. Others described having to organise training sessions for staff in their own time and trying to find a slot to give this training within a busy ISD framework. There was one comment that actually carrying out this activity and offering the training during in service days meant that the staff member is not able to avail themselves of other training opportunities that would have been programmed then.

This issue formed the basis of three statements about how Masterclass duties and responsibilities fit with the other activities of the Masterclasser. Participants and co-ordinators were asked to indicate to what extent they agree or disagree with each statement and the chart below presents the results of this analysis.

Chart 7.10

Views on the time pressures faced by Masterclassers

(Source : Q13)

A majority agree with all of these statements with the most widespread agreement arising for the statement 'to be able to do justice to Masterclass some dedicated time needs to be set aside for it.' Some 93% agree that dedicated time is needed of whom 58% agree strongly. As a group co-ordinators are more likely to agree with this: all who are in the co-ordinator only role agree and 93% of co-ordinator participants agree as do 93% of participants only. Those working in smaller schools are also slightly more likely to agree strongly with this statement than others. As one non Masterclass participant commented on behalf of their colleague 'you need time out of school to use what you learn, as you learn it.'

The other two statements look at the extent to which Masterclassers are able to find time to fulfil certain aspects of their Masterclass role. In each case around three quarters provide a response showing they find it difficult to do this; 76% say this in connection with 'finding time to share learning with others in the school or authority' and 74% find it difficult to 'be able to do Masterclass work alongside other duties.' Obviously, not all find this to be the case with around 1 in 4 in each instance disagreeing. Librarians are more likely to disagree with both these statements, as are local authority staff. Teachers, those in promoted posts and TEI lecturers are all more likely to agree.

Of the 7 respondents who were interviewed as co-ordinators only, 1 disagreed with these statements in each instance.

One of the key issues for schools and local authorities in releasing their staff to fulfil their Masterclass duties is the absence of suitable cover. In some locations we were informed during the qualitative interviews that even if funding were available, it could be impossible to find a qualified member of staff to offer cover.

'What I have is a band of very enthusiastic teachers who want to do work locally and I cannot find sufficient funding to release them for a time. They get back from training all fired up and then realise they have classes to teach, reports to do ..and they can't get a week off to action their plans. It would take substantial funding to release the Masterclassers without disrupting the children's education.'

Local authority co-ordinator

Funding also emerged as an issue in relation to being able to obtain some of the ICT equipment that is demonstrated though Masterclass.

'It's kind of like torture. They show you these things and you think 'that's really good.' They're wonderful, but you haven't got them.'

Teacher in Primary school

Clearly, schools do not have endless resources to devote to new equipment purchases and an issue faced by some Masterclassers has been being able to persuade the school management to make the investment in ICT equipment. While this is a challenge, and rightly so, schools have generally been increasing the amount of hardware and software in use - partly as a result of Masterclass activities ( see section 5.1). Nevertheless, accessing funding has been a difficulty experienced by just over half our Masterclassers with 54% saying that 'a key difficulty has been getting funding to buy the equipment that you see being used.'

Chart 7.11

Agreement that funding for equipment purchase has been difficult to achieve

(Source : Q13)

Co-ordinators as a group are less likely to say that this is an issue, with just 43% of those in the co-ordinator only role agreeing and 39% of co-ordinator participants. Librarians (67%) and those in promoted posts (58%) are more likely to agree that this is a difficulty they face.

Co-ordinators recognise the importance of being able to give staff time to make use of their Masterclass learning. When asked what additional support they would provide to support their Masterclassers if time and resources permitted, most of the comment focused on time and cover, having more meetings or getting funding to facilitate training and cover. Details are :

Time / cover

• Time out / time for visits etc (with 8 mentions)
• Provide cover (4 mentions)
• Funding to free up time for Masterclass work (1 mention)
• Funding to allow Masterclassers to do research (1 mention)
• Funding to release teachers for training (1 mention)

Meetings

• Face to face meetings (2 mentions)
• More frequent meetings (1 mention)

Funding for:

• Grants for training (1 mention)
• Grants for equipment (1 mention)
• Funding for Masterclassers (1 mention)

Other suggestions

• Training for the School Board (1 mention)
• Make them ICT support in schools (1 mention)
• A full time Masterclass co-ordinator (2 mentions)
• New software (1 mention)
• Central pool of resources (1 mention)
• Case studies and good practice (1 mention)