Scottish Government

Evaluation of the Masterclass Initiative

CHAPTER 5 : IMPACT OF MASTERCLASS

This chapter of the report attempts to assess the impact that Masterclass has had on the use of ICT in schools. A number of measures are considered in this review, but it must be borne in mind that other initiatives and support will have played their part in creating an environment conducive to the use of ICT.

The specific evidence reviewed in this chapter seeks to assess the physical impact that Masterclass has had on the presence and use of ICT equipment in schools, before looking at the extent to which specific expectations for Masterclass have been delivered. As outlined at chapter four, expectations centred on developments in personal skill levels for Masterclassers, and the extent to which Masterclass would build capacity to network, share good practice and become ICT champions.

5.1 ICT EQUIPMENT INTRODUCED IN SCHOOLS

Various questions were asked of Masterclassers and co-ordinators to assess the extent to which new ICT equipment had been introduced to schools since Masterclass began. The qualitative stage had indicated that there is already a vast array of ICT resources in schools, but also that there is a need for constant upgrades and improved supporting resources. Schools spoke of the difficulty of including ICT equipment in classrooms not designed with cabling in mind, while others suffer from being in a black spot for the use of wireless technology. There are also difficulties with integrating new equipment and software with old.

Two perspectives on the extent to which ICT equipment has been introduced in schools since Masterclass began were obtained - one from the point of view of local authority staff including all co-ordinators and one from school based staff. There are some slight differences in the patterns of equipment use detailed by these groups.

Table 5.1

ICT equipment introduced since Masterclass

• Both groups of respondents agree that the main items of equipment likely to have been introduced into schools since Masterclass began are whiteboards or smartboards, and digital projectors and cameras.
• Local authority staff are slightly more likely than school staff to mention most of the other types of equipment and this is because local authority staff are more likely to be aware of minority items. As such, if this piece of equipment has been introduced in just 1 school in the area, a local authority staff member would be aware of this, while school staff are likely to know mainly about equipment introduced at their own school.

If Masterclass participants are drawn from a good mix of schools throughout the area, then the data on the right of this table can be seen as representative of the spread of equipment types across schools. The data shown under the column for local authority staff is more representative of the range of equipment available for sharing across schools.

A range of other equipment was mentioned including scanners, webcams, printers and DVD/CD writers, servers and digipens.

To get a feel for the depth of investment involved, school based staff were asked to specify how many items of each type had been introduced since the start of Masterclass. The table below indicates firstly the percentage of staff saying each had been made available. It then shows the average number introduced into schools receiving this type of investment and the average number computed on the basis of all schools - which allows comparability as the base is the same for each calculation.

Table 5.2

Depth of investment in ICT equipment in schools

Although whiteboards and digital cameras, video recorders and projectors were obtained by a wide number of schools, they were bought in fewer numbers. In contrast, about 1 in 3 schools received a significant allocation of pc equipment with internet access (40 units per school receiving such an investment on average).

It was not thought that co-ordinators would be able to give figures on the numbers of different types of ICT equipment introduced to schools with any degree of accuracy during a telephone interview. A more worthwhile subject to pursue with them was to try and determine to what extent the investments made were part of an on-going programme of investment in any case or if Masterclass had paved the way for the introduction of the technology.

As the table below shows, the impact of Masterclass is more evident on some types of equipment than others. The first column indicates the number of local authority staff who say that the particular piece of equipment has been introduced in a school since Masterclass began. The next column shows what percentage of these introductions is attributed to Masterclass. By implication the balance is due to other plans that were already in place, or the origin of the investment is not known.

Table 5.3

Role of Masterclass in accessing the equipment

Masterclass is largely accredited with supporting investments of various types of digital equipment, associated editing and mixing equipment and whiteboards. Although more numerous in terms of the quantities obtained, more standard pc based investments arise from other plans and initiatives.

However, while current investment appears to have concentrated on specific areas and types of equipment, interest in new equipment and technology is widespread amongst Masterclassers and the qualitative interviews reveal that various new pieces of software / hardware are being piloted to assess their suitability for use. Indeed a majority of those taking part in our survey anticipate that further investments will be made with 74% of local authority staff foreseeing further equipment being introduced in schools in the next year (Q25g) or so and 71% of school based staff anticipating some investment in their school (Q25c).

Table 5.4 provides an analysis of the types of equipment which are believed likely to be introduced in the next year or so, from both the point of view of school based and local authority staff. It should be noted that the figures in italics are provided for comparative purposes and show what types of equipment have already been introduced.

Table 5.4

Anticipated investment in new ICT equipment

Again there are some differences between the patterns of future investment reported by local authority staff and those in schools and these will arise for the reasons outlined earlier.

• There is a consensus from both groups that the main area of new investment will be in smartboards / whiteboards with staff in over one-third of schools anticipating that they will get at least one, and almost two fifths of local authority staff saying the same.
• The same is true for digital projectors. Although one of the most common investments to date, around one fifth to one quarter suggest that this will feature in future investments.
• Beyond this local authority staff are more likely to report that other types of investment will be made on behalf of schools in the area, than is envisaged by schools themselves. In particular local authority staff anticipate a reasonable amount of investment in computer based equipment - either wireless technology or with internet access.
• There is some interest also in laptops and pc tablets, with more schools and local authority staff mentioning these as future investments than currently have these.

Again a wide range of other types of equipment are identified as future investments including servers, interactive systems, dataloggers and digipens.

5.2 ICT LITERACY AND SKILLS

One of the key motivators for staff taking part in Masterclass was the expectation that their skills in using ICT would be enhanced. However, three fifths of those taking part in Masterclass state that were already 'very ICT literate' at the time they started Masterclass while one third describe themselves as interested in ICT. Local authority staff are the group most likely to perceive themselves to be very ICT literate (82%) as do co-ordinators (78%) and those in larger schools of more than 700 pupils (67%).

This provides an interesting baseline position against which to determine the impact of Masterclass on ICT skill levels. When asked to assess their current skill levels, we do see some movement into the more literate classification and, at the most basic level, the survey indicates improvements in skill levels for some participants.

Table 5.5

ICT skills held

Local authority staff (85%) and co-ordinators (89%) remain as the groups who are now most likely to describe themselves as very ICT literate. However, the gap between these groups and school-based staff has narrowed. Furthermore, there is no real difference in the extent to which staff in different sized schools lay claim to being very ICT literate now, showing some evening up in skill levels across the sectors.

The qualitative research backs up the improvement in skill levels and many Masterclass participants spoke about the extent to which Masterclass had given them confidence in the use of ICT and encouraged them to try new things. For example, one participant spoke of having acquired

'an ability to use the net confidently, finding safe sites for children to access. I'm now able to take a risk and do new things whereas before I was cautious about what I would try. I'm willing to do things at home and I've been able to encourage others to try new things. Masterclass has raised my awareness of how important ICT is, but that it's not about your basic skills, it's what you get out of it.'

Nursery depute head

Two attitude statements (Q12) were used in the survey that examine specifically how Masterclass has impacted on ICT skill levels. These indicate that Masterclass is well regarded by participants in this respect. Firstly, 90% agree 'being a Masterclasser has helped develop my ICT skills' and 59% agree strongly with this. Of the 49 Masterclassers who disagree, 41 regarded themselves as very ICT literate at the outset in any case.

A similar proportion (89%) also agree that 'being a Masterclasser has made me more confident in my ICT skills and try new things' with 56% agreeing strongly with this. Again a large proportion of those who disagree were already very ICT literate at the outset (44 out of 54 people).

5.3 NETWORKING ACTIVITY

The earlier analysis indicates that, for many Masterclassers, the opportunity to network with others is one of the expected outcomes of participation. Again, this is an expectation which has largely been realised with 90% of Masterclassers saying that Masterclass has provided opportunities to network with others (Q7a). However, this did vary and TEI lecturers in particular were more evenly divided on whether they had realised this expectation (53% said they had).

A follow up question (Q7b) reveals, however, that networking activity is perhaps decreasing. When asked if they still network with people met through Masterclass, only 76% of all Masterclassers say that they do. Again this varies by respondent type with the drop off most pronounced amongst TEI lecturers. Thus of the 17 TEI lecturers interviewed, only 9 ever found Masterclass to help them network and and just 5 still network with people they met through Masterclass.

Co-ordinators were also asked to provide their views on the networking activity of Masterclassers both with others in the local authority area and across local authorities. From the results, networking activity does appear to operate to a degree within local authority areas, but is not widely found across areas.

Chart 5.1

Extent of networking by Masterclassers

(Source : Q30a / 30b)

As co-ordinators point out, the degree of networking varies from person to person. Some individuals make efforts to keep in touch and use the resources available to them while others will prioritise other activities.

It is interesting that there is relatively little belief that networking with staff from other local authority areas occurs. When conducting the qualitative interviews, we did speak with Masterclassers who had made contacts with those working in other local authority areas and had kept in touch. There were also comments made about it being useful to see what is being done in other locations and talk of the value of school visits, for example, as part of the training events. This suggests that there are likely to be benefits to wider networking than occurs at present and perhaps there should be some greater effort made to facilitate inter local authority networking. Some suggestions on how this might arise are provided in the recommendations.

The fact that much networking occurs within local authority areas reflects the fact that many Masterclassers depend upon the efforts of their co-ordinators to create networking opportunities.

'You need time to network properly and not enough is set aside for this. But we get some help through bi-monthly meetings facilitated by the council. There is also an element of sharing through these meetings.'

Primary depute head

There were comments from some of our qualitative respondents about the difficulties that they encountered in accessing lists of like-minded people to network with and so co-ordinator support is vital. One respondent stated that this had been a beneficial outcome from a conference he had attended as this had produced a list of contacts that he had not been able to gather in any other way.

When asked what routes or forums they could use to network with other Masterclassers a huge array of different channels is identified. The most commonly identified channel for participants is through email contacts and email newsletters, mentioned by 59% (Q7c). Meetings also are important for networking opportunities, with 31% of Masterclassers referring to this channel of communication.

Local authority staff refer to meetings arranged by the local authority co-ordinator (mentioned by 82%), the on-line community (73%) and electronic noticeboards (45%) as the three key channels of communication (Q7ci).

More formal events such as conferences and training/seminars are also mentioned by smaller proportions of both co-ordinators and participants.

Further information on networking opportunities was obtained from local authority staff by asking them what channels they have used and how often. Almost three in four (73%) have used meetings arranged by a local authority co-ordinator, and almost one in two (48%) have used the talk2learn community (Q7cii).

In terms of the frequency of use of different channels of communication, those most frequently used by local authority staff are email messages / newsletters and electronic noticeboards operated by the local authority (the former with an average use of 16 times per year and the latter 17 times a year). Meetings arranged by a local authority co-ordinator are less frequently used (around 7-8 times per year), and the on-line community is used almost once a month (Q7ciii).

Local authority education departments are credited by 79% of Masterclassers with supporting networking opportunities (Q7d). The ways in which this support is manifest is extensive (Q7e), but certain key approaches appear with some regularity. In particular, mention is made of the hosting of meetings - with half saying that this occurs. Beyond this, around 1 in 6 mention the personal support provided by the co-ordinator or other education department staff or the fact that the department is structured in a way that provides support. A further 7% mention just general support and encouragement (7%). More tangible aspects of support identified include funding (9%), the provision of cover (8%) or time out (6%), websites (6%) courses and training (4%).

5.4 SHARING GOOD PRACTICE

One of the key aims of Masterclass is to facilitate the dissemination of good practice in the use of ICT across Scotland. The vast majority of Masterclassers (88%) believe that Masterclass had provided them with opportunities to share good practice with others (Q8a). TEI lecturers are the group least likely to say this, although 71% did feel they had been provided with these opportunities.

Importantly, the extent to which sharing with other Masterclassers is going on appears, if anything, to be increasing as shown in the following chart.

Chart 5.2

Change in extent of sharing good practice

(Source : Q8b)

A relatively small proportion of Masterclassers (Q8b - 28 people) commented that the opportunities to share have decreased and analysis of some of the comments that they provide suggests this is due to :

• Time constraints
• Because they are sharing with non Masterclassers instead
• Or through lack of opportunity

Where sharing activity is seen to be increasing, this is attributed to a variety of reasons. Key ones mentioned are:

• Through regular meetings
• By passing on information
• Increased contacts and talking with different people
• Through shared resources and materials

As with networking opportunities there are a variety of channels that are available to Masterclassers to share their ideas and problems with other Masterclassers. Those identified with the greatest frequency are the same channels as identified for facilitating networking, that is meetings, email contact and through websites.

Chart 5.3

Key methods used for sharing good practice

(Source : Q8c)

Conferences and training courses also play a role, as do initiatives led by ICT groups and activities within the local authority.

5.5 CREATING ICT CHAMPIONS

One of the declared outputs of Masterclass is that it will create a group of individuals who can champion ICT within schools across Scotland and this section of the report reviews the extent to which this has been realised. It is important to note, however, that not all shared this vision. Of the 25 co-ordinators taking part in the survey, 3 (12%) state that creating a team of ICT champions was 'not a very important part of Masterclass as far as I was concerned' and 1 describes this as unrealistic (Q29). In the qualitative research, one co-ordinator stated that this was because he already had a system in place for disseminating good practice and supporting the ICT cause and did not need his Masterclassers to do this.

Of the remainder, around half (52%) of co-ordinators regarded the creation of a team of ICT champions as an important goal from the outset and 32% thought it was something that might be a possible benefit.

Co-ordinators views on the amount of championing activity that is going on varies, with one third saying it depends on the individual Masterclasser concerned. Their views also suggest that more championing is being done within schools, than has been achieved within the local authority area. Chart 5.3 indicates the extent to which co-ordinators believe Masterclassers are acting as ICT champions at a local and school level.

Chart 5.4

Extent of ICT champion activity

(Source : Q31a / b)

The most common constraint identified by co-ordinators as impacting on the amount of championing activity that is going on is time (72%). Lack of cover (28%) and cost (12%) also emerge as associated reasons.

Nevertheless, participants do regard themselves as having championed ICT in schools in a number of ways. Only 1 of the school based staff taking part in the survey say they have undertaken none of 11 suggested championing activities and high proportions of all other Masterclassers claim to have championed ICT in at least one way. The suggested activities and degree of involvement across all Masterclassers is shown overleaf.

Chart 5.5

Ways in which Masterclassers have championed ICT

(Source : Q9)

The ways in which equipment is used is one of the main areas of activity with 92% of Masterclassers in schools claiming they have helped introduce new equipment to their school and 88% saying they have been able to suggest new or better ways to use existing equipment. However, relatively few (29%) have been able to share equipment with those working in other schools in their area.

Masterclassers in schools also claim to have been able to improve knowledge of how ICT can be used - 90% have achieved this within their school, 90% have achieved this specifically with other teachers in the school and 83% have noted this impact on pupils. A further 73% have spread knowledge to those working in other schools in their area, but just 37% have played this role for schools outwith the area.

Over half the Masterclassers have introduced a new ICT initiative and shared it as good practice with other Masterclassers (55%) and 41% either gave or plan to give a talk at a conference about their experiences. More than half (55%) have had an impact on ICT policy in the area.

Staff working in local authorities tend to be similar in terms of their degree of involvement in different activities to those in schools. They are less likely to have introduced new equipment in schools (81%), but are more likely to have facilitated sharing between schools (64%) and to have influenced local authority policy (77%).

In addition to the above types of championing activity, minorities of staff lay claim to other activity in this area. Some examples of the types of action undertaken include through training and in service days, involvement in specific projects and producing examples of good practice. Further detail on the specifics of certain types of championing activity was obtained by asking staff in schools to what extent they had :

• Led an ISD training session (79% had). 55% had done this for staff from their school only, 37% for staff from other schools only and 8% for both their own staff and those from other schools (Q11a-11b)
• Given presentations to groups of staff at school (57% had). Again this was slightly more likely to be for staff in their own school only -51%, or from another school only - 40% (Q11a / 11c)
• Taken part in school meetings where ICT policy is discussed (62% had - Q11a)
• Been involved in or selected for other ICT type initiatives ie Headstogether or Early Learning Forward Thinking (34% had - Q11a)

Staff in local authorities had slightly higher levels of participation in each of these activities than those in schools, but the pattern of involvement is broadly similar:

• Led an ISD training session (83% had). This was usually joint training (77%)
• Given presentations to groups of staff (64% had), usually as joint training (78%)
• Taken part in local authority meetings where ICT policy is discussed (65% had)
• Been involved in or selected for other ICT type initiatives ie Headstogether or Early Learning Forward Thinking (43% had)

As mentioned already, Masterclassers are hesitant to lay claim to the title of Masterclass champion. However, they have been involved in a number of championing activities and it is largely time constraints that prevent them from doing more. Much of the activity carried out is through informal means with Masterclassers offering support to their colleagues and sharing their knowledge through in service days or semi-formal presentations and talks at school. As authority wide activity is dependent upon opportunities offered by the education department at present, this occurs less frequently. For championing activity to work well, it needs to be properly bedded down both within schools and across the area. There were comments from some Masterclassers, for example, of the need for the staff member chosen to be a Masterclasser to have a degree of seniority within their school. This is to give them access to decision-making when budgets are being allocated and to ensure they are part of school management groups and can actually play a leading role in the use of ICT. Further detail on who would make the best candidate for Masterclass is provided at chapter 7.