The Scottish Government

« Previous | Contents | Next »

Listen

Acknowledgements

The Project Team would like to thank all of the schools, teachers and pupils for their participation in the study. As always in such reports, they remain anonymous, but their co-operation has been of fundamental importance and we appreciate the full responses they made to the various elements of the project. Thanks also go to Iain Smith, Dean of the Faculty of Education, University of Strathclyde, for his work on the assessment statistics and to the Scottish Executive for their funding and support.

Executive Summary

Introduction

The Impact of Information and Communication Technology ( ICT) in Scottish Schools: Phase 3 study was undertaken in 2004 against a background of significant and fast-paced development in the range of technologies available for educational and home use. The third in a series of studies, the key aims were to determine the impact of a range of initiatives instigated by the Scottish Executive over the past 10 years or so which aimed to give pupils opportunities to learn about the technologies, gain experience of using them and to develop skills in using them to further their own skills and understanding. In addition, an appreciation of the social implications and the development of critical attitudes to the pervasive nature of technology were encouraged. The initiatives focused on the provision of hardware, software and staff development programmes for teachers and were complemented by the development of curriculum guidelines setting out the government's expectations for the acquisition and development of skills across the school system.

Three key groups of stakeholders were identified: the teachers, the learners and the ICT co-ordinator in each school, the last of whom, it was anticipated, would have oversight of developments at local levels and beyond. More specifically, the study focused on pupils in Primary 7 (11-12 years), Secondary 2 (13-14 years) and Secondary 4 (15-16 years); i.e. at the end of primary schooling, at the point when pupils are choosing subjects to study towards external certification and at the end of compulsory schooling.

The aims specified by the Scottish Executive were:

1. to assess the ICT knowledge and skills of a nationally representative sample of pupils at Primary 7 (P7), Secondary 2 (S2) and Secondary 4 (S4) in order to determine levels of attainment at each stage as well as differences across stages and within subgroups, e.g. gender
2. to gather information on levels of access to, provision of and use of ICT by teachers, school managers and pupils within and out of schools, as well as their attitudes towards its use in education
3. to assess the impact of training provided by New Opportunities Funding on teachers' knowledge, skills, understanding and attitudes
4. to assess the ways in which ICT initiatives have impacted upon pedagogy
5. to assess the impact of ICT initiatives on school management and administration
6. f. to ascertain and compare hopes and expectations of the uses and applications of ICT, and the role ICT will play in the future for teachers, pupils and school managers/ ICT co-ordinators.

2. Phases 1 and 2

The earlier phases of the project (1999 and 2001) provided baseline data on the impact of the initiatives as they were introduced into schools. As this third phase builds on that data, a brief summary of the main findings of Phases 1 and 2 is presented here. The earlier studies identified a number of strengths and limitations regarding the integration of ICT into the classroom ( SEED, 2002). Pupils' knowledge and understanding was strongest in aspects of hardware and software, e.g. knowing the names and functions of elements of each, but was less secure on some processes such as data handling and in identifying and using appropriate strategies to complete particular tasks. Some assessment was made of practical skills but the lack of facilities for Internet and email tasks in many schools made it difficult to determine levels of competence. School co-ordinators indicated an increase in their workload over the two years between the studies, although the main activities continued to be trouble-shooting technical problems and securing equipment and resources. The provision of appropriate resources had been a priority throughout and, as a result, little progress had been made in increasing the administrative use of ICT centrally, although the quality of reports had improved.

In 2001, teachers varied in their enthusiasm for and commitment to ICT in the classroom. Most had computers at home which they used for both personal and school-related tasks, the latter being concerned primarily with administration and with finding and creating curriculum materials. The use of ICT to support learning and teaching was little developed. Feedback from pupils varied little from the first to the second phase. Most pupils had access to a computer outwith school and reported reasonable levels of confidence and competence in many basic activities ( e.g. word processing, email, using CD- ROMs). The tasks they engaged in at home and at school varied, with activities at home tending to be more multimedia in nature and demanding greater creativity and imagination. Activities in the classroom tended to be more teacher-directed, task-oriented and curriculum-driven.

Overall, attitudes to ICT were mainly positive, although a significant number of teachers thought that the real and potential benefits of the investment in ICT had been greatly exaggerated. One of the tensions identified was the extent to which ICT was regarded as a subject in its own right or as a means to learning within other subjects and contexts, with the development of ICT suites or labs tending to support the former view.

3. Phase 3: Gathering the evidence

As in phases 1 and 2, information was gathered through a series of questionnaires to pupils, teachers and ICT co-ordinators as well as a package of assessment instruments for pupils. Nationally representative samples of primary and secondary pupils were sought but insufficient numbers of schools agreed to participate, leaving sample sizes slightly smaller than desired. This has implications for the extent to which it is possible to generalise across Scotland from the findings, although they are sufficiently robust to give an accurate picture of the impact of ICT on the schools involved.

Questionnaires and pencil and paper assessment materials were distributed to schools in early 2004 and most were returned just before or during the summer vacation period. These were analysed to meet the aims of the study as listed above. The key findings are presented here.

4. Phase 3: The findings

The knowledge and skills of pupils at P7, S2 and S4

Pupils at all three stages achieved greater success, overall, on the assessment element of the survey than they had in the previous surveys. They were most secure on aspects of word processing, graphics and communication skills, while information and data handling skills were less well-established (as in previous years). More pupils were using the Internet as a source of information for project work and similar tasks, and were familiar with many of its features. There was evidence of growing familiarity with peripherals and personal technologies such as mobile phones and MP3 players. While pupils performed well on items assessing knowledge and understanding, they were less successful on those that required the synthesis of ideas, critical thinking or problem-solving skills. Success rates were also lower on items that asked pupils to evaluate the use of various technologies and to consider the social and cultural impact of technological innovations.

The aspects of ICT where pupils were most successful were those that teachers and pupils said happened more frequently in the classroom, e.g. writing stories, producing newsletters, searching the Internet for research activities. Where they were less successful, the technologies involved ( e.g. video-conferencing) and/or the procedures assessed ( e.g. data handling) were less familiar. They were also the areas where teachers showed reduced confidence and familiarity.

Performance levels showed an increasing awareness and understanding across ages and stages, although the differences between P7 and S2 were not substantial and certainly less than might have been expected from 2 additional years of schooling. Performance levels at the two stages showed considerable overlap. (Similar findings have been made in the Assessment of Achievement Programme's surveys of performance in other subject areas.) The differences between S2 and S4 were significant and in line with the age difference.

Analysis for gender differences showed a slightly superior performance on the part of the girls across the stages assessed and in aspects where they have tended to dominate in the past, e.g. language-based activities. The differences were marginal, though, and therefore should not be over-emphasised.

Many pupils themselves felt that they had learned much of what they knew out of school. This was strongest at P7 but declined with the stage; at S4 the balance was slightly in favour of the school. This might be anticipated given the increasingly specialised support needed as pupils progress through secondary school. In the questionnaires, pupils across the stages were generally fairly confident of their ability across a range of applications and procedures. While this confidence was not always reflected in the assessment data, performance levels indicate that many were competent, if not proficient, with much of the technology.

Access to, use of and attitudes towards ICT

Access to computers out of school reached over 90% for teachers, pupils and co-ordinators, an increase over previous figures. However, there remain small but educationally significant numbers of pupils and teachers who do not have access out of school. Most of the home computers were connected to the Internet, many by broadband. The difference identified in the earlier phases, where teachers were more work-oriented while pupils tended to experiment and play out of school, was also in evidence in this third phase.

There is continuing evidence that much learning is taking place out of school and the opportunities for accessing a range of technologies in the home are increasing. The emphasis seems to have moved from computers to the range of peripherals that enable people to use their computers for a variety of purposes and more creatively, e.g. digital cameras (still and video), mobile phones, MP3 players. Some schools are developing strategies that capitalise on the powerful potential of these technologies and use them to support learning and teaching. While teachers tended to use their home computers for administration and producing resources for the classroom, they did appear to be more creative and use a wider range of technologies for personal use than previously.

In school, the use of computers within the classroom to support learning was very patchy, with small percentages of pupils experiencing this on a regular basis. Word processing, graphics and searching the Internet dominated. Much of this activity was concerned with writing essays or reports and producing presentations. A key obstacle to extending ICT-related activity is the reported lack of modern, up-to-date computers available in the classroom, particularly in the secondary school. Most primary schools have a number of computers for pupil use in the classroom as well as a computer suite/lab that can be booked. When computers are readily available, they tend to be used throughout the day and for a variety of purposes, becoming a relatively routine part of the day-to-day activities of the classroom. In secondary schools, fewer classrooms had more than one or two computers (other than computing or business studies classrooms) and most relied on scheduling time at the computer suite/lab. Technical failures and the inadequacy of what was available caused frustration for both pupils and teachers.

Learning about the new technologies continues to be an issue throughout school - upgrades, new peripherals, software and the pace of change all make it necessary to keep adding skills and knowledge. What would be an appropriate balance of teaching and learning about ICT and using ICT as an integral component of classroom practice, and how should the two be interwoven most effectively?

Learning about computers is almost essential for future employment, according to many of the pupils. It has other benefits, though - it can be fun, interesting and computers are everywhere so it you will need to know about them. Small numbers of pupils and teachers urged a little caution alongside the enthusiasm.

Staff development and training

Most teachers and co-ordinators had received ICT training of some kind, the most frequently cited being the New Opportunities Fund ( NOF) initiative. Staff development, of whatever form, had benefited them in various ways, the main one being increased confidence in working with technology. This confidence had led them to try new approaches and activities, many of which they felt had been effective and had benefited the pupils. In general, they recognised the potential for enhancing the pupils' learning experiences and identified a number of training needs related to specific equipment and software, e.g. interactive whiteboards, digital videos, video-conferencing and virtual learning environments. Teachers also sought staff development in embedding ICT in the learning and teaching process in the classroom. However, they did not want training for technologies that they could not access within the school, and the provision of the newer ones appeared to be very patchy.

The impact upon pedagogy

ICT had influenced a range of aspects of classroom practice, according to the teachers. A number of positive features were identified. Using a laptop and data projector in the classroom, for example, allowed animations and simulations that helped pupils access difficult concepts. In addition to helping pupils grasp new concepts, other benefits that teachers identified, perhaps more peripheral to learning, included increased interest and motivation, and greater independence in learning.

The focus in the classroom had shifted somewhat from learning aboutICT to learning with the support of or throughICT. Teachers wanted to know how best to use the technology available for the benefit of the pupils. While pupils might know more about ICT and be more comfortable with it, teachers did not see this to be as threatening as did the teachers in the previous surveys. However, the pressures of other priorities, on time and resources, is a significant obstacle to learning how to adapt old practices and adopt new ones in order to make the most of opportunities that the new technologies can offer.

It is difficult to discern any pattern in the impact on pedagogy. There is certainly evidence of changes in the activities teachers and pupils undertake but whether this means a fundamental change in the pedagogical strategies deployed is unclear and requires research focused on the interaction in the classroom across a range of technological innovations.

There are some excellent examples of good practice in schools. Where significant developments have occurred, this has often been down to the drive of one or more enthusiasts within a school or department, 'champions' who introduce new approaches to teaching and learning and influence their colleagues by example. This may take too long - a more effective method of dissemination, within and across schools, is needed. The role that Learning and Teaching Scotland has in this regard might be enhanced, given the numbers of teachers who said that they looked there for support and resources on a regular basis.

There remains a significant proportion of teachers who are less enthusiastic about, if not hostile to, the drive to introduce and develop ICT in schools and across the curriculum. For those who are ambivalent, the frequent failures of equipment and software to deliver and the lack of certainty that they will on any particular occasion are having a negative impact.

School management, administration and ICT

Many of the schools in the study had developed or were developing centralised, computer-supported systems for record-keeping, assessment data and reporting to parents. Whole school systems of electronic communication were more developed in the secondary sector, perhaps as a result of their size. Communication networks are now extending into the community and beyond, for both administrative and educational reasons.

Given the outward movement of electronic communication from the school, it is important that all involved are clear with regard to ethical and legal requirements and, while there are policies in schools and teachers are confident that many of the safeguards are in place, gaps remain. Most notably, pupils must be made aware of the consequences of their own actions and become responsible users of the technologies provided.

Where progress was not being made, teachers sometimes commented that ICT was no longer on the development plan. It may be that once schools feel that they have achieved certain (quantitative) targets, the rest will almost inevitably fall in to place. The evidence is that there is a lot more to integrating ICT into the educational experience of pupils than achieving a set ratio of computers to pupils and networking them.

Looking to the future

The general picture is one of teachers coming to terms with the (physical) impact of the government's ICT initiatives and the presence of the new technologies, and beginning to work out how best to use them. The majority of teachers were positive, if not always openly enthusiastic, about the potential for improving learning and attainment. As in previous surveys, primary teachers saw a greater opportunity for breaking down age- and subject-related barriers in schools than did their secondary colleagues, with almost half in each sector believing that the classrooms of the future would be significantly different from those of today, as a result of the impact of ICT.

With sufficient access to appropriate technologies, most teachers wanted to learn more about and to develop greater skill in using ICT to support learning across the curriculum, in new, relevant and exciting ways. The biggest obstacle to achieving their aims was time - time to learn new skills, find out about resources and technologies, plan and try out new approaches to teaching and learning and time to reflect upon and consolidate their experiences - and to share them with others. Competing priorities and, for some, a lack of access to relevant technologies outwith school continue to impede progress. To a certain extent, the teachers want what the pupils have had - time, preferably out of school, to play, experiment, try out new technologies and then, importantly, to have the facilities to use their new knowledge and skills in the classroom. For many teachers, learning from colleagues was the key way in which they developed new skills and learned of resources. There is considerable evidence that 'hot information' such as this is much more effective in bringing about significant change to practice than the 'cold' evidence of printed policy documents and guidelines, and ways of exploiting this should be considered.

5. Comparisons with Phases 1 and 2

Many of the questions and assessment items were common to the earlier phases of the study and a comparison across the three studies yields some interesting findings. There was clear evidence of change across most of the aspects investigated. Pupils' confidence and competence had increased alongside an increase in the range and frequency of activities within school, particularly in the primary sector and with regard to the Internet. However, pupils' interest in and enthusiasm for using computers was still not being met in school, although the situation had improved. In secondary schools, the picture was very patchy across subject areas.

Pupil and teachers tended to use computers more out of school, with a wider range of activities than previously, most notably in the use of email and the Internet. There was evidence of a growing interest in and use of peripherals for computers ( e.g. digital cameras, mobile phones, etc.). Staff in both sectors were more confident in using ICT across a range of technologies and teachers in 2004 seemed less concerned that pupils might be more competent than they were than had been the case in 2001. Work in schools is supported through various policies and guidelines designed to protect children and staff.

Most staff viewed the introduction of ICT into the classroom as a positive and potentially very beneficial development, although some remain to be convinced, particularly in the secondary sector. The main obstacles to further development and integration included funding for the new technologies and the infrastructure to support them, time to develop (staff) competence and confidence and the lack of a ready mechanism by which sharing of experiences and resources can be achieved.

6. Implications

Many of the basics are in place in terms of knowledge and skills for both pupils and teachers. For pupils, these were mostly acquired out of school and therefore they are unlikely to have been acquired systematically, leaving gaps and blanks within and across pupils. While it is important to attend to these, it may be that the best time is when there is a good reason to acquire the skill, i.e. within the context of a purposeful task. Beyond that, learning about the wider, social implications of ICT use, etiquette, protocol and security dimensions are all important components of being ICT-literate that pupils should acquire.

At present, pupils are developing skills on a range of technologies that have barely made any impact on the school as yet. Strategies for incorporating them into the educational experiences of pupils are developing. Mobile phones are used to compile weblogs when away from home on school visits and can be used as personal response systems (with some modification), similar to those used in several universities and on 'Who wants to be a Millionaire?' School radio broadcasts direct to MP3 players can carry information immediately and directly. These are aspects that could benefit from additional research and evaluation, drawing on work already going on in schools and colleges.

Alongside such developments, it is necessary to remember the small but important percentage of pupils without access to computers or other forms of technology and to ensure that they are adequately catered for through other strategies.

Staff development made an important contribution to the confidence and attitudes of many teachers, giving them some sense of security as they tried new strategies and technologies in the classroom. Most teachers have undergone some form of basic training. They are now looking for further staff development on specific technologies, software and electronic resources. There is a need for short, focused programmes that take account of teachers' existing levels of competence. However, these also need to be harmonised with the installation of the appropriate hardware etc., in order that teachers have a purpose for learning about it and the opportunity to practise in their own time.

There is evidence of a move towards integrating ICT into the everyday experiences of pupils through purposeful tasks and the effective use of different aspects of technology to support learning. The extent to which this has happened is limited as yet, both within and across schools. In previous surveys the key emphasis had been on acquiring the necessary hardware, networking, etc. and on accessing training to allow developments to begin.

Teachers reported a range of technology-related strategies that they had developed across the curriculum. Many pupils did not benefit from these, however. An effective strategy for the dissemination of good practice should harness the enthusiasm of the 'champions' out there to spread the word, to demonstrate what can be done. In addition, more focused research on such innovations with the aim of providing support for other teachers is required.

In summary, the picture presented in the Phase 3 findings shows that ICT is becoming embedded into the everyday experiences of many pupils and that they know and can do more than previously. However, many of the problems highlighted in the previous surveys remain. Both teachers and pupils are positive about the impact so far and believe that considerable potential remains. The tensions between ICT use in and out of school - the different tasks/activities, the quality of provision, nature of provision, time, the social context - are worthy of further exploration.

Introduction

This is the third in a series of studies into the impact of information and communication technologies ( ICT) in Scottish schools. The first was commissioned by the Scottish Executive in 1999 and followed a period of significant investment by the Executive into improving the provision of hardware, networking and software in schools. This provision was complemented by staff development programmes across the country. A second survey was undertaken two years later (2001) with the aim of identifying any change in practice in schools since the initial survey. This third survey, undertaken in 2004, provides further indication of progress, five years after the initial information was gathered. While the methods used in all three phases are virtually identical, the 2004 survey has tried to pick up on the issues raised by pupils and teachers, including co-ordinators, in earlier surveys and to determine the extent to which they are still pertinent.

One of the most significant changes in education in recent years has been the almost universal availability of a range of information and communication technologies at work, school and, significantly for the education system, at home. In Scotland, as in other countries, national education policy includes the expectation that all pupils will acquire and develop competence across a range of technologies, starting in their first experiences of formal education ( http://www.ltscotland.org.uk/5to14/guidelines/ict/index.asp). It is recognised that ICT literacy will be important in terms of future employment, given the ubiquitous nature of technology in business and industry, and that students will need knowledge and skills in handling the new technologies. While learning about computers is therefore important, learning through computers also features strongly. The increased access to information, knowledge and resources that the Internet offers has already transformed the world of work and is beginning to impact upon learning and teaching in schools.

The earlier phases of the study showed little change across the period following the introduction of the initiatives in ICT. The focus at the time tended to be on embedding the technology into the school and the classroom, with concerns over access to and the standard of the equipment provided. Both teachers and pupils used ICT frequently, in and out of school. The dominant technology used was the computer, usually connected to the Internet. Teachers tended to be 'task-oriented' in the activities they undertook, while pupils were more creative and explored the technologies, experimenting and learning in informal and less systematic ways.

Many teachers were ambivalent about the introduction of ICT and remained to be convinced. They became frustrated when equipment or materials did not work properly and felt that staff development had not kept pace with technological developments. They did feel that ICT could benefit learning and teaching but the practicalities had yet to be worked out. Almost all pupils saw learning about the new technologies as important, at school and in later life. They enjoyed activities that allowed them to use their imagination and develop their own ideas. Key benefits of using a computer, according to pupils, included improved presentation and the ability to make quick changes to drafts. Over the two years between Phases 1 and 2 there was evidence of increased use of the Internet, CD- ROMs and email by pupils in school. Phase 3 was primarily concerned with determining how much had changed, and in which aspects, since the earlier studies in 1999 and 2001.

The general aims of the project were:

• to provide information on the impact of the ICT initiatives on pupils' skills, knowledge, understanding, attitudes to and experiences of ICT five years after the implementation of the major ICT initiatives
• to assess the impact of ICT initiatives on teachers' attitudes to ICT, their perceptions of their technical skills and the use of ICT in the classroom
• to assess the impact of ICT initiatives on the use of ICT by school staff in a range of support activities.

2. Scope and design

The key participants in the ICT developments were the pupils, their teachers and the ICT co-ordinator/manager within their school. Materials designed to assess pupils' knowledge, understanding and skills were administered alongside questionnaires which sought evidence of access, usage and attitudes to ICT and aspirations and expectations for the future. Similar questionnaires were developed for teachers and co-ordinators. The questionnaires issued to teachers, pupils and co-ordinators contained a core of common items with additional cohort-specific questions.

The sample

Samples of pupils at three stages of schooling (Primary 7 - 11/12 years; Secondary 2 - 13/14 years; and Secondary 4 - 15/16 years) were drawn from the Scottish Executive's database of schools, using a sampling frame stratified by local authority and size of school. Within schools, pupils were selected on a random basis, while ensuring equal numbers of boys and girls at each stage. In the secondary sample, S2 and S4 pupils were drawn from the same schools and from across each stage cohort, i.e. not restricted to those taking computing courses. In the event, the number of schools that completed the various aspects of the study was lower than intended, mainly due to problems in securing the participation of schools. As a result, it cannot be assumed that the samples are nationally representative of pupils in Scottish schools, and therefore generalisability is limited, although the numbers are robust in terms of item analysis.

Methods

Three main types of data collection were used:

1. written assessment of pupils
2. practical assessment of pupils
3. questionnaires to pupils, teachers and ICT co-ordinators/managers.

The assessment framework

The 5-14 ICT Guidelines provided the framework for the assessment materials, written and practical, particularly for the two younger groups of pupils, with other curriculum guidance such as Standard Grade requirements taken into consideration. While the materials drew heavily on those of previous surveys, some modification of existing items and the development of additional items were required to bring the assessment package more firmly into line with curriculum statements and changes in ICT since 2001.

The assessment package

The assessment package included both written and practical modes of assessment:

i pencil and paper assessment - primarily for the display of knowledge and understanding, e.g. terminology, procedural knowledge, the purposes and capabilities of ICT, problem diagnosis and remediation

ii performance assessment - where pupils used ICT to perform specific tasks including word processing, data handling and various forms of communication (email, web searches and using websites).

The written papers contained a number of common tasks to allow comparisons to be made across pupil groups. In addition, approximately two-thirds of the tasks used in the previous surveys were retained to provide comparisons across the phases. Two booklets were compiled for each stage.

In the practical assessment, pupils were presented with a series of tasks within the context of a topic, e.g. 'Into Space', suitable for the stage of the pupils involved. The topic was presented as a simulated website with all of the relevant features such as forward/back buttons. This allowed demonstration of skills, e.g. email, within a safe environment that did not need access to the Internet. The websites were recorded onto a CD and the assessment was administered by trained assessors who observed the pupils at work and completed checklists accordingly.

Questionnaire surveys

Printed questionnaires were distributed to schools in the same package as the assessment booklets. Questionnaires for staff were also made available online. Information was sought from each group as indicated:

P7, S2 and S4 pupils

Information on: their attitudes; their ICT educational experiences; self-assessment of competence and confidence in ICT; access to a range of ICT facilities and resources (both at home and in school); their interests, attitudes, hopes and expectations for the future.

Primary and secondary classroom teachers

Information on: their attitudes, preferred and actual uses of ICT; their expectations and aspirations for the future; current levels of skills, access and level of use of ICT (area and local networks, Internet etc.); changes in pedagogy; staff development experiences, including strengths and weaknesses of NOF/Masterclass training, and future needs; and responses to the 'digital gap' between home and school.

School managers/ ICT co-ordinators

Information on: their awareness of and involvement in the initiatives; the existence of a school policy on ICT; awareness of and support for changes in pedagogy; their perceived priorities with respect to ICT and resource allocation, development planning, the impact of staff development ( NOF/Masterclass etc.); their intentions and aspirations for the future; responses to the 'digital gap' between home and school.

3. The involvement of schools

In total, 65 primary schools and 52 secondary schools agreed to participate in the Phase 3 study. They were sent parcels of assessment booklets, questionnaires and guidelines for completion. Schools were reluctant to become involved and a significant number declined the initial invitation to participate. The final sample included a good number of reserve schools. In the event, the final number of booklets and questionnaires was as in Table 1.

Table 1: Numbers of assessment booklets and questionnaires returned by October 2004

In May-June 2004, a sub-sample of the schools that had undertaken the written component was invited to participate in the practical phase of the survey. Several of the schools declined to participate and, as a result, the numbers involved in this phase were small, of the order of 12-20 pupils per activity. All, however, were pupils who had participated in the written component. The practical phase is therefore presented as a pilot exercise.

« Previous | Contents | Next »