The Scottish Government

« Previous | Contents | Next »

Listen

6. Performance on practical tasks

Assessment of pupils' practical skills in using computers was undertaken following the distribution of the questionnaires and assessment booklets. Trained assessors visited primary and secondary schools to assess a range of skills across a number of applications with pupils at P7, S2 and S4. In total, 10 primary and 13 secondary schools were visited. A larger number of schools was identified to participate in the practical phase but several declined to be involved when contacted by the assessors, mainly due to it taking place towards the end of the school year when it was difficult to accommodate additional events such as this. Assessment was undertaken on a one-to-one basis with pupils. The pupils assessed had already completed a questionnaire and a booklet in the main survey. They were identified by code number from the pupil list.

6.1 The assessment process

A series of four small (simulated) websites, each on a different topic, was developed: The Weather; Save Our Village; Into Space; and The Band. Each comprised a series of tasks, set within the context of the topic and targeted at the (st)age of the pupils involved. Each had an introductory or welcome page and a closing page.

Example 8: The front page of the Weather website

For all four websites, tasks covered the topics of:

1. setting up and inserting a CD
2. navigating the screen and the pages of the website
3. word processing
4. data handling
5. communication.

Websites were allocated to stages as shown, with one boy and one girl involved at each stage. Each pupil tackled only one website.

Table 21: Allocation of websites to pupils

The procedure was as follows. Pupils were asked to insert and start up the CD while the assessor observed and recorded their performance. On opening the CD, pupils were presented with a menu of the four websites and directed to one by the assessor. To navigate the website, pupils clicked on buttons on the bottom corners of each page ( e.g. back and next). The assessor prompted the pupil to move from page to page and to tackle the tasks they found there. There were basically three tasks on each website.

1. The first task on each website involved word processing. The task was explained to the pupil on the screen and, if s/he had any questions, the assessor was there to provide support. When the pupil was satisfied that s/he understood the task, s/he clicked where indicated ('click here'). This produced a Word document that could be manipulated by the pupil as indicated in the task. Pupils were also provided with hard copies of each worksheet on the website.

2. The second task asked pupils to handle some data, usually in the form of tables in a spreadsheet. At P7 and S2, pupils were normally required to insert additional data and to do some simple calculations e.g. total the numbers in columns. In websites targeted at S2 and S4, the drawing of graphs/charts was also required.

3. The third task on each website related to communication in some form or another - searching the Internet, interpreting websites and/or using email. Pupils were required to answer the questions by talking to the assessor. In two instances, the final tasks involved writing an email - pupils were provided with a template that could take text.

The final page in each website thanked the pupils for visiting the website and undertaking the tasks.

6.2 Assessing performance

Pupil performance was assessed using a detailed checklist for each of the websites, where each aspect assessed was measured on a 3-point scale:

1 managed competently
2 managed with some support
3 could not manage, even with support.

Assessors made note of any problems or difficulties that arose, whether with the assessment as a whole or on individual elements.

Setting up and navigating

Each of the websites required pupils to insert and open the CD and to navigate from page to page using the buttons provided. Although numbers were small, a pattern of progression across the three stages was discernible in the figures. Most pupils at P7 and S2 were successful in opening the CD and navigating the pages of the websites, with all of the S4 pupils coping successfully.

Word processing

Each of the websites involved word processing in some form. At P7 there were two websites and the pupils involved were able to complete most of the tasks successfully, although some were less sure of using a spell check. S2 pupils also tackled two websites and the majority of them completed most of the tasks successfully. Justifying text and managing graphs posed problems for a few. Almost all S4 pupils were successful on all aspects of the word processing tasks in the two websites they tackled.

Data handling

All four websites contained one task that focused on data handling in some form. P7 pupils were able to insert additional information into spreadsheets and tables but found the more complex tasks demanding. At S2, pupils coped well with most of the tasks in the spreadsheet section of the website. They were able to enter data and make simple calculations. They were unfamiliar with the graph wizard. At S4, pupils satisfactorily inserted data into charts and graphs but were less successful on the details of graphs and not all of them correctly completed the calculations involved.

Communication

The third of each series of tasks in each website focused on some aspect of communication: surfing the web; searching the web; using websites; and email. Most P7 pupils completed most of the tasks satisfactorily, although some lack of awareness of aspects of searching ( e.g. refining searches) was evident. More than half of the S2 pupils completed the tasks satisfactorily, while at S4 almost all pupils were competent across the range of tasks presented to them.

Summary

Although the numbers tackling each website were relatively small, the following observations are put forward for consideration.

• Performance on the practical tasks reflected performance data in the written assessment element of the survey.
• Most P7 pupils satisfactorily managed most of the basic tasks regarding using a CD and word processing.
• P7 pupils were less sure of spreadsheets, other than entering and reading individual pieces of information.
• Most P7 pupils were successful on the communication tasks, although interpreting web pages proved more difficult.
• S2 pupils performed a little better than P7 pupils in most instances.
• S2 pupils were successful on word processing and basic spreadsheet tasks, although they were unsure of turning data into graphical representations.
• S2 were, in the main, successful on the communication tasks.
• S4 pupils were very competent on most aspects of the ICT skills assessed; only data handling caused problems for some pupils.
• The informal feedback from pupils indicated that they found the websites interesting and were motivated to complete the tasks; assessors found them relatively easy to manage. As in previous surveys, access to appropriate computers and the Internet was limited.

7. Comparisons over time

In order that comparisons over time could be made, the assessment booklets and the questionnaires were designed so that they contained some questions that were identical to those used in Phases 1 and 2. It is on the responses to these questions that this section is based. The numbers of staff and pupils involved in each survey are shown in Table 22.

Table 22: Numbers of staff and pupils involved in 1999, 2001 and 2004

Pupils' knowledge, understanding and skills

In 1999, only P7 and S4 pupils were included in the survey. In 2001, the P7 pupils assessed in 1999 were in S2. A sub-sample of these pupils was traced and included in the second phase for both questionnaires and assessment booklets. This provided a longitudinal dimension. Therefore in 2001 three sets of comparisons were made: P7 in 1999 and 2001; S4 in 1999 and 2001; and the longitudinal comparison of performance in P7 with that in S2.

When the data from the first two surveys was compared in 2001, the overall finding was one of very little change. Although performance levels varied across the years, there were almost equal numbers of improvements and declines, with no real pattern of change discernible. At S4, while more success rates improved than declined, this appeared to be mainly due to a considerably improved performance on tasks assessing knowledge and understanding of software and its uses. Similarly, the longitudinal comparison yielded little pattern of any real change in performance levels. As a result, in this section, comparisons have been made primarily with the 2001 data, with reference to that from 1999 as relevant.

While attempts were made to retain as many as possible of the tasks that had been used in previous years, some modification was necessary for several reasons. Firstly, some were no longer appropriate ( e.g. the item on the millennium bug), while for others technology had moved on ( e.g. few pupils are now required to use a floppy disk or know how to format it). Other changes were necessary due to the evolution of hardware appearance; e.g. icons and command buttons have changed significantly in some instances and there has been a shift away from Apple Mac computers in schools, particularly primary, to the Windows environment.

In 2004, six items were common to all pupils who participated in the survey and these items had also been included in the 2001 survey. Comparisons of performance levels are made on this basis.

Table 23: Mean scores on each of the 6 common tasks, 2001-2004

Four of the six items show an increase in performance across the three years between Phases 2 and 3 (shaded) while two remain the same or slightly lower. The last two, both concerned with using the Internet, show the greatest improvement. This is one of the aspects of ICT on which both pupils and teachers reported increased activity over the years of the studies. An initial analysis of the total scores on other items common to 2001 and 2004 indicates little change or a slight improvement in performance across the two surveys. Further analysis by sub-question would be required before any firm conclusions could be made.

7.2 Pupils' experiences of and attitudes to ICT

Several sections of the questionnaires to pupils were common across the surveys, giving some indication of change over time at the three stages. A few questions appeared in only two surveys (2004 and 2001) and the data reflect this. (Unless otherwise stated, where data is presented in brackets, the first figure is the 2004 statistic, the second is 2001 and the third, if included, refers to 1999.)

i. Primary 7

P7 pupils were more confident in 2004 than previously, with 44% reporting that they 'knew a lot' or were 'real experts'; the figures for 2001 and 1999 had been 30% and 23% respectively. In school, they were using computers more frequently than before, with 'hardly ever' users dropping from 36% in 2001 to 21% in 2004. Out of school, they had also increased their use of computers. While the figures for use every day remained consistent at 31-32%, the figure for 3-4 times per week had risen from 17% in 1999 to 18% in 2001 and to 28% in 2004. There was, however, a decrease in the percentages using Nintendo, Playstation, etc. across the surveys, with using one more than four days a week falling from 44/45% in 1999/2001, to 10% in 2004 and figures for 'never' rising from 9% in 1999, to 11% in 2001 and 34% in 2004.

In the primary school, word processing and graphics were reported as the most frequent activities in all three surveys, with 80 to 95% of pupils experiencing these from time to time. Work on databases and spreadsheets became more common, with percentages of pupils reporting using them showing a rise of approximately 30% across the 5 years (between 40 and 50% in 1999 and 2001 to between 70 and 80% in 2004). More interestingly, use of the Web rose from 22% in 1999, through 48% in 2001 to 93% in 2004, and sending emails to pupils in other schools increased from 11% in 1999 to 31% in 2004. There was a slight decrease in the use of CD- ROMs for information, possibly influenced by the increase in using the Web for this. Most aspects of working with computers included in the questionnaires showed an increase across the three surveys.

Pupils increasingly acknowledged that computers helped them get on with their work faster (68%; 48%; 41%), that it helped them do better at school work (60%; 45%; 40%) and that the products (essays, presentations, etc.) were neater (90%; 76%; 75%). Computers made school work more interesting, according to over 80% in each survey, and there was an increase in the percentages who liked using computers because they got to use their own ideas and imagination (75%; 58%; 57%) Dislikes regarding using a computer changed little over the years, with only 'I don't get to use a computer often enough' showing a sizable decrease (46%; 59%; 58%).

In 2001, 73% of pupils reported that there was a computer in the house where they lived, rising to 88% in 2004, with figures for Internet connection rising from 71% to 88%. Percentages of pupils with computers in their own rooms rose from 33% (2001) to 87% (2004) over the three years between the studies. In 2001, 39% had a DVD player in the computer they used and in 2004 this figure had risen to 78%. It is possible that this has replaced games machines such as Playstation in the home for some children.

At home, almost all of the activities identified in the questionnaire showed an increase in frequency of use. Making drawings and designs (89%; 76%; 72%), sending email to family and friends (73%; 47%; 24%) and displaying or sending digital photographs (47%; 18%; 13%) showed substantial increases. In addition, 'doing my own programming' rose by 10% in 2004 from 28% in 1999.

Pupils indicated similar levels of support at home and school across the surveys and there is some indication that they are more able to try out different things on the school computers than before (2001 - 42%; 2004 - 55%).

ii. Secondary 2

Secondary 2 pupils were only included in 2001 and 2004 so only two sets of data are presented. The figures for confidence with computers show little difference across the two surveys, with a slight drop in those who know nothing at all, or very little (11% to 8%). Similarly, most taught themselves at home or with members of their family in both surveys. Some increase in frequency of use is indicated in the figures, however. In school, the 2004 figures show a slight increase in those using computers 3-4 times per week (10% to 22%) and 1-2 times per week (58% to 68%).

Some increases in what pupils used computers for in school were also in evidence when comparing 2001 and 2004. There were increases in:

• Writing essays, reports, etc. - from 66% to 81%
• Doing own programming - from 24% to 32%
• Displaying/sending digital photographs - 10% to 22%
• Searching for information on the Internet - from 65% to 93%
• Using email - from 34% to approximately 45%.

Use of databases and spreadsheets decreased slightly or stayed at the same level.

Pupils' reported 'likes' regarding using a computer showed some increases. In 2004, more pupils thought that they got on faster (63% to 70%) and thought that it helped them do better in their work (51% to 62%). More pupils thought they got to use their own ideas and imagination (54% to 64%). Increased percentages of pupils liked using the Internet to find information that they could not find in books (70% to 81%) and more reported that they enjoyed sending email on a school computer (34% to 53%).

Fewer complained about not getting to use a computer often enough (down from 69% to 55%) and slightly fewer thought that the work was too easy (33% to 26%). However, fewer also agreed that they got to try out different things at school (from 42% to 37%).

Slightly fewer in the 2004 sample had a computer in the house where they lived (2001 - 92%; 2004 - 89%) but more were connected to the Internet (2001 - 74%; 2004 - 86%). More had a computer in their own room (2001 - 41%; 2004 - 50%) and more had a DVD player in the computer they used (2001 - 42%; 2004 - 76%). As at P7, fewer were playing regularly on a Playstation or similar but the decrease was less dramatic.

Out of school, pupils used computers in similar ways to the 2001 sample of S2 pupils. Email and displaying or sending digital photographs showed sizable increases (58% to 80% and 24% to 56% respectively). There was some increase in the use of databases and spreadsheets out of school, but of the order of less than 8% in most instances. Support both in and out of school was at similar levels across the surveys.

iii Secondary 4

Confidence at S4 showed an increase across the surveys, with those who considered that they 'knew a lot' or were 'real experts' rising from 26% in 1999 to 34% in 2001 and 40% in 2004. Looking across all three surveys, S4 pupils were the only group who reported learning from school more than at home/with family members. There was a slight increase in 2004 over 2001 in the numbers who hardly ever used a computer in school or only used it a few days per month (20 to 23%), with a similar increase in those who used one 1-2 times per week (25 to 31%). For at home use, the figures are in line with other groups, with an increase from 58% to 65% in the percentages using one, either every day or 3-4 times per week.

In school, the figures for the various activities identified were very similar to those of previous years, with some evidence of reduced use of computers in some subject areas. Given the size of the S4 pupil samples, this may be relatively insignificant, but more research would provide information on this apparent trend. There was a slight increase in working with spreadsheets and databases. More pupils were doing their own programming (18% to 32%) and increases were also evident in searching for information on the Web (1999 - 36%; 2001 - 74%; 2004 - 86%) and using email (1999 - 10%; 2001 - 37%; 2004 - 51%).

Most of the figures regarding what S4 pupils liked about using computers stayed similar across the years, varying by only 1-5%. More did, however, think that it helped them do better at their school work (1999 - 43%; 2001 - 42%; 2004 - 53%) and that it could help find information that cannot be found in books (1999 - 55%; 2001 - 68%; 2004 - 84%). In addition, more pupils liked being able to send emails (1999 - 12%; 2001 - 41%; 2004 - 49%). Dislikes were also at a similar level across the years, although fewer complained that they did not get to use a computer often enough (1999 - 60%; 2001 - 54%; 2004 - 47%).

Ninety percent (90%) of S4 pupils used a computer out of school in 2004. Of these, 82% had a computer at home (93% in 2001). More had access within their own bedrooms (from 40% in 2001 to 54% in 2004), with Internet connection to this computer increasing from 35% in 2001 to 50% in 2004. In 2004, 73% had a DVD player as part of the computer, while the corresponding figure in 2001 was 42%.

There were increases in the following aspects of out-of-school use of computers:

• Using databases and spreadsheets (originally between 15% - 25%, the use of these has doubled);
• Doing own programming (18% to 40%).

Email (58%; 63%; 29%), using the Web (57%; 68%; 40%) and displaying or sending digital photographs (66%; 24%; 15%) all showed increases since 1999, though not necessarily since 2001.

Writing stories, letters, etc. showed a change from 69% in 1999 to 71% in 2001 and 52% in 2004; similarly, drawing and designing was reported by 61% in 1999, 59% in 2001 and 37% in 2004. Support in and out of school was at similar levels across the surveys, although there was a slight increase in the numbers who could spend as long as they liked on computers in school (6% to 14%) with a reduction in the number who thought they could try out lots of different things at home (92% to 84%).

Summary

• All stages showed an increase in pupils' self reports of competence on computers.
• There was an increase in computer-based activities in primary schools, with little change at S2 and evidence of some polarisation at S4 (into those who do use computers and those who do not). Unless pupils are taking business education, computer studies or a similar subject, there appear to be reduced opportunities for working with/through computers as pupils get older.
• For all three stages there was an increase in use of computers out of school in specific areas: sending email, using the Internet for information and sending/displaying digital photographs.
• There is some evidence that pupils, overall, were getting slightly more access in school in 2004 - but it is still not enough to satisfy them.
• While figures for access out of school have not changed much, those for connection to the Internet show a significant increase at all three stages. This is also reflected in the increased use of email and the Web.
• There was a decrease in the percentages using games machines such as Playstation and Nintendo across all three stages.
• Attitudes to computers in terms of likes and dislikes were fairly constant across the years, although more pupils appreciated some of the benefits they brought, e.g. impact on work, including presentation.
• The drive to greater integration of computer-based technologies in Scottish schools seems to have made more of an impact on the primary than the secondary sector, to date, in terms of the day-to-day experiences of pupils.

7.3 The co-ordinators

Co-ordinators in both sectors had seen the scope and nature of their role change over the period of the surveys. The sources of support and advice had also changed. In the primary sector, local authority staff (73%; 68%; 64%) were increasingly important, as were other colleagues in their own school (50%; 41%; 44%), conferences and seminars (25%; 23%; 17%) and the Web/Internet (37%; 19%; 4%). For secondary co-ordinators, subject teachers' networks (2004 - 66%; 2001 - 12%) and conferences and seminars (45%; 30%; 35%) had increased in importance. Most of the other sources had declined in importance, for both groups.

In 2001, the percentages of primary and secondary schools with written policies for ICT were 62% and 81% respectively, while in 2004 the figures were the same for both sectors at 75%.

In each survey, co-ordinators were asked to indicate progress on various aspects of ICT. The data reported here refer to the aspects categorised as 'established/well-established' in each of the three surveys - 2004, 2001 and 1999. The responses indicate considerable progress in a number of areas, with increasing proportions of schools indicating that the aspects identified were established/well-established. Increasing numbers of primary co-ordinators responded in this category for all but one of the aspects listed. The greatest increases were:

• securing opportunities for pupils and teachers to access curriculum material directly from the WWW (81%; 10%; 20%);
• setting goals for ICT experiences for all pupils at different stages (71%; 43%; 36%);
• giving specialist assistance to pupils with learning difficulties (69%; 39%: 33%);
• enhancing the quality of reports or profiles of pupils' progress and attainment (68%; 48%; 39%);
• centralising the assessment and testing records of pupils (65%; 26%; 25%);
• delivering in-service courses for staff in specific professional areas (65%; 35%; 30%);
• delivering in-service courses on ICT for staff (85%; 49%; 37%);
• recording test and exam results as part of the processes of monitoring and review (67%; 32%; 19%);
• communicating the school's identity and role in the community and beyond, e.g. through a WWW page (31%; 13%; 4%).

Three aspects were considered 'established/well-established' by less than 10% in each of the three years, i.e. showed little or no change across the surveys, even though they had started from a very low baseline: giving the local community access to school computers or the Internet; establishing collaborative projects with pupils in other schools; and giving pupils access to subject specialists outside the school.

In secondary schools, there were increases in a number of aspects, in particular:

• securing opportunities for pupils and teachers to access curriculum material directly from CD- ROM/ CDi (81%; 71%; 58%);
• securing opportunities for pupils and teachers to access curriculum material directly from the WWW (85%; 74%; 29%);
• giving specialist assistance to pupils with learning difficulties (79%; 74%; 56%);
• enhancing the quality of reports or profiles of pupils' progress and attainment (62%; 60%; 42%);
• centralising the assessment and testing records of pupils (47%; 30%; 8%);
• putting teachers in contact with other teachers and professionals locally (55%; 33%; 17%);
• delivering in-service courses on ICT for staff (70%; 61%; 52%);
• recording text and exam results as part of the processes of monitoring and review (53%; 42%; 29%).

The three areas with low responses in the primary sector had slightly higher response rates than in the secondary sector, with one area - giving pupils access to subject specialists outside the school - rising from 6% in 1999 to 10% in 2001 and 28% in 2004.

One section of the questionnaire asked how well established were some of the longer term goals or aspirations for the future. This analysis focuses on the percentages of responses in the 'already well-established' category. Table 24 shows the data for 2004 and 2001, indicating change.

Table 24: Progress on long term goals in 2004 & 2001 (% co-ordinators)

All aspects show progress other than communicating with parents.

One question asked co-ordinators to indicate which aspects of ICT pupils had experienced by the time they left the school, be it primary or secondary. Both sectors showed substantial increases in some of the responses between 2001 and 2004.

Table 25: Co-ordinators' reports of pupils' ICT experiences in school (%)

The areas where co-ordinators indicated the greatest increase in activity were similar for both sectors. The increases reported in word processing, information handling and using the Internet are in line with data gathered from teachers and pupils.

The final comparisons across the three surveys focus on obstacles to ICT development in order to determine the extent to which these have changed, if at all, over the 5 years of the surveys. Co-ordinators were asked to agree/disagree with a series of statements. Table 26 shows the percentages of co-ordinators who strongly agreed/agreed with the statements in each of the three years.

Table 26: Co-ordinators' views on obstacles to progress, 2004, 2001 and 1999

The figures indicate that many of the previous obstacles to development have reduced considerably, although some significant ones still remain, e.g. too many priorities competing for time/resources and the lack of time to preview software. The provision of hardware has improved over the years, but technical support, staff development and co-ordination of investment at school/local authority are still perceived as obstacles.

7.4 The teachers

Some changes were evident in teachers' uses of a computer at home, whether for personal use or for work. Primary teachers reported a shift away from using CD- ROMs for information (2004 - 64%; 2001 - 79%; 1999 - 49%) and towards using the web (92%; 82%; 24%, respectively). Sending email rose to 92% in 2004 from 80% in 2001 and 27% in 1999. Secondary teachers showed a similar pattern, with increases in the percentages sending email to friends and family (93%; 69%; 25%) and using the web, whether for professional or personal reasons, to 94% in 2004 from 72% in 2001 (27% in 1999). In 2004 almost all teachers (> 90%) used word processing for both personal and professional purposes, while in 1999 around two thirds of them reported using word processing programmes on a fairly regular basis.

Confidence in using various aspects of ICT had grown considerably over the 5 years, in both sectors. For primary teachers, using email and the web/Internet showed the greatest increases: over 90% of respondents were fairly or very confident users in 2004. In 1999, the corresponding figures were 28% (email) and 23% (web), rising to 64% and 68% respectively in 2001. Working with spreadsheets and databases showed increases since 2001 of at least 20% to over 60%, as did desktop publishing. Secondary teachers' confidence had also increased over the 5 years on all aspects of ICT listed. Email and searching the web/Internet showed the greatest increases, with figures for email increasing from 30% in 1999 through 73% in 2001 to 93% in 2004, and using the web increasing from 35% through 75% to 96%. Most other aspects showed increases of between 15 and 20% since 1999.

The figures for those aspects of classroom use of ICT reported by teachers as 'well established' showed some notable increases. In the primary classroom, word processing was well established for around the same numbers of teachers as in 1999 (>85%), with increases in making a newspaper/newsletter (2004 - 54%; 2001 - 21%; 1999 - 16%), working with databases (38% in 2004, 17% in 1999) and spreadsheets (36% in 2004, 10% in 1999). Sending emails to pupils in other schools increased to 20% in 2004 from 10% in 2001 and 4% in 1999, while using the web to find information rose to 84% in 2004 from 27% in 2001 and 3% in 1999.

Some of the newer technologies were first included in the 2001 survey and there were significant increases in the numbers reporting that they were well established in primary schools, with figures for displaying/sending digital photographs rising from 5% in 2001 to 28% in 2004 and those for multimedia presentations rising from 3% in 2001 to 37% in 2004. All aspects common to 2001 and 2004 showed an increase in the percentages of teachers reporting that they were well established, other than running simulations, which showed a decrease in the primary sector but an increase in the secondary sector.

In general, usage figures tended to be lower for the secondary sector, although there was an increase in the number of schools indicating that the aspects listed had become well established. Examples include:

• searching for information on the web (63%; 25%; 8%)
• emailing pupils in other schools (14%; 5%; 1%)
• running simulations (15%; 7%; 5%) and
• writing essays, poems or reports (38%; 24%; 24%).

Teachers were asked to indicate how well established various activities were in their own practice to give some indication as to how they had integrated ICT into the classroom. Table 27 shows the responses for all three years, for each sector.

Table 27: Teachers' views on how they are using ICT to support learning and teaching in the classroom

Increased numbers of teachers reported that many of the activities listed were well established, with differentiation generally and catering for individual needs of high attainers and those with learning difficulties more evident. Creating and sourcing resources also showed increases. Overall, the primary teachers reported greater increases than did their secondary colleagues. As with the co-ordinators, teachers turned increasingly to the web/Internet for advice and guidance on resources. Local authority staff and colleagues in their own institution decreased slightly. Subject teachers' networks were increasingly important to secondary teachers.

The extent to which ICT was well established in the monitoring, evaluating and assessing of teaching and learning increased in both primary and secondary responses for: producing reports/records for parents, profiling pupils' progress and recording and analysing test and exam results.

Teachers' views on the impact of ICT on pupils, teachers and the school generally had changed over the years of the surveys. In 2004, more primary teachers considered that ICT had contributed in positive ways to pupils' learning (2004 - 98%; 2001 - 94%; 1999 - 87%) and that the pupils showed a higher level of independence when working on computers (2004 - 93%; 2001 - 63%). The percentages considering that ICT had enhanced practice had also increased: 2004 - 85%; 2001 - 70%; 1999 - 69%. Slightly fewer teachers found it disturbing that pupils knew more than they did (2004 - 29%; 2001 - 36%; 1999 - 34%). With regard to teaching, fewer teachers felt that their staff development needs had not been well met (2004 - 31%; 2001 - 46%; 1999 - 60%) and fewer experienced classroom management difficulties (2004 - 30%; 2001 - 62%; 1999 - 45%). The figure for those who did not feel confident using ICT as a teaching aid increased in 2001 but by 2004 had dropped back below the 1999 figure (2004 - 31%; 2001 - 62%; 1999 - 45%).

Teachers in secondary schools were also more positive about the impact on pupils, greater percentages agreeing that ICT had contributed positively to pupils' learning (2004 - 96%; 2001 - 84%; 1999 - 74%) and that they showed a higher level of independence when working on computers (2004 - 77%; 2001 - 66%). As in the primary schools, smaller percentages of teachers were concerned that pupils knew more than they did (2004 - 27%; 2001 - 31%; 1999 - 42%), but the other figures were similar to those of previous surveys.

Secondary teachers were also more positive in their responses to some of the statements on the impact on teachers. Fewer felt that they did not have the time to become familiar with ICT resources (2004 - 54%; 2001 - 65%; 1999 - 67%), that their staff development needs had not been met (2004 - 46%; 2001 - 69%; 1999 - 68%) and that they found the classroom management of ICT difficult (2004 - 41%; 2001 - 47%; 1999 - 49%). Fewer also reported a lack of confidence in using ICT as a teaching aid (2004 - 22%; 2001 - 36%; 1999 - 49%). Secondary teachers' views on the impact of ICT on the school showed little variation across the surveys.

With regard to legal and ethical issues, both sectors reported increased availability of local authority/school guidance across the range of aspects identified in the questionnaires, as in Table 28.

Table 28: Teachers' reports, in each survey, of guidelines offered (%)

Summary of views of staff in schools, 1999 - 2004

The data from both sectors present a picture of the development of ICT as part of the learning and teaching experience in the Scottish schools involved in the surveys. Key features include:

• increased availability of modern computers, peripherals, networking and resources across the schools in the survey;
• greater staff confidence in using ICT and across a greater range of technologies;
• considerable increases in the percentages of teachers and their pupils using the web for communication (through email) and for research;
• more schools/local authorities having written policies for ICT and guidance for a range of ethical and legal requirements;
• increased percentages of staff with positive views on the introduction of ICT into the classroom;
• obstacles to development included the pressure of other priorities, time constraints and limited resources;
• staff in primary schools tended to be more positive than in secondary schools;
• co-ordinators tended to be more positive than classroom teachers.

8. Discussion and implications

This section pulls together the key messages from the questionnaires and assessment materials and looks across the surveys to identify current strengths and potential obstacles for the further development of information and communication technology in Scottish schools. The structure is provided by the original objectives of the project.

8.1 The Phase 3 findings

a. The knowledge and skills of pupils at P7, S2 and S4

While this discussion of levels of knowledge and skills draws heavily on the assessment data, there are also a number of issues from the questionnaire that have a bearing.

Pupils at all three stages achieved greater success, overall, on the assessment items than they had in the previous surveys. As before, competence on word processing, graphics and communication skills was most secure at all three stages. Information and data handling skills were less well established. Pupils, particularly at the older stages, were familiar with a range of technologies other than computers, particularly out of school. Success rates were highest on items that asked for the recall or recognition of knowledge; items that required the synthesis of ideas, critical thinking or problem-solving skills resulted in lower success rates.

The demands made of pupils in individual tasks influenced the success rates but it would be expected that pupils, particularly at the secondary stages, would have gone beyond the 'naming of parts' and begun to evaluate the use of various technologies, their appropriateness and the social and cultural dimensions of technological innovations. While most pupils made an attempt at items that addressed these aspects, success rates tended to be lower than on the procedural or knowledge-based items. A further influence on performance levels was the specific format of the item, the context in which it was set. The 5-14 Guidelines for ICT present targets that are broad, rather than specific, and de-contextualised. Assessment items require that knowledge and skills are exercised within a context, often a hypothetical scenario, and the choice of context can have a significant impact on success rates; this was most evident in the database items where the specific context appeared to make a difference.

Those qualifications aside, the performance levels of pupils in 2004 showed an increasing awareness and understanding which is clearly differentiated across the ages and stages of the three groups of pupils involved. The differences between P7 and S2 were not substantial and certainly less than might have been expected from two additional years of schooling. However, this finding is in line with Assessment of Achievement Programme surveys in other subject areas, where the overlap in performance at P7 and S2 is considerable. The difference between S2 and S4 was significant, indicating a real increase in knowledge, understanding and skill, more in line with a two-year age difference.

Gender differences indicated a slightly superior performance on the part of girls across the stages assessed and in aspects where they have tended to dominate in the past, e.g. language activities. The differences were marginal and, at this level, have little educational significance.

The aspects of ICT where pupils were most successful were those that teachers and pupils said happened more frequently in the classroom, e.g. writing stories, producing newsletters, searching the Internet for research activities. Where they were less successful, the technologies involved ( e.g. video-conferencing) and/or the procedures assessed ( e.g. data handling) were less familiar. They were also the areas where teachers showed less confidence and familiarity.

Almost all pupils at each stage had access to computers out of school and many had access to a range of other technologies that they did not experience in school. They were far more successful on Internet and email items than their experiences in school could account for; much of their skill had been learned informally at home/outside school. Many pupils themselves felt that they had learned much of what they knew out of school. This was strongest at P7, but declined with the stage; at S4 the balance was slightly in favour of the school.

Pupils were, on the whole, confident of their ability across a range of applications and procedures. This was not wholly realised in the assessment data, but five years after the first innovations began to appear in schools, many appeared comfortable and fairly competent, if not proficient, with much of the technology.

b. Access to, use of and attitudes towards ICT

Access to computers out of school reached over 90% for teachers, pupils and co-ordinators, and most were connected to the Internet, many by broadband. The difference identified in the earlier phases, where teachers were more work-oriented while pupils tended to experiment and play, was also in evidence in this third phase. A very small proportion of pupils and teachers continue to have no access to a computer out of school.

There is still evidence that much learning is taking place out of school and the opportunities for accessing a range of technologies in the home are increasing. The emphasis seems to have moved from computers to the range of peripherals that enable people to use their computers for a wider range of purposes and more creatively, e.g. digital cameras (still and video), mobile phones, MP3 players. Some schools are already developing strategies that capitalise on the powerful potential of these technologies and use them to support learning and teaching. While teachers tended to use their home computers for administration and producing resources for the classroom, they did appear to be more creative and use a wider range of technologies for personal use than previously.

In school, the use of computers in the classroom to support learning was very patchy, with small percentages of pupils experiencing this on a regular basis. When they did, it seemed to be concentrated on writing essays and reports as well as producing presentations of work. Thus word processing and graphics dominated. One of the key obstacles to further development is the reported lack of modern, up-to-date computers in the classroom. The position appears more critical in the secondary school, as primaries were less likely to raise this as an issue and primary pupils indicated greater use across the curriculum. Part of the explanation behind this may be a result of the differences in physical organisation. In primary schools, most classrooms have a number of computers for pupil use as well as a computer suite/lab that can be booked. The presence of computers in the classroom means that they are more likely to be used throughout the day and for a variety of purposes, i.e. they are more integrated into the day-to-day activities of the classroom. In secondary schools, fewer classrooms have more than one or two computers (other than computing or business studies classrooms) and most rely on scheduling time at the computer suite/lab. As a result, using a computer is rarely a spontaneous event.

Pupils and teachers criticised the access to and quality of ICT provision in schools. Both groups expressed concerns about technical failures and the age of some of the computers they had to use, and significant numbers of pupils compared the in-school provision very unfavourably with that available to them out of school.

Pupils indicated that they could usually depend on some form of support both in and out of school when they were working at the computer. However, it is possible (and likely) that the nature of that support is different. For example, is individual coaching or instruction more prevalent in the home than in school? It is possible that much of what a teacher is likely to do, working with between 20 and 30 pupils, is 'trouble-shooting' - giving quick helpful hints to get over obstacles and allow the pupil to proceed with the task in hand. Certainly this may well be the case in most classes where the purpose of the teaching is not the systematic acquisition and development of ICT knowledge and skills but about specific content such as biology or art. For teachers, learning about the new technologies continues to be an issue throughout school - upgrades, new peripherals, software and the pace of change all make it necessary to keep adding skills and knowledge. What would be an appropriate balance of teaching about ICT and using ICT as an integral component of pedagogical strategies, and how should the two be interwoven most effectively?

Most pupils were convinced of the value of learning about information and communication technologies, particularly how to use a computer, both for their personal lives and for future employability. Small numbers of pupils, and teachers, were concerned to point out that ICT should not sweep away all that is good in existing practice.

c. Staff development and training

Most of the teachers and co-ordinators in the study had undergone some form of staff development or training in information and communications technology, whether in-school, organised by the local authority or as part of a national programme. The most frequently cited was the New Opportunities Fund ( NOF) initiative which ran for approximately 5 years.

For those who had undergone some form of training, it had brought a number of benefits, the main one being increased confidence in working with technology. This confidence had, in turn, led to greater use of ICT in teaching situations. Most teachers felt that their practice had been influenced by their involvement in staff development events and could identify ways in which it had changed their practices to the benefit of the pupils and the school. Teachers were, in the main, positive about ICT, recognising the potential that it had for supporting learning and teaching more effectively, and were looking for additional training.

The majority of those who commented on the NOF training were very critical, feeling that it had been a general, undifferentiated experience which took no account of where individuals were in their understanding and skills. Some had found their own source of training, occasionally through award-bearing programmes, while others depended upon friends, colleagues and spouses. There is a demand for further training, but in specific technologies and applications or in embedding ICT in the learning and teaching process in the classroom - and working with the teacher from his/her existing level of competence and understanding.

However, the availability of training in itself was not the main impediment to developing greater confidence and competence. What many teachers were looking for was time - time to practise, explore and consolidate their own learning, and to be guided towards good resources and advice. They were also frustrated by a lack of access to the new technologies that they were hearing about (and sometimes being trained to use) but which were not readily available throughout their school. Examples included: interactive whiteboards, digital videos, video-conferencing and virtual learning environments.

d. The impact upon pedagogy

Teachers tended to report that the introduction of ICT into the classroom had influenced a range of aspects of their classroom practice. One of the key benefits of using a laptop and data projector in the classroom, for example, was that it allowed animations and simulations that helped pupils access difficult concepts, whether they were more abstract or referred to microscopic processes.

There was evidence from both the questionnaires and the assessment data that the focus in the classroom had shifted somewhat from learning aboutICT to learning with the support of or throughICT. The examples teachers gave of classroom practice using ICT indicated that, having got over the initial challenge of coming to terms with the technology, they wanted to know how best to use it for the benefit of their pupils. They readily acknowledged that the pupils were often more knowledgeable than they were, but this was not quite the threat that they had originally perceived it to be just a few years ago. There had not been a lot of progress in using ICT to help individual pupils or small groups with special needs (high attainers, those with learning difficulties, etc.), although some were looking for training in how this might be done.

The impact on pedagogy has been somewhat idiosyncratic and diffuse, with clear patterns difficult to discern. Teachers are slowly finding out which aspects of teaching can best benefit from ICT, but they need more time to do this. The pressures of other priorities, on time and resources, is a significant obstacle to learning how to adapt old practices and adopt new ones that will make the most of the opportunities that the new technologies can offer. In addition to helping pupils grasp new concepts, other benefits that were making an impact, perhaps more tangentially on learning, included increased interest and motivation leading to greater independence in learning and involvement with the task in hand (where those tasks were perceived by the pupils as purposeful).

There are some excellent examples of good practice in schools across the available technologies, but it is patchy and lacks coherence. Where significant developments have occurred, this has often been down to the drive of one or more enthusiasts within a school or department, 'champions' who introduce new approaches to teaching and learning and influence their colleagues by example. However, this is likely to be a slow process if all schools are to take on innovative practice, and a more effective method of dissemination, both within and across schools, is needed. Many teachers turn to Learning and Teaching Scotland and this is a role that they already undertake which could perhaps develop further. The various awards available through their website encourage teachers to take on new projects and try new approaches to learning and teaching.

There remains a significant proportion of teachers who are less enthusiastic about, if not hostile to, the drive to introduce and develop ICT in schools and across the curriculum. For those who are ambivalent, the regular failures of equipment and software to deliver and the lack of certainty that they will on any particular occasion are having a negative impact.

e. School management, administration and ICT

Many of the schools in the study had developed or were developing centralised, computer-supported systems for record-keeping, assessment data and reporting to parents. Whole school systems of electronic communication were more developed in the secondary sector. This may be due, in part, to the size of the typical secondary school compared to primary and the increased efficiency of information dissemination, etc. that ICT affords. Communication networks are now extending into the community and beyond, for both administrative and educational reasons.

The ubiquity of ICT means that it is important that all involved are clear with regard to ethical and legal requirements and, while there are policies in place, and teachers are confident that many of the safeguards are there, gaps appear to remain. Most notably, pupils must be made aware of the consequences of their own actions and become responsible users of the technologies provided.

The attention given to ICT, like other aspects of teaching and learning, depends to a large extent on the enthusiasm of staff and, in terms of the allocation of resources, on whether or not it is a priority on the school development plan (or the local authority equivalent). A number of schools commented that little progress was being made because ICT was no longer on the development plan. One interpretation is that schools feel that once they have achieved certain (quantitative) targets, the rest will almost inevitably fall in to place. The evidence is that there is a lot more to integrating ICT into the educational experience of pupils than achieving a set ratio of computers to pupils and networking them.

f. Looking to the future

The general picture is one of teachers coming to terms with the (physical) impact of the government's ICT initiatives and the presence of the new technologies, and beginning to work out how best to use them. The majority of teachers were positive, if not always openly enthusiastic, about the potential for improving learning and attainment. As in previous surveys, primary teachers saw a greater opportunity for breaking down age- and subject-related barriers in schools than did their secondary colleagues, with almost half in each sector believing that the classrooms of the future would be significantly different from those of today as a result of the impact of ICT.

With sufficient access to appropriate technologies, most teachers wanted to learn more about and to develop greater skill in using ICT to support learning across the curriculum, in new, relevant and exciting ways. The biggest obstacle to achieving their aims was time - time to learn new skills, find out about resources and technologies, plan and try out new approaches to teaching and learning and time to reflect upon and consolidate their experiences - and to share them with others. Competing priorities and, for some, a lack of access to relevant technologies outwith school, continue to impede progress. To a certain extent, the teachers want what the pupils have had - time, preferably out of school, to play, experiment, try out new technologies and then, importantly, to have the facilities to use their new knowledge and skills in the classroom. For many teachers, learning from colleagues was the key way in which they developed new skills and learned of resources. There is considerable evidence that 'hot information' such as this is much more effective in bringing about significant change to practice than the 'cold' evidence of printed policy documents and guidelines, and ways of exploiting this should be considered.

8.2 Comparisons with Phases 1 and 2

The use of identical questions and assessment items across the three phases of the Impact of ICT in Scottish Schools study provided the basis for comparisons with the situation in 1999 and 2001.

Pupils in 2004 demonstrated greater competence and confidence in and across a greater range of applications and technologies than in previous years. A particular development was the increased use of the Internet for research and learning in both primary and secondary sectors, with a corresponding drop in the use of CD- ROMs for such purposes. In 2004, pupils were more aware of the potential benefits of the new technologies and wanted more experience of using them in school, in purposeful ways. The responses from pupils indicated that the situation was improving across all stages but that the primary sector appeared to be making most progress. There was some evidence that, unless pupils take a technology/ computer-related subject for study to Standard Grade, their access in school is somewhat limited. Otherwise, there was little increase in the use of computers, for example, across subject areas.

Communication, within schools and with the wider community, was developing, particularly the use of school webpages and email facilities. In most instances, out-of-school use of such technologies was greater than in-school use for many staff and pupils. The access that pupils had to the various technologies and applications outside school was superior to that inside schools for many pupils; this is potentially a serious issue for schools, particularly in terms of motivation and bridging the gap between formal and informal learning.

Teachers and co-ordinators indicated that ICT was increasingly deployed to support learning and teaching in schools, with some practices becoming well established in many schools. Some of the aspects mentioned included accessing resources, specialist assistance to pupils with learning difficulties, communication, tracking and monitoring systems and the centralising of assessment and recording data. There were, however, a good number of schools who indicated that they had little intention of introducing many of the aspects identified.

While there was evidence that access was increasing, that new technologies were appearing in schools and that ICT-related activities were becoming embedded in the day-to-day experience of staff and pupils, there was some indication that achieving (quantitative) targets such as the expected ratio of computers to pupils and networking throughout the schools was considered sufficient. Teachers and co-ordinators were concerned that they be supported through good resources and staff development opportunities to develop their practice to make best use of the technologies available and to make the move from teaching and learning aboutICT to teaching and learning through and withICT. Targeted staff development opportunities that focused on using ICT within specific subject disciplines were requested by many.

8.3 Implications

Many of the basics are in place, but as they were mostly acquired out of school, they have not been acquired systematically. As a result, there are inconsistencies across what pupils, within and across stages, know and can do regarding the various aspects of ICT. While it is important to attend to these, it may be that the best time to do so is when there is a good reason to acquire the skill, i.e. within the context of a purposeful task. Beyond that, learning about the wider, social implications of ICT use, etiquette, protocol and security dimensions are all important components of being ICT-literate that pupils should acquire.

At present, pupils are developing skills in a range of technologies that have scarcely begun to make any impact on schools. Strategies for incorporating these into the educational experiences of pupils are developing. Mobile phones are used to compile weblogs when away from home on school visits and can be used as personal response systems (with some modification), similar to those used in some universities and on 'Who wants to be a Millionaire?' School radio broadcasts direct to MP3 players can carry information immediately and directly. These are aspects that could benefit from additional research and evaluation, drawing on work already going on in schools and colleges.

Alongside such developments, it is necessary to remember the small but important percentage of pupils without access to computers or other forms of technology and to ensure that they are adequately catered for through other strategies.

Staff development has made an important contribution to the confidence and attitudes of many teachers, giving them some sense of security as they tried new strategies and technologies in the classroom. Most teachers have undergone some form of basic training. They are now looking for further staff development on specific technologies, software and electronic resources. There is a need for short, focused programmes that take account of teachers' existing levels of competence. However, these need to be harmonised with the installation of the appropriate hardware, etc., so that teachers have a purpose for learning and the opportunity to practise in their own time.

There is evidence of a move towards integrating ICT into the everyday experiences of pupils through purposeful tasks and the effective use of different aspects of technology to support learning. The extent to which this has happened is limited as yet, both within and across schools. In previous surveys, the key emphasis had been on acquiring the necessary hardware, networking, etc. and on accessing training to allow developments to begin.

Teachers reported a range of technology-related strategies that they had developed across the curriculum. Many pupils did not benefit from these, however. An effective strategy for the dissemination of good practice should harness the enthusiasm of the 'champions' out there to spread the word, to demonstrate what can be done. In addition, more focused research on such innovations with the aim of providing support for other teachers is required.

In conclusion, the Phase 3 findings indicate that ICT is beginning to make an impact on practice, although some of the same problems that beset schools in previous studies remain. Pupils and teachers are increasingly using ICT in knowledgeable and creative ways, but practice varies across the schools sampled.

« Previous | Contents | Next »