The Scottish Government

« Previous | Contents | Next »

Listen

5. Performance on the assessment booklets

Six assessment booklets were developed, two at each stage, with items that primarily assessed knowledge and understanding across the ICT curriculum. This section describes the booklets and presents the findings for each of the stages as well as comparisons across stages and by gender.

5.1 The assessment booklets

Each of the assessment booklets contained a range of items assessing aspects of using information and communication technologies. Most items consisted of a series of smaller questions/tasks, as in Example 1. The total number of items per booklet is shown in Table 19.

Example 1: Item from P7 Booklet 1

Table 19: No. of items per booklet

¹ One of the items in Booklet 1 at each stage asked for views on the ethics of downloading music from the internet without paying. On initial analysis, it did not readily cluster with the other items. As it was primarily concerned with ethics/opinion rather than cognition or skill, it was excluded from further statistical analysis of pupil responses, although the findings are presented and considered later.

5.2 Reliability and consistency of the assessment instruments

The pupils' responses to the tasks in the assessment booklets were coded to give information on the numbers correct on each sub-question and to provide a limited error analysis. Overall scores were then calculated for each item. These scores formed the basis of further statistical analysis, i.e. stage and gender comparisons. They also provided the basis for consistency and reliability checks, with each booklet regarded as a separate 'test'. Analysis indicated that the tests were all internally reliable and consistent in terms of what they set out to measure. The vast majority of items resulted in scores that indicate that they were measuring a single construct. This, in turn, gives confidence in the accuracy of the findings presented within this report. The analysis also indicated that there were no significant differences in the two sub-samples of pupils at each of the stages. This means that they can be considered equivalent samples and that the data for both booklets at each stage can be considered together.

5.3 Performance on individual items and categories

The assessment booklets contained a mix of those items used in the previous phases (to allow for some comparison over time) and ones developed specifically for Phase 3. The first two surveys were undertaken prior to the introduction of the 5-14 Guidelines for Information and Communication Technology. In developing the items for the first two phases, a framework was constructed with 2 key categories, Knowledge and understanding and Personal appropriation (the ability to select and appropriate procedure[s] to complete a task) which could be applied across 4 aspects of ICT - Hardware, Software, Communications and Uses (including societal issues). At the start of Phase 3, the tasks to be retained were mapped on to 5-14 Guidelines for ICT. New items were developed to incorporate some of the technologies available since 2001 ( e.g.MP3 players, memory sticks) as well as reflecting some aspects of the 5-14 framework not previously assessed. All items have been matched to one of the 5-14 Strands.

The 5-14 Framework S trands

Matching the items to this framework was not straightforward for a number of reasons. Firstly, the 5-14 statements are very general in nature and this caused some discussion as to where individual items belonged. For example, 'browse records (in a database) and produce a simple report with support' seems fairly straightforward, but translating this into what you might expect a pupil actually to do to demonstrate it, is more complex - and opinions are likely to differ across teachers. Many of the items have also been assigned to a 5-14 Level (A-E). Determining the Level of individual items was not straightforward. Many of the items require pupils to grasp a context or scenario before answering any questions. Several questions are then asked of that context/scenario. Sometimes the questions range across two or more Levels, building in difficulty towards the end. Thus, where a Level has been assigned, it has tended to be the most difficult part of the task that has determined that Level.

A number of items asked pupils to name parts of hardware, aspects of software programmes and to identify what the various icons, etc. on the computer mean. Such questions do not readily fit the 5-14 framework, where knowledge and understanding of such detail are implicit in the completion of a task. These items have not therefore been assigned a Level, but relate instead to a basic understanding of how things work. As a result, the use of 5-14 Strands and Levels should be taken as indicative rather than definitive.

All of the sub-parts of an item were assessed independently and therefore each item had a maximum score representing the total possible on that item. The numbers of pupils scoring each of the possible scores for each item were analysed and an average score on that item was calculated. In Example 1, 'School shop sales', the maximum score is 5 and the average score for all P7 pupils in the sample was 3.9, which in turn equates to 77% of the item. These are the figures on which the following report is based.

5.4 Performance at Primary 7

This section presents the data for all items used in the P7 booklets. Where items were common to both booklets, the average score has been calculated across the entire sample of P7 pupils.

i. Using the technology ( UT)

Thirteen items were identified as fitting into the Using the technology ( UT) Strand, ranging from Levels C to E, but with most having no Level identified. Those with no Level therefore tended to ask for relatively simple responses: the naming of parts (hardware), icons and commands (software) and simple functions of each.

At P7, pupils scored at least half of the marks available on most of the items from this category, indicating that they knew the names and the functions of most of the hardware and software assessed. Performance on other items was lower. The majority of the items where P7 pupils were successful were of the 'naming of parts' variety, as in Example 2, which was included in all six booklets.

Example 2: Icon functions

Almost all of the pupils found this question straightforward and were successful on the individual parts. The average score at P7 and S2 was 88% and at S4 it was 95%.

One item sought understanding of the potential for abuse of mobile phones in school and the average score at P7 was 75%. On another item, where pupils were asked to give three rules for safe working at a computer, a quarter of P7 pupils could give one rule, fewer could give 2 and only 3% could give 3 acceptable responses to the question. The average score on the item was 18%.

ii. Creating and presenting ( CP)

Five items were concerned with aspects of Creating and presenting, focusing on the creation of graphics, the combining of text and graphics and editing text. Most were assigned to Level B of the 5-14 ICT Guidelines, although the reading demand and the specific content would have made them more demanding overall.

Most of these items were completed successfully at P7, i.e. with an average score of at least 70%, a good understanding of how graphics are created and edited. One task was more demanding, requiring awareness of databases, spreadsheets, word processing and graphics packages; P7 pupils scored an average of 35% on this task.

iii. Collecting and analysing ( CA)

Four items focused on databases and spreadsheets (5-14 Level C/D). Performance levels were relatively low on this Strand, apart from one item where pupils scored approximately 75% of the available marks. This involved reading information from a given spreadsheet.

iv. Searching and researching ( SR)

Six items were categorised as Searching and researching at Levels C to E. They focused on using the web, with one assessing understanding of using a CD- ROM for research. Performance on most of these items varied. Pupils were most successful on the CD- ROM item (Level C: 82%). Searching the web, particularly more advanced searches, scored lower (41 - 44%). One of the items in this category asked pupils to identify one advantage and one disadvantage of the wealth of information available on the web. Few pupils were able to give both, resulting in an average score of 38%.

v. Communicating and collaborating ( CC)

Eleven items assessed knowledge and understanding of various aspects of communicating using the new technologies. Overall, understanding of email and the Internet was satisfactory, although awareness of video-conferencing and some of the more technical terms, e.g.LAN, was relatively low.

Example 3: Email

In 'Email', pupils were asked to answer questions about using the email facility on the computer. At P7, between 51 - 79% of pupils were correct on each individual part of the question, with an average score for the item of 76%.At S2, the corresponding figures were 79 - 86%, while at S4, they were 82-90%. The average scores at S2 and S4 were 78% and 79% respectively.Items on long distance communication and the advantages and disadvantages of different forms of communication (video conferencing, faxing, email, telephones) all resulted in lower scores. Pupils coped quite well with most of the items matched to 5-14 Levels B and C, but found those associated with Level D more demanding.

vi. Controlling and modelling

Only two items focused on Controlling and modelling (5-14 Level C). Both involved pupils identifying the correct steps in a sequence to direct a model to a given spot. Scores on these items reached the 50% mark.

Summary

• Pupils had a good grasp of the names and functions of much of the hardware and software presented to them.
• They were less proficient at handling spreadsheets and databases.
• P7 pupils knew how to search using CD- ROMs and the Internet at a basic level although their awareness of some of the implications (safety, viruses, etc.) was less secure.
• Understanding of communications technology was around the middle range of the possible scores for most items, indicating a reasonable grasp of many aspects.
• Not all pupils were familiar with manoeuvring models through control technology. Just under two-thirds tended to be successful on individual steps, but fewer were able to string all of the steps together successfully.

5.5 Performance at Secondary 2

i. Using the technology ( UT)

Twenty-four items reflected knowledge and skills related to Using the technology. As at P7, several of these were concerned with knowing the names of parts of systems, hardware and software, and with being able to identify the function of various components and symbols, e.g. icons. Many did not readily match the 5-14 Guidelines for ICT, while the remainder covered aspects of Levels C to E.

The scores for each of the items listed indicate that S2 pupils had a very good grasp of most of the basic names and functions and recognised most of the icons used. Where they scored less than half of the available marks, this tended to be on items that were less 'mainstream', in that they were not explicitly mentioned in the 5-14 Guidelines. 'Hackers' is an example of such an item.

Example 4: Hackers

This item is more demanding than most of the others in this group in that it asks about a concept, 'hacking', that is not explicitly mentioned in the Guidelines but that is of general interest and relates to security. Approximately one quarter of pupils were able to give an acceptable definition of hacking, with around half able to give one protective strategy that banks employ, although few were able to identify two.

'Autoteller' and 'Barcode' assessed knowledge of magnetic strips on bank cards and the barcode on library books - neither of which are explicitly mentioned in the 5-14 Guidelines, but both of which are relevant technologies. In an item common to all three stages, 21% of S2 pupils gave three safety rules for using a computer, slightly better than the P7 performance.

ii. Creating and presenting ( CP)

Twelve items across the two booklets at S2 assessed aspects of Creating and presenting. As with P7, pupils showed a good grasp of the basics of graphics packages, spell checking and combining text and graphics. Pupils scored more than 50% of the available marks on those items assigned to Levels A/B (ranging from 55% to 86%) but scores on Level D items were lower.

iii. Collecting and analysing ( CA)

Ten items assessed aspects of Collecting and analysing, focusing primarily on the use of databases and spreadsheets. Items in this category were set at Level C/D and scores ranged from 28% to 64%. The items with the lowest scores tended to focus on databases.

Overall scores on Collecting and analysing were a little down on the previous two Strands, indicating that pupils were less secure on these items. While performance is stronger than at P7, it is still lower than might be expected from pupils at this (st)age.

iv. Searching and researching ( SR)

Seven items focused on aspects of searching CD- ROMs and the Internet, including using a search engine, refining searches and identifying relevant web pages. Items spanned Levels C - E of the 5-14 Guidelines, with performance levels best on the Level C questions. Overall, pupils scored at least half of the marks on all of the tasks in this section, with strengths in searching the web and using CD- ROMs.

v. Communicating and collaborating ( CC)

Fifteen items assessed aspects of Communicating and collaborating, ranging across Levels A to E, with most at C/D/E. Performance tended to be in line with the Level of the task, with scores ranging from 22% to 80%. Scores tended to be higher on the more straightforward items, e.g. 'Email' (Example 3 above), while others asked for more in the way of critical thinking skills and problem-solving, e.g. 'Retirement'.

In 'Retirement', pupils are asked to identify ways in which the Internet can provide information on various aspects of other countries and can help people to stay in touch when they live far away.

Example 5: Retirement

In this task, over half of the pupils were able to give reasonable answers for the question relating to 'how to travel' and were able to identify 2 ways in which they could keep in touch after they move away ( e.g. email, send attachments, use a web-cam, fax).

Only relatively small numbers gave acceptable responses to the other 3 questions (the weather, where to live, what to do there).

S2 pupils knew and understood aspects of email and communicating through the Internet satisfactorily, but scored lower on items where they had to use their knowledge to explain situations or solve problems.

vi. Controlling and modelling ( CM)

Four items were concerned with control technologies, two of which also appeared in the P7 booklets. A third item asked pupils to give two examples of when a sensor might be used and the fourth asked about the advantages and disadvantages of using simulated road crash situations to test cars. All were matched to Levels C or D. On average, pupils scored more than half of the marks available and seemed particularly aware of the use of sensors.

Summary

• S2 performance data show a good level of understanding of most of the basic names and functions of hardware and software.
• Some of the issues around the careful use of computers, security threats and counter strategies and the moral/ethical issues were not well-established.
• Similarly, some of the technologies other than computers were less familiar to pupils.
• As with P7, S2 pupils showed a good grasp of graphics, combining text and graphics and using different (multimedia) packages for different purposes.
• Collecting and analysing, which focuses on databases and spreadsheets, resulted in lower average scores, indicating that this is a less secure area for many S2 pupils.
• Most S2 pupils showed a reasonable level of confidence on items concerned with Searching and researching.
• S2 pupils understood the basics of the forms of communications included in the assessment tasks. Scores fell on those items which required critical thinking and problem-solving skills.
• Performance on Controlling and modelling items varied, but most S2 pupils achieved some success.

5.6 Performance at Secondary 4

The S4 booklets contained more items than those at the two younger stages and tended to contain more demanding questions in terms of analysing a situation and problem-solving. As with all booklets, they contained some 'easy' questions so that all pupils would have some feeling of success when tackling them as well as some which challenged their thinking.

i. Using the technology ( UT)

Twenty-nine items were matched to the category, Using the technology, although several of these were not assigned specific 5-14 Levels. Many of the items were also used at the other two stages and, as stated, are concerned with basic knowledge and understanding of details of hardware and software.

In the majority of instances, S4 pupils scored more than half of the available marks on any item. They were very confident on the 'names and functions' questions and understood the basic properties of a number of technologies e.g. magnetic strips, barcodes and memory sticks. As with the other two stages, the question asking for three rules for safe computer use was the least well done item (35%).

ii. Creating and presenting ( CP)

At S4, 14 items drew on aspects of Creating and presenting. Some focused on graphics, while others looked at combining text and graphics and multimedia presentations. S4 pupils were successful on the items concerned with graphics, handling text and combining text and graphics in a single document. One item, 'Road Traffic Accident', asked pupils to identify pieces of hardware and software that would be needed to produce a report on road traffic accidents in their area. Although 66% identified appropriate software and 45% made relevant suggestions for hardware, only 23% gave acceptable responses to both parts of the question.

iii. Collecting and analysing ( CA)

Thirteen items were included within this Strand, covering aspects of databases and spreadsheets. For most of these, S4 pupils scored more than half of the available marks, indicating a reasonable degree of familiarity with the procedures. In spreadsheet questions, pupils were adept at reading data but less familiar with more complex procedures such as using formulae. While most of the items were matched to Levels C/D, in terms of ICT targets, the specific content of some of the items makes them more demanding than this would indicate. This appears to be reflected in the success rates. Example 6, 'Criminal records', was the least well done item, with an overall score of 36%.

Example 6: Criminal records

All three stages performed similarly on this item, with decreasing success as they worked through the sub-parts of the item.

The table shows the percentages correct on each part of the item.

Table 20: Performance on 'Criminal records' (%)

The average scores at P7 and S2 were 25% and 31% respectively.

v. Searching and researching ( SR)

Seven items reflected targets within the Searching and researching Strand. Most were concerned with searching for and negotiating information on the web and matched to Level D of the 5-14 Guidelines for ICT. S4 pupils scored well on these tasks, with only one item falling below the 50% success rate.

v. Communicating and collaborating ( CC)

Fifteen items assessed aspects of Communicating and collaborating, including video-conferencing, faxing, email and using the Internet. The success rates at S4 fell below 50% on only two occasions and, on both, they were above 40%. As before, the more open-ended questions appeared to give pupils more difficulty. ('Retirement' has already been discussed above.)

vi. Controlling and modelling ( CM)

The same four items that were used at S2 also appeared in the S4 package. In one item, 'Sensors', only 14% of S4 pupils were able to identify two ways in which sensors might be used to gather environmental data, although a further 14% identified one way. S2 achieved 80% success on this item and it appears that many S4 pupils may have focused on the 'where' e.g. at work, in school, rather than what it might be used for measuring/perceiving, e.g. light, heat, pollution. In addition, a greater number of S4 pupils did not attempt this item. One explanation, at least in part, may be that this is a topic which is typically developed during S2 and so may have been more recently relevant to the S2 pupils involved. Otherwise scores were in the 50 - 60% range.

Summary

• S4 pupils were confident on items to do with the basics of Using the technology - knowing names and functions of frequently encountered hardware and aware of the icons and terminology used in software packages. They were also aware of a wide range of modern technologies.
• S4 pupils were also secure in basic aspects of graphics and text management and putting different media together into single documents, e.g. reports.
• Scores on Collecting and analysing items were reasonably high for most of the items at S4, although content sometimes influenced performance levels.
• S4 also scored relatively well on items in Searching and researching, indicating confidence in using the Internet and CD- ROMs.
• Scores on Communicating and collaborating were spread from 43% - 88%. The specific demand of the tasks influenced scores, with the more complex tasks gaining fewer marks.
• In the final section, Controlling and modelling, performance levels indicate that pupils had some awareness of the procedures involved.
• Overall, the figures indicate that most S4 pupils have competence across a range of technologies and their possible uses.

Postscript: the morality of downloading free music

One item at all three stages did not fit readily into analysis of items. This was 'Downloading', which focused on the morality/ethics of downloading music from the Internet without paying.

Example 7: Downloading

The percentages of pupils at each stage selecting either 'yes' or 'no' were calculated. Not all pupils were able to justify their choice but, where they did, their explanations were coded into a number of categories.

At P7, 37% of pupils answered 'yes', while 53% said 'no'. At S2 the figures were 36% and 50%, while at S4 they were 37% and 55% respectively. For those who did think it was a fair argument, the reasons given to support their opinion included: 'record companies charge too much/overcharge for CDs' (4% of P7; 8% of S2; and 8% of S4); 'most bands/singers are wealthy/rich and don't need the money' (2%, 4% and 0% respectively); and 'I cant afford to pay that much' (10%; 14% and 14% respectively).

Of those who said 'no', the reasons fell into a number of categories. These included: 'it's like stealing from the record companies' (5% at P7; 4% at S2 and 5% at S4); 'bands and singers should be paid for their work' (2%, 4% and 7% respectively); and 'it's illegal' (8%, 5% and 9% respectively).

Marginally more pupils did not think that downloading music and copying CDs was fair. They offered few explanations, with the most common being along the lines of 'it's illegal'. Of those who did think it was alright to get music without paying, it was mainly on the basis of a personal economic argument.

5.7 Comparisons by gender and stage

Comparisons of performance by gender and stage were undertaken. As approximately equal numbers of boys and girls tackled each item in the assessment booklets, it was possible to compare boys and girls at each stage on each of the booklets. Comparisons by stage were undertaken on the basis of the common items within and across the stage-related booklets. In addition, gender comparisons across stages were possible on the basis on the common items.

i. Comparisons by gender

The performance data for each booklet were analysed to provide evidence of gender-related differences at each stage. Comparisons were made between the two groups on individual item scores and on overall scores for the booklet.

The total numbers of significant differences by booklet and gender were:

While comparisons of performance levels by gender indicate a number of significant differences at each stage, it should be noted that by far the majority of items showed no significant differences, i.e. boys and girls performed at similar levels on most of the items in the booklets. There was some evidence of girls performing better than boys where significant differences were observed. In addition, boys' scores tended to be more scattered than those of the girls (a well-established gender difference on many tests).

Looking at the tasks involved, no real pattern of superior performance on, for example, software-related tasks can be discerned. There were two items where boys did better at two stages and two items where girls did better at all three stages. On another 9 items, girls at two stages did better than the boys. There is some suggestion that where boys did better the items were of a technical nature, e.g. 'hardware device' and 'voice'. With regard to the girls, the items where they did better spanned the applications assessed (word processing, graphics, Internet, email) and aspects of hardware and software (knowledge of icons, etc).

Summary

The key findings from the analysis by gender are:

• Overall, the differences do not point to a consistently superior performance by boys or girls across the categories assessed.
• Gender analyses showed a marginally superior performance on the part of the girls but this did not reach statistically significant levels in most instances.
• Boys' scores tended to be more widely distributed than those of the girls.

ii. Comparisons by stage

Comparisons were made across the three stages involved (P7, S2 and S4). The assessment booklets were designed to ensure that a significant number of items were common across stages within Booklets 1 and 2 (Booklet 1 = 22; Booklet 2 = 25). By comparing across stages, it was hoped that some evidence of 'progress' might be determined. In most well-standardised tests of achievement among school students, a cohort of 12-year olds will typically outscore a cohort of 11-year olds by _ to ¹/ ₃^rd of a standard deviation. For the purposes of this study, this was used as the basis of progress across stages for the tests used. However, the concept of 'progress' should be treated with a degree of caution in the following analyses in that this is a cross-sectional study rather than a longitudinal one and involves different cohorts at the same time rather than the same pupils at different times.

Comparing P7 and S2

Booklet 1 contained 22 items common to all three stages. Comparing P7 and S2 scores shows that S2 outperformed P7, but by an insignificant amount. The overlap in performance is significant, however, i.e. S2 and P7 pupils performed at similar levels on the majority of common items.

Booklet 2 contained 25 items common to all three stages. Comparisons between S2 and P7 showed a significantly superior performance at S2, in contrast to performance on Booklet 1 and more in line with the 2 years of additional schooling received.

Comparing S2 and S4

When comparing S2 and S4 on Booklet 1, S4 pupils outperformed S2 and the difference in scores is highly significant. While the older pupils demonstrated superior knowledge and understanding, the difference is, statistically, rather less than might be expected given the two additional years of schooling experienced by S4.

On Booklet 2, S4 pupils significantly outperformed S2, with a difference in performance levels that one might typically expect to see over a two-year period on well-standardised attainment tests.

Comparison by stage across 6 common items: P7 - S2 - S4

In total, 6 items were common to all six booklets (Booklets 1 and 2 at each of the 3 stages). While this is a small number of items for statistical analysis, the comparison does give some indication of difference across all three stages.

On this short test, S4 pupils outperformed S2 pupils and they, in turn, outperformed P7 pupils. The differences were statistically highly significant, although it should be noted that they provide limited evidence of stage-related differences and should not be over-emphasised. As at the individual booklet level, the difference between P7 and S2 was not a notably big gap for two groups with an age difference of two years. S4 pupils outscored S2 pupils by a greater margin, more like the gap one might expect to find between two groups of school pupils whose ages differ by two years. It should also be noted that there may be a slight ceiling effect operating at S4 as a result of attempting to design (some) items that are accessible to P7 but still challenging for S4.

Comparisons of gender-related differences across stages

Analysis by gender was also undertaken on the common items across stages.

Looking across the assessment booklets, there were no significant differences in the majority of items used in the study. Where differences were observed, they were more likely to show girls outperforming boys and tended to be from Booklet 1.

Comparisons across 6 common items by gender

An analysis by gender of the 6 common items was undertaken, combining the performance of each group across the three stages. On these items, girls scored about 1.4 points higher than boys, confirming a trend, although not a wholly consistent one, for girls to outperform boys. While the difference is statistically significant, it is small in absolute terms. The standard deviation for the boys' scores was higher by a significant amount (as consistently observed). Breaking the data down by stage, this gender difference showed at the level of all three age cohorts, being statistically significant at P7 and S4 levels, but not for S2.

The differences existed on all six test items. Girls knew more about the web; about CD- ROMs; about icons; and about commands. While a gender effect was observed, it should not be exaggerated. The differences were significant statistically and possibly educationally, but there were as many high-scoring boys as girls.

Summary

The key points arising from the analysis by stage are:

• Overall, performance levels on the items used reflected the differences in age and stage, i.e. the older pupils generally performed at a higher level.
• The differences between P7 and S2 performance levels were not indicative of 2 additional years of schooling, although the older pupils did turn in a marginally superior performance on the common items in Booklet 2.
• The differences between S4 and S2 were significant and showed a gap in attainment commensurate with the difference in age and stage.
• There was some evidence of marginally better performance levels by girls on common items across the three stages sampled, but the differences were not highly significant statistically and only of limited educational significance.

« Previous | Contents | Next »