| This paper summarises
aspects of ISEP that might be considered within the setting-targets initiative and also
within the broader context of evaluating school effectiveness and quality. |
|
| The messages from
ISEP underline the close relationship between research findings in school effectiveness
and school improvement and the Quality Initiative in Scottish Schools. The issues raised
should help schools to focus on improving attainment which is central to the targets
initiative. |
|
| Frank
Crawford, HM Chief Inspector of Schools, Audit Unit |
|
| One clear message
emerging from the ISEP research is that monitoring achievement and setting targets do not
occur in a vacuum. They rest on a climate of self-evaluation, a concern for evidence and a
willingness to use the evidence in improving standards. |
|
| Professor John
MacBeath, Quality in Education Centre, |
| University of
Strathclyde |
| August 1998 |
|
| Introduction
|
| How effective is our school?
|
| The Quality Initiative
in Scottish Schools has encouraged us to ask the question, 'How good is our school?'. |
| Is there a difference
between a 'good' school and an 'effective' school? Most people would say that the two
things are the same - that it is merely a matter of terminology. However, it is important
to understand the different uses of language. Researchers tend to use the word 'effective'
to refer to measured outcomes, usually pupil achievement. Effective schools are then
defined as those in which pupil achievement exceeds what might have been expected given
the background influences on the pupils in question. |
| In other words,
effective schools are those which: |
- use a reference point, or benchmark, to evaluate
progress
|
- achieve targets at, or above, the expected
benchmark.
|
| How far does this take
us in answering the question, 'How good is our school?'. The answer from most teachers is
- only part of the way. In other words, effectiveness measures are no more than the tip of
the iceberg. If we wish to gauge standards and set targets that are realistic and
attainable we need to know much more about what lies beneath the surface. What other
questions need then to be asked? What other areas of school life have to be explored? |
| The evidence from ISEP
provides some clear answers across 44 primary and 36 secondary schools. While the project
provides significant findings across all 80 schools, the most useful information to
schools was often their own. The experience of schools in the project shows that the
information, on both achievement and attitudes, provided an excellent focus for
professional development and school improvement. |
| Supporting
improvement |
| ISEP helps us to
understand the implications of different ways of looking at school quality and
improvement. It incorporates approaches from school effectiveness research which involve
looking at the relationship between pupils' measured attainment and influential factors.
The project also developed a range of strategies and instruments that can be used to
support school improvement. For example, pupil and teacher questionnaires were used in two
ways: |
- as research instruments, they contributed to a
nation-wide picture of many aspects of school life, such as teacher/pupil relationships
- the results from these questionnaires were given
back to schools, proving powerful in self-evaluation and review.
|
|
| Key
questions |
| The issue of school
quality is at the heart of the project - how should it be defined, measured and monitored?
This question is equally important for schools. In examining the general messages
presented in this paper, therefore, the questions for your school should include the
following: |
- Are the messages true of our school and how do we
know?
|
- Where might our school be different?
|
- Do pupils achieve more in some areas of the
curriculum than others?
|
- What are the differences in rates of progress
among pupils and what influences progress?
|
- How much is pupil achievement affected by the
peer group?
|
- Do the attitudes and expectations of teachers
make a difference?
|
- How important are pupils' attitudes - to school,
to teachers, to self?
|
- How do we measure aspects of school life that are
important but not easily quantifiable?
|
|
| What
difference do schools make? |
Guidance will be
developed from ISEP findings about how schools might add value, that is, how schools might
outperform expectations based on statistical predictions.
|
| ISEP addressed the
questions: |
- What effect do different schools have on the
level of measured pupil attainment?
- What value do schools add to pupil attainment?
|
| Many school
effectiveness studies show that school performance, when judged by examination and other
test results, largely reflects the prior attainment of pupils and their socio-economic
status. These are powerful determinants of attainment and progress but we also know from
previous research that schools themselves do perform differently even after these powerful
influences are taken into account. As that research has demonstrated, schools in similar
circumstances perform quite differently1. Because this is due to schools themselves rather than
extraneous or background factors, it is described as the school effect. This school
effect can be seen in ISEP schools. Page 3 outlines the effects of the school; of prior
attainment; and of pupil background factors in the ISEP study. |
| The activities taking
place in a range of ISEP schools were also examined using case study methods to find out
what might be associated with value added, for example: |
- What was the ethos like in high value-added
schools?
- How was it established?
- What approaches to teaching and learning were
predominant in schools that under-performed in value-added terms?
|
| |
| 1 Some of the
seminal work in this field was carried out by: RUTTER, M. (et al) (1979) Fifteen
Thousand Hours: Secondary Schools and Their Effects on Children, London: Open Books.
MORTIMORE, P. (et al) (1988) School Matters: The Junior Years, Salisbury: Open
Books. |
|
| Factors
influencing school performance |
| Current research
techniques are sophisticated and sensitive. They allow us to separate out different
factors at work in influencing achievement and to measure their effects. This in turn
allows us to build a value-added framework to estimate how these different factors
interplay and to look at their influence on attainment in different subjects or areas. For
example, the influence of prior attainment may be calculated independently of other
factors such as age, gender and socio-economic background. In the same way, we can
separate out socio-economic background. Free Meal Entitlement (FME) was used as the best
available proxy measure for socio-economic disadvantage. Although not a perfect measure,
it provides strong correlations with achievement. Of course, these two things - prior
attainment and socio-economic background - are clearly inter-related but we can find out
how they inter-relate and the extent to which they may actually work independently of each
other. The benefit of this approach is that we can identify the effects of three main
factors on pupils' achievement in ISEP schools. |
- Much of the variation among schools is accounted
for by prior attainment measured two years previously. This accounts for between
40% and 64% of the variation. In other words, what pupils achieved in 1995 is a strong
determinant of their attainment in 1997. However, it is not the only influence, and prior
attainment is strongly related to pupil background factors, mainly socio-economic
circumstances.
|
- Pupil background factors, separated from prior
attainment, have an additional influence accounting for up to 15% of the variation. In
other words, even after prior attainment has been accounted for, pupils' socio-economic
background still has a very substantial independent effect.
|
- Up to one third of variation in pupil performance
in 1997 was estimated to be due to the effect of the school after taking account of
prior attainment, socio-economic factors and other pupil background factors.
|
| |
| These findings show
that some schools have a stronger influence on attainment than others after the effects of
important variables are taken into account. The power of the school itself to influence
the level of pupil performance is clearly demonstrated. The findings also underline the
important role of pupil background and prior attainment factors in current achievement and
progress. |
|
| Factors and
their implications |
| Age differences within
a year group affect attainment at S2 and at Standard Grade. Generally, pupils who are
young in their year achieve significantly less well. |
|
This has
implications for the teaching, learning and assessment of pupils at all stages, especially
where age is not taken into account in internal and external assessment. |
| Girls as a group are
ahead in reading at S2. At Standard Grade, girls achieve better than boys in relation to
English (and average score for their 'Best 7' Standard Grades.) Differences in mathematics
in favour of girls just failed to reach statistical significance. |
|
There are
clearly implications for teaching and learning, particularly for boys. The finding that
the gap between boys' and girls' attainment continues over time up to Standard Grade
suggests that schools can do a great deal to raise achievement. Schools may want to
consider this in relation to the achievement of particular groups, for example, boys.
|
| Socio-economic
disadvantage as measured by free school meal entitlement has a powerful statistical
relationship with attainment across the curriculum. In secondary schools, it is also
associated with progress - pupils entitled to free school meals make poorer progress.
|
|
These
findings show that using raw school results alone (from examinations or tests) gives an
incomplete picture of school effectiveness and pupil progress. Taking account of
differences in pupil intake characteristics gives a more accurate picture.
|
| There is also a
compositional effect related to free school meal entitlement - in schools where there are
higher proportions of economically-disadvantaged pupils, results for all pupils tend to be
depressed. |
|
To raise
the achievement of all pupils there needs to be careful consideration of approaches to
school organisation, teaching and resourcing, especially in schools serving
economically-disadvantaged communities. |
|
| Adding
value: patterns of school effectiveness |
| Indicators of academic
school effectiveness were based on the progress of pupils from S2 to S4. Predictions were
made about pupils' performance at S4, having accounted for the effects of socio-economic
circumstances, prior attainment and other factors. Schools that exceeded these predictions
were classified as significantly adding value. |
| The main findings of
the project for secondary schools in relation to academic outcomes are summarised below. |
- In mathematics and average score across pupils'
best 7 Standard Grades, about one in four ISEP secondary schools could be said to be significantly
adding value to pupil achievement.
- In English, a slightly smaller proportion of the
whole sample could be said to be significantly adding value to pupil achievement.
- Approximately one in four secondary schools
performed significantly below expectation in mathematics.
- A smaller proportion performed significantly
below expectation in relation to Standard Grade English.
- The pattern of performance among schools was mixed.
For example, some schools performed close to expectation in relation to Standard Grade
English and to pupils' best 7 performances at Standard Grade; and just below in relation
to mathematics.
- One in 12 schools significantly added value
in all three academic measures - Standard Grade mathematics, Standard Grade English and
overall score for best 7 Standard Grades.
- The schools performing significantly above expectation
covered the whole range of socio-economic classification. Indeed, the school with the
highest free school meal entitlement in the project performed above expectation in two out
of three of the outcomes. Furthermore, one of the three schools performing significantly
well in all outcomes had relatively high free school meal entitlement.
|
|
| Extending
the quality framework: pupil and teacher perceptions |
| The targets that
will be set by schools under this framework are, however, not an end in themselves. They
are a focus for planning and improvement. Having set targets, schools will then have to
plan how they are to be achieved. Many factors will be important to how they approach that
critical task. |
| (SOEID: Setting
Targets - Raising Standards in Schools, March, 1998). |
| The above quotation
highlights the importance of school processes in relation to meeting targets. ISEP
confirms that schools do make a difference and that the study of school processes is very
valuable. It demonstrates, for example, how young people perceive their classroom
experience - how they view the levels of challenge in their work and the working
relationship between teachers and pupils. Likewise, the teacher questionnaire casts light
on many issues such as the management climate of the school. |
| Information about
pupil attitudes and perceptions was: |
- used to ascertain what change had taken place
over the lifetime of the project in the way pupils perceive school
- fed back to schools in the same way as attainment
information.
|
| The 'all schools'
information provides an interesting, nation-wide picture of pupil perceptions of school
life. |
|
| Value-added:
questionnaire responses |
| We can learn valuable
lessons from the ISEP research about how questionnaires can be used to understand the
general implications of pupil views for school improvement. |
- Schools find information about the attitudes of
pupils, parents and teachers useful in informing their priorities.
- Pupils' responses are generally positive. The
least positive responses related to classroom interaction.
- Pupils' attitudes to the quality of interaction
between teachers and pupils in class varied, particularly in teachers' use of praise
and pupils' attitudes to answering questions.
- Three separate items varied significantly across
the sample of secondary schools:
|
| feeling safe in the
playground |
| playing truant
|
| getting homework.
|
- Consensus varied. There was most consensus that
the level of work was usually about right. There was least consensus about truancy.
- One dimension, teacher support, varied
significantly across schools.
- Teacher support is significantly and positively
related to pupil academic achievement. It subdivides into five items, including:
|
| teachers help me to
understand my work |
| teachers tell me
how I am getting on with my work |
| teachers praise me
when I work hard. |
- The strongest associations between pupil
attainment and pupils' views were with frequency of homework and perceptions of the
pupil's own behaviour.
|
| As the relationship
between pupil attitudes and other factors is unclear from previous research, it is
important to be cautious about incorporating the results of attitude questionnaires into
value-added calculations. For example, existing research suggests that positive attitudes
decline as young people mature.2 |
2 Keys, W
and Fernandez, C(1993)
What do students think about school?: A report for the National Commission on Education.
Slough: NFER |
|
| Teachers'
views on their schools |
| The single most
essential requirement one needs when giving others an opportunity to comment upon aspects
of one's leadership and management style is resilience.... Criticism can be a constructive
catalyst for change and improvement. (Headteacher) |
| The teacher
questionnaire is a powerful instrument for analysis and for change. It asks teachers to
rate their school on two scales against statements of good practice: |
- this school now (presence)
- important for creating a better school
(importance).
|
| Some of the notable
findings from the teacher questionnaire relate to teacher expectation. A gap between the
way teacher expectation is perceived now (presence) and its perceived importance
for creating a better school (importance) is significant for educational development.
One example from the teacher questionnaire is shown below. |
|
| Teachers in this
school believe all pupils can learn |
Important for school improvement (importance) |
This school now (presence) |
67% agree |
93% agree |
| The responses paint a
picture of Scottish schools and help us to answer questions such as those posed in How
good is our school?: Self-evaluation using performance indicators (SOEID 1996):
|
- Where are schools now?
- Where do they want to go?
|
| Questionnaire
responses can be analysed in different ways to illuminate areas of strength and areas for
development. For example: |
In which areas is
school life meeting teachers' expectations?
(high presence and high importance) |
| |
Agree |
| |
Presence |
Importance
|
| Staff
encourage pupils to try their very best |
89% |
99%
|
| Teachers
regularly monitor the learning and progress of individual pupils |
84% |
95%
|
| Staff
ensure that pupils receive constructive feedback about their work |
74% |
97%
|
| Pupil
success is regularly celebrated in this school |
78% |
89%
|
| |
|
|
| In
which areas do schools fall short of teachers' expectations? (low presence but high importance) |
|
|
| |
Agree |
| |
Presence |
Importance
|
| Standards
set for pupils are consistently upheld across the school |
18% |
96%
|
| Pupils
in this school are enthusiastic about learning |
30% |
95%
|
| The
development plan includes practical ways of evaluating success in achieving goals
|
18% |
80%
|
| Pupils
respect teachers |
46% |
96%
|
| The items of the ISEP
teacher questionnaire fell into three distinct clusters or factors: |
- leadership and management, for example,
effective communication, availability of senior staff and staff participation in decision
making
|
- academic emphasis, for example, high
expectation of pupils, monitoring of pupil work and feedback and teacher collegiality in
the planning and review of learning and teaching
|
- behavioural standards, for example, clear
behavioural standards communicated to pupils and parents and respect shown by pupils.
|
| These factors, in
addition to individual items, can be used to measure and monitor many aspects of school
life. |
|
| Listening
to teachers and pupils |
| Teachers believe that
schools' most positive features relate to encouragement and praise for pupils and monitoring
pupil progress and giving feedback, yet the least positive of pupil responses related
to the interest level of school work and teachers praising pupils. What
accounts for these contradictory impressions amongst key players in school life? |
| Are there any answers
to this question in the aspects of school life that teachers see as neither manifest nor
important for improvement? These centre round the involvement of pupils and
non-teaching staff in school planning and peer review and sharing of practice. |
| How important might
such activities be in bringing the perceptions of pupils and teachers close together?
Dialogue about planning between teachers and young people and sharing of practice amongst
teachers might assist change in these very important areas of school life. The areas
defined by teachers themselves as fundamentally important to school improvement were: |
- clarity of standards in work and behaviour
- praise and encouragement for pupils.
|
| The project has
developed many approaches and instruments that can be used to monitor and support school
improvement. Two of the most powerful survey techniques are the pupil and teacher
questionnaires. When used in combination, these instruments provide a rich and useful
picture of school life. Listening to the voices of teachers, parents and pupils is a
fundamentally important exercise in seeking to improve the quality of education. |
|
| Key
findings |
| This summary is
based on findings from the 36 secondary schools involved in the Improving School
Effectiveness Project. The findings relating to academic outcomes are based on Standard
Grade Mathematics, Standard Grade English and pupils' average for their best 7 Standard
Grades. |
- Some schools performed significantly better than
statistical predictions.
- Schools can achieve value-added effectiveness
regardless of starting levels.
- A number of schools achieved better than
predicted results across all measures. This suggests that the remaining schools may have
an unrealised capacity to raise their performance.
- Most schools achieve mixed outcomes across the
curriculum.
- There were significant gender and age differences
in measured attainment at S2 and S4.
- Socio-economic disadvantage has a stronger
negative impact on language work than on mathematics.
- S4 pupils in some schools in disadvantaged areas
performed significantly above what might be expected, taking account of their attainment
in S2 and factors such as gender and socio-economic circumstances.
- Some schools in advantaged areas also exceeded
statistical predictions. Conversely, achievement in some schools in both advantaged and
disadvantaged areas was below expectation.
- Maximum levels of achievement should be sought
from the beginning of secondary schooling - the strongest predictor of achievement at S4
is achievement at S2. Success breeds success.
- The pupil questionnaire used in the project
provided very useful information for school planning.
- There was a significant link between the way
young people perceive the support available to them in the classroom (teacher support)
and levels of value-added achievement.
- Teachers praising me was one of the lowest
scoring items in the pupil questionnaire. It also is one of the most discriminating
amongst schools. This is powerful information especially when other findings suggest that
teachers see their schools to be strong in these aspects. Schools need to give more
consideration to dialogue about learning between teachers and young people, and in
particular to think about the meaningful use of feedback and praise.
- The teacher questionnaire is useful to schools in
evaluating quality. It is especially valuable because it uses a double scale, allowing
schools to identify areas where expectation is being met and how important they think
something is.
- The project demonstrates how schools might build
profiles for improvement based on both attitudinal and assessment data. Information based
on assessment data, whether 'raw' or value-added, can be seen as part of a larger picture
embracing the views of teachers and pupils. Although the relationship between these
indicators of quality is not straightforward, each has a valuable function to play in the
pursuit of improvement.
|
|
| The
Improving School Effectiveness Project |
| The purpose
|
- To understand better how schools become more
effective and 'add value'
- To develop approaches to self-evaluation and
improvement using information on attainment and attitudes
|
| The process
|
| A range of data was
collected in 1995 by the research team. It was analysed and fed back to schools and then
used in different ways by schools - for staff development, school improvement and setting
targets. In 24 of the schools (case study schools) a 'critical friend' worked closely with
the school to help in the process of interpreting and using data for development planning
and setting targets. |
| Data were collected
again in 1997, with pupils now in P6 and S4. This allowed the research team to examine the
progress made by individual pupils, by groups and by schools as a whole. These data
provided the basis for calculations of value added. |
| The data
|
| Data were collected
from P4 and S2 pupils. This allowed the research team to build a large database comprising
some 7000 pupils. Data included: |
- pupil background factors (including Free Meal
Entitlement)
- pupil attainment and attitudes
- teacher attitudes
- parent attitudes.
|
| The school and the
pupils |
- 80 Scottish schools - 44 primary, 36 secondary
- 24 of these were case study schools
- 1600 primary and 5500 secondary pupils.
|
| The research team
|
| Teams from the Quality
in Education Centre at the University of Strathclyde and from the Institute of Education,
University of London. |
| How to get further
information |
| Professor
John MacBeath |
Dr Pam
Sammons |
| Quality in
Education Centre |
Institute
of Education |
| University
of Strathclyde |
University
of London |
| Tel: 0141
950 3183 |
Tel: 0171
612 6342 |
| Fax: 0140
950 3178 |
Fax: 0171
612 6344 |
| E-mail:
p.m.robertson@strath.ac.uk |
E-mail:
tecupps@ioe.ac.uk |
|
|
