« Previous | Contents | Next »
Listen
CHAPTER THREE: TEACHER, PARENT & PUPIL SURVEYS
3.1 One of the key challenges of the study was to consider how the impact of DtS could be specified and measured on a quantitative basis. A significant problem, given the all- embracing and integrated nature of DtS activity. It requires a two-step approach. The first is defining and measuring 'determination' or the enterprising characteristics of the young people. The second is attributing any improvement in this to DtS.
3.2 This study has concentrated its activity on step one. Building upon the 7 generic themes of DtS delivery, discussed in Chapter Two. As part of this process, an analytical tool was developed which is referred to as the 'Index of Enterprising Attitudes and Behaviour' (the Index or IOEAB). It was formed by, and tested upon, a series of surveys with school pupils, parents and teachers, to calculate the Index and test its validity. Validity, in terms of predictive behaviour rather than statistical significance, is a key feature of Index analysis. Due to the nature of the Index structure, statistical significance cannot be easily calculated. This chapter sets out the rationale for this approach, the findings from the surveys and the future potential of the technique.
The Index of Enterprising Attitudes and Behaviour Concept
Why do we need an Index of Enterprising Attitudes and Behaviour?
3.3 The Index of Enterprising Attitudes and Behaviour emerged from a review of potential methodologies to measure enterprising behaviour among young people. While it is not be used directly elsewhere in the way we have assembled it, there are parallels in other studies in particular aspects. For example, the general Index approach, constructed from a discreet range of questions, is common in research in this sphere of educational psychology.
3.4 Our initial challenge was to find a way of encapsulating DtS impact. Essentially, DtS is about embedding enterprise into the day-to-day activity of school life. However, schools have significant flexibility on both relative emphasis and delivery structures. From an evaluation perspective, it is essentially a heterogeneous programme, integrated within the education system, with no qualitative benchmarks or baselines established.
3.5 DtS therefore represents a complex set of inputs which are difficult to define on a consistent and comprehensive basis. Figure 3.1 summarises the potential methodologies that one might adopt to assess the impact of the DtS programme.
Figure 3.1: Why do we need an index
3.6 One approach to assessing DtS impact is to assume that it is the sum of the inputs. In this case, it would be quantifying the extent to which, in individual school locations, plans have been developed, training has taken place, partnerships have been developed and materials prepared. This would be essentially an indicator of 'activity', but not necessarily impact.
3.7 Linked to the above approach is the assumption that Determined to Succeed activity is essentially an 'intrinsic good'. One would, by definition, want to have more of it, rather than have less of it. There is an assumption of 'faith' that inputs generate the required outputs. This model would be acceptable if DtS activity were a free good. As it is not, effectiveness remains an important issue, which in turn requires linking inputs and outputs.
3.8 A third approach is to decide that any quantitative assessment will be too difficult and subject to so many caveats that the answer just cannot be clearly found. On this basis, it may be better to rely on a combination of inputs and qualitative 'feel good' assessments.
3.9 The final approach, and the one that we adopted, is to establish the extent to which DtS activity is able to instil behavioural and attitudinal change in young people. The question then becomes how does one capture and measure behavioural change?
Segmenting DtS Inputs and IOEAB Outputs
3.10 The evaluation of DtS requires an assessment of the delivery activity, which schools and local authorities have developed as their response to DtS (the inputs) and the impact it has upon young people (the outputs). Figure 3.2 shows how the IOEAB focuses primarily on the output side of the model.
Figure 3.2: Segmenting Inputs and Outputs
3.11 Before one can measure DtS impact, it is necessary to define the characteristics it seeks to improve. If DtS is deemed successful, what will it look like? Can enterprising attitudes and behaviour be measured? The IOEAB, as the name implies, establishes a quantitative measure of this among young people. It can be used to assess enterprising attitudes and behaviour between different socio-economic clusters of young people in schools. It can monitor changes in the pattern and structure of enterprising attitudes and behaviour over time. It is important to note, however, that the IOEAB is not in itself a measure of DtS impact. In order to complete the link, it would be necessary to establish causality between DtS activity and any changes in IOEAB.
13
A Novel Approach
3.12 Enterprising attitudes and behaviour are not the easiest characteristics to measure, whether among young people or anybody else. Most approaches to this type of analytical area tend to be input focused or based on general stakeholder perception. The Index approach has been developed by York Consulting as a way of establishing a robust, quantitative measure. It is a new and novel methodology. As such, it needs to be thoroughly tested. The DtS evaluation has provided an excellent testing platform for the technique. However, there is potential for further work. The application of the approach is also potentially wider than DtS.
The IOEAB Concept Explained
3.13 The IOEAB is based on a perceptual assessment of specified enterprising attitudes and behaviour variables. It can be completed by a young person assessing themselves, or by someone who observes their behaviour, e.g. parent, teacher or employer. As part of the current DtS evaluation we have included assessments involving young people, teachers and parents. These are summarised in Figure 3.3.
Figure 3.3: Perception/Reality
3.14 The Index is an indicator of enterprising behaviour, rather than a direct measure of it. It is based on the assumption that perceptions of parents, teachers and pupils are close to reality. In order to assess consistency of approach, the perceptions of each group can be triangulated.
3.15 The IOEAB technique assumes the enterprising attitudes and behaviour of young people to be a function of the seven behavioural traits identified as linked to the overall DtS strategy:
- working with others
- personal effectiveness
- problem solving/creativity
- communication
- approach to learning
- confidence
- life beyond school.
- delegating
- challenging
- co-operation.
3.16 Separate questionnaires have been designed for young people, parents and teachers, and in each case they are invited to assess against each of the criteria and scored in terms of:
- strongly agree
- tend to agree
- tend to disagree
- strongly disagree.
3.17 An example of the parent questions relating to the 'working with others' behavioural trait is shown below.
Table 3.1: Parent view on Working with Others
Working with others | Score |
|---|
Strongly agree | Tend to agree | Tend to disagree | Strongly disagree |
|---|
1. When working with others, <insert child name> is happy to share tasks so people do the parts they are best at | | | | |
|---|
2. <insert child name> works well with people in a team | | | | |
|---|
3. <insert child name> often takes a lead in suggesting how projects should be done | | | | |
|---|
4. <insert child name> is good at getting along with all sorts of people | | | | |
|---|
5. <insert child name> doesn't like it when people who are in charge tell him/her what to do | | | | |
|---|
6. <insert child name> doesn't like asking for help, even when he/she needs it | | | | |
|---|
3.18 The Index is a weighted average of the response, based upon the following numerical weightings:
- strongly agree = 100
- tend to agree = 50
- tend to disagree = -50
- strongly disagree = -100
3.19 An important assumption here is that respondents perceive the distances between the categories to be equal. Essentially, ordinal data is being treated as cardinal. In the circumstances, we believe that this is a reasonable assumption to make.
3.20 Figure 3.4 shows the maximum positive score, reflecting maximum enterprising attitudes and behaviour, as 100, and the maximum negative score, indicating no enterprising attitudes and behaviour, as minus 100.
Figure 3.4: Creating the Index
3.21 For each dimension of perception, young people, teachers and parents, the IOEAB can be used to generate:
- an overall score
- a score for the 7 enterprising attitudes and behaviour components.
Interpreting the Results
3.22 The headline findings show the overall IOEAB for pupils to be 31.7. Is this good or bad? This is in the third quartile of the positive zone. A positive result, but with scope for improvement. It is difficult, at this stage, to say more. What we have produced is a relative baseline assessment. Until the exercise is repeated, and comparisons drawn with earlier results, conclusions cannot easily be drawn.
3.23 The power and usefulness of the IOEAB analytical tool can help to monitor perceived changes in pupils' enterprising attitudes and behaviour over time as well as changes in sub groups' perceptions (pupils, parents and teachers).
3.24 The analysis of the component themes of enterprising attitudes and behaviour can raise interesting questions. A variation in performance across themes can have implications for the emphasis and the type of DtS activity that may need to be expanded or contracted. These trends can be monitored over time and linked to particular inputs.
3.25 It is important to recognise that the measures of enterprising attitudes and behaviour generated from each of the three surveys are essentially relative. They are based on self-perception of enterprising attitudes and behaviour (pupils) or the enterprising attitudes and behaviour of others (when scored by teachers and parents). For example, within the pupil survey, a lower scoring of a 14 year old against a 12 year old does not necessarily mean that the 14 year old has less developed characteristics. It is the relative score in comparison to the age group that matters.
Application of the IOEAB
3.26 We recognise that the Index is in a development phase. We do not claim to have specified the definitive article and there is significant scope for further improvement and refinement. This might involve:
- expanding and re-defining the seven behavioural themes
- re-visiting and re-specifying the component areas within the behavioural themes
- equalising the number of component areas within each theme
- refining the calibration of the scales underpinning the Index
- applying weightings to particular themes. At present they are all deemed equal. This can only be achieved through further testing.
3.27 At present, however, and on the basis of the results presented here, we believe that the current IOEAB measure provides a good assessment of the current enterprising attitudes and behaviour of young people.
3.28 The IOEAB technique allows us to calibrate enterprising attitudes and behaviour on a consistent basis over time. The next step is to identify and classify DtS delivery or input activity and seek to correlate it with changes in observed enterprising attitudes and behaviour. This is potentially complex, as DtS does not have a monopoly on enterprising attitudes and behaviour improvement. Many other related activities and initiatives would claim to have an impact. One cannot simply attribute an improved IOEAB to DtS. Having said that, an increasing IOEAB is a positive policy outcome, whatever the relative mix of inputs. This, of course, takes no account of the resource input.
Overview of Key Findings
3.29 Outlined below are the key findings to emerge from the survey data. The basis for each group are parents (500), teachers (500) and young people (3,700). It should be noted that, due to the nature of the Index approach, statistical significance cannot be calculated.
3.30 The aggregate Index of Enterprising Attitudes and Behaviour for parents, teachers and pupils is set out in Figure 3.5. In line with the purpose of the index in aggregating and condensing responses on a relativistic basis, we do not report percentage responses to individual categories.
Figure 3.5: Overall Index
3.31 All three IOEAB scores are in the positive quadrant, but display significant variation. The highest level of young person enterprising attitudes and behaviour is recorded for parents (40.9), followed by pupils (31.7). The teacher assessment of a young person enterprising attitudes and behaviour, at 16.7, is below half the parental assessment. The relative positioning of the three groups is the same across all IOEAB themes.
3.32 From the outset we had predicted that the ordering of the three groups would follow this pattern. This, perhaps, reflects rose-tinted subjectivity on the part of parents, detached objectivity from teachers, with pupils positioning themselves roughly in the middle.
3.33 The differences between the three groups may also reflect differing methodologies employed to generate the IOEAB information. Parents were contacted by telephone and asked to assess one of their children in a specified year group. Teachers were interviewed by telephone and asked to conduct the assessment, for what they perceived to be an average pupil in a specified year group. Young people completed a self-assessment questionnaire in the classroom based on their own perceptions of themselves.
3.34 From an impact perspective, the most interesting dimension will be how each of the three groups' scores change over time against their own base position, rather than the relativities between them. This is something that could be addressed through subsequent surveys to be undertaken at a later date.
3.35 While the ranking/ordering of the three groups is common across all IOEAB themes, the relativities between them, in terms of variation above and below the average, is quite significant. Details of the observed variations across all six IOEAB themes are considered below.
Working with Others
3.36 The 'Working with Others' theme assesses the integrated working of young people with their peers. Specific areas addressed include:
- sharing tasks/comparative advantage
- working in a team
- taking the lead
- getting on with people.
3.37 The IOEAB scores for 'Working with Others' are set out in Figure 3.6.
Figure 3.6: Working with Others
3.38 The enterprising attitudes and behaviour score on 'Working with Others' was above the average for all three groups. In the case of teachers (30.5), it is almost double the average score. Parents recorded the highest score (48.2), followed by pupils (37). This is therefore an area of relatively high performance.
Personal Effectiveness
3.39 The 'Personal Effectiveness' theme focuses on the way that young people work. Component parts addressed within the theme include:
- planning work
- recording learning records
- learning re-enforcement
- seeking guidance
- time management
- working on own.
3.40 Details of the 'Personal Effectiveness' IOEAB scores are shown in Figure 3.7.
Figure 3.7: Personal Effectiveness
3.41 Levels of enterprising attitudes and behaviour on 'Personal Effectiveness' were below average for all three groups. Young people scored 21.2, which was the largest fall against the average, followed by parents (32.7) and teachers (13.8). This can be regarded as a relatively poor area of performance.
Problem Solving/Creativity
3.42 The 'Problem Solving/Creativity' theme addresses aspects of creative thinking and resolve to achieve outcomes. Specific areas addressed within this theme include:
- different ways of doing things
- good ideas
- careful planning
- volunteering
- solution seeking
- understanding others views.
3.43 The IOEAB scores for 'Problem Solving/Creativity' are set out in Figure 3.8.
Figure 3.8: Problem Solving/Creativity
3.44 The IOEAB scores on 'Problem Solving/Creativity' reveal a unique trend relative to other themes. Both the parent (46.5) and pupil (43.2) scores are above the average; particularly the pupil score. Teachers, on the other hand, allocated their lowest score of 9.1; only just in the positive zone. Teachers clearly have a very different perception of the problem solving ability of young people.
Communication
3.45 The 'Communication' theme addresses competency in expression and communication. Component parts include:
- public speaking
- written expression
- articulation/persuasion.
3.46 Details of the IOEAB scores, relating to 'Communication' are shown in Figure 3.9.
Figure 3.9: Communication
3.47 The 'Communication' enterprising attitudes and behaviour score was 30.3 for parents, 17.1 for pupils and 7.8 for teachers. The teacher and pupil scores were almost half their relative averages, indicating that communication skills are perceived as much less developed compared to other areas.
Approach to Learning
3.48 The 'Approach to Learning' theme assesses a young person's style and standard of learning. Component areas addressed include:
- learning by doing
- addressing choices
- enterprising approach to task completion
- high standards.
3.49 Figure 3.10 shows the Index scores for 'Approach to Learning'.
Figure 3.10: Approach to Learning
3.50 The parent (38.9) and teacher (14.8) scores on 'Approach to Learning' are a little below their respective averages. In the case of pupils (17.1), it is close to half their average score. Pupils clearly rank themselves as less developed/determined in relation to this theme.
Confidence
3.51 The 'Confidence' theme addresses the extent to which young people are comfortable in their surroundings and in themselves. Areas addressed within the theme include:
- enjoying school
- self confidence
- seeking to be best
- taking risks
- learning from mistakes
- self-ambition.
3.52 Details of the Index scores for 'Confidence' are set out in Figure 3.11.
Figure 3.11: Confidence
3.53 All three groups appear to be confident about confidence. Each group scores confidence above their own average; parents 45.6, young people 43 and teachers 19. Pupils appear to be particularly determined in this area, achieving the closest parity with parents.
Relative Rankings
3.54 The relative ranking of the Index thematic/behavioural themes are set out in Table 3.2.
Table 3.2: Relative Ranking of Behavioural Strands
| Young People | Teachers | Parents |
|---|
Working with Others | 4 | 1 | 1 |
|---|
Personal effectiveness | 5 | 4 | 5 |
|---|
Problem solving | 1 | 5 | 2 |
|---|
Communication | 7 | 6 | 6 |
|---|
Approach to Learning | 6 | 3 | 4 |
|---|
Confidence | 2 | 2 | 3 |
|---|
Life outside School | 3 | 0 | 0 |
|---|
3.55 The analysis shows that pupils rate themselves highest on problem solving and confidence, while parents and teachers allocated their highest scores to young people on working with others, confidence and problem solving.
3.56 The lowest enterprising attitudes and behaviour scores across all three groups are 'Communication', 'Approach to Learning' and 'Personal Effectiveness'.
Summary Points from Chapter Three
Table 3.3: Summary Points from Chapter 3
- The Index of Enterprising Attitudes and Behaviour ( IOEAB) seeks to define and measure the enterprising attitudes and behaviour characteristics among young people
- The Index at this point is not a measure of the effectiveness of Determined to Succeed. It measures the characteristics which Determined to Succeed seeks to influence, but at this stage there is no link between inputs and outputs
- The Index can be used to assess the relative 'levels of enterprising attitudes and behaviour' between different socio-economic clusters of young people in schools across time. It can also assess the pattern and structure of enterprising attitudes and behaviour
- The Index provides an approach to establishing a robust, quantitative measure of the 'enterprising' characteristics of young people. This particular approach is new and needs to be more widely tested and refined
- The IOEAB technique assumes the enterprising attitudes and behaviour of young people to be a function of 7 attitudinal and behavioural traits, linked to DtS strategy; Working with Others, Personal Effectiveness, Problem Solving/Creativity, Communication, Approach to Learning, Confidence, Life Beyond School
- The IOEAB tool has been tested out on data generated from surveys of parents, teachers and young people
- The IOEAB is a weighted average of response to attitudinal questions. A maximum positive score, reflecting maximum enterprising attitudes and behaviour is 100 and a maximum negative score indicating no enterprising attitudes and behaviour is minus 100
- The predictive power of the Index will be increased should the surveys be repeated at a future date
- The overall Index is positive for all three groups. Parents are close to the top of quartile one, pupils close to the bottom of quartile three and teachers midway through quartile four
- While it is not clear whether the recorded scores are good or bad, they are positive and there is significant scope for improvement
- The biggest single area of disparity across the groups is in relation to problem solving, where parents and pupils score significantly higher than teachers. This was the only theme where one group (teachers) came close to a negative score
- The lowest enterprising attitudes and behaviour scores across all three groups are 'Communication', 'Approach to Learning' and 'Personal Effectiveness.
|
« Previous | Contents | Next »