The Scottish Government

« Previous | Contents | Next »

Listen

2 METHODOLOGY

2.1 Any changes to the qualifications system in Scotland will have an impact on a wide range of stakeholders including the 'beneficiaries' (young people, their parents, their future employers and universities) and the 'deliverers' (secondary schools, colleges and local authorities) who have distinct interests and needs. Accordingly, it was important to capture the views of as wide a range of stakeholders as possible and to gain a deep understanding of any issues surrounding the proposed changes. This chapter describes the 2 strands of the consultation in detail.

The written consultation

2.2 The Scottish Government officially launched the consultation document, A Consultation on the Next Generation of National Qualifications in Scotland⁴, on 10 June 2008. The document contains 16 questions on the Government's proposed changes, the rationale behind each proposed change and the relevant policy background.

2.3 An electronic copy of the document was placed on the Scottish Government's consultations website so that all interested parties, including members of the public, had an opportunity to express their views.

2.4 Hard copies of the consultation document were circulated to all primary schools, secondary schools, colleges, universities and local authorities in Scotland. Additionally, 113 organisations and individuals representing a range of interests were sent a copy of the document (a full list can be found in Appendix 2). Responses to the consultation document could be submitted either electronically or by post.

2.5 The deadline for responses to the document was 31 October 2008.

Consultation responses

2.6 In total, 1807 responses were received from individuals and organisations and Table 2.1 provides a breakdown of responses by respondent type ⁵. The highest proportion of responses received was from secondary schools ⁶.

Table 2.1 Summary of responses to written consultation

2.7 The majority of respondents completed the structured questionnaire provided in the consultation document. Others submitted full written responses or emails. The length of the latter type of submission ranged from a single page commenting on selected issues through to very lengthy responses, some of which appended additional materials. Where permission has been given by the respondent, responses have been published on the Scottish Government's website ⁸.

Analysis of consultation responses

2.8 The analysis comprised several stages. Firstly, the Scottish Government created a database for recording all submissions. This contained basic data in relation to the identification and nature of the respondent, their address and whether their response was confidential. Respondents who requested that their responses remain confidential have not been directly quoted in this report.

2.9 Secondly, Ipsos MORI undertook the analysis of all responses. This involved reading each response in full and recording the views contained on a specially designed spreadsheet containing fields for each question in the consultation document.

2.10 Each respondent was assigned one of the 'respondent type' categories described in Table 2.1 above.

2.11 Individual code frames for each question in the consultation document were established using an initial batch of 50 responses.

2.12 Following this, responses for each question were read and assigned a code. The codeframes were amended throughout the coding to include any new codes emerging.

2.13 For each question, a general "count" of the number respondents was undertaken to reveal:

• the general level of support and opposition for the change considered
• the main reasons underpinning support and opposition
• the key caveats/concerns expressed.

2.14 Finally, subgroup analysis of these findings was conducted to identify any clear differences in opinion by respondent type.

A note on the interpretation of consultation responses

2.15 It should be noted that, although the method allowed for a general "count" of the number of respondents identifying overall support and opposition along with the key issues raised, the analysis of the consultation responses presented in this report is not intended to be a primarily quantitative account. It should be noted that:

• the point of a consultation such as this is not to be a 'referendum' on the different options but to inform and enhance the policy process by providing the opportunity for all interested parties to express their views and their reasons for holding these views
• many submissions represented the views of groups of individuals who had got together to submit a joint response and some were from organisations, sometimes representing the views of thousands of individuals. This makes it inappropriate to count the responses in this way (and impossible to count the number of individuals represented)
• a large proportion of responses to the consultation were from secondary schools, meaning that any count of responses is heavily skewed towards the opinions of this group
• respondents opted-in to the consultation and therefore are not necessarily representative of their sector.

2.16 Instead, the main focus of this analysis is on reflecting the range of issues which have emerged without attaching weight to particular viewpoints. Where proportions have been given, these are only intended as a broad indication of the level of support, subject to the limitations listed above.

2.17 The views presented in this analysis have not been vetted in any way for factual accuracy. The opinions and comments submitted to the consultation may be based on fact or, indeed, may be based on what respondents perceive to be accurate, but which others may interpret differently. The report, therefore, may contain analysis of responses which may be factually inaccurate or based on misunderstanding or misinformation on the issues, but nevertheless reflect strongly held views. In some instances, such inaccuracies and misunderstandings will be relevant findings in themselves.

2.18 It should also be borne in mind that as is the case with many other consultation exercises, detailed comment is generally given where there is disagreement or concern about a proposal. When they agree with a suggestion, consultees are less likely to elaborate on the reasons for their agreement or approval. The relative length of the sections on agreement and disagreement should not therefore be read as an indication of the level of agreement or disagreement.

Qualitative research

2.19 The aim of the qualitative research was to gather informed views of some key stakeholders. The specific objectives of the qualitative research were to:

• obtain the views of key stakeholders whose views it was felt were essential to the consultation, but were less likely to respond to the written consultation (secondary school pupils, college students, parents and employers)
• explore in greater depth reactions to the proposals from some key stakeholders (headteachers, teachers and college lecturers).

Composition of stakeholder groups consulted in the qualitative research

2.20 The selection of participants for the qualitative research was designed to capture a range of perspectives within each stakeholder group. This section details the composition of each stakeholder group consulted in turn.

Parents

2.21 Parents were selected on the basis of their views of their child's likely destination after they leave school, that is, whether they are likely to go straight into employment or to go on to Further/Higher Education. This was to capture any potential differences in the focus on different qualifications, for example, parents whose children are planning to enter straight into employment might focus more on the new proposals relating to the Standard Grade and Intermediate qualifications while those whose children will require Highers or Advanced Highers to access university might concentrate on the proposals affecting these qualifications.

2.22 Additionally, parents of primary school children and parents of secondary school children were consulted. The first group was important as their children will be affected by the changes while the latter group have experience of the current arrangements.

Employers

2.23 Employers within the tourism, retail, finance, construction, recruitment and public sectors were invited to take part in the qualitative research as these sectors tend to employ large numbers of school leavers and therefore place more emphasis on National Qualifications in recruitment decisions than employers who recruit those with higher level qualifications.

2.24 In order to consult with a wide range of employers selection also took into account business size and geographical location.

Schools

2.25 Ipsos MORI selected 8 secondary schools (6 local authority, 1 independent school and 1 special school) and qualitative research was undertaken with the headteacher, teachers and pupils. In the special school, only the teachers were asked to take part.

2.26 School location, both in terms of rurality and deprivation, was also taken into account in the selection of schools in order to ensure a wide range of school types.

Pupils

2.27 Within each of the selected schools, pupils in S4 and S5 were chosen as these pupils were either studying or had just finished studying at SCQF levels 3, 4 or 5 (the levels most affected by the proposed changes).

2.28 Half of the groups contained pupils who were likely to go on to Higher Education and half contained pupils who were 'disengaged with education'.

Teachers

2.29 Teachers of a range of different subjects were selected to take part in the qualitative research. All focus groups were intended to include English and Maths teachers as it was felt important to gather their views on the proposed introduction of literacy and numeracy awards although this was not possible in all schools.

2.30 Two primary school teachers from associated primaries were also invited to each focus group as it was felt that they might be able to provide a different perspective on the proposals than would secondary school teachers. Again, this was not possible at all schools.

College lecturers

2.31 College lecturers who delivered National Qualifications were selected to take part in order to ensure the relevance of the discussion topic. A range of college lecturers in different subjects were invited including those delivering English and Maths.

College students

2.32 College students who studied at least one National Qualification subject were chosen to take part in the qualitative research, again, to ensure the relevance of the discussion topic.

Recruitment

2.33 All recruitment for the qualitative research was undertaken by Ipsos MORI and the strategy varied for each stakeholder group.

2.34 Parents were recruited door-to-door by experienced Ipsos MORI recruiters. Recruiters were issued with a structured recruitment questionnaire, and corresponding recruitment instructions, which assessed the eligibility of people to take part in the qualitative research.

2.35 Businesses within the sectors outlined above were randomly selected using a business directory, and contacted by telephone by a member of the research team. The member of staff best suited to take part in the research was identified, such as an HR officer or the owner in smaller establishments. These individuals were provided with background details of the research over the telephone or by e-mail, and appointments were set up with those who agreed to participate.

2.36 Before inviting selected schools and colleges to take part in the research, letters were sent to the Director of Education of the relevant local authorities, the Scottish Council of Independent Schools and the Association of Scotland's Colleges. The letters provided information on the research and the opportunity to opt-out.

2.37 Letters were then sent to the headteachers of the selected schools and principals of the selected colleges, explaining the purpose and nature of the research. Follow-up calls were conducted by the research team and appointments made with the schools and colleges who were willing to participate. A liaison person within each school and college was identified and Ipsos MORI worked with them to identify the most suitable pupils or college students and teachers or college lecturers to take part.

Qualitative fieldwork

2.38 The research team at Ipsos MORI moderated all groups and depth interviews. All qualitative fieldwork took place between August and October 2008.

2.39 As the consultation involved a complex set of proposals, the qualitative research was designed to ensure that it gathered informed views from each stakeholder group. Discussions typically lasted around 2 hours in order to provide time for stakeholders to engage with background and additional information relating to the current system and the consultation proposals.

2.40 Parents were identified as a group not likely to have detailed knowledge of the existing qualifications system. To gather meaningful contributions from them, reconvened methods were used. Four groups of parents were invited to attend two evening focus groups with a break of three weeks in between. This allowed the research team to provide participants with appropriate background information and participants to absorb and reflect on this information between the two groups.

2.41 Twelve face-to-face depth interviews were conducted with employers. Employers were also provided with the background information as required during the interview.

2.42 Each school that agreed to take part in the research was visited by a researcher. During the course of the day, the researcher conducted two focus groups with pupils, an in-depth interview with the headteacher and a focus group with the teachers ⁹.

2.43 The colleges were also visited by a member of the research team. In one college, a focus group with college students and a separate one with the college lecturers took place. In the second college, a focus group with the college lecturers took place.

2.44 In keeping with practice in qualitative research, adult participants who took part in focus groups in their own time ( i.e. parents, teachers and college lecturers) received token remuneration from Ipsos MORI to cover their time and any associated travel or childcare costs. This approach enabled participation from a representative sample of the stakeholders.

2.45 A summary of the methods used with each stakeholder type is provided in Table 2.2 below.

Table 2.2 Methods used with each stakeholder group

Discussion guides design

2.46 Ipsos MORI, in collaboration with the Research Advisory Group, designed discussion guides to facilitate the qualitative research. In total, five discussion guides were produced: two for parents (one for each session), one for employers, one for pupils and college students ¹⁰ and one for teachers, college lecturers and headteachers ¹¹. Although all discussion guides were based on the questions asked in the official written consultation document, each was tailored to the experiences and interests of each stakeholder group to ensure that discussion remained relevant. This meant that not all proposals were discussed with all stakeholders. For example, pupils were not consulted on the proposal on the indicative timeline. An example of one of these guides can be found in Appendix 3.

Analysis of qualitative research findings

2.47 With the permission of participants, all discussions were recorded and then transcribed for analysis. XSight (a qualitative analysis software package) was used to organise, code, search and retrieve the text.

2.48 Throughout this analysis, findings were cross referenced with those from the written consultation to allow for a deeper understanding of any common themes and to highlight any diverging opinions.

A note on the interpretation of qualitative research findings

2.49 It is important to note that the aim of qualitative research is not to generalise to the wider population in terms of the prevalence of attitudes or behaviour, but to identify and explore the different issues and themes relating to the subject being researched. The assumption is that issues and themes affecting participants are a reflection of issues and themes in the wider population. Although the extent to which they apply to the wider population, or specific sub-groups cannot be quantified, the value of qualitative research is in identifying the range of different issues involved and the way in which they impact on people.

Consultation workshops

2.50 A series of 5 consultation workshops ¹² were organised by the Scottish Government and a range of representatives from across the education sector, including parents, were invited to attend. The workshops provided an opportunity for the attendees to find out more about the proposed changes and to discuss these in detail and ask questions of representatives from the Scottish Government. The events were intended to stimulate debate and it was hoped that attendees would encourage their colleagues to respond to the consultation.

2.51 Discussions during the workshops were facilitated by representatives from: the Scottish Government, Learning and Teaching Scotland and the Scottish Qualifications Authority. A researcher from Ipsos MORI attended each event to take notes on the key issues and themes emerging from the discussions. Three discussions from each event were recorded and transcribed. A summary of the themes emerging from these events was produced and the findings have been integrated into this report.

Presentation of the findings

2.52 In a summary document such as this, it is not possible to cover every detailed issue raised in every written response or discussion.

2.53 Rather, this report provides a summary of the general level of support and opposition to each proposed change, the key considerations and caveats underpinning support and opposition, and highlights any variation in the views of different categories of respondents.

2.54 This report also outlines some related issues which were not specifically addressed in the consultation document but were raised in the qualitative fieldwork or in the 'Any other comments' section of the written consultation document.

« Previous | Contents | Next »