The Scottish Government

« Previous | Contents | Next »

Listen

CHAPTER 2: METHODOLOGY

2.1 This chapter details the methods used to evaluate the free school meal trial for P1 to P3 pupils. A mixed methods approach was used. This comprised analysis of school meal uptake data, qualitative research with key stakeholders and a quantitative survey of parents of P1 to P3 pupils.

Analysis of school meal uptake data

2.2 A key objective of the study was to measure the change in update of school meals by P1 to P3 pupils as a result of the trial. In addition, it was recognised that increased uptake among P1 to P3 pupils may have knock-on effects for P4 to P7 pupils.

2.3 In order to analyse changes in the level of uptake, data was collected from each school in the 5 local authority areas before the trial started in October 2007, and again in late February 2008. In order to reduce the burden on schools, the February 2008 survey was carried out in conjunction with the annual School Meals Census that is undertaken by the Scottish Government. Both the pre-trial survey and the mid-trial February survey mirrored the methods employed by the School Census - a school level return being required from each school, giving figures for a "normal ³" day, with the data collection co-ordinated by the five local authorities involved in the trial.

2.4 In both surveys, mainstream schools where asked to provide the following details for each year group and split by whether registered for FSM:

• number of pupils on the school roll
• number of pupils absent from school
• number of pupils taking a school meal on census day.

2.5 Unlike mainstream local authority primary schools, in special schools, pupils are not always organised into strict year groups. An adapted version of the survey form was sent to special schools, asking for the details listed above, but broken down by those aged equivalent to Primary 1 to Primary 3 pupils and all other pupils of primary school age.

2.6 In the pre-trial survey, schools where asked to respond to a number of additional questions:

• whether school meals are cooked onsite or offsite
• whether the school envisaged that the extension of Free School Meals to all Primary 1 to Primary 3 pupils would create any practical difficulties.

2.7 Overall, there was a very high response rate to the surveys. In the pre-trial survey, we received 469 returns from schools, via the five local authorities. Table 2.1 gives the breakdown of returns by local authority. Three returns were excluded from the analysis due to obvious data inaccuracies. In total, 446 useable returns were received from primary schools and 20 from special schools ⁴. This represents a response rate of around 97% from mainstream primary schools. In the mid-trial survey, returns were made by the schools, via the local authorities and the Scottish Government. In total, 458 returns were received from primary schools and 19 for special schools.

2.8 In initial discussions before the fieldwork commenced, it was clear that some special schools already provided free meals for all of their pupils, whether or not they were formally entitled to receive FSM. Local Authorities and special schools decided whether they already provided FSM for all pupils in Primary 1 to Primary 3 before the trial commenced. Those that did were not asked to complete a questionnaire. This is the reason why the response rate was lower among special schools.

Table 2.1 Total number of school returns

2.9 Analysis of the two surveys of uptake is presented in Chapter 3. However, the change in uptake captured by the two surveys does not cover the changing pattern of uptake during the pilot period: was there a steep increase on the introduction of universal FSM, which then falls away slightly? Was there a steady increase over the pilot period? We therefore asked local authorities for some additional information throughout the pilot period on uptake. Analysis of this data is also presented in Chapter 3.

Qualitative research with key stakeholders

2.10 In each of the five local authorities participating in the trial, two case study schools were selected to take part in the qualitative research.

Sampling schools

2.11 Ipsos MORI selected the 10 case study schools.

2.12 The selection of schools was designed to ensure that a range of schools was covered in terms of population characteristics ( e.g. levels of deprivation) and school type/facilities. The sampling approach meant that:

• two schools were included from each local authority area
• the schools characteristics varied by:
• FSM eligibility (high, medium and low: against the pre-trial average for the five local authority areas).
• School size (above average, below average: against the pre-trial average for the five local authority areas).
• School meal uptake (above average, below average: against the pre-trial average for the five local authority areas).
• one school had a high proportion of pupils from ethnic minorities.

2.13 The chosen schools had a mix of facilities, including: whether all the children ate in a canteen or whether other areas were also used; on site and off site catering; breakfast clubs/after school clubs or not; and, whether schools envisaged difficulties with the extension of school meals. Matched reserve schools were also selected by Ipsos MORI at the outset.

2.14 After checking that none of the selected schools were due to be inspected by HMIE during the fieldwork period, local authority Directors of Education were afforded the opportunity to alert us to any other reason that the selected schools should not be approached to take part in the research. As a result, two schools that were currently going through a rationalisation process were replaced by reserve schools. Following this, the research staff approached the headteachers of the selected case study schools to invite them to participate in the research. One headteacher refused to participate due to the fact he was in an acting headteacher position at the school and did not feel that the school was well placed to take part in the research at that time. This school was replaced by its reserve school.

Qualitative fieldwork

2.15 During March 2008, each case study school was visited by a member of the research team. Over the course of a day in the school, the researcher spoke to the headteacher, the head cook, canteen staff, P1, P2 or P3 pupils and parents of P1-P3 pupils. In nine schools the researcher also spoke to class teachers. Although not a planned part of the research, in some schools it was clear the evaluation would benefit from speaking to classroom assistants, school administrators and dining room supervisors. This was done on an ad-hoc basis at the discretion of the researcher and the relevant headteacher. The researchers also undertook observation in the dining areas at lunch time.

2.16 On a different day, in-depth interviews were undertaken with the five local authority catering managers involved in the implementation of the trial - one interview was face-to-face while four were conducted by telephone.

2.17 In three local authorities, where there was sufficient interest from parents, an evening discussion group was held in a local hotel.

2.18 Full details of the methods used with each stakeholder type are in Table 2.2 below.

2.19 Parents were recruited to the qualitative research by means of a quantitative survey (see page 10). At the end of the questionnaire, parents were asked whether they would be willing to discuss the issues surrounding the trial in more detail and, if so, to provide their contact details. They were then invited to take part in a discussion group at the school in the afternoon or at a hotel in the evening. Giving parents the option of attending either an afternoon group or an evening group maximised the chance that they would be able to attend, ensuring that a range of parents in different circumstances could participate ( e.g. working full time, working part-time, non-working).

Table 2.2: Methods used with each stakeholder group

Discussion guide design

2.20 Ipsos MORI, in collaboration with the Research Advisory Group, designed discussion guides to facilitate the qualitative research. There were 7 guides in total: one each for pupils, parents, headteachers, class teachers, head cooks, canteen staff and local authority catering managers (although the headteachers and class teachers guides were very similar, as were those for the head cooks and canteen staff). The guides were designed around the key research questions (see Table 1.1 above) to ensure that the relevant topics were covered with the appropriate stakeholders.

A note on the interpretation of qualitative research

2.21 Qualitative research is often compared and contrasted with quantitative research. Qualitative research is less concerned with measurement ("how many?", "how often?" etc.) and more concerned with understanding motivations, attitudes and feelings ("why?", "how?" etc).

2.22 The qualitative component of the evaluation, like most qualitative research, involved substantially fewer people than the quantitative research, but it explores attitudes and experiences of participants in much more depth. The aim of qualitative research is not to generalise to the wider population in terms of the prevalence of attitudes or behaviours ( e.g. 'one quarter of pupils at Scottish schools do not take school meals because their friends don't have them', or 'girls are more likely than boys to be influenced by what their friends are doing for lunch'), but to identify and explore the different issues and themes relating to the subject being researched. The assumption is that issues and themes affecting the participants are a reflection of issues and themes in the wider population. Although the extent to which they apply to the wider population or specific sub-groups cannot be quantified, the value of qualitative research is in identifying the range of different issues involved and the way in which they can impact on people.

2.23 So, for example, one of the findings from this study was that some parents would like to have recipe cards for the school meals that their children enjoyed so that they could make them at home. Although we cannot extrapolate from this and say that X% of parents would like this, the implication is this suggestion should be considered if the provision of free school meals for P1 to P3 pupils is to be implemented across Scotland.

Quantitative research with parents of P1 to P3 pupils

2.24 A survey of parents of P1 to P3 pupils was also undertaken. This enabled us to collect some quantitative data on the trial whilst also allowing parents who did not wish/were unable to take part in a discussion group to provide some feedback on the trial.

Sampling schools for the parents' survey

2.25 The parents' survey was designed with the aim of achieving a broadly representative sample of 1,000 parents ⁵. Based on the average number of pupils per school and an estimated response rate of 50%, it was calculated that a further 17 schools (in addition to the 10 case study schools) would be required for the survey.

2.26 In order to ensure that the sample was broadly representative of the population regarding rurality and deprivation, details from the baseline uptake survey were linked to information from the School census, and the Scottish Index of Multiple Deprivation ( SIMD).

2.27 The urban rural classification was collapsed into three bands (urban, small towns and rural) and the SIMD into 3 bands (most deprived 20%, next 40% and least deprived 40%) to give a 9-cell matrix. Table 2.3 shows the distribution across all schools in the trial areas.

Table 2.3: Number of schools in pilot areas

2.28 In order to achieve a broadly representative sample, the sample design was disproportionate. It over sampled schools in deprived small towns and rural areas and under sampled schools from the most deprived urban areas (mainly in Glasgow). An alternative wholly proportionate design would not have provided such a useful sample, as in each of the small town cells, only a single school would be chosen, and no schools in the "Rural - most deprived 20%" cell would be included in the sample.

Response to the survey

2.29 The survey fieldwork ran from 11 February - 15 April 2008. The 10 case study schools and the additional 17 schools agreed to help with the administration of the survey. Schools received batches of stamped envelopes each containing a letter providing the background to the research, a questionnaire and a reply-paid envelope for parents to return their completed questionnaire directly to Ipsos MORI. Schools addressed the envelopes to the parents of their P1 to P3 pupils and posted them. A total of 926 parents completed and returned the questionnaire - an estimated response rate of 41% ⁶.

2.30 Midway through the fieldwork period, it was clear that the response rate from parents in deprived areas was significantly lower than in less deprived areas. In an attempt to boost the response rate in the most deprived areas, the following steps were taken:

• all parents in the most deprived areas were sent a reminder letter and a spare copy of the questionnaire. ⁷
• Ipsos MORI interviewers visited 9 of the 10 most deprived schools either in the morning when parents were dropping children off at the schools or in the afternoon when they were collecting them.

2.31 While this served to boost the response rate, differences by deprivation remained; the response rate was 46% in the least deprived areas, 42% in mid deprived areas and 37% in the most deprived areas.

Questionnaire design

2.32 The questionnaire for the survey of parents was designed by Ipsos MORI, in close consultation with the Research Advisory Group and was based on the key research questions (Table 1.1 above).

A note on measures of deprivation

2.33 Two measures of deprivation were used in the evaluation. For uptake data and other school level data, analysis by deprivation was based on the pre-trial level of registration for free school meals within each school. This is a commonly used proxy measure for the level of deprivation within the school population. For the data from the survey of parents, analysis was based on the Scottish Index of Multiple Deprivation ⁸ classification based on the parent's home postcode.

Structure of the report

2.34 Chapter 3 sets the research findings in context, exploring uptake of school meals, and associated trends, and identifies factors correlated with uptake. Chapter 4 discusses the process and practical challenges schools and local authorities faced in implementing the trial. Chapter 5 explores early perceptions of health and other benefits of the trial. Chapter 6 provides an analysis of the impact of the trial on the costs of providing school meals. Finally, Chapter 7 draws conclusions from the research, focusing on the implications for roll-out in other local authorities.

« Previous | Contents | Next »