The Scottish Government

« Previous | Contents | Next »

Listen

APPENDIX A: METHODOLOGY FOR PRIMARY RESEARCH

BMRB Scotland (referred to in this report as ' BMRB') and DTZ Pieda Consulting used a variety of methods to gather evidence for this evaluation including:

• Conducting qualitative interviews with partner organisations, the Head of the RAS and RAS customers
• Undertaking a desk research and benchmarking exercise
• Analysing the RAS management information data
• Conducting a survey of RAS customers
• Conducting mystery shopping of the RAS service

Web survey of RAS customers

Fieldwork for the customer survey took place between 2 ^nd June and 4 ^th July 2005. The sample was taken from the Relocation Advisory Service's database of customers. All RAS customers with email addresses were sent an email by BMRB, inviting them to take part in the research. The email explained the purpose of the survey, and included a unique link which they could click on to complete the online questionnaire.

The total number of customers included on the database provided by RAS was 3,529. However, 185 of these did not provide RAS with an email address, and were therefore excluded from the sample, reducing the total sample to 3,344.

A total of 863 questionnaires were completed. In order to estimate the response rate, a calculation can be made on the amount of 'deadwood'. Deadwood refers to cases where the respondent could not be contacted, and was calculated using the automatic mail delivery failure notifications which are produced when an email account has been closed. 394 of the contacts (12% of the total sample) were identified as deadwood.

After excluding deadwood, a response rate of 30% was achieved, which is a good response rate for a self-completion survey. However, this is likely to be a conservative estimate, as there may be other deadwood which we cannot identify, for example email accounts which have not been closed but are no longer used.

A breakdown of response is shown in the following table.

Mystery Shopping Survey

Fieldwork for the mystery shopping survey took place between 5 _th August and 24 _th October 2005. The fieldwork was conducted by interviewers from The Operations Centre. The mystery shoppers were fully briefed and provided with different 'scenarios' with which to contact RAS. These scenarios were designed by the research team in consultation with RAS, based on typical queries which they receive. The shoppers were provided with guidance on the basic details to include in their interaction with RAS, for example their name and nationality, along with specific questions relating to their scenario, such as queries about how to gain a visa.

Mystery shoppers were given assessment forms to complete. These forms contained questions such as how long RAS took to respond to their query, what information they were given, and an evaluation of the member of staff who dealt with their enquiry.

A total of 102 mystery shops were carried out. The following table breaks these down by contact method.

Of the 102 mystery shops conducted, 18% first made contact with RAS during July or August, and 82% in September or October. This balance of interviews was decided because RAS were facing a backlog of enquiries towards the beginning of the year and we did not wish to un-necessarily burden the staff.

When looking at the data from the mystery shopping exercise, it is important to bear in mind that some technical difficulties were experienced by RAS during the fieldwork period. The email address did not receive incoming emails in late September and early October. As a result of this, many of the shops conducted by email did not receive a response from RAS, as the emails were not received successfully.

Qualitative interviews

Sample

20 telephone discussions were conducted in September 2005 with users of RAS. 12 depths were conducted with those seeking employment, 5 with students and 3 with employers. Although the students came from a variety of countries, the majority of students were currently studying in Scotland. Those seeking employment were from a variety of countries, including America, India and Poland.

Recruitment of participants

RAS users who took part in the customer survey were asked if they would be willing to take part in further research. Individuals were then contacted by telephone by specialist recruiters. A recruitment-screening questionnaire was used to confirm individuals' eligibility for participation in the study, and to ensure the desired sample profile was achieved. Recruitment was undertaken by BMRB's specialist field and recruitment unit.

Conducting the interviews

Researchers from BMRB Social Research, trained in the techniques of non-directive interviewing, carried out all of the fieldwork. Each in-depth telephone discussion was guided by the researcher using a topic guide, or 'aide-memoire', which allowed for questioning which was responsive to the issues arising. A separate topic guide was used for each user group. Although the topic guides were kept as similar as possible, they explored the different experiences of each type of user.

Each in-depth telephone interview lasted between 30 minutes and 1 hour. Interviews were conducted at times suitable for the respondents, taking into account any time differences. Nevertheless some difficulties were encountered, such as the loss of telephone connections, the inability to get a connection at the specified time and poor quality telephone lines, which meant that a couple of interviews were slightly delayed or interrupted.

All telephone discussions were recorded and transcribed verbatim. The verbatim transcripts were then analysed using an in-house, inductive technique known as 'Matrix-Mapping'.

' Matrix-Mapping' begins with a familiarisation stage and includes an executive researcher's review of the transcripts. Based on the coverage of the topic guide, the researchers' experiences of conducting the fieldwork and their preliminary review of the data, a thematic framework is constructed. The analysis then proceeds by summarising and synthesising the data according to this thematic framework. When all the data have been sifted according to the core themes the analyst begins to map the data and identify features within the data: defining concepts, mapping the range and nature of phenomenon, creating typologies, finding associations, and providing explanations.

In order to illuminate and enhance some of the issues raised in the research, a number of quotations have been included in this report. These are taken directly from the verbatim transcripts of the interviews. All the quotations are anonymous.

« Previous | Contents | Next »