The Scottish Government

« Previous | Contents | Next »

Listen

2. Chapter Two Methodology

2.1. The research methodology consisted of the following elements:

• Analysis of the monitoring data collected by SE and HIE - providing information on company size, sector and the origin of the referral, i.e. which LEC referred the company onto the BLA programme. Information was available for 386 companies;
• A telephone survey of 100 companies out of a population of 225 companies which had either progressed sufficiently far in the BLA process to be able to usefully comment on it or had withdrawn from the BLA process altogether - withdrawn companies were included to allow for an analysis of reasons behind companies' decision not to continue with the programme;
• In-depth interviews with 60 companies to explore the interaction of issues at the level of the firm in more detail - in the initial phase of the evaluation 20 in-depth interviews were undertaken by means of telephone interviews, 5 for each pilot. During the main phase of the evaluation, the remaining 40 in-depth interviews were undertaken, mostly by means of face-to-face interviews (28 face-to-face interviews and 12 telephone interviews). These 40 interviews break down as follows:
  • 10 interviews for the HIE geographic pilot;
  • 9 interviews for the SE geographic pilot;
  • 9 interviews for the manufacturing pilot;
  • 8 interviews for the tourism pilot;
  • 4 interviews for the roll-out of the HIE geographic pilot.

Companies involved in the research through an in-depth interview are referred to as case study companies throughout the report to make a clear distinction with the telephone survey companies.

• Interviews with BLA stakeholders including:
  • Interview with the BLA managers in HIE and SE;
  • Interviews with the 4 BLA pilot managers in the lead LECs;
  • Interviews with staff of 10 other LECs (4 in HIE and 6 in SE);
  • Interviews with 12 local delivery agents;
  • Interview with the LDSBLA manager;
  • Interviews with 4 LDS training partners.
• A postal survey of individual learners with 2,112 questionnaires sent out to 300 companies. Individual learners could only be contacted through their employers because no direct contact details were available. Also, a substantial number of companies had not yet completed (or in some cases not yet started) their BLA training activities at the time of the postal survey. This resulted in a relatively low response rate of (4.1% or 87 questionnaires returned).

2.2. Unfortunately, the monitoring information available did not allow an analysis of the type and amount of training actually undertaken by the BLA companies. At best the databases contain information about the training included in companies' training plans; this means there is only information about training intentions.

2.3. A final methodological problem concerns the fact that the evaluation had to be completed before the end of the BLA programme (the fieldwork was completed in December 2005), when a substantial number of companies had not yet started or completed their training. The research team's previous evaluation of the Department for Education and Skills' (Df ES) Small Firm Development Accounts ( SFDA) programme showed that companies taking part in the initiative undertook most of their training in the final months of the programme and the evaluation has confirmed something similar is likely to happen for the BLA programme. This means that this evaluation unfortunately cannot address the full (longer term) benefits and impact of training activities on the companies.

Telephone survey

2.4. The sample size of the telephone survey is 100 companies, roughly evenly spread between the 4 BLA pilots. This sample size means that it has been possible to make fairly robust statements for the BLA programme in general. However, the sample size is too small to allow for a robust analysis at a more detailed level. The analysis has included a number of dimensions in exploring the relationships behind the answers given by employers (size, sector, etc.). All those where a significant (and robust) difference could be determined have been reported.

2.5. The 100 telephone interviews were achieved on a population of 225. The main reasons for companies refusing to participate were the following:

• Companies indicated they had only had limited involvement in the BLA programme and that they had little or nothing to contribute to the evaluation - in quite a number of cases, companies mentioned that they had received an initial phone call about the BLA programme but had heard nothing further about it;
• Companies did not recognise the BLA product.

Learner survey

2.6. The learner survey resulted in 87 questionnaires being completed. The low response rate of the survey means that the results of the learner survey had to be treated with caution.

2.7. The key characteristics of learners included in the learner survey were:

• Over half (53%) of the respondents fall into the 25 to 39 age group, and around one quarter are 40 to 54 year olds. Just 3% of survey respondents were 18 or under and a further 4% were over 55;
• Just over half of the respondents were female (54%); and,
• A significant proportion (40%) of BLA participants responding to the survey are in senior positions within the company, nearly one quarter are in the operations, assembly or other manual work, one in ten are in an administrative role, and nearly one in ten work in sales.

Table 2.1: Age of respondents

Source: CPC Postal Survey (Learners)
Note: The percentages do not add up to 100% because of averaging.

Table 2.2: Gender of respondents

Source: CPC Postal Survey (Learners)

Table 2.3: Staff grade of respondents

Source: CPC Postal Survey (Learners)
Note: The percentages do not add up to 100% because of averaging.

2.8. The majority of respondents (65%) had only attended one course as part of the BLA programme. Over one quarter had attended two courses, around one in ten had attended three courses and just 3% had participated in four or more courses.

Table 2.4: Number of courses per respondent

Source: CPC Postal Survey (Learners)

2.9. Of those respondents that recorded the total number of hours they were involved in BLA training, nearly half (46%) participated in 17 to 24 hours of training and just over 30% took part in BLA training for 5 to 8 hours. Just one respondent took part in a prolonged period of training, totalling over 80 hours of learning.

Table 2.5: Hours of learning

Source: CPC Postal Survey (Learners)
Note: The percentages do not add up to 100% because of averaging.

Monitoring information

2.10. All four pilots developed their own BLA monitoring database, using separate formats and variables. For the geographic SE pilot, the local delivery agent developed a web-based monitoring database containing company contact details (both addresses and telephone numbers), information about company size, turnover and sector of activity, details about the company's progress in the BLA programme (e.g. number of visits by the local delivery agent) and information about the training plan and the money drawn by the company (thus indicating that companies have undertaken at least some of their training). Finally, the database also indicates whether or not a company has withdrawn from the BLA programme altogether. In the second half of the programme the two other SE pilots started using the format of this database for reasons of consistency.

2.11. Whereas these databases contain useful and valuable information, two variables were omitted that could have added value to this evaluation: 1) names and contact details of individual learners and 2) information about training activities actually undertaken. The omission of individual learner contact details meant that it was only possible to contact the learners through the employer - which probably decreased the response rate to the learner survey substantially. The fact that no information on training activities was included hampered the evaluation in that it has not been possible to systematically analyse training actually undertaken.

2.12. The HIE pilot used its own database format, containing slightly less information; mainly the contact details of the companies and whether or not companies have drawn down any money (i.e. undertaken at least part of their training). The HIE database does not indicate which companies have withdrawn from the BLA programme, as HIE companies are never formally withdrawn from the programme.

2.13. Towards the end of the BLA programme LDS developed a comprehensive database putting together the SE and HIE datasets and complementing the available data with its own monitoring information. The LDS dataset combines:

• Company details (including size, sector and contact details);
• Data about the delivery agents and LDS training partners in touch with each company; and,
• Information about the training activities included in the company training plans (including e.g. duration of training and type of training provider).

2.14. Having said this, there has been no monitoring of the BLA process after the BLA account had been set up in terms of the type of training undertaken. However, it is expected that LDS will undertake a learner satisfaction survey towards the end of the programme.

« Previous | Contents | Next »