The Scottish Government

« Previous | Contents | Next »

Listen

CHAPTER EIGHT KEY PERFORMANCE INDICATORS AND PROGRESS TOWARDS TARGETS

INTRODUCTION

8.1 The previous chapter provided indications of the costs and benefits, and an assessment of the value for money provided by SUfI. Clearly, any consideration of SUfI's key performance indicators ( KPIs) needs to be couched in the context of those costs and benefits, and the underlying objectives which they reflect.

8.2 In doing so it should be emphasised that, at the time of the study, new KPIs were being developed with the intention of their being closely aligned to the SUfI corporate plan.

8.3 SUfI's overarching goals are set out in the form of six key objectives:

• stimulate the demand for learning
• enhance the skill and employability of individuals
• enhance the skills and competitiveness of SMEs
• provide information and advice
• improve access
• develop infrastructure and capacity.

8.4 In turn, progress towards achievement of these objectives is measured by means of KPIs, which, for 2005-06 comprised:

• to generate a set amount of individual enquiries from SUfI's target group through local and national marketing campaigns and partnership activity
• to generate a set amount of SME enquiries through local and national marketing campaigns and partnership activity
• to get a set rating for quality and effectiveness of the National Learning Opportunities Database ( NLOD)
• to get a set rating for customer satisfaction of the information and advice given
• to engage a set number of learners into learning through the network of quality assured learndirect scotland branded learning centres
• to engage a set number of SMEs into learning
• to partake in a set number of innovative joint/pilot initiatives to overcome barriers to learning.

8.5 The inter-relation between Objectives and KPIs is shown in Table 8.1 below.

Table 8.1: Relationship between Objectives and KPIs

Source: SUfI 2006

8.6 The extent to which each of the 2005-06 KPIs has been met, which was derived from material provided by SUfI, is set out in Table 8.2 below. For the purposes of discussion, the targets for 2004-05 are also shown, although the striking disparities between some of the two sets of figures require explanation. For example, while the target for individual enquiries stood at 300,000 in 2004-05, this was reduced drastically to 120,000 for 2005-06. This was because the 2004-05 figure was based on website hits, whereas the 2005-06 figure was based on unique enquiries. Clearly, this revised way of accounting for enquiries is preferable, as, under the previous method, an attempt at assessing the numbers of individuals who had been persuaded or encouraged to contact the service could be skewed by individuals generating multiple hits on the website.

8.7 A reduction in numbers also occurred in the target of SME enquiries to be generated, although the magnitude of the decrease was by no means as great (from 14,000 to 11,650). While this may have been a reflection of lower than anticipated responses from SMEs, the actual figures, at 19,428, point to further revisions being required for this KPI.

8.8 In contrast, the numbers of learners engaged through the branded learning centres had risen substantially, from 90,000 to 480,000. The targets for 2005-06 were formed on the basis of results from surveys, rather than from those of a sample audit, as had been the case previously. Thus, it would appear from the projected actual figures that those which were set for 2005-06 were far more realistic, and represented an accurate appraisal of the current situation and what might be considered to be challenging, but attainable levels of performance.

Table 8.2: Progress towards 2005-06 targets

Source: SUfI 2006

APPROPRIATENESS OF KEY PERFORMANCE INDICATORS

8.9 The following discussion assesses the appropriateness of different KPIs, and some issues related to their measurement. It should also be recognised that the Chairman's personal enthusiasm for reaching beyond the KPIs is well known within SUfI as the 'so what?' question. That is to say, numbers in themselves do not necessarily tell the full story.

Number of enquiries to the national helpline

8.10 In order to assess the extent to which it is raising awareness of, and demand for, learning SUfI currently measures the total number of calls to the learndirect scotland and learndirect scotland for business helplines. This, in itself, is useful, as it tracks the 'volume of traffic' to the helpline.

8.11 However, in both cases, these KPIs ,as they stand, are not sufficient to gauge the impact of SUfI. First, as far as the number of enquiries from individuals is concerned, more precise target-setting is required. For example, while achieving a set target of enquiries from members of key target groups, such as those with no or only low levels of qualifications, or those with little or no previous experience of engagement in learning, it is vital to identify the proportions who go on to undertake some form of learning as a result of their contact with learndirect scotland. While it is certainly a laudable goal to incite members of these groups to make initial contact with learndirect scotland, it is also important to assess the value which ensues from participation in learning, the acquisition of skills which enhance employability, and subsequent job entry. As stated earlier, there are, potentially, clear economic benefits accruing to: the individual, in terms of greater income; employers, through the reduction of skill shortages and enhanced productivity; and to society more generally, through a reduction of welfare dependency.

8.12 At the same time, as indicated in the previous Chapter, the magnitude of economic benefits derived is greater among those attaining higher level qualifications. Therefore, as well as retaining the focus on the key target 'hard to reach' groups, it is important to measure the impact at these higher levels, through providing targets for those already well-qualified. While the ability to attract 'new learners' and those from specific target groups should be built into the KPIs, it should be recognised that the greater, and often more immediate 'pay off' from the more highly qualified, represents considerable value for money.

8.13 Two key points need to be made when discussing a KPI which is based on identifying the number of enquiries from target groups, as a means of assessing the impact of marketing and awareness-raising activity. First, there is a need to provide targets for different categories of potential learners, based on characteristics such as socio-economic background, levels of qualification attainment, recent experience of learning etc. Second, it is not sufficient merely to know whether numbers of individuals from certain groups have made contact with learndirect scotland (although that does give an indication of the success of the marketing function). Rather, emphasis should be placed on the proportions who subsequently engage in learning activity, as that is when the economic benefits become apparent.

8.14 Similarly, with SMEs, identifying callers from industrial sectors which are generally less well represented in training activity, and those which have previously undertaken little workforce development may help to gauge progress against newly-specified KPIs. Importantly, however, as identified in the literature which points to significant productivity gains ensuing from investing in the training of the workforce, the value for money attached to employers' engagement in training activity is considerable. It is the measurement of the take-up and subsequent belief in training by employers which should be a focus for KPIs. As indicated in paragraph 5.15, the internal reviews of the Training Partners service which are conducted by SUfI provide robust evidence of the impact of the service.

Quality and effectiveness of the NLOD

8.15 Given that the currency and reliability of the NLOD depends upon individual providers updating their own course information, the database will never achieve 100 per cent accuracy. Providers are under no formal obligation to update their information. Nevertheless, the fact that the NLOD is such an important aspect of SUfI's remit means that it is important to maintain some form of indicator.

8.16 As with other aspects of the KPIs, the target for 2005-06 was considerably lower than had been the case in the previous year, reflecting the findings of the surveys, and therefore being aligned more accurately to the realities of available provision. In the current year, SUfI exceeded its KPI target for quality and effectiveness. However, it will be important to consider on an ongoing basis how best to maximise the accuracy of the NLOD.

Information and advice customer satisfaction rating

8.17 Again, although the 2005-06 target for customer satisfaction was exceeded, it was lower than the previous year's. Nonetheless, the findings of the SQW survey suggest that levels of satisfaction among individuals continue to be extremely high. This does not reduce the need to monitor these levels on an ongoing basis, and to include this issue in the KPIs.

8.18 Minor quibbles from the surveys concern callers not always receiving the follow-up information which helpline advisers are promising them by post, and a minority of respondents who rated the service as 'very poor'. Where such views are aired, further investigation may be required to gain a better understanding of any shortcomings in the service offered, and to develop appropriate responses.

Learners engaged in learning through the network of branded learning centres

8.19 The measurement of learners using branded learning centres is a major challenge for SUfI. Measurement relies entirely upon staff in branded learning centres providing monthly statistics to their Learning Centre Development Officer. Given that SUfI does not own the centres, and that centres are under no legal obligation to provide this data, it is possible that some centres will not provide the information required. The number of actual learners using learning centres (figures from those learning centres which do provide information), is extrapolated by SUfI in order to reach a total figure for all learning centres.

8.20 Again, further investigation may be required in order to better understand not only the levels and type of subsequent take-up of learning opportunities, but also the processes involved and any potential barriers to take-up.

8.21 Interviews undertaken with managers of branded learning centres revealed a cautious willingness on their part to provide additional learner data, if it would help SUfI to monitor its impact. Therefore, the possibility of obtaining further useful data from learning centres could be explored.

Innovative joint projects

8.22 SUfI has increasingly been involved in projects to stimulate demand amongst its key target groups. The 'Give It A Go' events in local communities have been a good example of this. Another example is SUfI's involvement in an ESF-funded project delivered by SSAScot which seeks to promote the benefits of learning to SMEs.

8.23 The number of such projects in itself does not, however, reflect what SUfI is trying to achieve. Therefore the KPIs should be related to the objectives of the initiatives. For example, the two initiatives mentioned above are geared to generating participation by individuals and SMEs respectively. The concern raised earlier about the high proportions of 'non-engagers' with few or no qualifications suggests that an initiative which attempted to address this issue could be valuable in retaining and facilitating the engagement in learning of some of these 'non-traditional' learners. If such an initiative were to be introduced, it could contribute to the achievement of this KPI.

ASSESSMENT OF PROGRESS

8.24 At the outset, it should be recognised that SUfI is still a relatively youthful organisation, which is seeking to make inroads into areas which are vital to Scotland's future prosperity, economic growth and social cohesion. If it is accepted that the targets for 2005-06 are realistic, then the fact that SUfI has not only succeeded in reaching all of its targets, but has exceeded them - in many cases by a wide margin - is extremely positive.

8.25 Certainly, given the size of Scotland's population, and of its business base, the target numbers of calls to be received, from both individuals (120,000) and SMEs (11,650), would seem, at first glance, to be ambitious. Therefore, the finding that the projected numbers show that both have been exceeded by some distance, is impressive.

8.26 However, this is overshadowed by the scale of the target (480,000), and projected numbers of learners (568,637) engaged through BLCs. This, in itself, suggests that progress is being made in effecting significant shifts in attitudes towards learning. While SUfI can only claim partial (and extremely difficult to quantify) responsibility for initiating this level of take-up, it is nevertheless important to place SUFI's work within this overall context. Moreover, the establishment of new KPIs should be an opportunity to set realistic and appropriate targets for the various aspects of SUfI's activities, whose progress will therefore be able to be monitored year on year.

8.27 As well as measuring the operational aspects, such as the number of callers, subsequent take-up of courses or, in the case of SMEs, investment in training, it will be important to be cognisant of the wider trends pertaining to learning activity. An essential part of SUfI's remit is to contribute to greater participation in learning among 'hard to reach' groups. Evidence of progress will be derived from national studies which chart trends in overall participation, and identify differential rates of participation between age groups, socio-economic groups etc.

FUTURE DIRECTIONS

8.28 As the organisation matures, it is likely that there will be a need to develop further the nature of the KPIs, ensuring a balance of those indicators defined on input/activity and output metrics with ones defined in terms of outcomes.

8.29 It will be important to recognise explicitly within an evolved monitoring and evaluation framework that similar desirable outcomes, either for individuals or for businesses, may come about through different access routes and activities. Such a framework may prove helpful in determining allocative efficiencies, offering further help in deciding how best to apportion and prioritise resources to achieve the desired outcome-based KPIs.

« Previous | Contents | Next »