The Scottish Government

« Previous | Contents | Next »

Listen

Working for Families: Lessons from the Pilot Projects (Stage 1)

CHAPTER FOUR Developing a Monitoring and Evaluation Framework

Introduction

4.1 Being able to demonstrate that the projects are reaching the target groups and moving them closer to being economically active is crucial. Accordingly this Chapter looks at the projects' targets and outcomes and progress in developing monitoring systems. It finishes by making a number of suggestions for the type of framework that needs to be put in place if data is to be captured that will demonstrate that the Scottish Executive's objectives are being met.

4.2 The Mid-Term Report, submitted to the Scottish Executive in July 2003, outlined the type of information that it would be useful to collect if the pilots were to be able to show that they were both reaching their target groups and moving them towards employability. Feedback received on this Report, along with comments made at the meeting of the first Advisory Group, has been incorporated into the Framework that is outlined here.

Project Targets and Outcomes

4.3 The various targets and outcomes for the pilot projects are summarised in Tables 4.1 and 4.2. These have been derived from the submissions made by the projects to the Scottish Executive. A number of conclusions can be drawn:

• Not all of the projects have quantitative targets. Their lack could make it difficult to assess project effectiveness or efficiency. Targets would seem to be a basic management tool. Their absence may indicate that there has been insufficient project planning. Alternatively a project may be so experimental that it may be very difficult, if not impossible, to derive targets that are anything more than guesses. However, the nature of the projects that lack targets makes the former interpretation more likely. It may also be the case that targets have not been given as there is a fear that failure to attain them may result in sanctions by the Scottish Executive; and
• There are greater concerns about the project outcomes. These fall into 3 groups:
  • Those projects that have quantitative outcomes that clearly relate to the Scottish Executive's objectives for the pilot projects, that is getting parents or carers either into, or nearer to accessing work, education or training. Of the 10 projects in the Tables only 3 fall into this category. There are, however, others that phrase their outcomes in terms of the numbers of people attaining the Scottish Executive's objectives but which fail to specify these numbers. The implication would seem to be that whatever numbers the project eventually achieves becomes the de facto outcome measure. This does not seem to be a very objective way of measuring progress;

Table 4.1 Glasgow Pilot Project Targets and Outcomes ¹

1. At the time of writing (October 2003) targets and outcomes were not available for the Castlemilk project.

Table 4.2 Dumfries and Galloway Pilot Project Targets and Outcomes

• Projects that have qualitative outcomes some, or all, of which relate to the Scottish Executive's objectives; and
• Projects that have outcomes that are essentially outputs, for example the numbers of families who benefit from support or enabling families to access services.

4.4 Rosemount is the pilot that has most clearly focused its targets and outcomes, in terms of their being measurable and closely linked to the objectives of the Scottish Executive. This no doubt reflects its awareness of monitoring and evaluation procedures, as a consequence of its activities having been subject to previous external evaluations.

4.5 What is surprising about the Outcomes is that none of the pilots refers explicitly to soft indicators, albeit that the Easterhouse pilot comes closest with its reference to the "increased ability of parents to take advantage of training, employment and educational opportunities". The reason for expressing surprise is that, both in the academic literature, some of the project descriptions and in interviews, the difficulty of demonstrating progress over 12 months in terms of hard outcomes, (such as securing a job) was commented upon. In part this reflects the variety of problems that the target group often faces, with childcare being only one of a number of barriers to becoming economically active. As such, progress to work may be slow. Given this, it would seem sensible for the pilots to begin to think of how they can demonstrate progress in a way that will show that the Scottish Executive's objectives are being met, if only partially, in the short term. This is something that has been raised with the projects in discussion and hopefully has been taken on board.

4.6 Examination of the project descriptions also indicates that some pilots need to tailor their activities more closely to the Scottish Executive's objectives ( see Paragraphs 3.17 to 3.18). For example, part of the Dumfries Community Transport Project description implies that its objectives are to allow childcare to be accessed rather than to access this for economic development ends. Even if this is reading too much into the description, this project, and others, would seem to require quite sophisticated monitoring systems if only to ensure that the appropriate target group is accessing the services. In turn this has implications for marketing and for the sensitive application of eligibility criteria when making decisions about who should be allowed to use a service.

Monitoring and Evaluation Systems

4.7 Given that not all of the projects have been set up, and those that have are mainly in their early developmental stages, it is unsurprising that progress on developing monitoring and evaluation systems is limited. However, given the short timescale and the need to show that objectives are being attained (if continuing funding is to be secured) setting up such systems should be a priority.

4.8 It seems that only 2 of the 11 projects have monitoring and evaluation systems. Not surprisingly these are the 2 established ones: Rosemount and Access to Work. However some of the others, for example Dumfries and Galloway's Project 5 (Expansion of Childminding Service), have drafted monitoring pro formas.

4.9 The Rosemount system has 4 elements:

• Profile information that is completed for each student. This covers things such as number of dependents, qualifications, employment status and benefits being received;
• A Student Referral Form, covering how the student heard about Rosemount, courses they are interested in, childcare needs and ability to communicate in English. Interestingly, in the light of the earlier comments about soft indicators (Paragraph 4.5), there is also a section on expectations covering such things as "Gain Skills", "Access Further Education" and "Gain Confidence". This is, however, simply a series of tick boxes, rather than a graded system which would allow measuring and tracking of progress;
• An Aftercare Guidance Form which summaries the student's needs and the action taken; and
• A Discharge Form which summarises why the student left, any referrals to other agencies and whether there is a need for follow-up.

4.10 At the moment the system is paper based and much of the information is collected in the form of open ended questions/statements. This would make analysis difficult as there would be an initial need to classify the responses. However, some of the pilot funding is to computerise this system. As part of this process Rosemount is currently revising its monitoring forms with the help of an external consultant and is trying to identify suitable software packages.

4.11 The Access to Work monitoring system is similarly comprehensive. Although some of the material collected is part of an interview process, designed to assess suitability for project entry, it could be used for monitoring purposes. The process starts with an interview which involves the Project Co-ordinator, members of the Accessible Transport Forum and local people asking the interviewee a number of questions covering such things as:

• The ways in which having a driving licence would be of benefit;
• Reasons why the interviewee feels that he/she should be given a place on the project; and
• The extent to which transport and childcare barriers restrict ability to take up employment.

This information is then fed into a scoring system which is used to make a decision on suitability for entry to the project. This is essentially a series of Yes/No answers to such questions as "Is the applicant actively seeking work?" and "Is childcare a barrier to labour market entry?"

4.12 Once recruited, clients go through an induction interview. Use is then made of the Rickter Scale, an evaluation tool, essentially a plastic board with sliders which allows clients to record their responses. The initial use asks for responses to the "Preparation for Work" Life Board. This asks a variety of questions covering such things as readiness for work, income needs from work, progress towards getting a job and the extent of job preparation. Use is also made of other boards as appropriate, for example that covering Drugs and Alcohol. This information then forms a baseline against which "Distance Travelled" can be measured when the questions are repeated at a later stage.

4.13 SCMA, that is responsible for delivering the Expansion of the Childminding Service (Dumfries and Galloway's Project 5), has produced a draft monitoring form that has been distributed to members of the Council's Working Group and feedback asked for. The intention is that information from individual advice sessions will be collated on a monthly basis. This will then be forwarded to the co-ordinator. The information to be collected includes:

• Numbers of attendees who were interested in becoming childminders;
• Their characteristics, such as being a member of an ethnic minority or a lone parent;
• Numbers receiving specific types of benefit; and
• Barriers to entering the labour market.

4.14 The other projects have made limited progress in developing monitoring systems. In Dumfries and Galloway the expectation seems to be that this will be the responsibility of the yet to be appointed co-ordinator, who is to oversee all 8 projects. In Castlemilk it is similarly expected that the secondee co-ordinator will have experience of monitoring and evaluation that will be used to develop a system for the pilot project.

4.15 Whilst this limited progress may be inevitable, given the development state of most projects, it does need to be urgently addressed. One reason for stressing this is that the discipline of developing these systems will emphasise to project staff the need to ensure that the pilots target the appropriate client group and are helping to move them towards work or training/education, rather than simply providing childcare support as an end in itself. In its turn this realisation should impact upon such things as marketing, eligibility criteria and measuring progress.

Developing a Monitoring and Evaluation Framework

4.16 On the basis of the examination of the stated targets and outcomes, and our understanding of the Scottish Executive's aims for the pilot projects, we would suggest that the projects need to -

• Revisit their targets and outcomes and develop quantitative measures, especially for the targets;
• Ensure that outcomes relate explicitly to the Scottish Executive's objectives. For some projects this might mean that they modify their activities;
• Give greater thought to the use of qualitative, soft, indicators to complement the hard measures in order to measure distance travelled as a result of project participation; and
• Develop monitoring systems that demonstrate that the target client group is being reached and show how this group has progressed towards employment or education/training.

These criteria should also apply to the main programme when it starts in 2004 and could usefully be specified in the Scottish Executive's guidance.

4.17 Whilst the projects are varied, in terms of the type of support they offer, they should all have 2 common characteristics:

• Be targeting a similar client group; and
• Be providing support to move this group towards work, education or training.

4.18 Given this, there should be a degree of similarity in the monitoring frameworks used, not only across the pilots but also across the projects that will form part of the main programme when it is launched in 2004. Our suggestion, therefore, is that any monitoring and evaluation framework fall into 3 parts:

• Client Profile information so that it is possible to ensure that the target client group is being reached;
• Hard Indicators relating explicitly to the Scottish Executive's objectives. These would relate to clients obtaining jobs and places on training and education courses. Voluntary activity should also be measured, as this may be a half-way house towards obtaining work, something that Rosemount stresses (Paragraph 3.45 ); and
• Soft Indicators that measure such things as changes in attitudes and expectations as a result of participation in a project. Given the potential problems in showing progress on hard indicators over a short time such information would seem to be important if progress is to be measured.

4.19 In addition to this information we assume that the normal financial monitoring and auditing will be undertaken to ensure that the money is being spent as agreed with the Scottish Executive. As such this does feature in our draft frameworks.

4.20 It is also likely that, as the 3 examples described above indicate (Paragraphs 4.9 to 4.13) the projects will all want to collect information that is specific to their own project. Given that this may be unique this is again not something that features in our draft framework.

4.21 The point was made in the Mid-Term Report that, in developing frameworks, there is a need to strike a balance between the theoretical ideal (which invariably tends towards the long and complex) and having something that can be used with minimal additional effort. Following feedback from the Scottish Executive the suggested monitoring framework was revised. However, these revisions were not to the overall approach, which was accepted, but to the detail. One consequence of this is that the content has increased. Its completion may, therefore, require additional work. However, if these requirements are made clear at the start of the funding period then projects should be able to incorporate them into their systems with limited additional effort.

4.22 As far as possible the contents of the pro formas consist of closed questions, although some open ones are inevitable in order to keep the length to manageable sizes. It would also be useful if they can be set up and distributed by the Scottish Executive electronically using an Excel spreadsheet. This should allow analysis of the closed questions to be undertaken automatically. However this would require that all projects be linked to the Internet. Whilst this would be no problem for local authorities and the larger projects, it might be for some of the smaller ones that may be involved in delivery. This could be overcome by making the provision of monitoring information electronically a grant condition.

4.23 In terms of the framework's content, for the Profile information, the following would seem to be required:

• Client's name and address;
• Gender;
• Age;
• Main language;
• Residency status which may be an important factor in certain areas, for example Glasgow, where there are large numbers of asylum seekers in some localities;
• Family status;
• Disabilities;
• Household composition;
• Non-child dependents;
• Qualifications held;
• Current economic status;
• If working, information about the job held;
• If not working, date last worked and information about this job;
• Benefits received;
• Dependent children (numbers and ages); and
• Other barriers to accessing work, education or training.

The detailed content of some of these questions was discussed with the Scottish Executive and some suggestions for additional information made. Most of these have been incorporated into the revisions. Some have, however, not been included as it was felt that they were too complex and would provide limited useful information. The main ones related to questions about the categorisation of jobs and industries. These have been left as open questions which may have implications for analysis.

4.24 The Hard indicators relate to the intended eventual outcomes of the Working for Families Funding, that is jobs, places on education and training courses or voluntary involvement. Our suggestions are that the following indicators be collected:

• Job secured, with information on its characteristics (full or part-time, temporary or permanent, level of skill, and gross weekly wage);
• Education course entered, with details of its title, provider, duration, time of attendance (Full-time, part-time, (morning, afternoon evening, weekend)), and qualification aimed at;
• Training course attended again collecting the same details as for education; and
• Voluntary work involvement, covering type of organisation being worked in, role and hours worked.

4.25 The main omission from the Hard indicators are any measures relating to support to enable people to sustain employment and develop a career path, something that Rosemount, for example, intends to do by working with low paid employees in entry level jobs. Although this is not one of the Scottish Executive's explicit target groups it may be that the indicators could be supplemented to cover this group.

4.26 Soft indicators are seen as important in order to try to capture project impacts that may not immediately result in a Hard outcome. We suggest that the focus be upon measuring Distance Travelled, that is changes in expectations and attitudes following the childcare support. This could be done in 2 main ways:

• By asking parents/carers to assess such things as their motivations and confidence at the start and end of support. However we recognise that this can be difficult, especially at the start of an initiative when clients may be very uncertain about participation and their expectations; or
• Towards the end of support when people are asked retrospectively to assess changes in motivation or confidence. The problem with this is that there may be an inbuilt tendency to overestimate project impacts, especially if participation has been a positive experience.

4.27 It may be that the most effective approach is to capture attitudes a short period into a project, when participants are clearer about what the project is offering. There would then be a follow-up at the end of the period of support that is 3, 6 and possibly 12 months after. This would allow Distance Travelled to be measured when the results are compared.

4.28 Most Distance Travelled methodologies, used in employment projects, capture information on some, or all, of 4 types of core outcomes: skills related to work; attitudes; personal factors; and practical abilities. Working for Families funded pilots are not generally being used to provide training (although there are exceptions, for example Childcare Training is part of the Roving Crèche project in Dumfries and Galloway). The projects are essentially about improving access by removing, or lessening, one barrier to labour force participation. As such, measuring core skills relating to work and practical abilities, and possibly even personal skills, may not be relevant. Accordingly the type of information that we suggest should be captured would cover:

• Motivation;
• Confidence; and
• Self-esteem.

Each of these dimensions would be measured on a progressive scale, with respondents being asked to give responses to set questions.

4.29 In terms of the timing of the implementation of the frameworks we would suggest that:

• Shortly after a client is recruited onto a project, and once they are aware of what support they are to receive, they be asked to provide the Profile and the Baseline Soft Indicator information by completing the appropriate pro formas. Examples of these are given in Appendix 2;
• Upon completion of their time on the project clients provide information that enables any Outcomes and distance travelled to be assessed. An example pro forma is shown in Appendix 3;
• This information is then collected again after 3 and possibly 6 and 12 months (Appendix 4). We are, however, aware that there are resource implications involved in tracking clients. If this is thought necessary, the Scottish Executive needs to make this a contractual requirement when funding is approved. To try to track beyond 12 months may be difficult if only because contact addresses and telephone numbers change. It may also be expecting too much of the intervention that, given all the other changes in the wider environment, it would still be possible to isolate its impact after a year.

4.30 In addition to completing these pro formas we would suggest that the process involve:

• An initial assessment to ensure that the potential client is committed to trying to secure a job or course place in the short term and to identify the barriers (other than childcare) that restrict access to labour market participation (Paragraphs 3.35 to 3.39); and
• Some form of Early Exit Assessment for those clients who leave the project before completion.

The process is shown in Diagram 4.1.

4.31 The intention would be that the Profile, Baseline and the Completion information pro formas be completed by the client with advice and support, as appropriate, from project staff. The tracking information will be more difficult to collect. Initial contact needs to be made by telephone when information on such things as current status can be gathered. Collecting the distance travelled information will require either (in descending order of preference) that:

• Clients be invited back to the project to complete the pro-forma;
• The pro forma be sent out for them to complete and return by post; or
• It is completed over the telephone.

The exact procedure to be used should be left to the projects, although the requirement to produce this information should be made a grant condition.

4.32 There are different methodologies for collecting soft indicators. It might be appropriate if further consideration were given to this before the main programme is rolled out. In particular the merit of using some "off-the-shelf" approach such as the Rickter Scale needs to be considered. Further information can be found at the following web sites:

• www.rickterscale.com
• www.getting-on.co.uk/toolkit/index.html

Summary

4.33 Effective systems for monitoring the pilots, and the main programme, seem essential. Increasingly there is recognition of the need for such systems and progress is being made in setting them up. The proposals contained in this Chapter would, if implemented, ensure that a standardised approach was put in place across all of the projects when implementation of the main programme starts. This will allow comparisons to be made and will make the eventual evaluation of the Working for Families Fund far easier and more economical. Accordingly we would suggest that this is something the Scottish Executive implement.

Diagram 4.1 The Assessment and Monitoring Processes

« Previous | Contents | Next »