Working for Families: Lessons from the Pilot Projects (Stage 1)

CHAPTER ONE Introduction

1.1 This is the First Report of the Working for Families pilot evaluation study, commissioned by the Scottish Executive from GEN Consulting. The Report covers 6 main elements, it:

Gives the background to the Study, looks at the extent to which the original brief has been conformed to and outlines the Study methodology (Chapter 1);
Gives a progress report on the pilot projects ( Chapter 2);
Looks at the findings and issues ( Chapter 3);
Looks in some detail at project monitoring and evaluation ( Chapter 4);
Looks at the good practices, mainly in project development, that have been identified ( Chapter 5); and
Provides guidance to the Scottish Executive on the main funding programme ( Chapter 6).

The Study Background

1.2 As part of the Scottish Executive's strategy for eliminating child poverty within a generation, in mid-2003 funding was given to 2 councils (Dumfries and Galloway and Glasgow) to set up pilot Working for Families projects. The projects were to provide childcare support in a variety of ways with the objective of helping those parents and carers who faced barriers to employment to move closer to employability.

1.3 The funding was to target clients whose needs were not being met by other providers but who had some motivation towards moving into employment, education or training. Thus it was aimed at lone parents, those on low incomes and households facing particular stresses, such as substance abuse, health problems and containing children with disabilities. The money was to be used to provide financial support for childcare, targeted at these groups in a way that would meet their specific needs. Additional support was to be accessed through specialist providers as needed.

1.4 A total of 11 projects were funded: 3 in Glasgow, albeit that most of these had a number of components, and 8 in Dumfries and Galloway. Although the original intention was that the money would be routed through local councils, one in Glasgow, (the Rosemount Transitions Project), that is run by a voluntary group, is being funded directly by the Scottish Executive. In total the 11 projects are to receive over 1 million for the 12 month pilots.

1.5 The main goal of this Study was to undertake a formative evaluation of the pilot projects in order to inform the development and subsequent evaluation of the main programme. This is to start in 2004/05 and has had 20 million allocated to be spent in the 2 years to March 2006. To inform the main programme the detailed Study objectives were, to:

Work with the initiatives in the pilot local authority areas to provide them with advice and support on the development of their delivery plans and monitoring and evaluation frameworks;
Work with the initiatives to evaluate their impact in meeting the needs of the target client group and moving them towards employability;
Identify and disseminate good practices between the initiatives;
Provide the Scottish Executive with detailed guidance on the implementation of the main programme from 2004-05. This was to cover such things as spatial targeting, use of local outcome agreements and mechanisms for good practice dissemination; and
Develop outline guidance for monitoring and evaluation of the main programme, both at a strategic and a project level. This was to include consideration of such issues as cost effectiveness and tracking of progress.

1.6 The assumption, of both GEN Consulting and the Scottish Executive, was that the pilot projects would be well developed by the time the Study started. It would therefore be possible to look at issues such as effectiveness in reaching the target groups and value for money. In the event this has not been possible. When work started in June 2003 on the Study it soon became clear that development was in its early stages. Even at the time of writing (October 2003) some of the projects have still to start. Accordingly it has not always been possible to do exactly what was outlined in the Study brief or in the GEN tender.

Changes to the Study

1.7 The main deviations from the methodology originally submitted to the Scottish Executive are as follows:

In Stage 3 we indicated that we would work with the projects to help them develop their monitoring and evaluation frameworks (Paragraph 4.13) ¹. This has not worked in the way we originally envisaged for 2 main reasons:
- 2 of the projects were developments of existing initiatives that already had sophisticated monitoring and evaluation systems in place; and
- Some of the projects were at too early a development stage with, in some instances, there being no delivery staff in place.
Accordingly we have, with the Scottish Executive's permission, discussed monitoring and evaluation with the 2 authorities and distributed to them the initial ideas of what a framework should include, as contained in the Mid-term Report. In the case of Dumfries and Galloway we were invited to attend a meeting of the Monitoring and Evaluation Sub-Group (set up to oversee monitoring of the pilots) where we distributed a paper outlining the likely monitoring requirements. In Glasgow these were discussed with the staff concerned and the same paper distributed. As well as making the projects aware of the Scottish Executive's likely requirements this process also raised the profile of monitoring with staff and consolidated the point that the objective of the pilots was to move parents and carers nearer to the labour market, with childcare being the means to doing this, not the end;
In Stage 3 (Paragraph 4.16) we envisaged the projects completing short questionnaires containing details of clients. We would then use this to assess the extent to which the projects were recruiting from the identified client group. As there are currently very few clients this has not been done;
In Stage 3 we intended to collect financial data from the projects in order to assess cost effectiveness. This has not been done as the projects are still mainly in the development stages;
In Stage 3 we intended to undertake a mapping of linkages (Paragraph 4.18). This has been undertaken in an impressionistic, rather than a systematic, way, as the projects are not yet fully operational. We have therefore explored with the projects the type of agencies they envisage developing networks with, rather than asking them to map this;
Stage 4 was to involve holding 2 Employment and Childcare Fora. In the event only one was held. The reason is the same as earlier: the projects are not sufficiently advanced to make 2 events worthwhile;
Stage 5 (Analysis and Final Report) was written on the assumption that the projects would have been running whilst the formative evaluation work was underway. As this has not been the case some of the things that we expected to cover have not been done. For example:
- It is not possible to say anything about the projects' effectiveness in reaching the target group nor about their impact upon employability; and
- As the projects are not, in the main, working with clients, it is difficult to say much about good practices for project delivery. However we can say a reasonable amount about good practices in project development, as this is something that we have tracked throughout the life of the evaluation.

The Study Process

1.8 The Study process involved 4 main elements:

Interviews with the 2 staff responsible for project co-ordination and liaison with the Scottish Executive in the 2 authorities;
Interviews with staff involved in wider economic development issues;
Ongoing contact with the staff responsible for running the pilot projects or, in the cases where the pilots had not been set up, with the staff who were responsible for establishing them. This was a mixture of face-to-face interviews and telephone follow-ups when it seemed as if there had been limited progress since the last meeting; and
Attendance at the 3 meetings of Dumfries and Galloway's Childcare, Employment and Training Working Group, set up to oversee the development of the various projects. We also attended the first meeting of the Monitoring and Evaluation sub-group, which was established to ensure that the projects developed a systematic approach to data collection.

The interviewees are listed in Appendix 1.

1.9 Each interview followed a semi-structured format. In addition to gathering information about the projects, a variety of background documentary material was collected. This is referred to as appropriate.

Formative Evaluation

1.10 The Study was essentially a formative evaluation of the pilot projects: that is it involved working with the projects and providing whatever advice and guidance we could, derived from our experience and knowledge of the other pilots, of monitoring and evaluation and of economic development generally. The degree to which this could be effective clearly depended upon the progress that the pilots had made. Given this, the extent to which we have been able to provide support has been more limited than was originally envisaged.

1.11 However we have been able to make some progress, for example putting Rosemount in contact with a project based in Stirling which has a well developed computer based monitoring system. We have also raised with Rosemount the possibility of it being represented on the Steering Group set up by the City Council to oversee the Easterhouse and Castlemilk projects. To some extent this would be formalising the links that already exist with the City. However this would enable the project to develop, not only links with the other Glasgow pilots, but also with senior officials whose departmental policies impact upon Rosemount's activities.

1.12 At a more general level we have been eager to stress to all interviewees the importance of being able to show that the pilots are both reaching the Scottish Executive's target group and are enabling them to move either into, or towards, work education or training. Originally we had some concerns that not all of the projects recognised the importance of being able to produce evidence to demonstrate progress in these areas. Accordingly this is something that we have devoted some time to. For example, with the Scottish Executive's permission, we distributed a note about monitoring and evaluation to the projects (Paragraph 1.7). Hopefully this has influenced the approach that will be taken to collecting monitoring information. The fact that Dumfries and Galloway, for example, has set up a sub-group specifically to look at monitoring the pilots would seem to indicate that this has had some impact.

The Report's Structure

1.13 The Report is structured as follows:

Chapter 2 gives an overview of the pilot projects;
Chapter 3 looks at the findings and issues that have emerged in the course of the Study;
Chapter 4 looks at project monitoring and evaluation;
Chapter 5 identifies elements of good practice for setting up and running the pilot projects; and
Chapter 6 draws upon this, and the earlier issues, to make a number of suggestions that could be incorporated into the guidance prepared by the Scottish Executive for the main programme when it starts in April 2004.

The 4 Appendices include a list of interviewees ( Appendix 1), and proposed forms for monitoring ( Appendices 2, 3 and 4).

1 The paragraph numbers in this Paragraph refer to those in the original tender "Working for Families Funding" 11 ^th June 2003 submitted by GEN to the Scottish Executive.

« Previous | Contents | Next »

The Scottish Government

Research

On this page: