« Previous | Contents | Next »
Listen
3. PROGRAMME MANAGEMENT AND IMPLEMENTATION
3.1 INTRODUCTION
This Section examines the management and implementation of the Phase Two Pathfinder Programme. In particular it examines: the process of allocating and administering grant funding through the Challenge Fund; the monitoring and support framework put in place; and the arrangements for programme management and administration.
The analysis presented draws on documented evidence made available by Communities Scotland and incorporates the views of Learning Connections staff, the external support contractor, and lead representatives from each of the eight Pathfinder projects.
3.2 CHALLENGE FUND PROCEDURES
3.2.1 INITIAL PROMOTION
Learning Connections initiated the Pathfinder Challenge Fund in April 2003. It was launched by way of an advert in the Times Education Supplement and through some 2,050 letters issued via a database of known individuals and organisations active in the literacies field.
While the Learning Connections network of contacts and distribution list was less developed in 2003 than today, this approach seems to have been handled in a considered way and proved successful in reaching a wide range of prospective applicants.
The interest generated from initial promotion is evident. In all, 296 requests for application packs were requested from organisations across Scotland. This included contact from 29 of the 32 Scottish local authorities, and interest from around 60 voluntary organisations.
3.2.2 THE APPLICATION PROCESS
The Challenge Fund was based on a two-stage application process. Initially, interested organisations completed a Note of Interest giving a brief outline of their proposed Pathfinder project. A smaller number of eligible organisations were then invited to submit a full application.
In response to the call for applications, 93 organisations submitted Notes of Interest, 18 of which were invited to submit a full application and offered ongoing guidance and support to do so. In all 16 full applications were received.
Overall, a robust and guided application process was put in place, which made use of appropriate application forms and supporting materials. No significant difficulties have been reported by the individuals involved in designing/administering this process or by the eventual grant recipients.
3.2.3 PROJECT APPRAISAL AND SELECTION
Staff within Learning Connections initially assessed Notes of Interest in order to ascertain the extent to which these were consistent with Pathfinder funding criteria and were unlikely to duplicate existing work nationally.
An independent funding panel, comprising representation from Learning Connections and external partners, then appraised full applications. Applications were assessed against an agreed scoring framework. Projects were ranked on the basis of scores allocated and the decision was taken to support eight projects within the available budget.
This two-stage process of project appraisal appears to have been appropriate and effective. The process was transparent (assessment criteria were agreed and circulated in advance) and decisions drew on a range of appropriate expertise from within and outwith Communities Scotland. The evaluation team has been made aware of no significant difficulties arising from this process.
3.2.4 FINANCIAL ADMINISTRATION
Applicants were notified of the outcome of the funding competition following assessment and there was an expectation that the selected Pathfinder projects would be up and running by January 2004. An award level and payment schedule was agreed with each Pathfinder host organisation.
In line with the terms of grant, payments were made quarterly in arrears on the basis of expenditure incurred and financial claims submitted. Expenditure was authorised by Learning Connections staff and monitored through the Scottish Executive TRS grant payments system.
Although a standard practice for the Scottish Executive, and agreed with project representatives at the outset, this retrospective payment procedure caused cash flow difficulties for one voluntary sector project host in particular. Arrangements for making payments 'in advance of need' should therefore be carefully considered in relation to voluntary sector grant recipients.
3.2.5 MONITORING AND EVALUATION SUPPORT
3.2.6 SUPPORT ARRANGEMENTS
The Phase Two programme aimed to generate maximum learning from the eight Pathfinder projects. As such it incorporated a formative evaluation framework designed to enable projects to track, review, share and improve performance on an ongoing basis.
In February 2004 an external consultancy, ODS Ltd, was commissioned to develop and support the implementation of the Pathfinder Evaluation Framework and Toolkit. The objectives of this commission were threefold:
- to develop a framework that could be used by all of the projects, acknowledging their different themes and based on common principles;
- to develop evaluation tools which could also be used by the projects at the different stages required by their 'Conditions of Grant' letters; and
- through ongoing support, to identify and implement any changes required in the evaluation process.
The formative evaluation framework operated in parallel to the delivery of the Pathfinder projects and was designed as complementary to the independent evaluation reported in this document.
3.2.7 THE EVALUATION FRAMEWORK
The formative Evaluation Framework was developed in a collaborative fashion with Pathfinder project representatives, who were given the opportunity to come together to collectively input at the design phase. As such it was a positive, participative process and arrangements were not imposed on participants.
The Framework provided an overall approach to ongoing self-evaluation, which attempted to cater for the diversity of project types and organisational capabilities. The Framework centred on ongoing Pathfinder reporting against eight agreed evaluation criteria.
A difficulty conveyed to the EKOS evaluation team by the support contractor was in tailoring this 'one-size-fits-all' model to all requirements. For one or two of the Pathfinder organisations, with limited performance management experience, the Framework was perceived as having been an additional and onerous burden. For the more experienced organisations the Framework was regarded as having been relatively straightforward, albeit not fully integrated into existing evaluation frameworks e.g. Learning, Evaluation and Planning ( LEAP).
The LEAP framework was considered as a possible model for the Pathfinder Evaluation Framework. Some of the terminology used in LEAP (such as 'outcomes' and 'outputs') was incorporated into the Framework. However, the Evaluation Framework Advisory Group decided that because the idea of using LEAP was introduced after the evaluation framework had been partly designed (with input from the Pathfinder organisations), any attempt to incorporate the LEAP model wholesale would only lead to confusion.
3.2.8 THE TOOLKIT
The Framework incorporated an Evaluation Toolkit, which provided a guided set of materials to be used by Pathfinder projects to assist performance management. These included pro formas that would be used to collate baseline data, establish an action plan, report quarterly on progress against key evaluation criteria, and undertake a detailed final self-evaluation.
While overall regarded as useful, Learning Connections and Pathfinder project staff have conveyed some slight concerns that the materials:
- were introduced during the busy set-up period for projects, meaning that some were playing 'catch up' from the outset to complete required paperwork;
- did not include any guidance or pro formas related to monitoring learner progression or outcomes; and
- centred on multiple and fairly complex evaluation criteria ( e.g. best value, innovation etc.), which were subject to variable interpretation by the Pathfinder projects.
It noting all of the above, however, it should be stated that the Pathfinder projects were very closely involved in developing and designing the evaluation criteria documents to be used in the toolkit.
3.2.9 ONGOING SUPPORT
The external support contractor to the Pathfinder organisations provided ongoing guidance and support in three main ways.
ONE-TO-ONE SUPPORT
Ongoing one-to-one consultancy support was offered to each Pathfinder, and was generally regarded by them as both helpful and informative. The input of the support contractor has mainly been used to compile baseline information, define project outputs and outcomes, and formulate dissemination strategies.
The use of support, however, was not a mandatory part of participation, and as such the take-up of advice varied substantially between projects, with most making little or no use of the support available. Despite efforts from the contractor to encourage the take up of support, Pathfinder organisations typically prioritised the delivery of agreed activities over the monitoring of project progress.
The scope of the support contractor's remit was also restricted within the terms of the contract from Learning Connections to advising on the use of the evaluation framework and tools. As instructed, their remit did not extend to other (potentially useful) aspects of project planning, the design of internal project monitoring systems, general management support, and the design of 'exit strategies'.
It should be noted that 'exit strategies' were for the Pathfinder projects to arrange themselves (planning exit strategies was an important part of the Challenge Fund application process). However, in the spirit of co-operation, the support contractor, did offer to facilitate additional themed workshops including one on 'exit strategies'.
WORKSHOPS
A series of five workshops were organised to provide the Pathfinder projects with the opportunity to come together and share common experiences. The structure and content of events was designed to enable projects to provide feedback to each other, to address issues of concern, and to learn about topics of interest.
Project representatives have reported that the introduction of workshop events was a welcome addition to the Programme. However, the commitment to these waned as project representatives, particularly those located in outlying areas, recognised the significant time involved in travelling to, and participating in, events. Concerns have also been expressed that the content of some events, one in particular, appeared rather unfocused and did not meet initial expectations.
WEBSITE
A dedicated Pathfinder website was established in early 2005 to enable users to download relevant programme materials, find out about other project activities, and share views with others via a discussion forum. The idea of having a website stemmed directly from early workshops facilitated by the support contractor and was welcomed by the projects. Access to the website was restricted to Pathfinder representatives and Learning Connections staff.
Project representatives have suggested that this website was an underused resource, which provided relatively static and non-interactive content. They have indicated limited value from the website, and consequently their use of the website and its content seems to have been minimal.
3.2.10 REPORTING AND DISSEMINATION
Despite input from participating organisations, the Evaluation Framework followed a relatively rigid written reporting format based on quarterly, annual and final reports. Where able to draw comparisons, Pathfinder representatives have reported little distinction between this and the reporting requirements of other grant funders.
All Pathfinder projects were required to complete: a baseline data form; a Project Action Plan; and Quarterly/Annual Progress Reports; and quarterly financial claim. The use of the other toolkit documents was optional, although it was thought that these documents might have assisted the projects when it came to compiling progress reports and in the final evaluation of the Pathfinder project.
In some cases, project representatives have reported that an excessive volume of Pathfinder reporting materials was required. The Support Contractor has also confirmed this assertion.
While helpful in reporting ongoing progress, and demonstrating accountability for the grant funding received, it has been acknowledged by Learning Connections staff that the Framework did not lend itself to ongoing critical reflection. This, and a natural inclination not to report negatively to funders, has meant that the reports produced are in many cases overly positive and offer very little critical analysis of the ongoing challenges arising from project implementation.
An important question is also the extent to which the reported information was used as a basis for ongoing learning and dissemination. Quarterly monitoring reports were posted on the Pathfinder website and experiences shared between projects at scheduled workshops. However, it seems that an opportunity was missed to systematically share the learning from these demonstration projects with a wider audience. That said, various Pathfinder dissemination activities did take place. For example, a small number of the Pathfinder projects were active in sharing their experience through local seminars and events. Learning Connections staff also ensured opportunities to take this experience onto a wider stage. A major Pathfinder dissemination event was also held in March 2006, as the culmination to the Pathfinder programme.
3.3 PROGRAMME MANAGEMENT AND CO-ORDINATION
3.3.1 THE MANAGEMENT FRAMEWORK
The Learning Connections Team within Communities Scotland managed the Pathfinder programme.
A review of management arrangements has indicated that the programme was managed as two separate but interrelated pieces of work:
- ongoing Pathfinder project delivery and support; and
- the external Pathfinder Evaluation Framework support contract.
Learning Connections staff have identified two main management issues arising at programme level:
- there were no regular dedicated programme management meetings that involved all relevant staff, including the support contractor, which meant that some confusion over roles and responsibilities emerged;
- while all of the reports were available on the website, there was no systematic process for making use of quarterly Pathfinder monitoring information, which meant that this intelligence wasn't used to influence programme delivery.
In addition, while the delivery of the external support contract benefited from ongoing oversight and guidance from a small Advisory Group, the Pathfinder programme itself did not. This meant that following the allocation of grant funding, there was no overall independent strategic co-ordination, guidance or input to the design or direction of the programme.
3.3.2 OPERATIONAL SUPPORT
Despite some challenges in establishing a clear and robust management framework, a number of staff within Learning Connections played an important and effective role in supporting the implementation of the Pathfinder programme.
Learning Connections staff fulfilled a range of responsibilities, with two staff tasked with managing the Evaluation Framework, and seven staff tasked with offering ongoing support to the Pathfinder projects.
In relation to this input, Learning Connections staff and Pathfinder representatives have found that:
- staff were given a degree of flexibility to undertake their roles and no significant tensions arose as a result of the dual role of staff, i.e. they acted both as grant administrators and as a source of ongoing advice and support;
- the approach and input of staff varied somewhat between projects, influenced by their background and interests of staff, although all appeared to fulfil their responsibilities with diligence; and
- the input of staff was generally valued by Pathfinder organisations, who welcomed the positive and supportive relationship that they established with their designated staff contact.
The support contractor was tasked with a specific remit, to provide support relating to the implementation of the Evaluation Framework. Where this direct support was utilised by the Pathfinder projects this was regarded as effective and of a high quality.
« Previous | Contents | Next »