« Previous | Contents | Next »
Listen
CHAPTER 4 THE LOGIC MODEL AND EVALUATION PLAN
The logic model
195 The research specification stipulated the use of a logic model (figure 4.1), to be developed by the researchers from the objectives set, and the work programme for the pilots. It was to be generic so that it could be used by any further pilots. Development of the model accounted for much of the early evaluation period, and was finalised in April 2006, after discussion with the research manager, and some recasting of the objectives of the pilots and the evaluation.
Figure 4.1 The logic model
196 The logic model was developed using the guidelines developed by the Kellogg Foundation (2004). It aims to show visually what is intended from a project or process, how this is to be done, and how progress might be measured. Logic models can be created at different levels of detail (for example, identifying the inputs or resources available) but the key components of the current model were
- the activities to be carried out (or outputs - the practical steps taken/services delivered);
- objectives (or outcomes - the specific aims, benefits or differences intended); and
- the overall goal (impacts/longer term outcomes).
197 A further step was to identify indicators to show whether the objectives were being achieved, and information sources that might provide these.
Indicators for objectives
198 Many of the activities identified in the model were those already laid down in the VWOs' work programme by the SE ( see previous chapter), rather than devised by the researchers, LSGs or VWOs to meet objectives. Indeed, to finalise the model, it was necessary to recast the objectives originally presented by the SE. These were reduced from 7 to 5 (see table 4.1), since some were seeking to achieve the same ends. Once this was done, it was possible to assign indicators that would help show whether each objective had been met, and sources of information for these.
Table 4.1 Indicators for objectives
199 As a final step in this part of the work, the evaluation plan was tailored to the objectives for VWOs, researchers, and steering groups. The plan set out activities and sub-activities, according to responsibility for and timing of these activities, and reporting dates. The finalised plan showed activities along a timeline, and split and colour coded to show activities by each partner (the VWOs, the researchers and national and local steering groups).
200 The final evaluation plan is shown in annex 6.
201 The overarching logic model produced, and the accompanying evaluation plan, formed the basis for evaluation of the pilots. Results from each information source and its implications for the evaluation are discussed in later chapters: the results of the pilot work programme (chapter 5); the agency survey (chapter 7); and interview study ( chapter 8).
202 One further major source of information, designed to show whether witness experiences had improved, was the witness survey. Following consultation with the VWOs and agencies asked to administer the survey however, it was decided to administer it for a one month period only. (A 12 month period was originally proposed, to gauge any changes in level of service provision.) This meant there was no prospect of forming a 'before' and 'after' view of witness satisfaction, but it was also decided that the pilot period was too short to produce and measure any changes in witness satisfaction during the evaluation.
203 This means there is no indicator of whether levels of witness satisfaction have changed during pilot operation. However, the decision was taken by the SE to consider this as a baseline survey that might be repeated in the future. The results of the witness survey are given in chapter 6.
204 In essence then, success against this objective cannot be assessed by the current evaluation other than by indirect evidence from agency surveys and interviews.
Reconciliation of the evaluation plan with VWO action plans
205 Certain activities were determined by the SE for the VWOs from the start, and the need for others was determined by the VWOs in tandem with the SE or LSGs to form, in total, the VWOs' action plans. One question was whether action plans developed would conform to or reflect fully the original objectives, which the researchers were trying to assess.
206 By June 2006, both pilots had agreed their initial action plans. The actions identified were those agreed with the LSG, which were not designed to reflect or measure progress against overall pilot objectives, but rather progress towards agreed actions. As such, the action plans were not related to the overall model evaluation plan. This entailed some work to reconcile activities against the overall aims of the pilots.
207 During the current evaluation, each of these initial action plans was 'back-referred' by the researchers to the pilot objectives shown in the logic model, by adding a column at the start of the table assigning overall pilot objectives. To illustrate this, the exercise completed for the Aberdeen plan is shown in table 4.2, with the generic objectives in the first column. It should be noted that some degree of interpretation was required, and that several of these activities might well be attributable to multiple objectives.
208 While such work may seem unduly time-consuming, it was designed to help the pilots maintain focus on the work as a whole. It also assisted in providing documentation of how far the activities were able to be completed within the time scales determined for them, and alerted VWOs and LSGs of any shortcomings in their action plans' ability to meet their objectives.
209 For example, undertaking these exercises indicated at an early stage that many of the priorities and activities identified by the VWOs and LSGs related mainly to the pilot objectives on identifying and investigating additional needs and priorities, and improving co-ordination between agencies. While this is perhaps unsurprising, and achievement of each of these objectives would contribute towards improving witness experiences, it meant that the objectives on evaluating the pilot, and putting a monitoring system in place, appeared to be less well addressed.
210 Relating activities to overall pilot objectives also alerted VWOs to work that was being done, even although it did not appear as a specific action point or was not envisaged by the SE at the start. Again, back referring by the researchers to the pilot objectives shown in the logic model, showed that the need for a monitoring system was poorly reflected in the initial work plans, although one means of monitoring is the LSG (and its meetings, minutes and reports from the VWO). This was work that the VWOs were largely already doing, in keeping the LSG informed and seeking their input, despite the absence of such work in the action plans.
211 Table 4.2 below provides the reconciliation for Aberdeen of the pilot evaluation plan and the pilot action plan as an illustration.
212 A further benefit of relating activities to overall objectives derives from the dynamic nature of the action plans. Since the plans developed by each pilot to guide their activities were not fixed at a particular point in pilot operation, ongoing work was needed to ensure that as further action points were identified throughout the course of the VWO work, their inclusion in the action plan was based on their fit with pilot aims and objectives.
213 Perhaps predictably, work on self-evaluation and monitoring systems continued to be less prioritised by LSGs when setting action points throughout the pilots. Over the bulk of the pilot period, activities related to monitoring and self evaluation were less well covered than for example, identifying gaps and working on agency co-ordination. Such activities were also perhaps less rewarding and difficult to progress.
214 For example, towards the end of the period and after strenuous efforts the VWOs obtained monitoring data from SCS on court users, but this was too late in the project period to inform the work, form time series, or be usefully analysed and interpreted. Given more time, such data might form the backbone of a local monitoring system.
215 The researchers were not in a position to require that the LSGs or VWOPSG considered fit against pilot objectives before adding tasks or activities to the VWOs' workloads. This resulted in heavy weighting of the objective to 'identify and investigate additional needs and priorities'. This was perhaps inevitable since this objective relates most obviously to 'getting things done' and also because the pilots were diverted into addressing implementation of the VWA.
216 Chapter 5 on the results of the work programme includes details of progress against the objectives in the logic model and chapter 9 (conclusions) summarises these.
Table 4.2 Illustration of reconciliation between evaluation plan and action plan
Summary of the logic model work
- A logic model was prepared, which involved simplifying the SE's original objectives.
- This showed that most of the VWOs' activities fell within the objective concerned with identifying and investigating additional needs and priorities. The successful establishment of the LSGs, their continuing existence, the action points progressed by members, and the general willingness of members to extend the pilot period proved some evidence that co-ordination of agencies had improved in each area.
- The other objectives, including self evaluation and putting a monitoring system in place, were less well covered and remained so throughout.
- The monitoring system was to prove an unrealistic objective given the state of national and local criminal and civil justice statistics, other than using information in reports to the LSGs.
- Progress against the objective on improving witness services was to prove impossible to measure within the lifetime of the pilots. Although the witness survey carried out can provide a baseline for future work, it will be difficult to attribute cause and effect of changes.
- The model would need to be re-cast were it to be used in future VWO pilot locations or roll-out. Notably, many activities may not need repeating in the light of findings, including the context paper and mapping exercises since the LSGs' action plans were remarkably similar ( see chapter 5). An activity plan based on these and the logic model might be useful.
- Despite intentionally robust evaluation methods, it proved difficult to provide hard evidence of VWO pilot effectiveness, so this evaluation needs to rely heavily on agency feedback.
« Previous | Contents | Next »