The Scottish Government

« Previous | Contents | Next »

Listen

CHAPTER SIX: COST-BENEFIT ANALYSIS OF THE PILOT

Introduction

6.1 ECOTEC was asked to evaluate the cost-benefit of the pilot project. Cost-benefits for this project were very hard to determine as benefits were hard to quantify. There were no formal cost benefit goals set at the start of the project, to the best of our knowledge (Whyte et al, 2006). This chapter reviews the broad range of targets that were identified by the project, looks at expenditure, and then sets the overall budget in context against activity levels on the websites. It draws on some comparator financial models to contextualise the cost-benefits. It is worth noting here that the remit of the pilot project was to investigate the use of ICT to facilitate the renewal of democracy at the community level; by delivering, inter alia, an e-democracy model and associated tools. In any attempted cost-benefit analysis therefore, it is right to distinguish between the investment required to develop an end-product and the cost of acquiring and running that product. We are concerned here with the balance of costs versus benefits of the entire project ( i.e. did the results of the project justify the cost and therefore represent value for money), and not directly with comparisons between the cost of a CC implementing the RLD model/toolkit compared with the cost of implementing an alternative product. However, we have included information on the costs of alternative models in order to explore options for follow-on actions to the pilot project.

Measuring cost-benefits

6.2 In the International Teledemocracy Centre's ( ITC) final report (Whyte et al, 2006) benefits were mainly mentioned in qualitative, community and policy contexts, but there was no information produced about any formal cost-benefit expectations against which the project outcomes were assessed, and from the material provided to us the pilot project was not required to undertake a formal cost benefit.

6.3 The Project Mission Statement (Renewing Local Democracy Working Group) identifies only very generic targets:

The overarching aim of the Renewing Local Democracy Working Group [….] is to renew democracy by enabling people and communities throughout Scotland to have greater influence over factors affecting their lives.

6.4 More detailed goals were listed in the Project User Requirements Document (relevant quantitative targets are extracted and underlined below) (Whyte et al, 2005):

Benefits sought:

The overall success criteria reflect the needs of the Community Councils, the communities they represent and the funders of the project, from the perspectives of the community councillors participating in the project.

The 5 criteria and a summary of the indicators to be used are as follows:-

1b. Used on at least a monthly basis by 3 or more members of each Community Council
2b. Number of councillor-user downloads of documents
3b. Continued growth in 'unique visitors' over 3 months.
4b. Continued growth over three months in: -

i. the number of consultations responses that members of the public have contributed to;
ii. the number of items received on other matters of local concern.

6.5 It was significant that the targets were over a period of three months, rather than for a longer period in which it would have been easier to show significant improvements.. While accepting that impacts must be evaluated against a balance of both quantitative and qualitative methods, there was not a clear section in ITC's final report (Whyte et al, 2006) that systematically noted statistics against these criteria, and in interviews, some stakeholders noted that it was challenging to find measurable outputs that could be reported to the Scottish Executive to demonstrate the impact of the pilot.

What was spent on the project

6.6 The project budget set by the Scottish Executive was £168,569. The outturn from Phase 1 was as follows: £62,998 spent, £63,998 projected, against the budget maxima of £65,000. The Phase 2 budget was projected at £105,446, and the Scottish Executive confirmed that the final spend was £98,327.34, as the full amount for dissemination was not spent.

6.7 Stakeholders reported that the budget was spent broadly according to plan and only some minor deviations were needed at the end of each phase (for example where IT equipment costs had fallen). Most stakeholders felt the budget had been well-planned and well-managed, however some felt that the training aspects of the pilot project (responsibility for which fell to the Community Council partners on the Steering Group), had not been well planned and delivered inadequately.

Comparator financial models

6.8 The two comparator models we draw upon were noted in the earlier chapter: the 'Your Community Council' initiative, with an expenditure of around £6,000, for 37 websites, and the 'Local Channel' model, which offers a zero-cost start-up cost, with full documentation and online advice/training.

6. 9 Again, we acknowledge that these two models were not designed explicitly as participatory democracy websites. However, their low end-user cost, and the fact that they do include participatory activities such as forums, was set against the £161,000 investment in this pilot project, and the low level of activity that was identified in following sections. Clearly, the development or investment costs of the comparators used are not known. Normally such investment would be re-couped through income from roll-out.

Assessment of value for money

6.10 ITC's final report did not identify cost benefits explicitly. We also noted earlier that the broad targets were quite restrictive, focusing on athree-month period, agreed with the project partners and the Scottish Executive. ITC at Napier provided updated web statistics, for the period December 2005 to August 2006. They acknowledged that the August 2006 statistics may be distorted by ECOTEC evaluation activity, so we have looked at the profile to July 2006. Between December 2005 and July 2006 the number of 'visitors' to the site ranged from 1,500 to 3,100 per month. The average time online ranged from 111 to 144 seconds, and on average each visitor viewed around four pages per visit. We note also that there was a considerable increase in activity over the summer 2006 on the Bannockburn site, with many responses relating to 'Housing Stock Transfer'. We note also that some other sites had an increase in material being posted in July and August 2006, but that the number of participatory responses was very few. However in themselves these statistics do not tell much about participatory activity, so we examined activity on the sites using a surrogate measure.

6.11 A surrogate measure of activity was taken on August 20-21. ^st All the websites were visited and website activity noted. On all of the websites there were not clearly identified responses to the comments that would show they have been considered. Therefore, closure was not evident for citizens. ( The details of this process are provided in Annex Four.)

6.12 In terms of levels of usage, a comment on one of the Bannockburn website by 'Den' (a registered user) notes 300 responses. However our analysis by respondent name for that website indicated only 106 responses, and adding this to the consultation responses on other websites ends only with some 120 responses evident.

6.13 It could also be reasonably expected from other experience that the overall 'audience' would be larger than the posted responses; nine people would be passive observers of the responses:

It's an emerging rule of thumb that suggests that if you get a group of 100 people online then one will create content, ten will "interact" with it (commenting or offering improvements) and the other 89 will just view it (Arthur, 2006).

6.14 It would not be sensible at this stage to start dividing the finance by responses to give a cost per participation - for example allowing generously for 1000 participations the cost would be £165 per participation. However, the crucial indicator must be meaningful participation:

We need studies to identify participation mechanisms suited to each phase of the overall process. Does the suitability of a mechanism to a phase of the process also depend on the context of the underlying societal decision?" (Bayley and French, 2005).

6.15 In that context the lack of any detailed demographic information about participants makes it difficult to assess whether marginalised or excluded groups were brought into the democratic sphere. As noted elsewhere, the project partnership chose to focus on the community as a whole, since, traditionally public participation in Community Councils has been low. In this context the RLD Final Evaluation Report provided a rudimentary analysis based on the demographic characteristics of respondents to the Citizen Panel survey who showed an interest in engaging with CCs. However there are emerging demographic classifications for internet usage that may help to benchmark e-inclusion targets, for example, recent research at University College London presents a national classification of the UK Information Society showing the diversity of ICT use throughout society (see table 6.1 below).

Table 6.1 Demographic classification of internet usage

Source: ECOTEC adapted from CASA (2006)

6.16 From our review of the website activity it seems that the activity has declined substantially on most of the websites. If the project was funded to experiment with online e-Democracy and provide valuable learning lessons then it is difficult to argue against the project delivering those aims. That would corroborate with experience of e-Democracy projects elsewhere. In a study of local e-Participation and e-Democracy in the USA, Donald Norris notes that only 22% of projects had a formal planning scenario, and "few involve citizen desires or needs in any such planning (about one in five), and hardly any conduct prospective cost-benefit (4.0 percent) or retrospective impact analyses (3.2 percent)" (Norris, 2006).

6.17 In another review of projects in the USA, Steven Clift noted the following challenges for any project:

"Democratic necessity does not guarantee the use of ICTs based on their demonstrated or potential value. The use of ICTs in democracy does not guarantee their success or a positive impact. Success in one country or government agency does not guarantee its spread nor its sustained use even where clear value is demonstrated. The tenuous nature of democracy requires continuous improvement and sustained enrichment with the newest tools available. Therefore, one needs to articulate the necessity, demonstrate and document success towards desired democratic outcomes, and work deliberately to ensure its spread" (Clift, 2004).

6.18 Lastly, The Scottish Executive, in the review of Internet Access Points in 2004 (Aiken, 2004) also noted that unexpected outcomes need to be assessed:

"In evaluating the impact of public expenditure it is useful to look at:

Deadweight - to what extent would the outcomes and impacts have occurred without the funding;

Displacement - the extent to which the impacts have been additional rather than having been displaced from other projects or interventions; and

Leakage - the extent to which the funding has resulted in impacts to groups and individuals who were not being targeted."

6.19 Several positive impacts on the community were evidenced by stakeholder interviews, including improved awareness of CCs, improved understanding of the CC's roles, improved transparency and local people raising and discussing local issues. In addition, unexpected benefits should also be taken into account, the recruitment of new Community Councillors for example. However, from the comparative assessment of other websites, and levels of activity on the pilot website, we would conclude that much the same activity could occur through low level resource from other sources ('Deadweight'). We cannot say whether 'Displacement', in the context of significant additional activity has occurred. 'Leakage' was also very difficult to assess, since specific e-inclusion targets (targeting disadvantaged groups) were not specified.

Summary

6.20 In summary, when set against the stated goals of the pilot project, a simple financial cost-benefit analysis produces figures which could be easily challenged. Overall, ITC's final report honestly addresses the low levels of use, and critiques the approaches that were taken.

6.21 While some of the initiatives such as 'Your Community Council', and the Local Channel, may not have been active at the time of the project inception, they generally were active at the time of the project funding ending. They may also help to set in context the relatively low level of activity on most of the pilot websites, for example asking whether slow initial take-up eventually creates a critical mass for wider meaningful participation:

Largely, people socialise online with the same people they would otherwise offline or via telephone. But where a community has very high levels of access, with content suited to local uses, then potentially interesting outcomes arise. … Equally, websites which operate nationwide, but which are tailored to help people communicate locally, such as UpMyStreet, UK Villages Online and The Local Channel, may start to affect social capital as internet penetration rises" (Davies, 2005).

6.22 Overall, the information we have gathered from the review of the pilot website indicates that usage was not growing towards a critical mass. In value for money terms the pilot was an ambitious, but high-cost attempt to raise significant growth in participatory democracy.

6.23 The fairly low-levels of overall activity, make it difficult to justify this funding model for the future, particularly when set against the low-cost, and low-risk models now available elsewhere. To an extent, outcomes were predictable and the recommendations produced appear anodyne, albeit with a degree of hindsight. Equally, no technical solution was developed that was significantly better than any that could be sourced easily from elsewhere at modest cost. Indeed, there remain significant technical issues to address before a finished product is available.

6.24 In conclusion, funding high-cost projects like this, that return uneven levels of activity that do not clearly show how local democratic participation is enhanced (and again we must note the lack of any demographics about participants), is a high-risk activity for the Scottish Executive. Any funding may be better focused on encouraging a significant participation in low-cost approaches that will be able to build capacity, and grow organically to embrace participation and debate.

« Previous | Contents | Next »