« Previous | Contents | Next »
Listen
ANNEX 2: METHODOLOGICAL ANNEX
This annex contains full details of the methodology used in this evaluation. The methodology included a combination of primary and secondary research as described below.
A review of web traffic / usage statistics
We conducted a review of web statistics provided by the International Teledemocracy Centre ( ITC) at Napier University for the three months that the pilot project was functioning to the date of the evaluation (Sept 2005 - Aug 2006). Our web expert also reviewed the website in terms of usability and accessibility. We were also provided with a username and password to access the 'private' administrator area of the website in order to allow us to assess the private as well as the public areas. The purpose was to explore the usage of the website and, where possible, compare this with relevant comparator websites.
Content analysis
We conducted a content analysis of the website. Firstly, we counted up the numbers of 'threads' and 'messages' posted on the website for the period September 2005 (when the pilot website was fully operational) through to July 2006 (the time of the evaluation) in order to provide a broad overview of the level of usage of the pilot website. This allowed us to draw conclusions about the pilot website while being funded by Scottish Executive funds and beyond the life of those funds.
Selection of sample
We then reviewed a sample of these messages selected from the pilot website. We selected a sample of messages to be analysed by randomly selecting three threads of messages from each Community Council ( CC) area of the website from three separate months.
- One month was selected from the time the pilot was fully operational and funded (Sept-Nov 2005).
- One month was selected fairly recently (July 2006) in order to assess any change in patterns of usage since the end of funding. For each thread we analysed the first message (or post) and all responses to that post.
- One other random month was selected.
ECOTEC developed an analysis grid, using the evaluation framework which looked at the following aspects of a sample of the messages:
- Author (councillor, members of public)
- Date and time
- Topic and content ( e.g. litter, cleaning up local wood, traffic issues)
- New thread or reply to previous message
- Relevance to local community ( i.e. was it a locally relevant issue)
- Aim of message (to inform, agree, disagree, make request, give advice)
- Evidence of impacts on local democracy (has the message has been listened to, responded to, acted upon and has it led to any change in local democracy)
- Disadvantaged groups (any evidence that disadvantaged or unrepresented groups were using the website)
The full analysis was conducted in Excel and is available upon request. A summary of the findings is included in chapter three.
A desk review of pilot project information including models, secondary data, reports and existing evaluations
We reviewed a set of project documentation provided by the Scottish Executive including project plans, financial statements and progress and final reports. This aimed to explore the project management aspects of the pilot project and assess whether the project achieved its aims and objectives. A full list of the materials reviewed is included in the reference list in Annex Four. We used an Excel pro-forma grid to undertake the primary analysis, and the findings from this were written up into chapter two of this report.
A desk review of external literatures and websites relating to other e-community engagement initiatives
Mike Blakemore, the academic expert in our team, reviewed a selection of websites from across Scotland and the rest of the UK in order to draw comparisons with the pilot website. This aimed to place the pilot project within the wider context of e-democracy initiatives helping to draw conclusions about the possibility of rolling out the pilot project, and is presented in chapter four. The method was to firstly establish an inventory of community-based or engagement focussed websites across Scotland, then select case studies for further analysis.
Inventory of e-democracy websites
The inventory included a series of projects funded by the Office of the Deputy Prime Minister ( ODPM) 38, which reported in 2004 39 as well as an extensive list of case studies provided by The LT Scotland Online Service ( LT Scotland, 2006), funded by the Scottish Executive Education Department ( SEED). This list indicated considerable activity at CC level. We reviewed all the websites in the list to identify those that were dormant, or those that were purely informational and passive.
Further projects were reviewed that aimed to support local participation including community toolkits, which were produced by VOICE40 for English Local Authorities, and are provided free of charge. Villages Online 41 was more of a resource centre that provided structured advice for those building their own websites. The inventory is available in Annex Three.
Selection of case studies
From the detailed inventory we selected a sample of websites with interactive facility to explore in greater depth. ECOTEC contacted a sample of those websites with a short questionnaire, sent by email. To date, very few responses were received, and information was included in this report from those that were received to date.
Two further initiatives were selected as case studies and assessed in more detail using web based information and details provided by project staff:
- The Local Channel, with over 150 CCs in Scotland using its low-cost/free Website provision.
- The 'Your Community Council ( YCC, 2006)' initiative, which was created by the West Lothian Association of Community Councils.
A cost-benefit analysis of the pilot toolkit
Mike Blakemore, the academic expert in our team, reviewed the financial information provided by the Scottish Executive to assess whether the pilot website had delivered value for money. Full details of the approach to this are presented in Annex Four. The financial model used in the pilot project was compared with other financial models available for similar websites with other forms of e-engagement in order to assess whether the project delivered value for money.
In-depth interviews with pilot partners and stakeholders
The purpose of the depth interviews was to explore stakeholders' perceptions of the pilot project and provide detail on each of the key issues in the evaluation framework. We conducted a series of eleven in-depth interviews with project stakeholders using a topic guide agreed in advance with the Scottish Executive. Interviews were conducted with representatives from:
- Napier University's International Teledemocracy Centre ( ITC)
- members of the six CCs (Bannockburn, Cambusbarron, Strathfillan, Thornhill & Blairdrummond, Drymen and Torbrex)
- the Stirling Assembly
- the Association of Community Councils for the Loch Lomond and Trossachs National Park area
- the Association of Scottish Community Councils ( ASCC).
We also interviewed a representative of the Scottish Executive in order to place the pilot project within the wider strategic context. In-depth interviews were tape recorded with permission, written up into detailed notes and analysed during the secondary analysis stage. Anonymous write-ups can be made available on request.
Interviews with the community
We conducted 15 semi-structured interviews with members of the local community who had used the pilot website, as well those who had not, in order to build an understanding of how effective the pilot was in engaging with the local community. We recruited users of the website by placing a notice on the websites asking for volunteers to take part, and by asking the councillors for referrals. We recruited non-users of the website by making contact with other community groups in the areas in order to tap into existing community networks to recruit volunteers. General public respondents were paid £20 as a thank you for taking part. Interviews were conducted by telephone and were written up into an Excel analysis grid.
« Previous | Contents | Next »