« Previous | Contents | Next »
Listen
3 Assessment methodology
3.1 Overview
The study required the project team to develop a realistic methodology to assess pilot scheme performance and the cost-effectiveness of rolling out of each scheme to a large number of households. Currently in the UK, monitoring of recycling services tends to be patchy. Effective monitoring and evaluation is often hampered by lack of access to data, available resources, and differences in approach. The collection systems employed in multi-occupancy properties provide additional challenges.
Figure 3-1 shows a schematic of the residual and recyclate collection prior to implementation of the pilot schemes.
Residual collection is typically via on-street communal bins or backcourt wheelie bins. Recyclate collection is via recycling centres and points. Residents will either take their recyclates to a recycling centre or point, or dispose of them in the residual bin. Those who recycle might not participate in the full range of materials accepted by the recycling centres and points, nor recycle all the recyclate material produced by their household. The recycling centres and points are likely to be used by households spread over a larger area than the pilot scheme areas. The recyclate collection is effectively an 'open system'. Therefore, it is difficult to quantify the recyclate diversion rate in the pilot scheme areas prior to the introduction of the schemes.
Figure 3-1 Pre-scheme residual and recyclate collection
Figure 3-2 shows residual and recyclate collection after implementation of the pilot scheme. In this situation the residents have a choice to either:
1. Not participate in recycling at all
2. Recycle using the pilot scheme
3. Continue to use the recycling centres and points
4. Use a mixture of (2) and (3)
5. Continue to use the recycling centres and point for materials not collected in (2)
Figure 3-2 Pilot scheme residual and recyclate collection
As for the pre-pilot situation, the recycling centres and points will be utilised by other households not in the pilot scheme area. In addition, for those schemes that implemented 'on-street' facilities, households outside of the pilot scheme area may use the new recyclate bins.
3.2 Pilot scheme performance
Pilot scheme performance was assessed against the five performance measures. These are summarised in Table 3-1. In addition to these measures a detailed waste analysis was carried out on each pilot scheme to provide assessment of scheme efficiency and householder behavioural change.
Details relating to the operational aspects of each pilot scheme would be valuable to other local authorities that may be considering commencement of similar schemes. During the operation of the pilot scheme, operational staff in each local authority were asked to complete a monthly questionnaire which recorded observations and experience of scheme operation. At the end of the study, a face-to-face interview was also carried out with the responsible managers from each council to review scheme operation.
Details of the data collection and analysis methodology for each performance measure are provided in the following sections.
Table 3-1 Pilot scheme performance measures
Performance Measure | Definition |
|---|
Materials Collected | Recyclate materials collected by the pilot scheme |
|---|
Householder Participation | Estimate of the proportion of households utilising the pilot scheme, based on public survey data and waste analysis data |
|---|
Householder Satisfaction | Estimate of the proportion of households who were satisfied with the pilot scheme, based on public survey data |
|---|
Percentage recycling rate | The amount of recyclate collected as a proportion of the overall household waste stream |
|---|
Diversion Rate | The amount of material (kg/hh/week) diverted by the pilot scheme |
|---|
Recycling scheme cost | Cost per tonne of material diverted |
|---|
3.3 Assessment timescales
Due to delays in commencement of the pilot scheme, and the need to complete the study by March 2006, the pilot schemes were only assessed over a period of 4 - 5 months of operation. This did not allow for the assessment of seasonal impacts on scheme performance, nor any performance improvements resulting from the scheme being operational for a longer period of time.
3.4 Householder participation and satisfaction
The standard definition for recycling scheme participation is the proportion of households putting out a container for recyclate at least once a month ( WRAP, 2003). In this study, five of the seven pilot schemes were communal facilities, either on-street or backcourt. Therefore, it was not practical to measure participation in the standard manner.
Participation, both before and after implementation of the pilot schemes, has been assessed in this study via waste analysis (see section 3.7) and public surveys carried out by SWAG2. Householder satisfaction is based on the percentage of respondents to SWAG surveys who said they were using the scheme and were satisfied with the service provided.
SWAG ran a public awareness campaign in conjunction with the introduction of the introduction of the pilot schemes in each of the seven local authority pilot areas. Two public opinion surveys were also taken: one before the publicity campaign began, and one following the publicity campaign and the introduction of the new recycling schemes. The pre-publicity surveys assessed baseline attitudes to recycling; the post-publicity surveys assessed changes in attitudes and behaviour following the publicity campaign, and attitudes to the new recycling schemes. The results of these surveys have utilised in assessment of the pilot scheme performance. The key reports in this respect are:
- Public Attitudes and Behaviour Towards Recycling Services for Multi-Occupancy Properties (Prior to introduction of multi-occupancy feasibility pilot services) A SWAG Survey, April - July 2005. Final Report; and
- Public Attitudes and Behaviour Towards Recycling Services for multi-occupancy properties A SWAG Survey, October 2005-January 2006. Final Report.
This method of measurement provides an indication of participation in each pilot scheme area. There are obvious potential errors in the householder providing incorrect answers to survey questions.
The publicity campaigns varied between areas but generally involved introductory leaflets, main leaflets, fridge magnets, decals, and follow up leaflets/postcards.
More details on the SWAG survey methodology can be found in the above reference reports.
3.5 Diversion rate
The diversion rate is defined as the amount of material that is recycled instead of going to landfill. It is measured in kilograms per household per unit of time, typically per week or per year (kg/hh/wk or kg/hh/yr).
Each local authority provided a monthly data return on the amount of recyclate collected by the pilot scheme. Data was collected from the first month from when the scheme started until January 2006.
The diversion rate achieved by a pilot scheme was thus calculated as follows:
To undertake the cost effectiveness analysis it was necessary to calculate the amount of recyclate that each scheme would collect over the course of a year. Data from the pilot schemes were collected during the period July - January. However, it is common to see seasonal variations in waste and recyclate arisings. A methodology was derived to estimate the yearly arisings from each pilot scheme that would take into account potential seasonal variations. This was carried out by analysing collection data for an existing recycling scheme which as been operational for a calendar year. Full details of this methodology can be found in Annex 2
3.6 Cost effectiveness analysis
3.6.1 Definitions
Cost effectiveness analysis compares the cost of alternative ways of producing the same or similar outputs. In terms of costs, each pilot scheme collection system has been compared by its net cost per tonne of recyclate material collected.
The net cost of recycling will be equal to the total cost minus the avoided costs of disposal and revenue from sales of recyclate material (see Figure 3-3).
Figure 3-3 Net Cost of Recycling Scheme
Each local authority was requested to provide the following costs:
- Capital and operational expenditure ( CAPEX and OPEX) resulting from the pilot scheme
- Projected CAPEX and OPEX costs of rolling out the pilot scheme collection system to a large number of households
To ensure consistent reporting of costs, the Project Team assigned standard definitions of the items to be included under CAPEX and OPEX headings (see Table 3-2).
The contribution of CAPEX to the net cost of recycling was calculated by apportioning the cost of a capital item over its estimated lifespan. The yearly contribution of a capital item, for example a refuse collection vehicle, to net recycling costs was thus calculated as follows:
The estimated lifespan of various capital items are shown in Annex 3.
The total cost of recycling was defined as the sum of the yearly OPEX and yearly CAPEX contribution, divided by the projected tonnage of recyclate collected during that year.
| Total cost of recycling (£ / year)= | net OPEX per year + CAPEX contribution per year |
| tonnage of recyclate collected per year |
Where:
3.6.2 Pilot Scheme Cost Data
Each local authority was requested to provide monthly OPEX pilot scheme costs for the period July 2005 - March 2006. It should be noted that those given for January - March were projected costings. CAPEX costs incurred at the start of the scheme were also to be provided.
3.6.3 Rollout costs
The number of households to which the pilot schemes were implemented ranged between 168 and 5,000. It was thought likely that the net cost of each scheme would be less if it were rolled out to a greater number of households.
The Project Team liaised with each local authority to develop scheme costings for large-scale rollout. Large-scale rollout was defined as the point at which no further reasonable economies of scale would be realised (e.g. at least one collection route). In the case of some local authorities, costs were based on rollout to all of the suitable multi-occupancy households in the local authority's area.
Net costs per tonne were calculated using the methodology set out in Section 3.6.1. These were based on 2006/07 prices (landfill tax set at £21/tonne).
Table 3-2 Definitions of capital and operational expenditure
Cost Heading | Notes |
|---|
Operational Expenditure ( OPEX) - costs per month / year (£) |
1. Collection Cost | |
Staff | Crews; staff training; overtime; recycling officer / waste manager time |
Container Replacement (including distribution) | Includes on-street bins and bags / bins given to householders and distribution costs |
Transport & Vehicle | |
Vehicle hire / leasing costs | Ignore if vehicle has been purchased (see capital costs). Cost for 100% utilisation |
Vehicle maintenance costs | |
Total fuel costs | |
2. Treatment Costs | |
MRF Gate Fee for recyclates | Gate fee paid by Local Authority to MRF operators (if applicable) |
Disposal of contaminated recyclate | Landfill tax and gate fee. |
MRF Operation Costs | If the MRF is in the ownership of the Local Authority |
3. Additional activity | |
Awareness Raising activities | |
4. Revenue from sale of recyclate | Any income resulting from the sale of recyclates |
5. Avoided costs of disposal | Landfill tax and gate fee |
Capital Expenditure - investment costs required in year 1 (£) |
1. Containers | Includes on-street bins and bags / bins given to householders |
2. Cost of Vehicle | Cost of vehicle if purchased |
3. Container Distribution & installation | |
Distribution Costs | |
Road Markings | If applicable |
Fixings | If applicable |
Signage | In addition to (3) |
3.7 Waste analysis
Waste analysis provides a breakdown of the component materials that are in a given waste stream. This analysis technique was applied in this project principally to assess:
1. Behaviour Change:
Has the scheme affected a change in behaviour with regard to the disposal of the scheme's target recyclates? Specifically, is less target recyclate being disposed of through the residual waste stream (i.e. the domestic waste bin)?
2. Scheme Efficiency
How much of the target recyclate is being collected by the scheme?
3. Collateral Effects - Modal Shift in Target Recyclate Disposal
To what extent has the scheme shifted existing target recyclate collection from one scheme (e.g. recycling centres and points) to another (i.e. the pilot scheme systems)?
It should be noted that the estimation of the latter effect inevitably has to be by indirect inference, since there was no monitoring - for obvious practical and financial reasons - of the behaviour of the pilot and control areas with respect to the recycling centres and points.
Although it provides valuable information, waste analysis is not an exact science; it deals with the outputs of (relatively) unpredictable human beings and is only a snapshot of waste production at any one point in time.
- For practical and financial reasons, it was only possible to undertaken sampling over a period of two weeks before and implementation of each pilot scheme. Ideally, a longer period of sampling would have been useful to assess temporal variability.
- It was not possible to assess seasonal impact on waste arisings and recyclate.
- It should be noted that residual waste is collected from householders in the pilot scheme areas in communal bins. In some areas, for example Edinburgh and Aberdeen, the bins are situated on street rather than in backcourts. Therefore, it is impossible to confirm that waste collected for sampling has come from specific tenement building or the number of households that have contributed to that sample.
A standard methodology was developed and utilised across all seven pilot schemes. This methodology has been based on the SEPA draft guidelines 'Protocol for the Analysis of Municipal Solid Waste'. Reference was also made to guideline produced by Defra (Defra 2004). The methodology utilised is summarised below.
3.7.1 Determining sample size
Determining the number of households to be sampled is a balance between available resources for analysis and the need to obtain statistically robust data. The required sample size will depend upon the expected degree of change and the variance in the data. This requires evaluation of data from existing waste studies. Analysis in the SEPA draft guidelines of data from studies in Wales shows that to determine a practical 90% confidence interval for the mean weight per household per week a minimum sample of 100 households is required. In this project, it was thought prudent to aim to collect samples from 150 households in order to make allowance for the inherent uncertainties associated with waste collection e.g. householder fails to put waste out or waste is collected by RCV Crews. This number of households was thought to generate between a 1 and 1.5 tonnes of waste, which is achievable for a waste analysis team to sort in one working day.
3.7.2 Selection of sample households in each pilot area
The objective of this research project is to evaluate different methods of recyclate collection from a particular housing type. However, it is recognised that waste composition may also be influenced by socio-economic factors. For the purposed of this project households selected for sampling from each pilot area were therefore selected on the basis of:
- Housing Type: are the households selected representative of the pilot area (e.g. tower blocks, traditional tenement buildings).
- Operational Issues: do the selected households form a distinct block (e.g. all lie within one street or area) that would make collection of samples more straightforward?
- Socio-economics: do the households reflect the wider social-economic characteristics of the pilot area?
The socio-economic characteristics of each area where assessed using the ACORN classification system. This classifies households (by postcode) into one of 56 socio-economic types. The ACORN classifications are shown in Annex 4. n some cases, existing waste collection is via communal on-street bins. It is therefore not possible to ascertain the exact source of any waste sampled from that bin. In these cases the waste analysis team have selected bins that they consider most likely to be used by the households selected for sampling. However, without disruption to the household method of waste disposal, it is impossible to remove this error.
3.7.3 Selection of control areas
For each council a suitable sample of control households were selected. These control areas were selected on the basis that they equivalent to the pilot scheme areas in terms of housing type, residual waste collection system, and ACORN classification.
3.7.4 Analysis protocol
A standard Waste Analysis Protocol was developed for the project. A copy of this is enclosed in Annex 5. Each council and/or their waste analysis contractor were provided with a day of training in the procedures detailed in the protocol. Waste Analysis was carried out over two consecutive waste collections.
The protocol makes use of a set of 18 primary material categories, some of which are further divided up into a set of sub-categories ( see Annex 5). The primary categories are the same as presented in the SEPA draft methodology. The use of the SEPA standard conforms with UK best practice and thus the results are likely to be more widely accepted and more widely compared to other studies in the UK.
The sub-categories reflect the materials that are being collected by the pilot schemes. For example category 3 - Card and card packaging - is further divided into Boxes and Packets and Drinks Cartons.
3.8 Operational questionnaires
A copy of the operational questionnaire distributed to operational staff within each council is provided in Annex 6. Local Authorities were requested to have this questionnaire completed once a month between August and December 2005.
« Previous | Contents | Next »