7. Precision and Accuracy
7.1 Mapping of peat erosion features
Both a qualitative (visual) and quantitative (statistical) accuracy assessment have been carried out. The results of this are presented below.
To present the qualitative accuracy of the classifications, a series of comparison maps are presented. At the scales presented, they demonstrate similarity of pattern and colour. Looking at the full classification in a GIS system will allow for a more detailed qualitative accuracy assessment. The following maps are presented;
- Figure 19. Erosion risk and aerial photography classifications in comparison with the 25 cm aerial photography
- Figure 20. Earth Observation classification in comparison with the ASTER data
- Figure 21. SPOT and aerial photography data with related classifications.
- Figure 22. Gully field sampling and comparison with gully classification at the Earth Observation Level
Figure 19. Erosion risk and aerial photography classifications in comparison with the 25 cm aerial photography
Figure 20. Earth Observation classification in comparison with the ASTER data
Figure 21. SPOT and aerial photography data with related classifications
Figure 22. Gully field sampling and comparison with gully classification at the Earth Observation Level
In Phase 2, following the successful completion of the field work and production of the Definiens classification, it was possible to establish statistical measures of accuracy.
Background to quantifying accuracy
Within accuracy assessment for a land cover maps there are several layers of uncertainty. The primary layer of uncertainty is found in the field due to ecotones between one cover type and another e.g., where a degraded bog becomes a peat erosion feature is not necessarily a 'hard' line on the ground. In many cases the slump at the end of a peat hag has resulted in clumps of bog vegetation within the bare peat area. Where these are frequent the area is eroding bog, where they are infrequent the area would be considered 'bare peat', but the land in-between has a degree of uncertainty. This uncertainty is mirrored within the fuzzy mapping of the vegetation and peat classes. In areas where the vegetation meets the whole criteria of one class we are very certain about it group, where the criteria are less definite we are less certain of it. A map could therefore be drawn which moves the line on the ground from 'most likely area of bare peat' to 'any likely chance of bare peat'. This uncertainty is also reflected in the various type of bog vegetation especially when the frequency of a cover plant makes the difference between one vegetation type and another. For example intact bog will have a moderate amount of Eriophorum vaginatum with it, but eroding bog is likely to have more. Where the imagery is identifying Eriophorum vaginatum both types of bogs are possible, one being just slightly more likely than another.
A further level of uncertainly is added by the process of capturing information in the field. We will have accurately identified a peat erosion feature, but in collecting the data we may have recorded it up to 5 metres away from its 'actual location' (due to GPS accuracies). Five metres is good when working in an environment like the Monadhliath, but as the imagery is very well georeferenced, this inaccuracy in recording the field data leads to a further source of uncertainly and error.
In order to build an error matrix that takes these issues into account it is necessary to move away from straight boolean classes, as they do not exist on the ground. Instead we need to introduce the concept of plausibility. For the example explained above; on the edge of the eroding bog, a field data collection point that recorded bare peat, eroding bog, or tussocks would have an equal chance of being a 'correct' interpretation of the situation on the ground. Added to this is the scale of mapping issue; for example our minimum mapping unit within the air photo level is a metre; that means any feature of less than a metre is going to be consumed within another class. Within the satellite data, the minimum mapping unit is 5 m; that is anything of less than five square metres will be consumed into a surrounding class.
These two scale issues mean it is incorrect to use point data for accuracy assessment as points in themselves would not exist as a class in their own right. Instead features of the size of the minimum mapping unit must be created, so that classes can be accurately described. Within our field data collection we also have to deal with the accuracy of the field collection devices, therefore any features within a five metre radius of our mapped point is also a 'plausible' class.
When we come to examine the accuracy therefore, we could have several data tables or matrices. One is for where the field class is exactly the same as the map class or is a plausible explanation of the map class. Another table would show where the map class seems to be correct and the field class incorrect or visa versa, or there is a possibility that neither of the data are correct. In order to resolve these disagreements a further data set would be needed. This could be an air photo interpretation exercise.
Once each of the matrices has been resolved, matrix algebra could be used to report the outcome of the error investigation. Within this report we have just reported the basic error with those that agree or are plausible against those that disagree. We have not tried to delineate whether the disagreement arose from the field or map data being incorrect. As the accuracy assessments of these types of maps is very much still a research topic we are aware that this is a simplistic explanation of a more complex truth. The result (and current view of on going research) is that any accuracy tables shown are likely to understate the potential of the map to describe the situation on the ground in a useful way; that is real accuracy is likely to be higher than the ones reported here.
An error matrix allows for the comparison on a category by category basis of the relationship between known reference data ('ground truth'), and the corresponding results from the classification process (Lillesand and Kiefer 2004).
Multiple descriptive measures can be derived from these error matrixes which include overall accuracies, user and producer accuracies and the kappa coefficient. Overall accuracies are computed by dividing the number of correctly classified pixels or objects by the total number of reference pixels or objects. An overall accuracy of 75% when using a direct pass/ fail accuracy approach suggests that the accuracy of the classification was good, but not perfect within confusion existing between bare peat and eroding blanket bog for example. However when bringing a plausible fail logic into the assessment, the accuracy increases to 84%. As each object is composed of representative pixels, there is the implicit assumption that the image is composed of pure pixels. Remotely sensed data is often dominated by mixed pixels that contain more than one class and therefore this is a major problem when obtaining accuracy. This explains the confusion that exists between bare peat and blanket bog - eroding classes.
User accuracy is obtained by dividing the number of correctly classified pixels/ objects in each category by the total number of pixels that were classified in that category. Producer accuracies however are obtained by dividing the number of correctly classified pixels per category by the number of training samples per category (Lillesand and Kiefer 2004). Good producer accuracy indicates that the reference data was correctly mapped, whereas high levels of user accuracy indicate that the land cover map has a good correlation with the reference data.
The classes of rocky heath and blanket bog - eroding, both had high producer and user accuracies which indicates that there is a very good correlation between the reference data, the classification produced and the land cover within the classification area. The class of blanket bog - stable however had much higher producer accuracy in relation to producer accuracy which suggests that although the reference data was correctly mapped, the relation between the reference data and the land cover map was not as good.
The kappa coefficient, describes when the value is closer to one then there is a true agreement regarding the extent to which the percentage correct values of an error matrix are compared to. When it's closer to 0 then it's seen as being 'chance'. As the kappa coefficients achieved in this classification are 0.68 and 0.80, it can be said there is a true agreement and therefore the classification is between 68 and 80% better than one resulting from chance.
Accuracy statistics were calculated for the map from in-situ field data collected by the project team (see Table 3).
Table 3. Classification accuracy
7.2 Evaluation of suitability for purpose
Evaluation of the effectiveness of satellite and aerial data
The purpose of this study is to assess the erosion of organic soils in Scotland using EO. As set out earlier in this report, EO is any system that remotely collects information about the surface of the earth; whether it is space or airborne. To begin with the focus of this work was largely on the spaceborne EO component due to the greater complexities of the pre-processing and interpreting. In practice, equal focus was been placed on both the air and space-derived information.
The scale of working that is required for mapping the erosion features ( e.g. the peat gully 'jump test') means that VHR and HR satellite imagery can contribute to the mapping process (in particular by providing a time series of thematic information) but to get the best possible delineation of peat erosion features currently airborne provides the optimum scale.
Historically this required manual digitisation and interpretation, which is both time consuming and costly. The opportunities presented by applying satellite-derived methods of processing ( e.g. image segmentation and feature extraction) mean that aerial photography can now be processed in an efficient and controllable manor to provide the topographic structures for classification in areas that are often devoid of any features mapped by the Ordnance Survey.
Evaluation of satellite data for classifying peat erosion features
Specific small scale peat erosion features could not be established through the use of EO data such as SPOT and ASTER. These sensors do not possess the spatial resolution to permit the identification of peat gullies. It was however possible to establish the larger areas of bare peat which are present. An example of the classification results achievable through the use of EO data is shown in Figure 23. Potential areas of peat are clearly distinguishable (purple) with the areas of bare peat shown in black.
Evaluation of aerial photography for classifying peat erosion features
The use of aerial photography enabled the smaller scale peat erosion features to be classified (Figure 24). Features such as peat gullies could be classified and to a high level of detail. The use of aerial photography though did not enable the discrimination between areas of bare peat, and dark vegetated areas containing species such as Calluna vulgaris, therefore it was necessary to use the satellite -based EO to initially target areas, within which the finer detailed imagery could be applied.
Figure 23. Peat classification (left) of ASTER (right)
Figure 24. Peat classification (left) of aerial photography (right)
The use of the CIR (colour infrared) aerial photography though did enable this separation between bare peat and heath species such as Calluna Vulgaris. CIR data was tested for an area within the Scottish Borders (White Coomb) within an area containing evidence of peat erosion. A clear separation can been see in Figure 25 whereby peat is classified in yellow, cleared areas in orange, and Calluna Vulgaris in purple.
Figure 25. Definiens airborne infrared classification in the Scottish Borders.
(a) original infrared aerial image
(b) Classified image
Evaluation of Object vs Per Pixel Classification
It was apparent that the classifications produced using an object orientated rule based approach were far more representative of the Monadhliath landscape. They allowed for the use of many external datasets and derived layers/ indices within the classification process which removed the many classification confusions that existed through using the pixel based classification approaches. This was noted by Smith and Fuller (2001) who attributed this confusion to noise in the data, atmospheric effects and natural variations within the land-cover type. These factors will therefore adversely impact upon the spectral information present within each pixel, and ultimately the accuracy of the classification produced.
By incorporating this information, knowledge on the context of the landscape used within the classification process will permit accuracies attainable to be considerably higher. The object based classification produced within Definiens Developer made use of features present within the landscape such as a DTM and MasterMap stream vectors in the classification of specific land-cover classes.
Gaps and opportunities
Phase 1 of the project has been based on the use of multispectral satellite and airborne imagery, combined with GIS data. It is considered that this is a robust basis for operational mapping of peat erosion; however consideration has also been given to the opportunities presented by incorporating;
Figure 26. TerraSAR-X image
- Synthetic Aperture Radar ( SAR) imagery ( e.g. TerraSAR-X, Figure 26) to help characterise and extract information on the structural attributes of the terrain and its land-cover characteristics.
- Radar expertise to help with the processing and development of algorithms for integrating SAR into a classification production chain. SAR processing is inherently different to that of multispectral imagery.
- Unmanned Aerial Vehicles ( UAVs). A UAV is an aircraft designed or adapted to operate with no human pilot on-board which, when combined with some form of payload, avionics, and appropriate (ground-based) infrastructure offer a flexible and relatively low cost EO data source. UAV technologies and wider operational infrastructure are developing rapidly e.g. the miniaturisation of sensors and data storage. Currently the main limitation for routine deployment is the regulation of civil airspace, requiring UAVs to be flown within direct visual range of the 'pilot'. It maybe a few years yet, but in the medium term this is likely to be an important tool in the box.
- Airborne hyperspectral has not been explicitly studied in this project, but existing knowledge of the offering makes it clear that the technology provides opportunities as a source of a large amount of spectral data, but over specific targeted areas e.g. strips of 500 m. The additional information provided by a hyperspectral sensor would be valuable over sites at very high risk for detailed monitoring.
- A systematic acquisition programme of all available SPOT5, ASTER and IRS imagery for Scotland. This acquisition programme (an equivalent is currently underway in Wales) provides a library of information about the surface of the earth in Scotland. Allowing for both a multi-faceted and ongoing and retrospective viewing and analysis.
- All image pre-processing (geometric, ortho, radiometric and topographic corrections) should be undertaken to a high specification. Off-the-shelf services are available from the image suppliers, but consideration should be made of the input data used ( e.g. topographic maps) in these processes. This is of particular importance when subsequently undertaking automated processing ( e.g. multi-stack image segmentation and classification).
- Topographic correction should be performed when undertaking a systematic, operational programme of EO-based mapping in areas of varying terrain.
- Cloud masking should be considered to enable the use of partially cloudy images in the mapping chain.
As the reflectance of vegetation (from EO imagery) varies as a function of, for example, season and the physical environment, the division of the landscape into distinct biogeographical regions that best captured this variation is desirable. It is proposed that a series of biogeographical zones are established, based on knowledge of habitat distributions and environmental influence across Scotland. Between and even within these regions, several habitats could vary in terms of timing of leafing, flowering and senescence but also their overall structure and species composition.
The biogeographical zones, unless they exist already for Scotland, can be produced primarily from national data on soils, elevation and rainfall. They would only need to be broad (estimated at around 50-60 to cover Scotland) to allow for the logical handling of how habitats present in EO and also in the logical sub-division of Definiens image processing tasks.
Assessing peat erosion is inherently time-based i.e. a time series of information is required to be able to assess the change in habitat cover over time. Currently the best available dataset specific to this is the Grieve study. As part of the assessment on this project, the original stereo air photo prints (see Figure 27) were examined and their possible role in monitoring was discussed. The study, originating from 1995, has an information rich dataset in the stereo pairs of black and white aerial photo prints. These were interpreted using a stereoscope and consideration was given to how the information could be used;
- Mapping of peat gullies and bare peat from manual stereo air photo interpretation.
- Scanning, georeferencing and the digital mapping of peat gullies and bare peat from air photo interpretation.
- Targeting areas of change for identification of a time series of air photos.
Figure 27. Comparison of (a & d) air photo prints, digital air photos (b & e), classification (c & f)
(a) 15/6/1989 Stereo Photograph
(b) 2006 Aerial Photography
(c) Bare Peat and Peat Gully classification
(d) 15/6/1989 Stereo Photograph
(e) 2006 Aerial Photography
(f) Bare Peat and Peat Gully classification