« Previous | Contents | Next »
Listen
ANNEX 6 INTRA-OBSERVER, INTER-OBSERVER AND INTER-PANEL VARIATION
1 Checks on intra-observer variation (the consistency of each expert's scoring over time), inter-observer variation (the consistency of experts' scoring within each panel) and inter-panel variation (the consistency of the scoring of the 2 sub-panels) were carried out.
2 Intra-observer variation was examined by sending the same case to each expert on 3 occasions. Comparisons across the 3 time periods identified one expert who scored somewhat inconsistently. A comparison of the final analysis with and without this expert's scores showed that exclusion of this expert made little difference to the results. This expert had given valuable and constructive qualitative assessments and it was therefore considered appropriate to include this expert in the analysis.
3 Inter-observer variation was examined by looking at each expert's score in relation to the average (median) score of the 6 experts for each case. Some differences of assessments and judgements between reviewers had been anticipated. An examination of the spread of the differences for each expert, and between experts, did not identify any marked inconsistencies in any expert's scores. Consequently, no expert was excluded.
4 To investigate inter-panel variation, 6 cases were sent to each sub-panel and panel members' scores for these cases were compared. (The 6 cases were those felt to be more complex, rather than being randomly selected.) Analysis indicated that both panels were scoring to the same standard. Consequently the data for all the cases could be collated together in tables and charts, rather than being reported by sub-panel.
« Previous | Contents | Next »