The Scottish Government

« Previous | Contents | Next »

Listen

CHAPTER 6 UNDERSTANDING SCRUTINY REPORTS

Summary

There was widespread concern with the level of detail in scrutiny reports and with their accessibility. Participants thought that reports should either be kept very short or that a summary, which would be sufficient for most people, should be produced along with the full report, which could be made available for those who wanted it. The impact of reports was felt to be reduced by abbreviations, jargon and a lack of clarity about how measures were calculated and what they meant. There was no consensus on whether descriptive reports were more useful than reports that are based on ratings or percentages or summary measures. Most people felt that both were useful and that elements of the two were necessary.

Assessments of scrutiny reports

6.1 As discussed in Chapter 3 above, there was little familiarity with inspection reports or other documentation on scrutiny outcomes. In the reconvened groups, therefore, we provided participants with examples of short inspection reports, or excerpts from longer reports, on the relevant service. We gave them a few minutes to look at the reports and then asked them to comment on how informative and user-friendly they found them. Two main concerns emerged from the subsequent discussions: the level of detail and with the accessibility of the reports.

Level of detail

6.2 There was a widespread view that reports should either be kept very short or that a summary of the report, which would be sufficient for most people, should be produced with the full report being available for those who wanted it. There was an assumption that, in most cases, people would want to read the fuller report to get more information on any weaknesses or problems that had been identified, but that they would be satisfied with the summary report when the assessment was entirely positive. Even relatively short reports were seen as being too long. For example, a nine page report on a nursery was felt to be too long by one participant who thought that two pages would have been more useful. Participants also suggested one or two page summary sheets or an assessment of 'ten things that are basic'.

Accessibility

6.3 It was clear from the discussions and the questions asked about the example reports, that their accessibility was reduced by abbreviations, jargon and a lack of clarity about how measures were calculated and what they meant. The following types of comments were fairly typical:

Now that you've explained it, it seems fairly straight forward but if I looked it up on the internet I would probably give up after about two seconds. (User of council services)

That goes over my head 'National Care Standards'.

It's written very much for a business person. You know, as a mother of a two year old, I haven't got a clue. (Parents of children in a nursery)

6.4 It was also evident that some participants misunderstood aspects of the reports. For example, the recent NHS hospital patients discussed the following excerpt:

6.5 On the previous page, the report indicated that 'standard almost met' meant 'minor shortfalls: no major deficiencies and required levels of performance seem achievable without extensive extra activity'.

Looking at that one, it says patients receiving treatment using laser and intense pulse lights, 'standard almost met'. I would have thought something like that would have to be either a 100% standard or not. It's like X-rays and all that, that's a lethal situation to be in.

[…] how can you have a standard almost met?

Is like saying somebody is 'almost pregnant' isn't it! (Recent NHS hospital patients)

6.6 One parent of a child at a nursery had read a line in an inspection report that indicated that improvements were to be made within 6 months of publication of the report. She then commented that one requirement was that stair gates needed to be in place at the top of the stairs, and felt that sort of safety issue should have been addressed much more quickly. In fact, the report indicated that that particular requirement had to be implemented within 24 hours although this was not clearly stated immediately next to the requirement.

6.7 These two examples illustrate the scope for misinterpretation when definitions and explanations are not clearly spelled out on each page or even immediately next to a finding or recommendation.

6.8 There was also a view that the narrative, descriptive elements of reports could sometimes be 'woolly' and written (deliberately, it was sometimes suggested) in a way that required a certain level of sophistication or a degree of familiarity with the subject matter in order to understand the real message. A parent of a secondary school pupil described how he had read a report and thought it had not been critical. He then attended a meeting where someone talked about how bad the report had been, and on re-reading the report with that in mind, realised that it was actually quite critical about certain aspects.

6.9 Similarly, another participant commented:

…you kind of have to read through it three or four times and like highlight things before you pick up where things aren't quite to standard. (Parent of child in a nursery)

6.10 In terms of whether narrative, descriptive reports were preferable to reports based mainly on quantitative measures (ratings or percentages) or summary measures (such as 'excellent' or 'adequate'), the consensus of opinion was that elements of both were required. Summary measures were useful for an 'at a glance' assessment, but some element of description was also needed to provide the context, a feel for the overall judgement of the service provider, and an assessment of less tangible or measurable aspects of service delivery. Verbal descriptions or definitions of any summary measures were also seen as important.

Rating systems

6.11 While it was generally felt that quantitative or summary measures were useful, no preferred rating or scoring system emerged from the discussions. Various suggestions were made: a 'tick' against key criteria to show that a service provider had met minimum standards in particular areas; marks out of ten or marks out of 100 for each area; bronze, silver and gold awards to show what level of service had been reached. However, participants were not too concerned about the particular system that was used, or whether the ratings represented an 'overall' score or a score for particular elements of service delivery. What they felt was more important was:

• clarity about what was checked
• clarity about why a particular rating/score was selected
• regular scrutiny to keep the ratings up-to-date
• a 'strictness' about the standards expected so that they did not become meaningless because they were too easy to achieve
• a recognition that ratings could not tell the whole story
• a recognition that comparisons between service providers in different circumstances, e.g. a large service provider and a small service provider, or an urban service provider and a rural service provider, were problematic.

6.12 In relation to the last point, there was a common view that comparisons between service providers could be helpful. This was not so much to inform choice of service provider but to put the assessment in context and to guide service users on whether their service provider was doing relatively well, or whether it was doing relatively poorly and they should press for improvements. However, participants felt strongly that such comparisons could only be a rough guide and that different circumstances made 'fair' comparisons difficult. There was also a view that 'league tables', based on scrutiny outcomes (or any other measures), were simplistic and could be unfair. It was apparent that many participants had a fair understanding of the potential and limitations of standardised measures.

« Previous | Contents | Next »