The Scottish Government

« Previous | Contents | Next »

Listen

CHAPTER 9: CONCLUSIONS AND IMPLICATIONS FOR SCRUTINY

9.1 The overall aim of this study was to inform the work of the Scrutiny Review by increasing its understanding of public perspectives of the external scrutiny regime.

9.2 In this context, the "public" encompassed three different sets of stakeholders with interests in scrutiny - users of public services, carers and guardians of those users (who may also be users in their own right) and the members of the public or citizens. The study was designed to explore the perspectives of all three groups.

9.3 More specifically, the study explored:

• awareness, understanding and experience of current scrutiny arrangements
• expectations of the purpose and nature of scrutiny
• views on public involvement in scrutiny
• views on how the current external arrangements might be changed to better reflect service user or citizen interests.

9.4 Participants were more familiar with inspections than most other types of scrutiny and could more easily grasp how inspections might be carried out. The group discussions therefore tended to focus very much on inspections - even where participants referred to 'scrutiny', it was clear that they very often meant 'inspection', and when the researchers tried to move the discussions on to other types of scrutiny, they often reverted back to inspections. This focus on inspections is inevitably reflected in the findings.

9.5 Where there were clear differences between stakeholder groups, these are discussed in the report. However, these were less striking than might have been expected.

• Direct service users and indirect service users: there were few differences between the direct service users and the indirect service users. What differences there were appeared to relate to the fact that the indirect service users were discussing services where the direct service user ( e.g. an older relative in the case of care homes or a young child in the case of day nurseries) was particularly vulnerable, rather than reflecting any differences in perspectives or priorities between the two groups.
• Service users and citizens: there were likewise not that many differences between service users and citizens. The main difference was that citizens placed a higher priority on scrutiny ensuring value for money than service users. However, like users, citizens felt that the main priorities were to ensure minimum standards were met and encourage service improvements. Both thought that the involvement of service users in scrutiny was more useful than the involvement of citizens.

Awareness, understanding and experience of current scrutiny arrangements

9.6 The participants were more familiar with, and more knowledgeable about, inspection and complaints handling than about regulation and audit. This resulted from their greater experience and involvement in these processes, with many participants having been consulted as part of an inspection, used the complaints procedures or known someone else who had done so. Audit was the aspect of scrutiny with which participants were least familiar. Levels of knowledge and awareness of scrutiny methods varied greatly, with relatives of people in care homes and parents of secondary school pupils being among the most knowledgeable and users of council services and citizens (talking about the coastguard service) among the least knowledgeable. There was a general awareness of several scrutiny bodies, in particular, the Care Commission, Her Majesty's Inspectorate of Education ( HMIE), the Health and Safety Executive and the Scottish Public Services Ombudsman ( SPSO).

9.7 However, even among those who were least knowledgeable, there was a general assumption that things were being checked, that they 'must be', even when they did not know how or by whom. People only really acquired more detailed information when they or their relative began to use a particular service. Therefore it seemed that people gain their knowledge through experience, or from to talking to others in a similar position to themselves.

9.8 While some service users wished they had known a bit more about scrutiny, particularly about scrutiny outcomes, when they were choosing a service, in general participants seemed content to get the information they needed ( e.g. to make a complaint) when they needed it. This is supported by the sentiments of the users of council services and the citizens who were talking about services which they did not use regularly or with which they had not encountered problems. These groups were quite happy in the knowledge that these services would be scrutinised and had given little thought to the way in which this was done prior to their involvement in the study. It appears that people do not want or need to be overloaded with information about services in general. What they need is accessible and useful information at the right time. An important time for all services is the point at which users first need them. Certain services, such as care homes and day nurseries could be required to provide prospective users with their most recent inspection report and in an accessible format.

9.9 The surprisingly high level of awareness of the SPSO, given that it was set up in 2002, also has implications for the Review. Its high profile makes it pivotal in referring members of the public to appropriate complaints procedures in cases that it cannot itself investigate and a strong candidate for offering more general information about scrutiny.

Expectations of the purpose and nature of scrutiny

Purpose: what scrutiny should do

9.10 Participants identified four roles for scrutiny, the first two of which are the main priorities:

• ensuring that service delivery meets minimum standards and adheres to statutory requirements
• promoting the continual improvement of services
• ensuring value for money
• providing information to inform choice of service.

9.11 A secondary, but still important, role was to ensure value for money. The distinct contribution of audit to the overall system of scrutiny is implicitly suggested by this concern. Although value for money was more of an issue in relation to what might be termed 'background services' ( e.g. the coastguard service and council services such as roads, refuse collection and parks, where everybody is a service user or potential service user, but where the quality of service provision, unless exceptionally poor, tends to carry less salience in daily life), the relatives of people in care homes were clearly very concerned to discover that care home fees can vary greatly for no apparent reason. That said, it is important to stress that ensuring minimum standards and improvements in quality of service were regarded as the primary roles for scrutiny of all services.

9.12 It was also acknowledged that there was a role for scrutiny in helping inform choice of service provider. However, this was seen by participants as less important than the other roles - partly because in practice 'choice' of service provider is often limited or irrelevant and partly because, where there is a choice, other sources of information on quality of service were seen as more valuable. The dominant view was that people find scrutiny reports useful in the absence of other information that can inform choice. So, for example, it might be useful for people new to an area who cannot rely on 'word of mouth' information from local contacts. Scrutiny information might also be used to back-up or confirm judgements based on other sources of information (typically visits or word of mouth).

What scrutiny should look like

9.13 Participants' views on what an effective scrutiny mechanism should look like followed naturally from what they felt were the main purposes of scrutiny. In order to fulfil the four roles outlined above, a number of features of an ideal scrutiny regime were identified. These can be categorised under who scrutineers should be, what they should do and other views on the system as a whole.

Who scrutineers should be

9.14 Independent and objectiveIndependence from the service provider was seen as a very important quality for scrutineers. Independence enabled scrutineers to be objective about the service and provide an honest assessment of its quality. This was seen as a key advantage of external scrutiny over internal scrutiny: internal scrutineers may be tempted to cover things up but this is less likely with external scrutineers.Independent scrutineers can also act as a 'fresh pair of eyes' and, in contrast to service staff, are better able to take a step back and identify problems and more effective ways of doing things.

9.15 People with expert knowledgeMore specifically, in relation to standard setting but also in relation to scrutiny (participants were mainly referring to inspection), the involvement of independent 'experts' - who are usually professionals with experience of the relevant field but with no connection to the service being scrutinised - was seen as crucial. These experts were generally afforded the lead role in both standard setting and scrutiny teams (although the NHS patients group were more in favour of a lead role for patients and/or members of patient representative organisations). The value of these independent experts was their expert knowledge of the relevant service, their knowledge of best practice, their awareness of statutory requirements and their independence from those responsible for service delivery. It was also felt that they could use their expert knowledge to inform service users and members of the public who were involved in standard setting or in scrutiny teams.

9.16 However, it was recognised that independent experts had their limitations: there was a view that they can sometimes be blinkered as a result of their extensive experience in the field and may not be able to 'think out of the box', they may be somewhat out of touch with the day to day realities of service delivery, and - in most cases - they will not have personal experience of receiving the service themselves. For those reasons, effective scrutiny needs the input of a range of other experts and stakeholders.

9.17 Service users and service providers The importance of independent experts in standard setting and inspection teams has already been discussed. However, it was also seen as important that service users and service providers were involved in standard setting. Service users can provide input based on their needs and expectations. Service providers can provide input based on their knowledge of service delivery and of practical issues and it was felt that their involvement was likely to encourage their commitment to the enforcement of standards.

9.18 In relation to inspection teams, a range of independent experts from different fields was deemed necessary e.g. professionals with experience of fire safety, other health and safety experts, plus education or health or social work professionals as appropriate. Again, there was strong support for service user involvement - assuming careful selection, appropriate training, and guidance from the experts. Less commonly, there was a view that trained or knowledgeable members of the general public who were not service users had a useful role to play. It was felt that service staff should not be members of scrutiny teams because they have a vested interest in the outcome. However, it was also felt that they should have the opportunity to provide evidence to the scrutineers.

9.19 In summary, participants felt that mixed teams of standard setters and scrutineers, involving a range of experts and stakeholders, was the best way to ensure effective scrutiny. However, in some circumstances, e.g. fire safety, participants did not expect scrutiny bodies to physically undertake specialist inspections, but thought they should check that services possessed the relevant up-to-date certificate indicating that they had been inspected. In effect, therefore, they thought some aspects should be scrutinised by other bodies, e.g. the fire service or the environmental health department, rather than what they saw as the 'main' scrutiny body for that service e.g.HMIE in the case of schools. This appears to open up a model of scrutiny by 'issue' as contrasted with an approach to scrutiny based on the nature of the service provided.

9.20 Qualities of a good scrutineer Participants identified a variety of desirable qualities and qualifications that effective scrutineers should possess. These included:

• first-hand experience and/or expert knowledge of the service being scrutinised (at least some members of the team should have this)
• independence from the staff, management and ownership of the service being scrutinised
• an ability to form objective judgements
• training in scrutiny methods ( e.g. inspections).

What scrutineers should do

9.21 Focus on benefiting the service user Service users (both direct and indirect) were seen as the most important beneficiaries of an effective system of external scrutiny and, because they were directly affected by service provision, external scrutiny should focus primarily on their needs. Although service staff and service managers could also benefit ( e.g. through being given advice and support to improve, through protecting frontline staff from being asked to do things they know they should not be doing, and through higher levels of morale stemming from increased service user satisfaction and trust) one consequence of this was thought to be the knock-on benefits for service users: happier staff provide a better service. Something of a virtuous circle was therefore identified whereby external scrutiny increased service user trust and satisfaction, which increased staff morale and motivation, which improved standards of service provision, which in turn led to increased service user satisfaction, and so on.

9.22 Focus on poorer performing service providers and higher risk situationsIt was generally agreed that scrutiny resources e.g. inspections, should focus on poorly performing services or high risk situations. These might be identified from the results of previous inspections, or from 'triggers' such as a change of management, a large number of complaints, unusually high staff turnover, or 'whistle blowing' by staff. That said, a basic level of scrutiny should apply to all services at all times.

9.23 Facilitate the sharing of best practice In order to promote the continual improvement of services, scrutiny should facilitate the sharing of best practice between service providers. This is one of the reasons for involving 'expert' scrutineers, who have experience in the relevant field and are involved in the scrutiny of other service providers.

9.24 Conduct unannounced inspectionsThe importance to participants of unannounced inspections cannot be over-emphasised. There was universal support for this approach, and it was the issue that came up more than any other in discussions and in recommendations. The main reason was the perception that an unannounced inspection was the only way for scrutineers to obtain a true picture of service delivery: with an announced inspection (even if the service provider is given only a few days notice), participants thought problems could be covered up and things could be put in place 'for show'. In other words, they felt it would be easy to 'pull the wool over inspectors' eyes'. They were not at all convinced by the researchers' suggestion that experienced inspectors would be able to see past these attempts to cover up the service's shortcomings. Although "contrived randomness", as Christopher Hood, one of the leading analysts of new public management ( NPM) describes it, has been a technique of control in public administration for some time, this enthusiasm for unannounced inspections is a significant departure from the status-quo.

9.25 When the researchers put forward some other arguments for having announced inspections ( e.g. that they lead to a less hostile, more collaborative relationship between inspectors and service providers; that they give service providers time to collate relevant documentation; and that they allow for a more in-depth and thorough inspection) participants sometimes conceded that there may be a case for having both announced and unannounced inspections - but the latter were still seen as fundamental to an effective system of scrutiny.

9.26 For the same reasons, it was felt that the time of day that inspections were carried out and the frequency of inspections should be unpredictable. More specifically, for services with 24 hour provision such as care homes, it was thought that some inspections should be carried out during the night, when service users may be particularly vulnerable.

9.27 Scrutinise internal processes as well as outcomesThere was some debate about whether scrutineers should focus purely on outcomes for the service user or whether they should also look at the internal processes. The more common view was that they should scrutinise both. It was felt that only by looking at internal processes could scrutineers effectively identify areas where practice could be improved and assess value for money.

9.28 Consider complaints informationComplaints information was seen to play a useful role in the scrutiny of public services. More specifically, there was widespread agreement that services receiving a large number of complaints should be investigated. No participants suggested that a large volume of complaints could be seen in a more positive way e.g. as an indicator of a successful complaints procedure and an indicator of confidence in it. It was felt that it would be useful for scrutineers to have data from any external body that was responsible for handling complaints as that would allow them to place the results of a particular service in context. It was also suggested that inspectors should be examining the content of individual complaints. More commonly, however, it was felt that they should concentrate on whether effective complaints handling procedures were in place - rather than on the content or outcome of individual complaints.

9.29 Scrutinise complaints handling procedures While participants were generally comfortable with current complaints procedures, when prompted, they identified five types of barriers which might prevent or deter service users from making a complaint:

• a lack of knowledge about how to complain
• a lack of ability to complain e.g. being unable to express oneself in writing
• a concern about repercussions of the complaint
• a fear of appearing to 'nag'
• scepticism about whether anything would happen as a result of the complaint.

9.30 Participants provided several suggestions as to how such barriers could be overcome. There was a consensus that, where possible, complaints should be made to the service involved in the first instance and that only if the complaint was seen to be very serious should the complainant go directly to an external body. As noted above, it was generally agreed that scrutineers should examine the effectiveness of complaints handling procedures.

9.31 Participants were introduced to the idea of a single complaints phone line that dealt with all public services in Scotland. Overall, the perceived problems associated with the it outweighed the perceived benefits. Some of the perceived problems were problems associated with centralised phone lines in general such as having to press numerous buttons before reaching the correct person and being 'cut off'. Other potential problems related to the fact that the number would cover all public services in Scotland. This would mean that it was unlikely that staff would have expertise in all areas and also that the lines would be blocked by people with very minor concerns. Perceived positive features of the complaints line were that service users would be in no doubt about to how to complain, there would be no need to put things in writing, and service users would not be passed around several internal departments.

9.32 Produce accessible reports If scrutiny reports (inspection reports in particular) are to be accessible to service users and the general public they need to:

• be kept very short ( i.e. one or two pages) or there should be a summary report, which would be sufficient for most people, and a readily available fuller report for those who want more detail
• be in plain English and avoid abbreviations/jargon with which most service users/members of the public will not be familiar
• be very clear about how any summary measures or quantitative measures are calculated and what they mean (and the explanation needs to be right next to the measure, not in an introduction or an appendix)
• contain both narrative descriptions and summary/quantitative measures. In terms of whether narrative, descriptive reports were preferable to reports based mainly on quantitative measures (ratings or percentages) or summary measures (such as 'excellent' or 'adequate'), the consensus was that elements of both were required. The latter were useful for an 'at a glance' assessment, but some element of description was also needed to provide context, a feel for the overall judgement of a service, and an assessment of less tangible or measurable aspects of service delivery. Verbal descriptions or definitions of any summary measures were also seen as important.

9.33 There was a view that comparisons between service providers could be helpful in putting the assessment in context and guiding service users on whether their service provider was doing relatively well, or whether it was doing relatively poorly and they should press for improvements. However, participants felt strongly that such comparisons could only be a rough guide and that different circumstances made 'fair' comparisons difficult. There was also a view that 'league tables', based on scrutiny outcomes (or any other measures), were simplistic and could be unfair.

9.34 Follow up on requirements and recommendations Scrutiny was seen as pointless unless action was taken to sort out any problems identified. Unannounced inspections should be undertaken to check whether service providers had carried out actions they had claimed were 'in hand' or which had been required or recommended by scrutineers. However, while participants thought that service providers should constantly be striving to raise standards (as discussed above, they felt that one of the main purposes of scrutiny was to improve services) they were in favour of a supportive climate where providers are normally given a reasonable amount of time, advice and support to improve. They identified practical problems with more serious actions such as the imposition of new management or closing down services. The latter, in particular, was seen as a last resort which should happen very rarely if there is an effective scrutiny regime in place.

Views on the system as a whole

9.35 Costs of scrutiny met by the public purse The dominant view was that the costs of external scrutiny should be met from the public purse - variously described as 'the government', 'taxpayers' or 'all of us'. There were two main reasons for this. The first was that it was the only feasible option. The second, equally important reason, was that it was the only fair way - because everyone benefits from the service and from the scrutiny of the service.

9.36 The disadvantages of over-scrutiny While there was universal support among participants for at least some degree of external scrutiny, it was recognised that there were some potential disadvantages. These disadvantages related, in particular, to a notional level of what might be seen as 'over-scrutiny' - although it would be very difficult to identify what this level might be. The main problems were thought to be: the effect on staff time (in particular, time spent preparing for inspections and time spent taking part in inspections), increased bureaucracy, undermining trust in professionals, costs, and private sector providers opting out of provision if it became too 'difficult' .

9.37 A sensible limit to scrutiny There was widespread agreement that an additional level of scrutiny, to scrutinise the scrutineers, was going too far. Participants recognised that there was a potentially infinite number of scrutiny levels. 'Where do you draw the line?', 'how far do you go?' and 'bureaucracy gone mad' were typical comments.

9.38 It was felt that scrutineers should be accountable but that the features already identified as important for effective scrutiny - training, independence, objectivity, consistency, transparency - would be sufficient to achieve this end.

The role of the public in external scrutiny

Level of involvement

9.39 The role of service users and the broader public in external scrutiny can be placed on a spectrum of increasingly active models from mere information provision to representation in scrutiny work. The consensus among most groups was that service user/general public involvement should be towards the more active end of this spectrum and that, for example, service users should be involved in inspection teams.

9.40 Everyone thought that involvement should be at least at the level of consultation with, or gathering evidence from, the service users: no-one thought that simply providing information on scrutiny outcomes was sufficient. It was generally acknowledged that some individual service users may not even be interested in receiving information about scrutiny outcomes, but the opportunity for greater involvement should be there for those who want it.

The benefits of service user involvement

9.41 The benefits of service user involvement identified by participants fell into three categories: independence from those who run the service, understanding of the service and the importance to service users of high quality service delivery. These applied to both direct and indirect service users.

Concerns about public involvement in scrutiny

9.42 Despite the perceived benefits and the general consensus in favour of public involvement, there were still some concerns about it and a certain lack of support for the lay model of involvement. The main concerns mirrored two of the perceived benefits and were about lack of expertise/understanding (which might apply to some service users but applies to a greater extent to the general public) and lack of independence (in the case of service users). There was also a concern that the types of people who would get involved would not be representative.

Barriers to public involvement

9.43 Regardless of the perceived benefits or concerns about public involvement, two main barriers to public involvement in scrutiny were identified: lack of interest and lack of time.

9.44 Lack of interest was perceived as more of an issue in relation to the 'background' services, e.g. the coastguard and council services, where everyone is a service user or potential service user, but the quality of service provision, unless very poor, tends to have less salience in people's lives.

9.45 Lack of time was viewed as major barrier to involvement at all levels: affecting the attention paid to scrutiny reports and the willingness/ability to be involved in scrutiny teams. When discussing the types of people who might become lay assessors, for example, having the time was a key criterion.

Ways to overcome concerns and barriers

9.46 Various suggestions were made about how some of the concerns about public involvement and the barriers to it could be overcome. In terms of involvement in scrutiny teams (mainly as lay assessors), the dominant view was that teams should be mixed, with input from experts (independent from the service provider), service users and, in some cases, members of the public who were not service users. The experts could then provide guidance for the service users/members of the public and, to a certain extent at least, compensate for their lack of knowledge or expertise in particular areas. Appropriate training was also viewed as essential for anyone involved in inspections, as was payment of lay assessors in order to compensate people for loss of earnings.

9.47 A number of suggestions were also made in relation to better publicising scrutiny outcomes and thereby encouraging more service users to read them (and, perhaps, to become more involved in the scrutiny process). Suggestions included readily accessible reports at the service itself, e.g. in a care home or at a day nursery; on the internet (one suggestion was for an independent site comparing outcomes from different councils, similar to house price comparison websites) and in libraries. All of these methods are currently used, at least sometimes, by scrutiny bodies.

Views on how the current external arrangements might be changed to better reflect public/service user interests.

9.48 It is encouraging that participants' views on the purpose and nature of external scrutiny, and their views on the role of the public in it, accord very well with much current practice and thinking among scrutiny experts. More specifically, their comments and suggestions on the following issues were in line with what many scrutiny bodies are already doing:

• what should be scrutinised (adherence to regulations, quality of service, value for money, complaints information and handling processes)
• scrutinising both internal processes and outcomes for the service user
• the qualities of a good scrutineer (first hand experience and/or expert knowledge of the service being scrutinised; independence from the staff, management and ownership of the service being scrutinised; an ability to form objective judgements; training in scrutiny methods)
• a prominent, often lead, role for trained experts
• collecting evidence from service users and service staff
• the powers of scrutineers (helping services to improve, imposing new management, 'naming and shaming' and, as a last resort, closing down services)
• where scrutiny reports should be made available (service provider premises, on the internet, in libraries)
• who should pay for scrutiny (there is little option other than the public)
• the role of the public (at least the level of being consulted in the scrutiny process but preferably some involvement in scrutiny teams).

9.49 There were therefore relatively few innovative suggestions - this is perhaps not surprising, given participants lack of prior awareness and experience of scrutiny arrangements - and few suggestions that are at odds with (some) current practice. The most notable were:

• the shared perception that service users and their relatives should be seen as the most important beneficiaries of an effective system of external scrutiny constitutes a major challenge for scrutiny bodies
• the suggestion that a service should have the opportunity to feedback on the experience of an inspection and on the report. Where inspectors were consistently getting negative feedback or having reports disputed, action would be taken
• producing very short (one or two page) summary inspection reports - it was felt that these would be more accessible and sufficient for most people
• ensuring that reports are written in plain English and avoiding abbreviations/jargon with which most service users/members of the public are not familiar
• being very clear about how any summary measures or quantitative measures are calculated and what they mean (with the explanation appearing right next to the measure, not in an introduction or an appendix)
• in relation to the scrutiny of council services, a more stringent 'test' for independence which would require scrutineers to be from a different local authority area (as opposed to the usual current test of 'not employed within the body being scrutinised')
• there was a view that 'naming and shaming' - publicising negative scrutiny findings e.g. in the press, - would be an effective way of ensuring improvements, either because the service would be shamed into taking action or because the general public would force them to take action. However, the opposing view was that this would have negative consequences by demotivating staff and, in the case of private sector providers, leading them to close down their business. Therefore care is to be taken with this suggestion
• a stronger emphasis on unannounced inspections including inspections at all hours of the day and night for round-the-clock services (as an alternative to that, it will be necessary for scrutiny bodies to attempt to convince a very sceptical public of the advantages of announced inspections)
• inspections triggered by 'alarm bells' such as a change of management, an unusually high volume of staff turnover, an unusually high volume of complaints, and staff reports of incidences of non-compliance or bad practice.

9.50 In conclusion, we think that the views expressed by those who took part in this study have provided the Review with some very useful information on the experiences that different groups of service users and members of the public have had with a number of scrutiny bodies and processes, on their knowledge of different types of scrutiny, and on their views regarding public participation in the scrutiny of public services. We hope that the study will inform the Review about the impact of scrutiny on the public and the recommendations it makes for improving the current arrangements to better reflect the interests and concerns of the public.

« Previous | Contents | Next »