The Scottish Government

« Previous | Contents | Next »

Listen

DEVELOPMENT OF TOOLS TO MEASURE SERVICE USER AND CARER SATISFACTION WITH SINGLE SHARED ASSESSMENT

CHAPTER EIGHT PILOTING THE TOOLKIT

8.1 A pilot or field test exercise for the draft Toolkit was set up from the end of January.

Aims

8.2 The aim was to test how the various elements of the Toolkit performed with service users and carers across a range of personal circumstances and in different areas where differing approaches to single shared assessment might be expected as well as variations in monitoring resources and geographic factors. Specifically we wanted to find out if the 4 methods were likely to

• enable everyone who wishes to give their views

• enable authorities to collect a wide range of views for general and specific monitoring purposes.

8.3 Specifically, the pilot aimed to test the following in each of the main approaches:

The self administered questionnaire

• were the questions understood as intended?

• did they elicit accurate and full answers?

• was the design of the questionnaire easy to follow and inviting to complete?

• was the length acceptable?

• what were preferred alternative formats for receiving the questionnaire (e.g. on tape)?

• what were effective mechanisms for enabling service users and carers who did not speak English as their first language to receive the support needed to complete the questionnaire?

The administered questionnaire

• were the questions understood as intended when administered face to face, with the possibility of giving fuller explanation and prompts?

• did they encourage service users to say what they wanted to say?

• was the length of interviews acceptable to service users?

• was the questionnaire easy to administer face to face?

• did the use of prompts (linked to the Service Users' Guidelines) enable people to give fuller answers?

The core questionnaire or guided discussion

• were the questions understood as intended?

• how effective were the communication methods chosen?

• did the guidance in the Toolkit assist communication with the service user?

• what resources are available to service users (e.g. someone who best understands the user's unique communication mode)?

• did all or any of these factors combine to enable the recording of service users' views?

The carers' questionnaire

• the same tests as the questionnaire when self administered or by interview, as above

• effective ways of introducing the questionnaire to carers, given relationship sensitivities where a family member or friend was also acting as an unpaid carer, and possible lack of recognition of this role

8.4 In addition the pilot aimed to explore the following issues which would impact on future use of the Toolkit:

• the effect of giving people a choice between a self administered questionnaire and an interview

• the use of interviewers who could present themselves as independent and whose interview skills came from a range of backgrounds including trained laypeople

• the use of a range of interviewers (for people who might present communication challenges) similar to those Joint Future Partnerships might be likely to commission to carry out evaluations.

8.5 Other critical issues identified earlier in the research literature, the survey and the development phase were also taken account of in the pilot strategy. These were:

• people's difficulty in identifying their experience of assessment

• reconciling a standards and measurement approach to monitoring with the individuality of experience and expectations (as identified in SPRU Outcome Programmes, see Literature Review above)

Size and sampling

8.6 The plan agreed with the Steering Group was to pilot the materials with a total of 48 service users plus 20 carers. 50% of service users should be over 65 whilst the remaining 50% should ideally be representative of different care groups. People from minority ethnic communities should be included across the total sample.

8.7 It was decided that the sample should include 24 service users who had filled in self-administered questionnaires. A key element in validating the self administered questionnaire in the pilot would be for the Project Team to conduct follow up "cognitive" interviews. This type of interview focuses on checking what interviewees understand by the questions within the questionnaire; and finding out how easy or difficult questions were to answer. Within a cognitive interview, other questions elicit the interviewee's feelings about other features of the questionnaire: e.g. its layout; whether instructions, information and introduction to sections had been read and understood. It is generally well established (Qureshi and Rowlands, 2003) that understanding of the questions and design of the questionnaire were the elements that required the widest possible testing within time and resources available.

8.8 The rest of the sample (24 service users) was to be split evenly between interviews and Guided Discussions conducted by local people outside the Project Team. This allocation between local and Project Team resources was seen as most likely to produce sound test results, as well as spreading the burden of work in a limited time.

Location of pilot

8.9 The pilot was originally planned to encompass 4 Joint Future Partnerships (JFPs). It was considered that this number would limit the burden for participating authorities whilst also producing sounder results by introducing varied local circumstances. The invited authorities would have indicated both willingness to participate and confidence that they would be carrying out single shared assessments across several client groups by March/April 2004. At the time of giving this information in the September 2003 survey, only 5 authorities were certain, a further 8 said they probably would meet this latter criterion.

8.10 It was further decided to aim for participating authorities which were within easy reach of each other, to reduce time taken in briefing and feed-back activities.

Interviewers

8.11 Time constraints also affected the approach to selecting interviewers who would be independent of the community care authorities, have the necessary skills, and who would be coming to the task with the objectivity of no involvement in developing the survey materials. It was decided to seek assistance from appropriate local organisations which had shown their interviewing capability in the survey, and to negotiate with authorities on the basis of agreeing interviewers from these sources or such other sources as were acceptable to them and fitted with the pilot criteria. Briefing and/or training would be provided by the Project Team. A small budget was allocated to pay interviewer expenses and fees when appropriate.

Obtaining consent from service users and carers

8.12 Enlisting the co-operation of service users and carers in the pilot presented a major challenge, involving issues of informed consent and how to introduce the topic in a manner which was linked to their assessment experience but independent of it. To the known difficulty of lack of understanding of the assessment process, there would be an added complication in explaining that this exercise was a pilot of an evaluation of their assessment.

8.13 In an attempt to minimize delays in the process of gaining consent, it was decided that community care staff carrying out single shared assessments should be asked to explain the pilot exercise at the end of their assessment meeting with service users (and in some cases carers), and seek to obtain their informed consent to take part. They were to reassure people that their decision would have no effect on services they might receive. On balance, for the limited purposes of piloting the materials, the advantages of establishing the link with assessment, quickly, were considered to outweigh the obvious disadvantages of this method not being highly desirable in real evaluation exercises.

Implementation

8.14 5 Joint Future Partnerships were approached by telephone and email at the beginning of February. They were given introductory information which

• referred them to their response to the survey and any other earlier expressions of interest by them

• updated them on the project's work

• outlined the aims of the pilot, using definitions of single shared assessment recently obtained from the Joint Future Unit

• detailed what participation would involve them in doing.

8.15 By late February 2 Partnerships had agreed to participate, the other 3 eventually declined. Further enquiries were made to other JFPs but were unproductive. Reasons given for declining were that Partnerships did not feel sufficiently advanced in implementing Single Shared Assessment or that they were currently participating in other initiatives (e.g. SSA-IoRN). The former reservation was also expressed by JFPs who did participate, one of whom said "this exercise is premature to capture the outcomes of all the hard work that is going into SSA and the Joint Future agenda."

8.16 Following the Practitioners Meeting on 24 February, 2 other JFPs were enlisted. One of them brought in 2 neighbouring Partnerships with which they worked closely on an NHS area basis. The other JFP worked with a local organisation which provided advocacy and participation development (also represented at the Practitioners Meeting) and was interested in carrying out interviewing.

8.17 4 introductory meetings were set up in March, 3 with individual JFPs and one with 2 out of the 3 Partnerships which worked together. The third, more remote, JFP in this group was unable to attend the meeting but agreed to participate through their Learning Disabilities team, on the basis of telephone and email briefings. Between them the participating Partnerships provided a good representation of types of geographic area and approaches to SSA.

8.18 The initial briefing meetings were set up in consultation with the lead JFP officers in each of the 4 Partnerships/Partnership group. A common set of briefing papers was prepared for all of them, with some customising for individual authorities according to how they wished to negotiate engagement and what they had agreed to do. For example, one Social Work Department had a research access process involving an application form channelled via a research officer, and the briefing meeting was held with this officer and the JFP lead officer.

8.19 In all other Partnerships/Partnership group the JFP lead officer brought together community care team leaders and potential interviewers to be briefed by 2 members of the Infusion team. All had previously received information about the project and outline information about the requested involvement of assessment staff and role of independent interviewers. At the meetings detailed information was provided, general briefing papers and papers specific to assessment staff and to interviewers were distributed and talked through and further explanations given in response to questions.

8.20 The involvement of assessors was mainly readily accepted by staff at the meetings, although a few expressed reservations about assessors being able to add another and alien task to what might have been a long and stressful meeting with a service user. However, a key device intended to help people to associate the research visit or questionnaire with their assessment was for both to be accompanied by a photo of the assessor. This device turned out to be more problematical, as several people believed this would be unacceptable to assessors, or at best their agreement could not be relied on.

8.21 The approach to recruiting interviewers was different across all 4 participating groups, with time constraints being the major factor in decisions. One decided it was not possible to be involved with the interviewing part of the project at all, using either the full questionnaire or the guided discussion. One, having tried unsuccessfully to involve voluntary organisations with relevant interviewing experience, brought in social work students. Another preferred not to involve voluntary organisations and recruited a retired community nurse and members of the Council Quality Assurance department. The fourth as mentioned above had an active association with a local voluntary organisation who produced people with relevant experience for interviewing across the planned sample of people and communication modes.

8.22 It was acknowledged that there might be varying degrees of actual and perceived independence amongst this group of interviewers, although these could be addressed in further briefing and training.

8.23 Some groups expressed reservations about involving Health partners in the short time available to them. No NHS personnel attended the briefing meetings.

8.24 Agreement was reached at the meetings:

1. that staff carrying out completed (or nearly completed) single shared assessment over a 2 (or if necessary 3) week period would seek service users' and carers' (where present at meetings) consent to take part in the pilot, using forms and information as supplied to assist people to decide

2. they would inform Infusion Co-operative of all people who gave consent so that Infusion could select the required sample across all participating JFPs

3. on the method of assessment staff contacting Infusion, to ensure speed, client confidentiality, and transfer of necessary information for purposes of sampling and further contact by the research team with the service user, and carer when appropriate

4. on target quotas for each Partnership ranging from 4 of each of the 3 questioning modes for service users to 12 self administered questionnaires for the Partnership which did not wish to undertake the other 2 modes

5. that the approach to carers should be with the agreement of the service user and that they should follow the service user's preference for self administered questionnaire or interview, although carers could opt to complete a questionnaire or be interviewed separately at the point of contact with an interviewer

6. that all interviewers would act as members of the independent Infusion research team, and would present ID cards to clients stating this

7. on liaison between Infusion and interviewers, and between interviewers and assessors to obtain any further information needed to facilitate communication with interviewees

8. on further briefing and/or training for interviewers from Infusion

9. that where interviewers encountered service users with very serious concerns about their assessment, they would invite the service user to use a form to communicate this directly with a senior officer as nominated by each JFP

10. on arrangements for feedback on the research

8.25 All JFP lead officers were confident they could fulfil their target quotas on the basis of a rough estimate that over twice the number of consents would be needed to produce the required sample. Some doubt was expressed about meeting quotas for Guided Discussions.

8.26 Because of these doubts and the withdrawal of one JFP from working with the Guided Discussion method, it was decided to approach another Learning Disability Partnership which had contributed to the September 2003 survey. They were very confident that they could assist and cooperated at short notice and in a very tight timeframe, receiving briefings by telephone and email. Interviews were arranged to be carried out by a learning disabilities professional engaged by Infusion and by a resource worker attached to the Learning Disability team who was not known to the interviewees.

8.27 The other JFP which had offered to seek access to people with learning disabilities chose to use a member of the Council's quality assurance staff who had carried out a recent survey of people with learning disabilities' views on services.

8.28 The withdrawal of the JFP from offering a choice of survey mode presented an opportunity for testing how people responded if they were only offered a self administered questionnaire.

8.29 Documents were further customised to meet individual JFP's preferred ways of working and the different ways in which they believed service users and carers in their area understood "Single Shared Assessment".

8.30 Most interviewers attended one of 4 local briefing sessions held by Infusion, and the 3 who were unable to attend were briefed by telephone after they had studied the documentation.

Findings

8.31 By the end of the period for obtaining consents to take part in the research a very small number had been received. Meetings were carried out with all people giving consent as below:

Table 8.1 Breakdown of participating service users and carers by method of enquiry

8.32 The reasons for the low number of people involved in the pilot are discussed below on p57-8.

Effectiveness of tools:

Self administered questionnaire, interviews based questionnaire,

carers' questionnaire and interviews based on it

8.33 The small number of interviews and returned questionnaires precluded any definitive analysis at this point of how these tools had performed. However, indicative findings regarding interviews are given below.

8.34 A particular difficulty is the absence of self administered questionnaires and, consequently, of cognitive interviews. This was planned as the principal means of checking on the clarity of the questions and how they were understood as self administered and as administered in interview.

Service user interviews

8.35 7 service users (5 women and 2 men) were interviewed using the questionnaire as the interview schedule: this was 5 short of the target of 12 interviews. 6 of the people had physical disabilities whilst 4 had sensory impairments. 2 of the people (both of them women) were under 65.

8.36 Interview times ranged from 40 minutes to 1 hour 30 minutes. All except one person found the time acceptable. One person found an hour long interview too long.

8.37 2 people did not have difficulty in understanding the questions. 4 people found some of the questions repetitive.

8.38 2 people felt they could have completed the questionnaire themselves with some help. 2 others liked company, so preferred an interview: it is not clear whether they would have completed a questionnaire on their own if requested. One person said they could not have completed the questionnaire without it being administered as an interview.

Carer interviews and questionnaires

8.39 5 carers were interviewed; one chose a self administered questionnaire following a Guided Discussion for the person they cared for; and one chose a questionnaire as did the person they cared for. This total of 7 was 13 short of the target of obtaining the views of 20 carers.

8.40 Most carers understood the questions as intended. In 2 of the interviews explanations had to be given of SSA and Carers Assessments. The carer who completed the questionnaire along with the service user may not have been entirely clear about these terms when completing the questionnaire.

8.41 In almost all cases, the questions elicited full answers which also appeared to be fully relevant. Some answers were critical: e.g.

• "call from Social Work informed me of the meeting so was prepared, but was not given prior notice of assessment, so was not prepared for that"

• "more time [needed] to discuss things, felt rushed"

• "[carer] wanted to get it over and done with- lead in period made me anxious"

8.42 The yes/no/please comment format appeared to assist.

8.43 All carers said they found the questions easy to answer

8.44 The 2 people who completed the self administered questionnaire said they found it straightforward.

8.45 All carers said that they found the length of the interview or the questionnaire acceptable.

Interviews based on Guided Discussion schedule

8.46 The 8 Guided Discussions conducted were 4 short of the target of 12. 2 unsuccessful attempts at Guided Discussions also produced useful information. It was not possible to select people presenting a wide range of communication challenges and most people who were interviewed did not present unusual challenges. However feedback on the interviews that did take place does allow some limited findings to be made.

8.47 The sample included 3 people over 65, one with memory problems and sight impairment; one with severe dementia, whose carer had informed the interviewer that it would not be possible for her to communicate with; and one with mental health issues. The other 5 people were all under 65: all of them were described by assessors as having learning disabilities. 2 of the older people had carers present and one of the younger people had a carer present, all carers having been present during assessments.

8.48 The Guided Discussions were carried out by -

Advocacy worker from local organisation (2)

Learning disabilities professional (1)

Resource worker (3)

Research officer (1)

Public information and resource worker (1).

8.49 The consents to Guided Discussions all came from pilot areas which had been confident that their implementation of SSA with relevant service user groups was well established. This included the 2 Learning Disability Partnerships which had come in at a late stage of the pilot and where there had been necessary adjustments to the standard approach of engaging with assessment staff and of assessment staff obtaining consent. In both areas an individual care manager who had carried out assessments was directly involved in finding people to give consent. In one area which had a particularly tight timeframe all people who took part had apparently good relationships with the assessor. In the other area one of the assessments had not been considered satisfactory - by the carer.

Engaging service users: what helped and hindered

8.50 All interviewers had information about interviewees from assessors. This included information about communication needs, about carer or other support, and about when and what assessment had been carried out with them. All interviewers said that this was essential. One said "It helped me to prepare for the interview and helped when carrying it out". The above comments applied equally to the guided discussions and the interviews.

Guided Discussion

8.51 In the case of 3 people with learning disabilities and one person with mental health issues the assessor introduced the interviewer because no carer or support worker present at the assessment was available to help the service user link the interview with their assessment. The interviewers involved reported that this worked well. In one case the assessor stayed for most of the interview, which the interviewer reported did not appear to bias the service user's responses but did give the opportunity to observe the relationship between the assessor and service user.

8.52 The other 2 interviewers who had carers present during interviews found this an essential link to orient the person to the purpose of the visit. One interviewer who knew a carer had been present at assessment but was not available for the interview believed that the service user was less able to engage without this link and support.

8.53 A blank SSA form was found to be an effective aid to engaging interviewees in the topic. Interviewers who were not introduced by the assessor found it was helpful to refer to the assessor by name. No photographs of assessors had been produced for any of the interviews.

8.54 3 interviewers had to administer consent forms at the start of interviewing people with learning disabilities, as it had not been possible to do so prior to their visits. 2 of the interviewers found this got the interview off to a bad start, causing confusion and taking up a lot of time, for one where carers were also present and one where the assessor was present. One interviewer commented: "I think the forms could be confusing. .. . it was really difficult to know his real understanding and I gathered from [assessor] that he is inclined to agree with things rather than say 'no' and he thinks that 'people with forms know everything'".

8.55 An interviewer in another area also had to administer the consent form, but the service user and his parent/carer had already had the explanatory leaflet and had agreed to participate when requested by the assessor. There had been earlier contact with the interviewer to arrange the interview. Both service user and carer understood the purpose of the interview by the time the form was presented and it was "easy" to go through it at the start of the interview.

8.56 In one interview it had been relatively easy to engage with the service user but impossible to discuss his experience of assessment because it had taken place a year ago. Length of time since the assessment was one of the impediments of another interview with a "naturally shy" person, who also found the conversation with a stranger inhibiting. All other interviewees had been assessed over a recent period.

8.57 In one interview, the interviewer recognized that the service user was quite anxious; the interview was taking place at the time when she expected to go home from her day centre. The interviewer decided to abandon the interview as this seemed to be evidence of withdrawal of consent. Because of the time constraints in arranging this interview, the interviewer had not had much prior information about the service user. She had seen the person's Essential Lifestyle Plan, which had been central to the assessment and thought it would have been very helpful in conducting the interview.

8.58 Another interview could not be carried out with the service user because the parent/carer refused to allow access. It appeared likely that this had also happened in the single shared assessment as the carer had taken away the form to complete following a "review meeting" with an assessor and the service user's key worker. The carer made it clear that she had not felt at ease with the assessor but had made sure she had her say.

How the toolkit performed

Service user Interviews

8.59 Tips in the toolkit and in briefing meetings helped interviewers to establish the purpose of their visit with the service users and to retain the focus of the interview. This element of the interview sometimes took a significant amount of time. One factor in this may have been that either assessors had not used the word "assessment " when talking to service users about the process they were going through; or the description of assessment by the assessor had not registered with service users. This then left interviewers with an added difficulty of establishing with service users that they were asking users about a process which some of them did not readily recognize. This difficulty was also acknowledged by practitioners in their workshops.

8.60 Some service users found some of the questions repetitive.

8.61 Some interviewers commented about their concern that the person may have been giving answers they [the person] thought the interviewer wanted.

Carer interviews

8.62 All interviewers indicated that interviews based on the carers' questionnaire were easy to facilitate and provided information readily.

8.63 There was some indication that a number of carers may prefer an interview even though they may be able to complete a self administered questionnaire.

Guided Discussion

8.64 All interviewers reported that the Guided Discussion schedule enabled them to obtain good quality information from service users.

• "It provided a good framework to the discussion".

• "The interview schedule reinforced the importance of planning prior to interview so that communication was as clear and precise as possible."

• "By starting out asking broader, more open questions, I then used the guided discussion questions to ask the service user to contribute his thoughts on SSA and how he found the experience. After a yes or no answer it was quite simple to open up the question for further input through general conversation."

8.65 In one interview the service user clearly understood and gave short answers to the questions, which were then amplified by his parent/carer.

8.66 Interviewers had not found they had to make much use of other Toolkit resources for the people they knew they were going to interview. One interviewer commented: "I think if someone has particular communication requirements it is preferable to have another person present to help with interpretation. The Toolkit was useful in describing the available methods and how to obtain more help".

8.67 Another said that she had found helpful prompts in the Toolkit "about assessor's name, when visits had taken place, if carer was present."

8.68 One added that "the Toolkit helped to explain the reason for the interview"

8.69 Most of the interviewers said that information from the assessor and carers and prompts from them during the interview gave them most help in communicating effectively. One noted

"The use of reality orientation to time and place was also important to get

feedback about the assessment process that had taken place"

8.70 A blank SSA form was also found to be useful in re-focusing service users during the interview - where the form had been used overtly during their assessment.

8.71 All interviews were under an hour, the 2 shortest being 20 minutes. One interviewee found this too short so the interviewer went over his answers with him.

Experience of interviewers

8.72 Interviewers were asked what past experience had helped them in carrying out the interviews. In all cases this had been working with the service user group, mostly in the course of their everyday work, and in one case carrying out a similar research exercise. All but one interviewer had presented as independent of social work and health services. One interviewer had been introduced to 3 interviewees with learning disabilities as a member of social work staff. This did not appear to affect how people responded to him compared with how people responded to interviewers who presented as independent.

8.73 Interviewers using the questionnaire as a schedule experienced some difficulty in using it flexibly. This may reflect the relatively short time they had to assimilate the approach they were asked to adopt and the small number of opportunities they had to practice using the toolkit (the maximum number of service users interviewed by any one interviewer was 3).

Conclusions and possible adjustments to the Toolkit

8.74 From this limited exploration of setting up, monitoring and testing the Toolkit the following conclusions may be reached:

• The co-operation of assessors with people carrying out monitoring is important in enabling the latter to carry out their task effectively

• More consideration is needed of how to involve assessors so that sampling is not biased towards people who feel well served by them, but the benefits of assessors' involvement are maximised where possible

• The Guided Discussion questions and approach are effective in enabling people with moderate learning disabilities and people with severe dementia to give good quality information on the main areas of enquiry of the guided discussion (which equate with the general questions in the full questionnaire)

• It is more difficult for evaluators to establish the purpose of an evaluation visit where an understanding about the assessment process has not been established with service users

• This can be achieved by interviewers who feel confident of working with the service user group

• Interviewers need better guidance in the way they use the questionnaire as an interview schedule

• The Toolkit background information helps interviewers to understand and prepare for their task

• Individualised information from assessors and carers is essential for preparing for discussions where there is a significant degree of communication impairment

• The presence of a carer or other supporter contributes enormously to the effectiveness of the monitoring interview.

Setting up monitoring exercises

8.75 A clear lesson from the pilot is that very careful consideration has to be given before adding extra burdens associated with monitoring to the assessment task . When the tiny number of consents was confirmed JFP lead officers and assessors were asked to suggest reasons. Helpful responses were received from all main pilot areas. The difficulty had not been service users' unwillingness to take part but staff difficulties in making the request for participation. These difficulties were:

• "Team leaders felt they had insufficient time themselves to grasp the scope or scale of the request or to discuss its implications" (team leaders had not attended briefing in this area, although similar comments came from other areas where they had)

• Team leaders said timescales were too short to prepare and brief staff properly. As a result some assessors were not involved at all. Others had to absorb quite extensive information from the documentation without attending a briefing session and did not have time to do so properly

• Many other requests were being made of staff at the time: one area said that "staff are beginning to implement a new assessment tool and system, and considerable requests were being made for statistical information for the end of the year"

• The request was made over a holiday period when people felt even more pressurised than usual when at work

• People who have SSAs are likely to have complex needs and are tired and feeling frail by the end of assessment meetings, not in a state to have something extra and complicated explained to them

• For all of these reasons many team leaders did not feel they could make the request, others that they could not insist on compliance

• In one area many of the assessments carried out within the pilot period were "flare ups of existing cases" rather than new SSAs

• In the area which did secure most consents, people who were approached were "simpler cases"

8.76 In following up the limited consents that were produced, the benefits became clear of direct work with individual assessment staff who appeared to have some commitment to the aims of the project. Through direct working, the research team gained a greater understanding of the local assessment process, and it appeared that assessment staff also developed a better understanding of the monitoring approach.

8.77 Another lesson from the pilot is that when setting up a new approach to monitoring a new way of working, the time scale must be realistic in order to allow players necessary to its success to sign up to it.

8.78 In developing local monitoring schemes it will not be necessary nor desirable to involve assessors to the extent to which this project sought to involve them (largely because of the time factor). However the pilot has shown albeit in a limited way, how assessors could and should be involved, and that this is likely to vary with service users groups and the purpose of monitoring.

8.79 The pilot has also indicated that other groups who need to be involved in setting up local monitoring schemes are service users' and carers' organisations and groups, including advocacy organisations in the broadest sense. The aims of involving them should include

• Endorsing and/or adjusting the guidelines to local use

• Creating wider understanding amongst service users and carers of the SSA changes and the importance of service user and carer involvement in monitoring them

• Developing their capacity to assist with the monitoring task, as interviewers.

ANALYSIS OF PILOT SERVICE USERS' SATISFACTION WITH THEIR SINGLE SHARED ASSESSMENT

8.80 This worked example is an analysis of the interviews and guided discussions carried out with service users.

8.81 Responses to interviews carried out with 9 people living in four pilot areas have been brought together for the purposes of the example.

8.82 The Toolkit methods were developed to elicit reliable person centred information about service users' and carers' satisfaction with their SSA for Joint Future Partnerships. Evaluation questions were developed with service users and carers with a wide variety of perspectives and personal circumstances and were designed:

• to facilitate the inclusion of people who are often not included in evaluation exercises because of language, cultural communication and cognitive issues

• to elicit information about satisfaction from a) each individual respondent's perspective and b) from the perspective of service users and carers more widely.

8.83 The premise for analysis of the responses is that the evaluation questions reflect good practice from service users' perspective and that

• answers demonstrating that the guidelines have been followed

• and positive answers to questions about whether respondents feel that their views, hopes and wishes have been central to the assessment

are primary indicators of satisfaction.

8.84 Initial pilot testing revealed difficulties with a small number of the questions (3) in the long questionnaire/interview schedule; these questions are asterisked and will need to be revised before further piloting and/or dissemination of the Toolkit. However responses to these questions have been included - for illustrative purposes.

8.85 Whilst numerical analysis of indicators shows overall results in relation to satisfaction, a more detailed analysis of individual comments is essential to gain full value from the information gathered and to fully reflect service users' views.

8.86 The guided discussion and interview questions have been paraphrased in the results tables below for brevity.

8.87 The carers' questionnaire/interview results can be analysed using the same model.

8.88 It should be noted to that this is a preliminary approach to analysing and presenting results from interviews/discussions with service users and carers. Further developments to be tested with larger pilot numbers might include: totalling the number of respondents who gave all positive answers; the number who had specific criticisms, etc.

8.89 A second pilot would also work with pilot sites on how the analysis might be used.

Sample

8.90 6 Guided Discussions with service users

3 Interviews ( using questionnaire) with service users

Service Users' satisfaction with SSA using Guided Discussion

8.91 Total: 6 respondents

Note: 1 respondent had significant memory impairment and could not remember their SSA and was able to answer question 4 only

Table 8.2 Service users' responses to Guided Discussion questions

** question to be revised/omitted

Overall results

8.92 Overall, most respondents gave positive answers about the explanation they had received and about how the assessment had been carried out. There had been changes as a result of the SSA in the services being provided for 4 people but not for 2 people; however from other responses we know that of these 2 people one person was happy with the services s/he was receiving and the other said there had been no change for the better or for the worse. ( ** NB it may be necessary to omit this question where the evaluation interview takes place very soon after the assessment meeting(s) has ended, but use it in follow up enquiries related to target timescales for delivery of services).

Additional comments, suggestions, learning points

8.93 One respondent said that s/he always finds having to answer questions intimidating and easily feels 'stupid' and at a disadvantage; s/he had found the assessor understanding and supportive and a positive relationship had developed, however the number of questions still seemed too long for him/her.

8.94 One respondent asked that his/her carer were always informed of any visits being made to self and that both s/he and carer were kept up to date with all developments.

8.95 Three respondents spoke particularly warmly about the attitude and approach of their assessor, e.g.

"x was very good , h/she made sure I asked him question about the SSA and what I felt I needed"

"the assessor was friendly and polite and very informative, s/he explained everything."

Service Users' satisfaction with SSA using Interview/Questionnaire

8.96 Total: 3 respondents

S1. Information and preparation for assessment meeting

Table 8.3 Response to General Questions in Section 1 of Interview Schedule

Table 8.4 Response to Detailed questions in Section 1 of Interview Schedule

** question to be revised

Overall results

8.97 Responses to Questions 1-6 indicated that whilst no respondents felt totally unprepared for their assessment meeting, there was insufficient information and preparation for all respondents in some respects. Results indicate that this is an area where improvements should be prioritised.

Other comments, suggested changes, learning points

8.98 One person answered No to Q6 but said that they had their carer with them at the assessment visit.

One person (yes to Q1) explained that the assessor had got to know him/her before doing the assessment.

One person felt that procedures could have been clearer.

S2 How the assessment was conducted

Table 8.5 Response to General Questions in Section 2 of Interview Schedule

Table 8.6 Response to Detailed questions in Section 2 of Interview Schedule

Other comments, suggested changes, learning points

8.99 One person said one visit would have been easier than two but did not suggest this as a 'change'.

S3 More about how the assessment was conducted

Table 8.7 Response to General Questions in Section 3 of Interview Schedule

Detailed questions

Table 8.8 Response to Detailed questions in Section 3 of Interview Schedule

Overall results Sections 2 and 3

8.100 Overall, responses about the way assessments were conducted were overwhelmingly positive. The only area where response suggested room for improvement was in relation to individual's hopes being included as part of the assessment process and information about services being provided to ensure that service users could identify services they needed or wished to use, rather than 'working in the dark' (Q 27 and 28).

S4 What the assessment led to

Table 8.9 Response to General Questions in Section 4 of Interview Schedule

Table 8.10 Response to Detailed questions in Section 4 of Interview Schedule

** question to be revised

Comments, suggested changes learning points

8.101 All respondents reported that they had not been asked the same question by other community care or health professionals. ** NB this question should first establish whether or not respondent has been assessed over the same period by other health and community care professionals.

8.102 Two people used the last question (Q41 any other comments) to express frustration with being assessed for services when the specific services they needed were not available: "What's the point of it if there's no carers (tea time) anyway. It gets our hopes up"; "Things are not happening quick enough. It's frustrating. It's hard".

Overall results Section 4

8.103 Responses indicated that service users were satisfied in large part with outcomes of their assessments but that there is room for improvement in relation to the accessibility of the copy assessment or summary which is provided to service users and information provided about the main contact person.

Conclusion

8.104 Despite the low numbers of interviews and guided discussions available for analysis, it is clear that a mixture of quantitative and qualitative analysis of responses as described above can provide Joint Future Partnerships with information which would enable them to improve the standard of assessment in their areas. For example, if a significant percentage of users were giving negative responses to questions in Section 1, this would indicate that effort needed to be focused to ensure that users had the opportunity to be well prepared for their assessment meeting(s). The material could be analysed to identify whether performance varied between area teams, assessments for specific client groups or even individual assessors. There would be a further option of weighting particular questions: e.g. one could argue that the answers to questions 3 (being asked about your communication requirements before any meetings) and 26 (being asked about what mattered most to you) would be key determinants of the quality of the assessment.

8.105 As noted earlier, a second pilot would work with pilot sites on how the analysis might be used and developed to ensure that the evaluation task was achievable within available resources.

Summary

8.106 The pilot set out to test the usefulness of the methods developed and the toolkit by involving 48 service users and 20 carers. In the event 16 service users and 7 carers took part in the pilot: these people were drawn from 4 Joint Future Partnership areas. A major reason for the low take up in the pilot was that many assessors felt unable for a variety of reasons to ask users and carers to take part.

8.107 Key themes from the pilot were:

• The co-operation of assessors with people carrying out monitoring is important in enabling the latter to carry out their task effectively. A timescale which takes into account the other pressures on assessors' time is vital. Asking assessors to obtain consent was not an effective way of engaging service users and carers with the pilot, as it was clear that many assessors did not feel able to ask people to participate. There is no evidence that service users and carers were unwilling to participate if they were asked

• More consideration is needed of how to involve assessors so that sampling is not biased towards people who feel well served by them, but the benefits of assessors' involvement are maximised where possible

• Assessors' experience of working with individual service users and carers was very helpful in assisting evaluators to decide which toolkit method should be used

• More consideration is also needed to maximize service users' understanding of the assessment process so that the purpose of evaluation visits can be more readily established

• Insufficient people filled in the postal questionnaire or were interviewed using the postal questionnaire as the interview schedule to enable firm conclusions to be drawn about the 2 methods' effectiveness. However, there are some indications that some adjustments are needed to the interview questions and/or the way they are presented to service users

• People who were interviewed generally expressed satisfaction with their assessment

• The Guided Discussion questions and approach are effective in enabling people with moderate learning disabilities and people with severe dementia to give good quality information on the main areas of enquiry of the Guided Discussion (which equate with the general questions in the full questionnaire)

• This can be achieved by interviewers who feel confident of working with the service user group

• There are indications that the carers' questionnaire, whether self completed or used as an interview schedule provided an effective means for carers to give their views on the SSA

• There was anecdotal evidence that many users and carers enjoyed an opportunity to talk to the interviewers

• The Toolkit background information helps interviewers to understand and prepare for their task

• Individualised information from assessors and carers is essential for preparing for discussions where there is a significant degree of communication impairment

• Service users can be helped to remember their assessment meetings by skilful interviewers who have obtained relevant information from assessors.

• Service users' recall of the content of their assessment meetings was more difficult when there was a large time gap between the assessment meetings and the evaluation meeting

• The presence of a carer or other supporter contributes enormously to the effectiveness of the monitoring interview.

« Previous | Contents | Next »