The Scottish Government

« Previous | Contents | Next »

Listen

DIVERSITY IN THE PUBLIC APPOINTMENTS PROCESS IN SCOTLAND

ANNEX TWO METHODOLOGY

The main aim of this research was to explore why the public appointments process is not as effective as it should be in producing a diverse range of candidates and to identify recommendations for improving the process. This was undertaken through a qualitative study of the workings of the process with regard to diversity issues, exploring in detail the perceptions of all stages of the public appointments process by all of those involved.

The key aims of the research were to:

• Explore the activities of Scottish Executive sponsor teams during an appointments round and identify their perceptions of the process.
• Explore experiences and perceptions of the application and appointment process from samples of successful and unsuccessful applicants.
• Explore the role and activities of Independent Assessors in the appointments process and their perceptions of the process.
• Explore the role of other organisations which may be involved in the appointments process.
• Identify current practice in the public appointments process in relation to diversity and highlight good practice and areas for improvement.
• Make recommendations for improvements.

Overall, it was anticipated that the findings would be used to review and improve the Executive's existing processes and also inform the development of a diversity strategy for public appointments in Scotland.

The research focused on a number of under-represented groups identified in the original brief as those defined through the data collection and monitoring process used in public appointments. This includes minority ethnic people, disabled people, women, young people (under 25) and people living outwith the central belt of Scotland.

It was recognised that the exploration of diversity issues in the public appointments process would require the use of a range of methods. These methods are described below.

CASE STUDIES

A large amount of detailed information was gathered by conducting a small number of case studies of recent public appointments processes. It was considered inappropriate to carry out case studies of current appointments processes (as there would almost certainly have been a "researcher effect" on these), but four case studies were undertaken of appointments processes which took place in the preceding year, providing an in-depth account of participants' perceptions.

These were selected from all of the appointments processes which were carried out in that year, with an attempt made to include the following:

• A cross section of different types of organisation and different sectors.
• Service delivery organisations and 'board' type organisations.
• Organisations of different sizes.
• Rural, semi-rural and Scotland-wide organisations.

Thereafter, the organisations were selected at random.

Each case study involved a number of strands. Firstly, the documentary material relating to the appointment(s) was requested and gathered (including background material and the information which was sent to applicants, application forms used, monitoring forms etc.).

In addition, face to face interviews were carried out with the range of participants in the process. This included, in each case, members of the Scottish Executive sponsor teams, the Independent Assessor and members of the selection panel. Successful and unsuccessful interviewees were also asked to participate in face to face discussions and interviews were carried out, in each case, with candidates.

Only a small number of candidates (2) opted out and one of the Scottish Executive panel members did not wish to be interviewed. One sponsor team member had since moved on. A further two candidates made arrangements which were then cancelled and could not be rescheduled in the timescale and one other respondent was unable to identify an appropriate date. In one case, as a very large number of candidates took part in the application process, a sample of candidates (7) was interviewed, while the remainder were given the opportunity to take part through the completion of a postal questionnaire. A total of 36 detailed face to face interviews were carried out in the case studies.

The case studies explored, with all of these participants, the ways in which all stages of the public appointment process were carried out. Interviews with those involved in selection also covered preparation received, awareness and perceptions of diversity issues, types of support provided to the process and the use of training and guidance. Interviews with candidates also explored their views of the ways in which diversity issues were reflected in the process. Suggestions were sought from all of those involved of means of improving the public appointments process.

This part of the work provided a very large amount of data. The interviews were considered successful and generally lasted 1-1.5 hours (although a small number lasted longer and the respondents in these cases agreed to continue). Generally, participants in the case studies were both helpful and willing to discuss their views and many made a range of suggestions relating to future practice.

POSTAL SURVEY OF RECENT PROCESSES

While the case studies provided detailed examples of experiences of specific public appointments processes, it was also recognised that there was a need to gather information from a broader range of participants.

A postal survey was carried out of participants in additional recent appointments processes, to supplement this work. It had originally been agreed that this would involve a further 16 processes, making a total of 20 including the four case studies. At an early discussion, it was decided to approach contacts in 20 additional processes (in case some were unable to respond), making a new total of 24 including the case studies. This proved to be a useful addition, as two processes were subsequently excluded (one which was about to carry out a new appointment and another which was also in the process of change), leaving a total of 18, as well as the four case studies.

Of these, one of the organisations involved also took part in a case study and sent details only of the case study. (It would, in any event, have led to confusion to carry out a simultaneous postal survey.) One other round provided no contact details, leaving a total of 16 processes which were included in the survey, and the four case studies (20 rounds in total). This was considered to provide a good number to allow the examination of perceptions of public appointments.

In each of these cases, questionnaires were sent to the sponsor team and selection panel members, which includes the Independent Assessor and the NDPB representative. Postal questionnaires were also sent to all of the interviewees for these processes for whom contact details were held by the Scottish Executive (96) and the same questionnaire was sent to a random sample of 75 applicants for these appointments who had not been selected for interview. These questionnaires were prepared by Reid - Howie Associates and distributed by the Scottish Executive, in order to comply with data protection requirements. The completed questionnaires were returned to Reid-Howie Associates to protect the confidentiality of the candidates.

In the case of both the Independent Assessors and the Scottish Executive sponsor teams, some of the respondents were involved in a number of separate appointments rounds. For this reason, two questionnaires were developed. One covered their general perceptions of the public appointments process, while the other examined practice in relation to the specific round for inclusion.

Clearly, it was not anticipated that this would generate comprehensive data in relation to each of the processes, in the way that the case study material did, as it was likely that a number of respondents would not return their questionnaires. It was considered likely, however, that any participants who wished to raise specific issues about any of these processes would take the opportunity to do so. It was also considered that the respondents who did participate would provide an indication of the roles of different participants and suggestions for development.

Data (from at least some of the participants) was received in relation to all 16 of the processes in the postal survey. An excellent response was received from candidates themselves, with 78 responses received, representing respondents from all of the rounds. This constituted around 46% of all questionnaires sent to candidates (a very high response for a postal survey) and was encouraging, as candidates' views of diversity (particularly those who believe that there is scope for development of work) are clearly a key aspect of identifying actions which can be taken.

Amongst the candidates who responded to the postal survey, the inclusion of under-represented groups was found to be as follows:

Table 1: Candidate respondents who considered themselves to be a member of an under-represented group

There was also an excellent response from Independent Assessors, who provided detailed material for all but one of the 16 processes selected.

Most of the senior Scottish Executive staff who participated on interview panels (and who were still in post) returned their questionnaires (70%), providing details of three quarters of the processes which they covered.

There was also a good response from the members of sponsor teams responsible for the actual organisation of the processes. Responses were received from sponsor team members representing 12 of the processes (75%).

Lastly, around a third of the NDPB representatives involved in these appointments rounds responded.

COLLECTION OF OTHER VIEWS

The material relating to the specific processes was supplemented further through the collection of additional information in a number of ways.

Interviews with key organisations

Interviews were also carried out with key national equalities organisations working with under-represented groups in Scotland covered by the remit of this research, in order to identify their views of the key barriers to diversity and suggested changes.

Interviews were held with the Disability Rights Commission, the Equal Opportunities Commission, the Commission for Racial Equality and the Scottish Trades Union Congress. The Youth Parliament was also approached and invited to participate.

Interviews were also held with members of the Public Appointments Team and with a representative from the Scottish Executive Equality Unit.

Postal survey of other relevant contacts

It was also recognised that it was important to provide the opportunity for any other NDPBs, Independent Assessors or sponsor team members who were not involved in the selected processes to provide their general views of diversity issues. For this reason, short postal questionnaires were sent to the remaining NDPB Chairpersons in Scotland, the remaining sponsor team contacts and the remaining Independent Assessors. It was not considered likely that a high number would participate in this way (given that the focus was on diversity more generally, rather than on a specific process involving them), but again it was recognised that any with specific views of developments required would have the opportunity to submit these.

A list of equalities organisations was also identified in conjunction with the Scottish Executive Equality Unit and a postal questionnaire circulated to those working with the under-represented groups identified for inclusion in the research. This gave them the opportunity to provide additional equalities expertise to assist in identifying any barriers and suggested improvements.

A total of 51 responses were received from others with an interest in public appointments who were not involved in the specific processes. This was a higher number than expected and produced a large amount of additional, valuable information and positive suggestions.

DOCUMENTARY EVIDENCE/GOOD PRACTICE

A number of relevant documents were also identified which helped to provide the background and context for the research, as well as assisting in the identification of good practice. These documents were:

• Cabinet Office (2001): "Making Public Appointments: A Best Practice Guide for Departments".
• Cabinet Office (2002): "Public Bodies - Opening up Public Appointments 2002-2005".
• Cabinet Office (2002): "Public Bodies and Public Appointments: Frequently Asked Questions and their Answers".
• Cabinet Office (2003): "Report of a Short Life Working Group on Improving Diversity in Public Appointments".
• Council of Europe (2002) "Going for Gender Balance: A Guide for Balancing Decision Making".
• Council of Europe (2000): "Positive Action in the Field of Equality Between Women and Men".
• Department for Transport, Local Government and the Regions (2001): "Making a Difference: Women in Public Appointments".
• Housing Corporation London (2000): "Recruitment of BME Board Members".
• Office of the Commissioner for Public Appointments (2001): "Code of Practice for Ministerial Appointments to Public Bodies".
• Office of the Commissioner for Public Appointments (2001): "A Guide for Independent Assessors in the Public Appointments Process".
• Race Equality Advisory Forum (2001): "Recommendations on Public Appointments".
• Scottish Executive (2000): "Equality Strategy: Working Together for Equality".
• Scottish Executive (2001 and 2002): "Appointments to Non-Departmental Public Bodies in Scotland".
• Scottish Executive (2001) "Public Bodies: Proposals for Change".
• Scottish Executive (2001): "Preliminary Report on the Equality Strategy".
• Scottish Executive (2002) "Public Appointments and Public Bodies (Scotland) Bill: Consultation Paper".
• Scottish Executive (2002): "A Guide to Public Bodies in Scotland".
• Scottish Executive (2002): "Working Together for Race Equality - The Scottish Executive's Race Equality Scheme".
• Scottish Executive (2002): "Closing the Opportunity Gap".
• Scottish Executive (2003): Making Progress: Equality Annual Report.
• Scottish Executive (2003): "On Board: A Guide for Board Members of Public Bodies in Scotland".

DATA ANALYSIS

This research generated a large amount of material which required detailed analysis. It should be stressed here that, although there was an excellent response to this study, the actual response rate was less important than the provision of the opportunity to all participants to give their views. It was not intended that this should be a quantitative study of the public appointments process. The focus of the analysis was upon the identification of the experiences of those who expressed their views on diversity issues, on the identification of good practice, and on their suggestions for future development.

Perhaps the most important issue to stress in relation to the data is that the actual number of respondents expressing particular views is not a measure of the strength of impact, or value, of many of the issues raised. In this context, it was inappropriate to express the findings quantitatively, other than to give an overall flavour of the responses, for the following reasons:

• The purpose of the research was to identify details of the types of issues which affect under-represented groups, and the range and depth of views, not to quantify the number of participants affected by them.
• In the promotion of good practice in equalities issues, the number of people experiencing discrimination or barriers is irrelevant. The important issue is the identification and elimination of these barriers, wherever they are experienced.
• The research focused largely upon the views of participants in the process who, by definition, were not excluded. In order to comment meaningfully on the level of experiences of discrimination, it would have been necessary to explore the views of those who had not participated in the process.
• Some organisational responses (e.g. equalities organisations) presented the views of a number of individuals, making it impossible and inappropriate to count the number of people represented. Thus, while a response may constitute only one actual "submission", it may provide the views of many individuals whose interests the organisation represents.
• In the case of diversity, an individual's level of understanding of the issues has a significant impact upon their perceptions of the nature of a process. The perceptions of a process, for example, by a white male with a professional background, no physical impairments and experience of applying for many high level posts may be very different to those of members of under-represented groups. These issues can also affect Independent Assessors, panel members, sponsor team members and NDPB representatives, as can the level and quality of diversity training which they have received.
• A number of respondents (who were not surveyed as candidates), whilst recognising that they themselves had not necessarily faced barriers, were nonetheless able to identify a range of issues affecting other groups. A quantitative account of those affected by specific issues would not take account of these perceptions.

The main consideration was to identify areas in which any respondents experienced or identified barriers, believed that there were ways in which the approach to diversity could have been improved, or highlighted ways in which particularly good practice had been promoted, and to use this to develop recommendations for future good practice.

In the light of the above, it should be emphasised that the focus of this research (and the final report) was not upon identifying whether individual processes were fair and appropriate per se, and the report does not attempt to assess whether each of the separate appointments was conducted in such a way. Neither was the purpose to identify the proportions of different respondents in each case expressing particular views. The overall aim was to identify general barriers and suggested solutions, in order to inform the improvement of the existing process and the development of a diversity strategy to be led by CPAS. For these reasons, there was no need to identify the four case study organisations and the other 16 rounds which were considered.

The material from the case studies and face to face interviews was analysed manually and in detail, with the identification of issues from scrutiny of all of the findings.

The material from the postal surveys, although containing a large amount of qualitative detail, was input electronically. This was, however, analysed qualitatively, through the identification of key themes and issues from each of the groups of participants.

PERCEPTIONS OF METHODOLOGY

A small number of respondents expressed views about the methodology or overall conception of this research. This almost inevitably happens with work which focuses on sensitive issues such as equality (compounded by personal issues such as interview success or otherwise) and can generate strong views (both supportive of the research and more negative). A small number of practical issues were also raised which were dealt with in the course of the study.

In addition, some respondents expressed their views about the public appointments process as a whole (rather than focusing on diversity issues), sometimes relating to their own experiences as a candidate. Where these may have had relevance to diversity issues, their experiences have been included.

It was also recognised that people with a high level of knowledge of the issues may have found the shorter postal questionnaire too general to deal with some of the subtleties of diversity/equality issues (and this point was raised by one NDPB Chairperson). It was important, however, to try to strike an appropriate balance in the questionnaire, and most respondents found it possible to complete this and to offer their views, providing supporting information where necessary.

Telephone conversations were held with respondents who requested these.

Two other issues were raised in relation to the methodology:

• There was no involvement in the research of Ministers, who take the final decisions in relation to public appointments. This was because they generally have no direct contact with candidates and do not become directly involved in selection (and the focus of the research was primarily on the processes). It was acknowledged in the report, however, that this is not always the case, and many of the recommendations made are also relevant to Ministers' role in the process.
• The specification of particular under-represented groups by the Scottish Executive excluded other relevant groups (such as LGBT people and people experiencing material deprivation). The Scottish Executive specified the groups for inclusion in the research by identifying those groups for which current information is available (in terms of data collected on monitoring forms). It has been recognised in the report, however, that there are other equalities groups which are under-represented, and many of the lessons learned are transferable to these groups (although there will also be particular issues for some). The good practice recommendations should address many of the issues for other groups, and additional suggestions to address specific barriers can continue to be added to good practice guidance as this develops.

OVERVIEW

All of these methods, in combination, provided a detailed insight into the public appointments process in Scotland. The research allowed the identification of barriers to participation for a number of groups, and highlighted many suggestions which will help to inform the improvement of the existing process and the development of a diversity strategy to be led by CPAS.

« Previous | Contents | Next »