Scottish Government

MONITORING AND EVALUATION OF THE SCOTTISH COMPACT Baseline Results 2004

APPENDIX FIVE: MONITORING SURVEY METHODOLOGY

INTRODUCTION

A5.1 The purpose of this Appendix is to outline the monitoring survey methodology and look at how representative the responses are. The survey was designed to establish a baseline against which knowledge, awareness and use of the Compact could be measured. The survey also examined relations between the sector and government and the degree of partnership working which exists between them. The survey was sent out by email and post between the 20 ^th July and 2 ^nd August 2004 to a variety of voluntary sector organisations, Executive Departments, agencies and NDPBs.

GOVERNMENT

A5.2 As far as was possible a census of government was undertaken with questionnaires being sent to all Executive departments, agencies, NDPBs and the health service, 323 in total. The survey was sent to named individuals within the Executive from a list of Departmental Voluntary Sector contacts supplied by the Voluntary Issues Unit. The breakdown by Executive and non-Executive responses and the response rates are shown in Table A5.1. In the government survey, the Executive is under-represented in terms of responses, accounting for 46% of distributed questionnaires yet only 34% of returns. However, 10% of returns did not indicate where they were from.

Table A5.1 Government Survey Responses

Note:-
1. The overall response rate was 30%.
2. A ratio of 1 indicates that exactly the same percentage of responses was received from the indicated area as were distributed. A ratio above 1 indicates an over-representation, whilst a ratio below 1 is an under-representation.
3. The balance was made up of returns for which no respondent details were given (10).

THE SECTOR

A5.3 The survey of the sector was based on a stratified random sample of SCVO's database. The database provides the only comprehensive list of Scottish charities and voluntary organisations and contains some 25,000 entries. The strata used were based on those employed by the database:-

• Location by local authority area;
• Income band; and
• Area of activity.

This was used to generate a random representative sample of 700 organisations. This represents about 1.4% of the 50,000 organisations in the sector that SCVO identified in research undertaken in 2002.

A5.4 Table A5.2 compares the sample with the responses by location, according to the council groupings used by the Scottish Household Survey (TNS Social and MORI Scotland, 2004, p. 149). It can be seen that:-

• There is an over-representation for responses from Central, Dunbartonshire and Grampian; and
• An under-representation from Ayrshire, Lothians, North Lanarkshire and Renfrewshire and Inverclyde.

However, overall the distribution of responses seems reasonable, given that the variations are relatively minor and, in absolute terms, could be easily corrected by the addition of a small number of additional responses.

Table A5.2 Sector Survey Responses by Location

Note:-
1. Eighteen respondents did not give an address.
2. A ratio of 1 indicates that exactly the same percentage of responses was received from the indicated area as were distributed. A ratio above 1 indicates an over-representation, whilst a ratio below 1 is an under-representation.

A5.5 Table A5.3 looks at responses according to the main area of activity, grouped using the categorisation used by SCVO. It will be seen that:-

• Arts, sports and culture and learning are under-represented; and
• Children and families, social care and youth are over-represented.

However, as with the geographical breakdown, the sectoral distribution of responses seems reasonable overall, given that the absolute variations are relatively small and at least half of the responses match (in relative terms) the numbers distributed.

Table A5.3 Sector Survey Responses by Area of Activity

Note:-
1. Nine respondents did not specify a sector.
2. A ratio of 1 indicates that exactly the same percentage of responses was received from the indicated sector as were distributed. A ratio above 1 indicates an over-representation, whilst a ratio below 1 is an under-representation.

A5.6 Table A5.4 looks at the survey responses according to the combined numbers of employees and volunteers involved in each organisation. It will be seen that, for example, 15% of respondents had in excess of 100 staff and volunteers whilst 35% had 10 or less. Unfortunately comparators are not available from the SCVO database. What the database does provide is data on the financial shape of the sector. This shows that 72% of organisations had an income of less than £25,000 in 2001. Given that this is likely to correlate with a limited number of staff and volunteers it may be that the survey responses are biased towards the larger organisations.

Table A5.4 Sector Survey Responses by Combined Staff Numbers

Notes:-
1. Overall staff numbers are the sum of paid employees and unpaid staff and volunteers.

CONCLUSION

A5.7 There is a slight under-representation of responses from the Executive and a corresponding over-representation from agencies and NDPBs.

A5.8 Although the distribution of responses is not an exact match for the distribution of the sector, either on a geographical or an activity basis, the variations are not extreme. It does, however, seem to be the case that the responses are biased towards the larger groups. This is to be expected, given that they are more likely to have heard about the Compact and are therefore more likely to be willing to respond to the questionnaire. When invitations to the Focus Groups were issued, care was taken to include small scale voluntary sector organisations and community groups given that they were relatively under-represented in the sample.

Reference

TNS Social and MORI Scotland, 2004, Scotland's People: results from the 2003 Scottish Household Survey, Scottish Executive National Statistics Publication, Edinburgh.