The Scottish Government

« Previous | Contents | Next »

Listen

5.0 Findings

The pilot, which was conducted as a core component of the research, is a test of the instrument for two different reasons:

1. To test the instrument for validity and to identify ways in which it offers reliable results, and

2. To understand the impact of CLD activity in the pilot population.

Understanding whether this instrument has merit for use in measuring the outcomes of CLD activity is a function of understanding what the data generated by completion of the instrument tells us. Placing the data and the results under the scrutiny of analysis allows us to test its validity and the reliability of findings. We ask questions of the data, analyse the answers and then ask questions of the answers in order to test out the validity of the instrument itself.

Therefore, a discussion of the findings based on an analysis of data with respect to the impact of CLD activity is presented first, and a discussion of the validity of the instrument follows.

Impact of CLD Activity

Looking at the data from a variety of perspectives allows us to see the impact of CLD activity

• toward each outcome;
• from across personal development outcomes, end outcomes, community capacity building outcomes and different groupings of these;
• based on demographic considerations of age, gender, ethnicity and the like;
• across the range of roles participants play within CLD settings; and
• in broad CLD settings categories of youth work, adult learning or community groups.

While each of these perspectives might suggest nuances in impact, the primary analysis for the purposes of this report is based on gender and the setting for CLD activity - i.e. youth work, adult learning or community organisation.

Population and Sample: The population for the pilot test included people within Aberdeen and Aberdeenshire who participate in CLD activity. CLD workers in the area were asked to provide information about the survey to the participants in CLD activity they come into contact with through their work. The workers were identified by CLD managers across the area. Access to the survey was provided both
on-line and on paper, with participants directed to the survey during their contact with the CLD workers. Some participants completed the survey during their CLD time and some completed it on their own time. All survey respondents were offered support to complete the survey from their CLD worker or from Avanté consultants.

Those who completed the survey from the total population included in the pilot were a convenience sample based on the following:

1. those who were connected with a CLD worker who was committed to contribute to the pilot, and

2. those who were willing and able to complete the survey.

The total number of respondents to the pilot survey is 231, and demographic information about the sample is as follows:

In addition, describing the type of CLD activity and the nature of the role of the participant in the activity is as follows:

Note that just over half of those who completed the instrument are not involved in any particular role with the activity, but would classify themselves as participants only.

This shows that the pilot participants were predominantly involved in adult learning with two-thirds of the sample engaged in this activity.

Respondents who completed the assessment for the Community Capacity Building indicators also completed the Personal Development section; but those who completed the Personal Development section could 'opt out' of completing the Community Capacity Building section. Of the 231 total responses, 115 included responses to both Personal Development and Community Capacity Building, and 116 included responses to only Personal Development indicators. In the discussion about the impact of CLD activity below, responses are compared between those who completed only the Personal Development section of the instrument and those who completed both the Personal Development and Community Capacity Building section.

It is important to note that some people did not answer some or all of the questions. For Personal Development outcomes, a total of 79 of 231 people (34%) did not respond to one or more of the indicators for the start of CLD activity, and 2 people completed registration information but did not provide any responses. For Community Capacity Building outcomes, a total of 35 of 115 people (31%) did not respond to one or more of the indicators for the start, and 5 people completed the registration information but did not provide any responses.

Most significantly, while non-response was less than 9% across all other Personal Development indicators and between 8% and 18% across all Community Capacity Building indicators, non-response was between 19% and 25% for End Outcome indicators related to employment. Given that 24% of the participants were between the ages of 10 and 25 (6.9% between 10 and 15 years of age; 9.5% between 16 and 20; and 7.8% between 21 and 25) and 33% were 60+, the analysis should take into account whether the employment indicators are simply not relevant for some or many of these young and older people.

A closer look at the responses by age show that a significant number of older people did not respond to these questions. For people who are 60+, the non-response was between 40% and 54%, a much higher percentage than for the entire sample. However, for young people, non response was between 11% and 15%, an average rate similar to the entire sample. Taking the young people and the older people out of the sample and looking only at the responses for participants between the ages of 26 and 59, the non-response rate for the four employment and economic activity indicators is between 8 and 12%. This suggests that the employment and economic activity indicators and outcomes are not as relevant for people of pension age, although entirely relevant for many others.

Indicators and responses: The instrument uses 36 indicators across 4 Personal Development and End outcomes and 40 indicators across 6 Community Capacity Building outcomes, for a total of 76 outcomes indicators.

Each indicator had two responses, one for the participant's perspective about when they started with CLD activity and one for what the participant thinks now.

In order to quantify the responses to the survey, each response was assigned a numerical value, where

1 = hardly ever
2 = some of the time
3 = most of the time
999 = no answer.

Averages: In order to make sense of the entire data set, it is useful to look at average responses of groups of individuals for each indicator. The most common types of analysis to find an 'average' response are the mean, median and mode. The mean is an arithmetic average, where the responses are added up and then divided by the number of responses. The median is the response that is in the exact middle, where 50% of the responses fall below it. The mode is the most frequent response. For the purposes of this analysis, we determined that the mean would not be very interesting as the 231 respondents had a choice of only three possible responses. The median and the mode provide more interesting results as we can see the level of 'skew' for each indicator, or where responses are concentrated across the three possibilities.

An analysis of the data shows that the median and the mode are very similar across the entire data set. Therefore, for the purposes of the discussion that follows, we use the mode as the average measure.

Change and Starting Point: An important objective for developing this instrument is to attempt to measure the distance travelled toward the desired outcomes by participants in CLD activity. In other words, to measure the amount of change participants report. The instrument measures change by asking participants to provide an assessment of where they think they were when they started the activity and where they believe they are now.

In order to understand the distance travelled, or the amount of change from the start of the activity to now, the amount of change was also assigned a numerical value, where

-2 = change from 3 to 1
-1 = change from 3 to 2, or from 2 to 1
0 = no change
1 = change from 1 to 2, or from 2 to 3
2 = change from 1 to 3
999 = no answer.

It is important to note that when respondents report change, the direction of change is predominantly positive. In other words, participants report a positive impact from CLD activity, with movement toward the outcomes rather than away from the outcomes. Across all 76 indicators and 231 respondents - a total of 17,556 possible responses - a negative change was reported only 38 times, or 0.22% of the total possible. Most of these negative changes were for indicators related to the End Outcomes, as discussed below.

The discussion below describes a significant number of responses indicating no change. While this could suggest that CLD activity does not have an impact on the outcomes, it must be viewed based on the starting point for each indicator. Distance travelled is relevant when we understand where participants begin in their journey as well as the direction and distance of their travel. For example, for the Personal Development indicators many people are starting at 'most of the time', and therefore only negative change is possible. Therefore, no change is a positive indication.

End Outcomes

Economic Activity and Employment: When taking into account the non-response rate to these indicators as discussed above, and therefore changing our analysis to exclude those who are age 60+, we see that no change was indicated for 58% of young people (between 10 and 25) and one level of positive change was indicated for 20% and two levels of positive change was indicated for an additional 4%. This means that nearly one-quarter of the young people in the pilot group experienced a positive change in employment and economic activity.

For people who are between the ages of 26 and 59, no change is indicated for 66%, one level of change is indicated for 14% and two levels of change is indicated for 5%.

In addition to the high number of non-responses for people age 60+, no change was indicated as the average response for 47%. However, 3.5% indicated one level of positive change and 2.5% indicated two levels of positive change. None indicated negative change.

These four indicators elicited the highest number of negative change responses. Females indicated the highest incidence of negative change in this category, with each indicator in this category having at least one incidence of one or two levels of negative change across all ages. For the people age 26 through 59, negative change of one or two levels was indicated 1.5% of the time and for young people between 10 and 25 one or two levels was indicated for 2.3%.

Overall people across all ages indicate some change toward being in a job or in training. A large number of 103 of 231 people (45%) indicated 'hardly ever' at the start and 79 of 231 (34%) indicated 'hardly ever' now. This is a positive change for 24 people, which is just over 10% of the total.

Similarly, indicators about job and working satisfaction as well as income levels show progress toward the outcome but at lower rates than in other outcome areas. Those reporting 'hardly ever' for these other indicators are as follows:

Learning and Education: For this measure, 146 people (63%) indicated no change, with 18.6% indicating one level and 7.3% indicating two levels of positive change. The number of people who started at 'most of the time' was 81, and this number grew to 117 now - a positive change of 44% from the start and 16% of the total sample. The number of people who (started) at 'hardly ever' went from 79 to 45 now, a positive change of 15% of the sample.

However, it must be noted that five people (2%) indicated either one or two levels of negative change.

Democratic Participation and Engagement: This measure provided the least evidence of change between the start of activity and now, with 196 respondents (85%) indicating no change 8% indicating one or two levels of positive change and 1% indicating one or two levels of negative change.

Health: Approximately 80% and 82% of responses for the questions related to physical health indicated no change. 108 people (47%) report being physically healthy 'most of the time' at the start, and that measure grows to 130 (56%) now. Only 69 people (30%) report not engaging in harmful activities 'most of the time' with the number growing to 80 people (35%) now. This is a positive change of 20% and 16%, respectively.

For the indicator related to well-being, 60% indicated no change but 31% indicated one level of positive change and another 2.6% indicated two levels of positive change. The change for those who report 'most of the time' for this indicator from the start to now is from 76 (33%) to 122 (52%), or a change of 60.5%.

Personal Development

Confident Individuals: While no change between the start of activity and now was the average response for two-thirds of the respondents (153 of the 231), a large number (47%) indicated starting at 'most of the time'. The numbers who indicated most of the time at the start and at now was 109 and 181, respectively, an increase of 66%. Those who reported hardly ever at the start to now went from 33 to 1.

Women were more likely to report lower confidence levels at the start and experience more change in confidence due to CLD activity. Nearly 16.5% of females but only 9% of males reported 'hardly ever' feeling confident, and six female respondents indicating two levels of change and only one male reporting such change.

Effective Contributors and Responsible Citizens: Again, an average of 67% (153 of 231) of responses indicated no change between the start of activity and now. The number of people reporting most of the time at the start and the number reporting most of the time now went from 85 to 140, a change of 65%. Though only 35% indicated starting at 'most of the time', an additional 41% reported starting at 'some of the time'. The average response for those who report hardly ever for the indicators in this category went from 39 at the start to 8 now.

For this indicator, the responses patterns are more similar with very little difference in percentages of responses between males and females.

Successful Learners: The indication of no change between the start of activity and now was the average response for 62% of the respondents (144 of the 231). While only 35% of the total indicated starting at 'most of the time', that number was 64.5% now - an increase of 84%. The number reporting hardly ever went from 40 at the start to 0 now.

More females than males were likely to indicate 'hardly ever' at the start (nearly 19% vs 14%) and indicate less experience of 'no change' (59% vs 69%), thereby reporting more change toward the outcome of being a successful learner. Nearly 5% of females indicated two levels of change and 32% indicated one level of change, while less than 2% of male respondents reported two levels of change and 21% reported one level of change.

All of the indicators taken together show the average change toward the Personal Development outcomes by gender below. The overall average indicates 170 of the 231 respondents (74%) indicate no change, and 46 of the 231 (20%) indicate one level of positive change (from 'hardly ever' to 'some of the time', or from 'some of the time' to 'most of the time').

It is interesting to note that there is not a big difference between the responses in the Personal Development section provided by those who completed only this section and those who completed both the Personal Development Section and the Community Capacity Building section. More people (6% vs 3%) who completed both sections provided no response and fewer people (72% vs 75%) indicated an average of no change.

In order to understand the high level of no change for this category, it is interesting to note the starting point for individuals across all Personal Development indicators as follows:

Community Capacity Building

To begin the discussion about the Community Capacity Building indicators, we review an aggregate of the indicators. This shows the average change toward the Community Capacity Building outcomes by gender. The overall average indicates 77 of the 115 respondents (67%) indicate no change, and 22 of the 115 (19%) indicate one level of positive change (from 'hardly ever' to 'some of the time', or from 'some of the time' to 'most of the time'). This compares with the 74% and 20% for Personal Development indicators as shown above.

Again, to understand the high level of no change reported in this category, it is good to note the starting point for communities as reported for each of the Community Capacity Building indicators as follows:

Confident, Skilled and Active Members of the Community: Of the total 115 respondents, 52% report an average starting point of 'some of the time' and 23% indicate one level of positive change. More than 10% of all respondents indicate positive movement from an initial report of 'hardly ever'.

Active Communities with More Influence: For these respondents, 49% report 'some of the time' for this indicator; and while 59% indicate no change, a positive change of one or two levels is reported by 29%. Interestingly, just less than 2% (2 people) indicate one level of negative change for this indicator. Responses for 23% of people who reported hardly at the start changed to 7% now, and those who reported 'most of the time' changed from 18% at the start to 40% now.

Community Organisations with Access to Resources and Deliver Services Effectively: For this category, many people reported 'most of the time', with 42% responding this way at the start and 63% now. Positive change of one or two levels was indicated for 22%, and those who reported 'hardly ever' decreased by 10% of all responses - a reduction of 60% from the start.

Community Organisations Plan, Manage and Assess their Work Effectively: Approximately 70% of respondents indicate no change in this area, though the number who report 'most of the time' grew from 41% of the total to 54% of the total, a change of 32% from the start. At the other end of the scale, respondents report 61% fewer assessments of 'hardly ever' between the start of CLD activity and now.

Community Organisations Include a Wide Range of People in their Work: At the start of CLD activity, 39% of the participants reported 'most of the time', 33% reported 'some of the time', and 18% reported 'hardly ever'. For now, those numbers changed to 56%, 24% and 10%. With only 1 person reporting negative change, and 23% reporting one level of change, this indicates a significant change from 'some of the time' at the start of activity to 'most of the time' now.

Productive Networks and Relationships: This category of indicators provided evidence of the largest amount of two levels of change through over 4% of the responses. Responses indicate a growth of 53% in the responses for 'most of the time' and a reduction of 60% in the responses for 'hardly ever'.

The Instrument

The above analysis of the data provides information about the impact of CLD activity. Understanding whether or not the information is useful is a function of the validity and reliability of the instrument that gathers the data. Validity for this instrument is a matter to be reviewed on different levels:

Internal validity is a question of whether there is a causal relationship between the CLD activity and the outcomes.

Construct validity is a question of whether the instrument is measuring what we think it is measuring. In other words, are the indicators good representations for the outcomes? Do they operationalise the constructs of confidence, effective contributors, responsible citizens, and so forth?

External validity is a question of whether the results can be generalised to other persons, places or times.

This instrument does not contain a sufficient test for internal validity. Because the instrument was constructed under the assumption that there is a relationship between CLD activity and the outcomes, and because that relationship is considered to be causal, participants were continually prompted to remember that they are to be thinking about their participation in CLD activity and its impact as they responded. However, as described above in the constraints section, CLD activity is one of many factors and variables in a person's life that may have an impact on these outcomes. Therefore, the amount to which CLD can be credited or blamed for the positive or negative impact remains unclear. This instrument is insufficient to determine the exact nature of the relationship between CLD activity and the outcomes, and particularly whether that relationship is causal.

Construct validity was tested by a thorough review of the documents and information relating to this study, specifically Delivering Change, HGIOCLD?2, and the literature review related to these outcomes completed by The University of Edinburgh. In addition, the focus groups that were conducted prior to completing the development of the instrument for the pilot were aimed at exploring the ways in which various indicators operationalised the outcomes. These focus groups resulted in culling the indicators from more than 100 to the 76 that became part of the pilot instrument. This culling was based on information from the focus groups that some of the indicators were too general so that they applied to all of the outcomes and some being too vague so that they did not apply to any of the outcomes.

A review of the impact analysis presented above provides an additional analysis of the internal and construct validity by looking at the number of non-responses for each indicator and comparing those responses across all responses to find patterns of interest. This analysis suggests consistency of non-response patterns across all indicators except for the employment indicators for the End Outcomes. Non-responses are less than 9% across all Personal Development indicators and between 8% and 18% across all Community Capacity Building indicators. However, non-response was between 19% and 25% for End Outcome indicators related to employment. This suggests that these particular indicators, the measure, or both were not valid.

An analysis of the non-response that indicates more blank indicators at the end of the instrument than at the beginning would suggest that people got tired or bored with completing the instrument and simply didn't finish. However, the non-responses do not follow that pattern, which suggests that people who didn't fill in answers to indicators did so purposefully and not just through attrition.

Finally, there are some people who filled in all the demographic information but who did not fill in any answers, or only completed a few. This suggests some problem or difficulty with the indicators and the measure against each indicator, but there are not enough of these to indicate a problem with the internal or construct validity.

With respect to external validity, or whether the results could be generalised to other people, places and times, the conclusions and recommendations section of this report suggest changes to the instrument that would enhance its external validity as well as make the data more useful for analysis and understanding the impact of CLD activity. In order to understand how well the results could be generalised, it would be helpful to understand the representativeness of the sample against the population of CLD participants across all of Aberdeen and Aberdeenshire. This analysis was not part of this study, so whether the sample is representative or not is unknown. It is important to note that it wasn't intended for the sample to be representative of the entire population, but simply to be a subset of the population that could be adequately described by certain identifying characteristics.

The people who completed the survey were those who were made aware of the survey by workers committed to participation in the study. This could mean that the responses to the survey are skewed by the level of commitment of the worker. Again, this analysis was not part of this study, so this level of influence is unknown.

However, the sample size of 231 is a fairly substantial number of respondents and therefore is a good starting point to suggest external validity.

Feedback about the Instrument

People who completed the survey were asked for their perspectives about it - including the content and the form - by way of additional questions at the end of the survey and through interviews of participants after they completed the survey. In addition, CLD workers participated in a focus group where they shared what they had learned from the people who they had asked to complete the survey.

Responses to the additional questions contained at the end of the survey were as follows:

The responses for the top three things that would improve the survey were:

A total of six participants were interviewed after they completed the survey to ask what they thought about the instrument overall, the indicators, and the way in which the survey was constructed. These participants were all adult learners and they all completed the survey on-line.

All of those who were interviewed indicated that they really enjoyed completing the instrument as it gave them a good opportunity to reflect on their progress as a result of participating in CLD activity. They did not think it was difficult, though a couple of them indicated it was hard to think back on a time when they were not doing as well as they are now. While they all were concerned that it would take a long time, they were surprised that it didn't take more than '20 to 30 minutes'. However, when asked for suggestions for improving the survey, they all said to make it shorter. All of those interviewed indicated they were glad they had agreed to complete the survey.

A selection of CLD workers who disseminated the survey to participants for completion agreed to participate in a focus group to provide feedback about the survey on behalf of the participants. A total of 11 workers participated in the focus group. The results of this feedback session are as follows:

What worked:

1. Commitment from workers keen to have evidence of the impact of their work

2. The on-line version for a lot of CLD participants that come to Learning Centres where IT is commonly used and available; also those with access to IT facilities in community centres

3. The time it was 'open' was about right - possibly 8 weeks rather than 6

4. Participants of all ages really enjoyed self-reflection

5. Emphasis on confidentiality

6. Choice of on-line or paper

7. Updates and reminders were helpful for workers

8. Clear process for completion

What didn't work:

1. Employment related questions that didn't relate to older retired people

2. Many learners expressed a desire to have more support to complete

3. The scale and layout of the paper questionnaire was off-putting

4. Participants needed a stronger understanding at the outset as to what was in it for them

5. Some participants were keen to please the worker or made a strong assumption that they needed to say good things in order to secure funding - this was hard to dissuade

6. Some of the questions were considered irrelevant by participants

7. Annoyance at some of the registration questions - especially sexual orientation and faith

8. Some would have liked to skip questions - you could do this on the paper version but not on-line

9. Participants would have liked to have had a 'score' at the end of the task

10. Much of CLD is supported by p/t sessional staff so the workers briefings, while essential, meant that they then need to be the people that go and enthuse the sessional staff about the relevance and importance of the task - there was a mixed response to this. It would be good to get them involved in the same kind of briefings

11. The 'one size fits all' approach was not welcome with workers feeding back that some of the language used in the questions needed to be different for different audiences e.g. young people

12. There needs to be a clearer break between PD and CCB sections in order for clarity for those respondents who would have no interest in continuing into the CCB section

13. Too many questions

14. More options needed for scoring

15. Not all scoring headings worked for each question

16. The format was boring - needs colour, graphics, audio, movement, variety of formatting, 'help bubbles', stronger visuals that stimulate and retain attention

17. Easier to self-complete for those that are more confident and very problematic for those that have literacy needs

18. The categories used to describe the CLD activity wasn't always recognised by those completing

19. The length of the introduction

Finally, unsolicited feedback was received from workers who posted paper copies of the survey to the consultants or the client for data input. This feedback suggested that a group of elderly participants were put off by the length of the introduction and the number and type of registration questions. They were offended by being asked about sexual orientation and almost did not go on to complete the survey because of this.

« Previous | Contents | Next »