« Previous | Contents | Next »
Listen
ANNEX 1: GUIDE TO STATISTICAL
RELIABILITY
The variation between the sample results and the "true"
values (the findings that would have been obtained if
everyone had completed the questionnaire) can be predicted
from a knowledge of the sample sizes on which the results
are based, and on the number of times that a particular
answer is given. The confidence with which we can make this
prediction is usually chosen to be 95%, that is, the
chances are 95 in 100 that the "true" values will fall
within a specified range.
The table below illustrates the required ranges for
different sample sizes and percentage results at the "95%
confidence interval":
Approximate sampling tolerances
applicable to percentages at or near to
these levels | Actual Sample Size | 10% or 90%
+ | 30% or 70%
+ | 50%
+ |
|---|
Overall | 500 | 2.3 | 3.5 | 3.8* |
|---|
*For example, if 50% of all respondents were to give a
particular answer, the chances are 19 in 20 that the "true"
value will fall within the range of
+3.8 percentage points from the sample results.
Comparing percentages between sub-groups and
overall total
When results are compared between separate groups within
a sample, different results may be obtained. The difference
may be "real", or it may occur by chance (because not
everyone completed a questionnaire). To test if the
difference is a real one -
i.e. if it is "statistically significant" - we
again have to know the size of the samples, the percentages
giving a certain answer and the degree of confidence
chosen. If we assume "95% confidence interval", the
difference between two sample results must be greater than
the values given in the table below:
| Actual Sample Size | 10% or 90%
+ | 30% or 70%
+ | 50%
+ |
|---|
Overall (500) vs: |
|---|
Sub-groups of: | 50 | 8.6 | 13.1 | 14.3 |
|---|
| 100 | 6.2 | 9.5 | 10.3 |
|---|
| 200 | 4.6 | 7.0 | 7.6 |
|---|
| 300 | 3.9 | 5.9 | 6.4 |
|---|
| 400 | 3.5 | 5.3 | 5.8* |
|---|
*For example, if 50% of the total sample (500) give a
particular answer, and 55% of respondents in a sub-group of
400 give the same answer, there is
not a statistically significant difference
between the responses of the two groups.
Looking at the fifth column of the above table shows
that there needs to be a difference of
+5.8 percentage points between the two results in
order for the difference to be statistically
significant.
Therefore, if 56% of the latter group give the same
answer, then this
is a statistically significant difference
(since there is a 6 point difference between the two).
Comparing percentages between
sub-groups
The following table indicates differences required for
significant comparisons between sub-groups.
Approximate sampling tolerances
applicable to percentages at or near to
these levels | 10% or 90%
+ | 30% or 70%
+ | 50%
+ |
|---|
Sub-group of 50 vs: |
|---|
100 | 10.1 | 15.4 | 16.8 |
|---|
200 | 9.2 | 14.0 | 15.3 |
|---|
300 | 8.9 | 13.5 | 14.8 |
|---|
400 | 8.7 | 13.3 | 14.5 |
|---|
Sub-group of 100 vs: |
|---|
200 | 7.0 | 10.7 | 11.6 |
|---|
300 | 6.6 | 10.0 | 10.9 |
|---|
400 | 6.3 | 9.7 | 10.6 |
|---|
« Previous | Contents | Next »