« Previous | Contents | Next »
Listen
APPENDIX 2: GUIDE TO STATISTICAL
RELIABILITY
STATISTICAL RELIABILITY
The respondents to the questionnaire are only a sample
of the total 'population'. It is not certain therefore that
the figures obtained are exactly those we would have if
everybody had been interviewed (the 'true' values).
However, we can predict the variation between the sample
results and the 'true' values from a knowledge of the size
of the samples on which the results are based and the
number of times that a particular answer is given.
The confidence with which we can make this prediction is
usually chosen to be 95% - that is, the chances are 19 in
20 that the 'true' value will fall within a specified
range. The table below illustrates the predicted ranges for
different sample sizes and percentages results at the '95%
confidence interval', based on a random sample.
Size of sample on which survey result is
based
Approximate sampling tolerances applicable to
percentages at or near these levels
| 10% or 90%
+ | 30% or 70%
+ | 50%
+ |
|---|
100 interviews | 6 | 9 | 10 |
|---|
200 interviews | 4 | 6 | 7 |
|---|
300 interviews | 3 | 5 | 6 |
|---|
400 interviews | 3 | 4 | 5 |
|---|
500 interviews | 3 | 4 | 4 |
|---|
Source:
MORI
For example, on a question where 50% of the people in a
sample of 500 respond with a particular answer, the chances
are 95 in 100 that this result would not vary by more than
four percentage points, plus or minus from a complete
coverage of the entire population using the same
procedures. However, while it is true to conclude that the
"actual" result (95 times out of 100) lies anywhere between
46% and 54%, it is proportionately more likely to be closer
to the centre of this band (
i.e. at 50%).
When results are compared between separate groups within
a sample, different results may be obtained. The difference
may be "real", or it may occur by chance (because not
everyone completed a questionnaire). To test if the
difference is a real one -
i.e. if it is "statistically significant" - we
again have to know the size of the samples, the percentages
giving a certain answer and the degree of confidence
chosen. If we assume "95% confidence interval", the
difference between two sample results must be greater than
the values given in the tables below:
| Actual Sample Size | 10% or 90%
+ | 30% or 70%
+ | 50%
+ |
|---|
Overall (500) vs: | | | | |
|---|
Sub-groups of: | 100 | 5 | 8 | 9 |
|---|
| 200 | 3 | 5* | 5 |
|---|
| 300 | 2 | 3 | 4 |
|---|
| 400 | 1 | 2 | 2 |
|---|
*For example, if 30% of the total sample (500) give a
particular answer, and 34% of respondents in a sub-group of
200 give the same answer, there is
not a statistically significant difference
between the responses of the two groups.
Looking at the third column of the above table shows
that there needs to be a difference of
+5 percentage points between the two results in
order for the difference to be statistically
significant.
Therefore, if 36% of the latter group give the same
answer, then this
is a statistically significant difference
(since there is more than a 5 point difference between the
two).
« Previous | Contents | Next »