The Scottish Government

« Previous | Contents | Next »

Listen

WELL WHAT DO YOU THINK (2004): THE SECOND NATIONAL SCOTTISH SURVEY OF PUBLIC ATTITUDES TO MENTAL HEALTH, MENTAL WELL-BEING AND MENTAL HEALTH PROBLEMS

ANNEX B: STATISTICAL SIGNIFICANCE AND STATISTICAL RELIABILITY

Statistical significance

The formula used for calculating significant differences between sub groups is as follows:

The standard deviations for two sub-groups are calculated as SD ₁ and SD ₂

1. Calculate an "overall" or "pooled" SD for the two groups together. This is very close to the weighted average; weighted by the relative sizes of the sub-groups in the sample.

2. Use this pooled measure to calculate the Standard Error of the Difference (SED) between the sub-group means, i.e.:

3. Divide the difference between the sub-groups scores that you observe, by the SED. If the size of this result (technically referred to as the "t-score") is greater than 1.96 (i.e. either less than -1.96 or greater than +1.96), then the difference is statistically significant at the 95% confidence level. In other words, there is sufficient evidence that scores in the underlying population are different for the two sub-groups. Thus:

Statistical Reliability

B.1 The respondents to the questionnaire are only a sample of the total 'population'. We cannot therefore be certain that the figures obtained are exactly those we would have if everybody had been interviewed (the 'true' values). However, we can predict the variation between the sample results and the 'true' values from a knowledge of the size of the samples on which the results are based and the number of times that a particular answer is given.

The confidence with which we can make this prediction is usually chosen to be 95% - that is, the chances are 19 in 20 that the 'true' value will fall within a specified range. The table below illustrates the predicted ranges for different sample sizes and percentages results at the '95% confidence interval', based on a random sample.

Table B.1: Predicted ranges for different sample sizes at the 95% confidence interval

Size of sample on which survey result is based	Approximate sampling tolerances applicable to percentages at or near these levels
	10% or 90% +	30% or 70% +	50% +
100 interviews	6	9	10
200 interviews	5	7	7
300 interviews	4	6	6
500 interviews	3	5	5
1,000 interviews	2	4	4
1,401 interviews	2	3	4

Source: MORI

B.2 For example, on a question where 50% of the people in a sample of 1,401 respond with a particular answer, the chances are 95 in 100 that this result would not vary by more than four percentage points, plus or minus from a complete coverage of the entire population using the same procedures. However, while it is true to conclude that the "actual" result (95 times out of 100) lies anywhere between 46% and 54%, it is proportionately more likely to be closer to the centre of this band (i.e. at 50%).

B.3 Tolerances are also involved in the comparison of results from different parts of a sample. A difference, in other words, must be of at least a certain size to be considered statistically significant. The following table is a guide to the sampling tolerances applicable to comparisons.

Table B.2: Sampling tolerances

Size of samples compared	Differences required for significance at or near percentage levels
	10% or 90% +	30% or 70% +	50% +
100 and 100	8	12	13
200 and 200	6	8	9
200 and 400	5	7	8
200 and 500	4	7	7
500 and 500	3	5	5
700 and 300	3	5	6
700 and 400	3	5	5
1,000 and 100	6	9	10

Source: MORI

Table B.3: Demographic sub-group comparisons

Size of samples compared	Differences required for significance at or near percentage levels
	10% or 90% +	30% or 70% +	50% +
Males vs. females (663 vs. 738)	2	3	4
Age 54 and under vs. 55+ (937 vs. 464)	3	4	4
Annual household income of less than 26,000 vs. 26,000 plus (761 vs. 307)	3	5	6
Good general health vs. poor general health (1162 vs. 147)	5	7	8

Source: MORI

« Previous | Contents | Next »