METHODOLOGY
The Scotland Performs website provides trend information on the indicators and targets set out in the Government's National Performance Framework. This note describes the methodological approach taken for the evaluation arrows shown on the website - including some of the technical considerations we faced in developing this new approach to reporting. Further background information on the statistics is provided in the Scottish Government Statistics web pages at:
http://www.scotland.gov.uk/Topics/Statistics/
All the data underpinning Scotland Performs are publicly available in the Scottish Government Statistics web pages and the High Level Summary of Statistics Trends has been updated to ensure that this summary information is easily accessible. We have also introduced a new standardised report on the Scottish Neighbourhood Statistics website which gives access to all the data on each indicator including any data available at local level. Technical notes on each of the targets and indicators provide more information on how each is measured at: http://www.scotland.gov.uk/Publications/2007/11/30090722/0
We welcome comments on the methodology. We will be developing these issues further over the coming months and are keen to involve others in this process. We will set out proposals for this in the near future.
Time period of reporting
It is important to note that current data availability means most reported changes are for periods which are wholly or mainly before mid 2007.
We have shown the change between the latest two dates for which data is available, rather than longer term trends. These dates vary, but the section on each target/indicator provides more information on the period covered in each case. Setting arrows based on longer time periods will not provide clear separation between pre and post baseline changes in future. However, charts in each section show changes over a longer period of time.
As the data series develop, measuring change from the baseline will become increasingly relevant but when this becomes possible will vary from indicator to indicator. Subsequent updates to Scotland Performs will include progress towards targets since the baseline, as well as the latest annual change.
What do the arrows show?
The arrows show most likely situation on the basis of the latest figures, in other words, whether they indicate that the position is improving, worsening or stable.
We have set thresholds for what we consider to be stable and these are explained under each indicator/target. In doing so we have deliberately stuck to a very limited change constituting "stable". We have also taken into account the scale that figures are usually reported to and the nature of past and possible future changes. All these cut-offs will be reviewed and we would welcome comments on what is considered appropriate.
In line with the aim of detecting change coming through the figures (whether positive or negative) the direction of the arrows does not mean we are making a statement about how statistically significant such a change is. This is reported on separately as the figures are published.
All statistics involve a degree of uncertainty. Where data are drawn from sample surveys we can assess the statistical uncertainty. But data, whether from surveys or other sources can involve respondent error, administrative error, processing error etc. which in most cases we cannot quantify although our processes are designed to minimise this. Other factors which can affect uncertainty about changes include the size of the number being measured and how separate the processes of measurement are over time.
Proxies, gaps and provisional data
For some of the indicators and targets it has been necessary to develop new data sources which are not yet available. In the meantime we have included proxy measures where possible to minimise the number of indicators where there are no data. Some data are provisional and subsequent revisions may alter the direction of arrows.
Decisions on the direction of arrows
Final decisions on the direction of the arrows have been made by a group chaired by the Director of Corporate Analytical Services and involving chief professional officers. This ensures that the assessment of direction on each arrow has been made on an objective and professional basis. We will explore the possibility of external involvement in this group.