« Previous | Contents | Next »
Listen
ANNEX 1: FRAMEWORK FOR ASSESSING THE QUALITY OF EVIDENCE IN EVALUATION STUDIES
Based on Spencer et al (2003). 27
To receive a GOOD grading all of the following criteria need to be fulfilled:
1. Clear statement of study aims & objectives
2. Evaluation addresses its aims and objectives
3. Discussion of rational for study design
4. Description of sample selection & sample profile
5. Clear description of procedures for data collection
6. Alternative perspectives in the data have been presented and explored.
7. In quantitative studies, outcomes are measured in a valid and reliable way 28
8. Findings are credible and are supported by data/study evidence which is visible to reader
9. In qualitative studies, clear links between data, interpretation and conclusions, as well as discussion of explicit & implicit explanations within the findings
10. Discussion of the underlying assumptions / theoretical perspectives / values which have shaped the form and output of the evaluation
To receive a POOR grading at least 4 of the following criteria need to be fulfilled:
1. No statement of study aims & objectives
2. Evaluation does not address aims & objectives
3. In RCTs or quasi-experimental designs, study groups not treated equally
4. In RCTs or quasi-experimental designed, baseline study differences
5. Missing or incomplete information about data collection procedures
6. Inappropriate methods of data collection / analysis 29
7. Significant omissions or errors in analysis
8. Incomplete / unclear reporting of findings
9. No discussion of limitations / contextualisation of findings
10. Interpretation and conclusions not based on the data presented.
11. No discussion of underlying assumptions / theoretical perspectives / values that have shaped the evaluation.
All other studies are graded as FAIR
« Previous | Contents | Next »