Judging the quality of systematic reviews and meta-analyses for policy analysis: an exploratory study of utilization in three ministries in British Columbia




Malange, Ramsay

Journal Title

Journal ISSN

Volume Title



Public policy analysts are often tasked with reviewing research or other forms of evidence in order to provide advice for policy decisions. Many have argued that systematic reviews that include meta-analyses (SRMAs) are the most rigorous forms of evidence, and thus, when possible, should form the basis of policy decisions. However, it is not yet clear to what extent policy analysts are aware of systematic reviews and meta-analyses, or to what extend they use them to inform policy work. Moreover, given the importance of evaluating the quality of research before using it for policy, it is not clear to what extent policy analysts feel able to judge the quality of systematic reviews and meta-analyses. An online survey was used to provide initial estimates of the extent to which policy analysts a) are familiar with SRMAs; b) use these reviews to inform their policy work; and c) are able to evaluate them. It further sought to explore other correlates of use, barriers to use, methods to increase use, and knowledge of factors that influence quality. Thirty-nine Ministerial policy analysts responded to the survey, 18 from the Ministry of Health, 9 from the Ministry of Environment, and 12 from other ministries. Policy analysts reported being fairly familiar with both systematic reviews and meta-analyses, although they were more familiar with systematic reviews than with meta-analyses. There were no differences between the Health, Environment, or Other groups with respect to familiarity. Respondents reported moderate scores on most indicators of use, with results suggesting the Health group having the highest rates of use, followed by the Environment group and then the Other group. Finally, there were relatively high self-ratings on ability to judge the quality of SRMAs, with no differences found between groups. The results of other exploratory analyses are also presented, and implications and recommendations are discussed.



Meta-analysis, Systematic Review, Research Quality, Policy Analysis, Evidence-based policy, policy makers