Assessing robustness to worst case publication bias using a simple subset meta-analysis.

Abstract

This article discusses a simple method, known as a meta-analysis of non-affirmative studies, to assess how robust a meta-analysis is to publication bias that favors affirmative studies (studies with significant P values and point estimates in the desired direction) over non-affirmative studies (studies with non-significant P values or point estimates in the undesired direction). This method is a standard meta-analysis that includes only non-affirmative studies. The resulting meta-analytical estimate corrects for worst case publication bias, a hypothetical scenario in which affirmative studies are almost infinitely more likely to be published than non-affirmative studies. If this estimate remains in the same direction as the uncorrected estimate and is of clinically meaningful size, this suggests that the meta-analysis conclusions would not be overturned by any amount of publication bias favoring affirmative studies. Meta-analysis of non-affirmative studies complements an uncorrected meta-analysis and other publication bias analyses by accommodating small meta-analyses, non-normal effects, heterogeneous effects across studies, and additional forms of selective reporting (in particular, P-hacking).