Null misinterpretation in statistical testing and its impact on health risk assessment.

Abstract

Statistical methods play a pivotal role in health risk assessment, but not always an enlightened one. Problems well known to academics are frequently overlooked in crucial nonacademic venues such as litigation, even though those venues can have profound impacts on population health and medical practice. Statisticians have focused heavily on how statistical significance overstates evidence against null hypotheses, but less on how statistical nonsignificance does not correspond to evidence for the null. I thus present an example of a highly credentialed statistical expert conflating high "nonsignificance" with strong support for the null, via misinterpretation of a P-value as a posterior probability of the null hypothesis. Reverse-Bayes analyses reveal that nearly all the support for the null claimed by the expert must have come from the expert's prior, rather than the data, there being no background data that could support a strong prior. The example illustrates how inattention to the actual meaning of P-values and confidence limits allow extremely biased prior opinions (including null-spiked opinions) to be presented as if they were objective inferences from the data.