Misinterpretation of P Values and Statistical Power Creates a False Sense of Certainty: Statistical Significance, Lack of Significance, and the Uncertainty Challenge.

Abstract

Despite great advances in our understanding of statistics, a focus on statistical significance and P values, or lack of significance and power, persists. Unfortunately, this dichotomizes research findings comparing differences between groups or treatments as either significant or not significant. This creates a false and incorrect sense of certainty. Statistics provide us a measure of the degree of uncertainty or random error in our data. To improve the way in which we communicate and understand our results, we must include in reporting a probability, or estimate, of our degree of certainty (or uncertainty). This will allow us to better determine the risks and benefits of a treatment or intervention. Approaches that allow us to estimate, account for, and report our degree of uncertainty include use of confidence intervals, P-value functions, and Bayesian inference (which incorporates prior knowledge in our analysis of new research data). Surprise values (S values, which convert P values to the number of successive identical results of flips of a fair coin) express outcomes in an intuitive manner less susceptible to dichotomizing results as significant or not significant. In the future, researchers may report P values (if they wish) but could go further and provide a confidence interval, draw a P-value function graph, or run a Bayesian analysis. Authors could calculate and report an S value. It is insufficient to mindlessly report results as significant versus not significant without providing a quantitative estimate of the uncertainty of the data.