Outliers (typically) cannot cause type I errors in one-sample/paired t-tests.

Abstract

The presence of outlying data points can have a significant impact on statistical modeling and significance testing. In the specific context of one-sample t-tests, prior studies have shown (primarily through simulations) that outliers make it more likely for t-tests to fail to reject the null hypothesis. In this study, we investigate the opposite scenario: when an outlier can cause the rejection of the null hypothesis. While it may seem intuitive that outliers aligned with the direction of an effect strengthen that effect, prior studies have shown that this is not always the case. Towards this end, we introduce mathematical bounds on how large outliers can be while still increasing the t-statistic in a given sample. These bounds are validated and supported using Monte-Carlo simulations and a survey of available data sets. From these results, we find that although it is not impossible for outliers to cause significant results in paired or one-sample t-tests, it can only occur under rather narrow circumstances. Specifically, it requires a concordant outlier, a minimal sample size of ([Formula: see text]), and a sufficiently small effect size ([Formula: see text]). Based on these findings, we argue that the risk of isolated outliers causing type I errors is low in many practical situations, especially when sample sizes are small.