Combining exchangeable <i>P</i>-values.

Abstract

The problem of combining <i>P</i>-values is an old and fundamental one, and the classic assumption of independence is often violated or unverifiable in many applications. There are many well-known rules that can combine a set of arbitrarily dependent <i>P</i>-values (for the same hypothesis) into a single <i>P</i>-value. We show that essentially all these existing rules can be strictly improved when the <i>P</i>-values are exchangeable, or when external randomization is allowed (or both). For example, we derive randomized and/or exchangeable improvements of well-known rules like "twice the median" and "twice the average," as well as geometric and harmonic means. Exchangeable <i>P</i>-values are often produced one at a time (for example, under repeated tests involving data splitting), and our rules can combine them sequentially as they are produced, stopping when the combined <i>P</i>-values stabilize. Our work also improves rules for combining arbitrarily dependent <i>P</i>-values, since the latter becomes exchangeable if they are presented to the analyst in a random order. The main technical advance is to show that all existing combination rules can be obtained by calibrating the <i>P</i>-values to e-values (using an [Formula: see text]-dependent calibrator), averaging those e-values, converting to a level-[Formula: see text] test using Markov's inequality, and finally obtaining <i>P</i>-values by combining this family of tests; the improvements are delivered via recent randomized and exchangeable variants of Markov's inequality.