Beyond reweighting: On the predictive role of covariate shift in effect generalization.

Abstract

Many existing approaches to generalizing statistical inference amid distribution shift operate under the covariate shift assumption, which posits that the conditional distribution of unobserved variables given observable ones is invariant across populations. However, recent empirical investigations have demonstrated that adjusting for shifts in observed variables (covariate shift) is often insufficient for generalization. In other words, covariate shift does not typically "explain away" the distribution shift between populations. As such, addressing the unknown yet nonnegligible shift in the unobserved variables given observed ones (conditional shift) is crucial for generalizable inference. In this paper, we present empirical evidence from two large-scale multisite replication studies indicating that covariate shift can help predict the strength of unknown conditional shift. Analyzing 680 studies across 65 sites, we find that even though the conditional shift is nonnegligible, its strength can often be bounded by that of the observable covariate shift. This pattern only emerges when the two sources of shifts are quantified by our proposed standardized, pivotal measures. We then interpret this phenomenon by connecting it to similar patterns that can be theoretically derived from a random distribution shift model. Finally, we demonstrate that exploiting the predictive role of covariate shift leads to reliable and efficient uncertainty quantification for target estimates in generalization tasks with partially observed data. Overall, our empirical and theoretical analyses highlight an alternative perspective on the problem of distributional shift, generalizability, and external validity.