ReLU integral probability metric and its applications.

Abstract

We propose a parametric integral probability metric (IPM) to measure the discrepancy between two probability measures. The proposed IPM leverages a specific parametric family of discriminators based on single-node neural networks with Rectified Linear Unit (ReLU) activation. The proposed IPM yields estimators with good convergence rates and can serve as a surrogate for other IPMs that use smooth nonparametric discriminator classes. We present an efficient algorithm for practical computation, offering a simple implementation and requiring fewer hyperparameters. Furthermore, we explore its applications in various tasks, such as covariate balancing for causal inference and fair representation learning. Across such applications, we demonstrate that algorithms with the proposed IPM provide strong theoretical guarantees, and empirical experiments show that they achieve comparable or even superior performance to other methods.