Theoretical analysis of the denoising autoencoder using Tweedie's formula.

Abstract

The denoising autoencoder captures the score, that is, the gradient of the logarithm of the data distribution. This result holds only when the corruption process is Gaussian, however many theoretical and practical studies consider the non-Gaussian setting. In this paper, we extend the classical analysis to an exponential family by leveraging Tweedie's formula. This empirical Bayes estimator yields a new formulation of the denoising autoencoder. Our analysis elucidates that the denoising autoencoder captures the conditional expectation of the sufficient statistic of the underlying stochastic denoising process: an estimator of an information-theoretically compressed quantity that bridges the original and corrupted data. Furthermore, we demonstrate that this quantity implicitly contains the score of the corrupted data distribution. Therefore, the denoising autoencoder simultaneously captures both the estimated sufficient statistic and the corrupted data score even if the corruption process belongs to an exponential family. Numerical experiments support our theoretical findings.