Ape Optimizer: A p-Power Adaptive Filter-Based Approach for Deep Learning Optimization.

Abstract

Deep learning has been widely applied in various domains. Current widely-used optimizers, such as SGD, Adam, and their variants, are designed based on the assumption that the gradient noise generated during model training follows a Gaussian distribution. However, recent empirical studies have found that the gradient noise often does not follow a Gaussian distribution. Instead, the noise exhibits heavy-tailed characteristics consistent with an $\alpha $ -stable distribution, casting doubt on the performance and robustness of optimizers designed under the assumption of Gaussian noise. Inspired by the least mean p-power (LMP) algorithm from the field of adaptive filtering, we propose a novel optimizer called Ape for deep learning. Ape integrates a p-power adjustment mechanism to compress large gradients and amplify small ones, mitigating the impact of heavy-tailed gradient distributions. It also employs an approach for estimating second moments tailored to $\alpha $ -stable distributions. Extensive experiments on benchmark datasets demonstrate Ape's effectiveness in improving both accuracy and training speed compared to existing optimizers. The Ape optimizer showcases the potential of cross-disciplinary approaches in advancing deep learning optimization techniques and lays the groundwork for future innovations in this domain.