A hybrid adaptive preconditioned gradient method with momentum for deep learning.

Abstract

Deep neural networks are typically trained using stochastic optimization methods, which leverage approximate gradient information to update network parameters. While first-order optimizers are prevalent in this field, the potential of second-order gradient methods for faster convergence is well-recognized. However, the application of traditional second-order optimization algorithms in deep learning is hindered by significant per-step overhead. In this paper, we introduce a novel second-order adaptive optimizer, named the adaptive preconditioned gradient method with momentum (AdapGradm), specifically designed for deep learning tasks. AdapGradm constructs a diagonal approximate Hessian matrix based on first-order derivatives, enabling it to capture second-order curvature while maintaining computational efficiency comparable to first-order optimizers like Adaptive Momentum estimation (Adam). Moreover, we propose a hybrid AdapGradm (HAdapGradm) to facilitate a seamless transition between AdapGradm and stochastic gradient descent (SGD). The convergence of AdapGradm and HAdapGradm is rigorously established under mild conditions. Experimental evaluations are conducted to evaluate the performance of these optimizers in training three common neural network architectures on application tasks related to image classification and natural language processing. The results demonstrate that HAdapGradm achieves lower training errors and exhibits superior generalization capabilities compared to Adam and baseline optimizers.