SSR1M: A stochastic SR1 method with momentum acceleration for non-Convex optimization.

Abstract

Despite the strong performance of matrix-based second-order optimization methods, their high computational and storage costs make them challenging to scale to large machine learning problems. To mitigate these limitations, we propose a stochastic symmetric rank-one (SR1) algorithm with momentum acceleration, referred to as SSR1M. This method relies on vector-level point-to-point operations rather than matrix products, significantly reducing both storage and computational overhead. Specifically, we substitute the batch gradient with a stochastic gradient and simplify the correction terms into hyperparameters for improved practicality and ease of use. Furthermore, we incorporate momentum, exponential moving averaging, and normalization techniques to enhance the algorithm's performance. We provide a detailed convergence analysis of SSR1M in non-convex settings and evaluate its empirical performance on benchmark tasks for non-convex machine learning. Experimental results demonstrate that SSR1M outperforms state-of-the-art methods in terms of both function value and accuracy, demonstrating its effectiveness as an efficient and practical approach for optimizing non-convex multinomial regression models and deep neural networks. The implementation of SSR1M is available at: https://github.com/xudp100/SSR1M.git.