Adaptive Learning Rate Methods for Complex-Valued Neural Networks.

Abstract

Artificial neural networks (ANNs) have become a popular tool in digital signal processing (DSP). Among the widespread ANN architectures, complex-valued neural networks (CVNNs) have been extensively studied in image processing and telecommunications. Unlike their real-valued counterparts, CVNNs can handle signals directly in the complex domain. Due to this capability, CVNNs usually exhibit higher accuracy and improved convergence compared to real-valued neural networks (RVNNs). Despite their improved performance in several applications, CVNNs still lag behind RVNNs in terms of learning techniques and heuristics. In this context, we propose adaptive learning rate approaches for CVNNs, extending the well-known adaptive gradient (AdaGrad), root-mean-square propagation (RMSProp), AdaMax, AMSGrad, softplus AMSGrad (SAMSGrad), Nesterov-accelerated adaptive moment estimation (Nadam), and DiffGrad to the complex domain. Computational complexities of the proposed optimizers are analyzed for CVNN architectures. Results are compared in terms of mean-squared-error convergence.