A novel backpropagation algorithm based on negated kurtosis loss for training shallow, convolutional, and deep neural networks.

Abstract

The conventional backpropagation (BP) algorithm remains the most widely used approach for training neural networks (NNs), including shallow NN (SNN), convolutional NN (CNN), deep NN (DNN), and deep CNN (DCNN), due to its easy implementation and well-described mathematical structure. However, the BP algorithm has several drawbacks, such as slow convergence and high steady-state error arising from its mean square error (MSE) loss function. For this reason, in this paper, we propose a novel BP algorithm to improve the training and testing efficiencies of SNN, CNN, DNN, and DCNN architectures. This is achieved by minimizing a loss function defined as the negated kurtosis of the error of the output layer. Moreover, the proposed kurtosis-based BP algorithm, which originally employs stochastic gradient descent (SGD), is extended to incorporate more advanced optimizers, such as root mean square propagation (RMSProp) and adaptive moment estimation (Adam). The experimental results on the regression and classification problems indicate that, in the training and testing procedures of all the NN architectures, the proposed kurtosis-based BP algorithm not only increases the convergence rate but also decreases the steady-state error when compared to the well-known competitive methods.