Bi-PIL: Bidirectional Gradient-Free Learning Scheme for Multilayer Neural Networks.

Abstract

Training deep neural networks typically relies on gradient descent learning schemes, which is usually time-consuming, and the design of complex network architectures is often intractable. In this article, we explore the building of multilayer neural networks based on an efficient gradient-free learning scheme offering a potential solution to the architectural design. The proposed learning scheme encompasses both forward and backward training (BT) processes. In the forward process, the pseudoinverse learning (PIL) algorithm is employed to train a multilayer neural network, in which the network is dynamically constructed leveraging a layer-by-layer greedy strategy, enabling the automatic determination of the architecture across different hierarchies in a data-driven manner. The network architecture and connection weights determined in the forward training (FT) process are shared with the backward process which also conducts gradient-free learning to update the connection weights. After the bidirectional learning, a neural network comprising two twin subnetworks is obtained, and the fused features of subnetworks are used as inputs for downstream tasks. Comprehensive experiments and detailed analyses demonstrate the effectiveness and superiority of the proposed learning scheme.