Theoretical Convergence Analysis and Initialization Comparisons of Deep Soft-Thresholding Networks.

Abstract

Soft-thresholding (ST) has been widely used in deep neural networks. Its fundamental network structure is a deep soft-thresholding fully connected network (ST-FCN). However, training deep ST-FCN to achieve convergence remains time-consuming or even encounters gradient explosion, in part because the convergence behavior is not fully understood. To address this issue, this article proves the relationship between the convergence of deep ST-FCN and the values of network weights and biases. Theoretical analysis shows that, as the number of network layers approaches infinity, deep ST-FCN converges when the network weights tend to an identity matrix, while the biases tend to zero. Following this guidance, we initialize the network weights as the identity matrix, compare it with other representative initialization methods (Gaussian, He, LeCun, Xavier, and Uniform), and quantify their effects on network convergence. Extensive results on a synthetic spectrum dataset and real-world datasets (MNIST and CIFAR-10) demonstrate that initializing the weights to the identity matrix and the bias to zero leads to fast and stable convergence. These conclusions are further supported by additional experiments and statistical analysis on deeper ST networks (with more than ten layers) and other representative architectures (DenseNet-161, ResNet-152, and VGG-19), and more challenging benchmarks (CIFAR-100, STL-10, and Tiny ImageNet). This work provides a theoretical foundation for understanding the convergence of ST neural networks. Furthermore, convergence theory analysis for deep recurrent neural networks (RNNs) with ST is deduced.