Recursion Newton-Like Algorithm for l_2,0-ReLU Deep Neural Networks.

Abstract

Rectified linear unit (ReLU) deep neural network (DNN) is a classical model in deep learning and has achieved great success in many applications. However, this model is characterized by too many parameters, which not only requires huge memory but also imposes unbearable computation burden. The l_2,0 regularization has become a useful technique to cope with this trouble. In this article, we design a recursion Newton-like algorithm (RNLA) to simultaneously train and compress ReLU-DNNs with l_2,0 regularization. First, we reformulate the multicomposite training model into a constrained optimization problem by explicitly introducing the network nodes as the variables of the optimization. Based on the penalty function of the reformulation, we obtain two types of minimization subproblems. Second, we build the first-order optimality conditions for acquiring P-stationary points of the two subproblems, and these P-stationary points enable us to equivalently derive two sequences of stationary equations, which are piecewise linear matrix equations. We solve these equations by the column Newton-like method in group sparse subspace with lower computational scale and cost. Finally, numerical experiments are conducted on real datasets, and the results demonstrate that the proposed method RNLA is effective and applicable.