Convergence Suppression and Divergence Facilitation: Minimum and Joint Use of Hidden Units by Multiple Outputs.

Abstract

Pruning of multi-layer neural network structure is not a simple problem if two or more output units are present; unlike the single-output case, removing a hidden-output synaptic link does not necessarily mean elimination of a hidden unit. A new pruning algorithm called CSDF is presented. Its aim is to minimize the total number of active hidden units, thereby making some hidden units utilized jointly by more than one output unit. In addition to the backpropagation (BP) term, two antagonistic actions are introduced for controlling the synaptic development of hidden-output links; the incoming pathways to each output unit try to prevent each other from growing, whereas those outgoing from each hidden unit help each other for growing. CSDF works by dynamically balancing these two lateral interaction effects in the presence of the BP action, and hidden units not indispensable for any output unit will disappear. The process has some parallel with recent findings in developmental neurobiology. The CSDF pruning is applied to some multi-task problems, including one that exemplifies the importance of "structural compatibility" and demonstrates easy discovery of the optimum modular architecture which is difficult to deduce from separate structural optimization for each task alone. In most cases examined here, CSDF outperformed other pruning algorithms such as "weight decay" and "optimum brain damage." In particular, no existing algorithm other than CSDF can deal with pruning of auto-encoders. Some examples are also given for such paradigm of input-output identy mapping. Copyright 1997 Elsevier Science Ltd. All Rights Reserved.