MD-LSM: an enabling tool for real-time monitoring linear separability of hidden-layer outputs of deep networks.

Abstract

The linear separability of hidden-layer outputs plays a key role in understanding the working mechanism of deep networks. However, it is still challenging to develop the linear separability measure (LSM) that satisfies the following requirements: 1) it should be an absolute measure; 2) it should be insensitive to the outliers; 3) it should be affine invariant; and 4) its computational cost should be low. In this paper, we propose the Minkowski difference-based linear separability measures (MD-LSMs) that just meet the first three requirements. Moreover, we also introduce an approximate calculation method to significantly decrease their computation costs with only a slight precision sacrifice. As an application, we conduct the experiments on the real-time monitoring for the hidden-layer behavior of several popular deep networks, and show that the outputs of the hidden layers adjacent to the output layer have higher linear separability degrees. We also observe that the change of linear separability degree of hidden layers (especially the ones are adjacent to the output layers) is in sync with the change of the training accuracy of the entire network. These findings imply that MD-LSMs serve as reasonable and enabling tools of monitoring the training status of deep networks.