Training sparse convolutional deep predictive coding networks with attention.

Abstract

This paper proposes a novel training methodology for sparse convolutional deep predictive coding networks with attention (DPCN-SCA). In the same spirit of self supervised learning, the method fully exploits the bidirectional architecture, and takes advantage of the extra top-down structure that resembles the autoencoder. We modify the traditional equations to accommodate this extra top-down flow, and also show the importance of this modification for new applications of DPCNs as visual memories for object-level information compression, as well as for improved interpretability of the learned features. The effectiveness of a DPCN-SCA with six convolution layers (similar to AlexNet) is verified on several popular benchmark datasets. In the visualization results, we observe that DPCN-SCA starts by extracting detailed features in the first layers until focusing on contours in deeper layers, mirroring the behavior of supervised convolutional neural networks (CNNs), in spite of avoiding explicit labels.When the deepest feature maps are constrained to retain only 1-5% non-zero activations, DPCN-SCA substantially surpasses all unsupervised sparse coding baselines, yielding more than a 20% absolute gain in classification accuracy under comparable sparsity levels. In addition, we introduce a new visualization method for the deep layers of DPCN. This approach projects activations from the deep receptive fields back onto the input space, allowing for clear observation of the attention patterns captured by individual filters. Codes can be found at https://github.com/hongmingli1995/DPCN-SCA.