Dense Optimizer: An Information Entropy-Guided Structural Search Method for Dense-Like Neural Network Design.

Abstract

Dense convolutional network has been continuously refined to adopt a highly efficient and compact architecture, owing to its lightweight and efficient structure. However, as the current dense-like architectures are mainly designed manually, it becomes increasingly difficult to adjust the channels and reuse level based on past experience. As such, we propose an architecture search method called dense optimizer that can search high-performance dense-like network automatically. In dense optimizer, we view the dense network as a hierarchical information system, maximizing the network's information entropy while constraining the effectiveness and the distribution of the entropy across each stage via a power law, thereby constructing an optimization problem. We also propose a branch-and-bound optimization algorithm that tightly integrates power-law principle with search space scaling to solve the optimization problem efficiently. The superiority of dense optimizer has been validated on different computer vision benchmark datasets. Our searched model DenseNet-OPT achieved a top-1 accuracy of 84.3% on CIFAR-100, which is 5.97% higher than the original one. Specifically, dense optimizer achieves high-quality search results while only requiring 4 h of computation time on a single CPU.