Lightweight Reparameterizable Integral Neural Networks for Mobile Applications.

Abstract

Standard integral neural networks (INNs) employ continuous integration layer representation across the kernel and channel dimensions. However, they neglect the reparameterization problem of continuous integration layers, making it difficult to deploy INNs on resource-constrained mobile devices by decoupling their training-time and inference-time structure. We propose a continuous reparameterization strategy that reparameterizes the train-time multiple integration layers into a feed-forward structure at inference time to address this issue. Then, we extend the vision transformer (ViT)-like MetaFormer structure to the continuous integration layer design and leverage an overparameterization integral branch to improve the representation capacity of INNs. Last, exploiting the above innovative techniques, we establish a family of lightweight reparameterizable INNs (RINNs) to achieve strong performance on resource-constrained mobile devices. The results of extensive experiments show the superior performance of RINNs to state-of-the-art lightweight ViTs and favorable zero-shot transfer performance in downstream tasks. On the ImageNet dataset, our RINNs achieve over 79.1 top-1 accuracy with 0.87-ms latency on mobile devices. Moreover, our RINNs maintain the same performance at up to 50% of the rate of structural pruning, without fine-tuning, compared with the 25%-50% accuracy loss of the state-of-the-art discrete models. The codes are publicly available here: https://github.com/ljh3832-ccut/RINN.