Two-phase collaborative model compression training for joint pruning and quantization.

Abstract

To reduce the storage and computational complexity of neural network models, various model compression techniques have been proposed in recent years, including pruning and quantization. However, due to the lack of interconnection among different type of methods, it is difficult to effectively integrate the advantages of these diverse techniques. This paper proposes a novel two-phase collaborative training framework for joint pruning and quantization to achieve synergistic optimization of multiple compression techniques. This framework combines pruning, quantization operations, consisting of two phases: collaborative constraint pre-compression and post-training compression refinement phases. In the collaborative constraint pre-compression phase, a novel unified constraint loss function is designed to ensure that weights are close to quantization values, and sparse regularization is utilized to automatically learn the network structure for pruning. It can effectively combine pruning and quantization operations, avoiding the potential negative impacts of separately implementing pruning and quantization. By calculating the difference between the current parameter values and the target quantization values, quantization errors are reduced through iterative optimization during the training process, making the parameters closer to the selected 2ⁿ values. The pruned network has a regular structure, and quantization to 2ⁿ values makes it highly suitable for hardware implementation as it can be achieved using a shifter. In the post-training compression refinement phase, joint compression operations including channel pruning and low-bit quantization are completed. Experimental results on benchmark datasets such as MNIST, CIFAR-10 and CIFAR-100 show that the framework generates more concise network parameters while maintaining considerable accuracy, demonstrating excellent effectiveness in terms of compression ratio and accuracy. The proposed framework can integrate the complementary aspects of quantization and pruning, and effectively minimize the possible adverse interactions between quantization and pruning.