Directly training on quantized model via gradient scale correction for edge device.

Abstract

The deployment of pre-trained models on edge devices often necessitates quantization due to computational resource constraints. Adapting these models to new private data typically involves data uploads for retraining or fine-tuning, which raises significant security concerns. Existing training methods struggle to directly train quantized models on edge devices. To address this, we analyzed the training process of quantized models and proposed a novel training method based on Gradient Scale Correction and Activation Loss (GSC-AL). Initially, we focused on the gradient and weight mismatch in quantized models during training. The proposed Gradient Scale Correction (GSC) efficiently scales the gradient to a defined range, enabling effective weight updates in quantized models. Furthermore, we introduced the activation Loss (AL) in the loss function, effectively alleviating the non-Gaussian activation distribution phenomenon caused by the accumulation of quantization errors and activation truncation problems in the training process based on GSC. Finally, we validate our method on seven common datasets using ResNet18 and MobileNetV2 as baseline models. Experimental results demonstrate that our GSC-AL method significantly enhances model predictive performance, achieving a training accuracy improvement of 41% compared to existing methods. The proposed GSC-AL method is presented at https://github.com/Zdwsan/GSC-AL.