MixGradient: A gradient-based re-weighting scheme with mixup for imbalanced data streams.

Abstract

A challenge for contemporary deep neural networks in real-world problems is learning from an imbalanced data stream, where data tends to be received chunk by chunk over time, and the prior class distribution is severely imbalanced. Although many sophisticated algorithms have been derived, most of them overlook the importance of gradient information. From this perspective, the difficulty of learning from imbalanced data streams lies in the fact that the gradient estimated on an uneven class distribution is not informative enough to reflect the critical pattern of each class. To this end, we propose to assign higher weights on the training samples whose gradients are close to the gradient of corresponding typical samples, thus highlighting the important samples in minority classes and suppressing the noisy samples in majority classes. Such an idea can be combined with Mixup, which exploits the interpolation information of data to further compensate for the information of sample space that the typical samples do not provide and expand the role of the proposed re-weighting scheme. Experiments on artificially induced long-tailed CIFAR data streams and long-tailed MiniPlaces data stream show that the resulting method, termed MixGradient, boosts the generalization performance of DNNs under different imbalance ratios and achieves up to 10% accuracy improvement.