Average Top-k Aggregate Loss for Supervised Learning.

Abstract

In this work, we introduce the average top- k ( AT_k) loss, which is the average over the k largest individual losses over a training data, as a new aggregate loss for supervised learning. We show that the AT_k loss is a natural generalization of the two widely used aggregate losses, namely the average loss and the maximum loss. Yet, the AT_k loss can better adapt to different data distributions because of the extra flexibility provided by the different choices of k. Furthermore, it remains a convex function over all individual losses and can be combined with different types of individual loss without significant increase in computation. We then provide interpretations of the AT_k loss from the perspective of the modification of individual loss and robustness to training data distributions. We further study the classification calibration of the AT_k loss and the error bounds of AT_k-SVM model. We demonstrate the applicability of minimum average top- k learning for supervised learning problems including binary/multi-class classification and regression, using experiments on both synthetic and real datasets.