Hierarchical Active Learning with Label Proportions on Data Regions.

Abstract

Learning classification models from real-world data often requires substantial human effort devoted to instance annotation. As the instance-based annotating process can be very time-consuming and costly, we propose a novel active learning framework that builds classification models from human-annotated <i>regions</i>. A region is defined by a set of conjunctive patterns that are formed by value ranges over the input features. A region label is a human assessment of the class <i>proportion</i> in the data population covered by the region. By leveraging <i>learning from label proportions</i> algorithms, regions and their class proportions can be used to train instance-based classification models. However, the key challenge is that in practice, very few regions are defined already. Therefore, to identify regions important for model learning, we design a <i>hierarchical active learning</i> (HAL) framework, which actively builds a hierarchy of regions. Similar to the decision-tree learning process, our approach progressively divides the input data space into smaller sub-regions, solicits labels for the new regions, and retrains the base classification model with all the leaf regions. And we further develop a <i>multi-hierarchy</i> (forest) solution, which builds multiple shallower hierarchies that have more informative, diverse, and simpler regions. We evaluate our HAL framework on numerous impactful classification datasets as well as on a real user study - on the survival analysis of colorectal cancer patients. The results demonstrate that region-based active learning methods can learn high-quality classifiers from very few labeled regions. Hence, our framework is shown very effective in reducing the human annotation effort needed for building classification models.