Multi-modal feature alignment networks for multi-label image classification.

Abstract

Multi-label image classification is a classification task that assigns labels to multiple objects in an input image. Recent research ideas mainly focus on solving the semantic consistency of visual features and label features. However, since images contain complex scene content, the features captured by visual feature extraction networks based on grid or sequence representation may introduce redundant information or lack continuity when identifying irregular objects. In order to fully mine the visual information of complex objects in images and enhance the inter-modal interaction of images and labels, we introduce a flexible graph structure to explore the internal information of objects and design a multi-modal feature alignment (MMFA) network for multi-label image classification. To enhance the context awareness and semantic association of different patch regions, we propose a semantic-augmented interaction module that combines two kinds of visual semantic information with label embeddings for interactive learning. Finally, we refine the dependence between local intrinsic information and overall semantics by redefining semantic queries through semantically enhanced visual spatial features and graph aggregation features. Experiments on three large-scale public datasets: Microsoft COCO, Pascal VOC 2007 and NUS-WIDE demonstrate the effectiveness of our proposed MMFA and achieve state-of-the-art performance.