Balanced multi-modality knowledge mining for RGB-infrared object detection.

Abstract

RGB-Infrared object detection aims to fuse the complementary information of two modalities to improve the accuracy and robustness of the detector. Given the advantages of transformer in modeling long-range dependencies, transformer-based cross-modality fusion methods have been continuously proposed and achieved satisfactory results. However, existing methods face two major challenges: 1) it is difficult to balance the mining of intra-modality specific knowledge and inter-modality complementary knowledge; 2) a single attention layer only models the relationship between token features of the same receptive field, thus failing to capture the intrinsic relationship between objects at different scales and lacking the ability to focus on both local and global information. To this end, we propose a balanced multi-modality knowledge mining method. Specifically, we design a dual attention knowledge mining (DAKM) module, which explicitly mines intra- and inter-modality key knowledge through self-attention and cross-attention, respectively. In addition, we introduce multi-scale information into the attention layer of DAKM, which not only extracts multi-scale object features but also retains both local and global information. Then, we fuse the intra- and inter-modality features obtained by DAKM using the scene-aware adaptive interaction module. The module employs differential and scene information to focus on object-related feature fusion. Finally, the cross-layer feature refinement module is utilized to aggregate different fusion layers to further enhance the feature representation. Extensive experiments in multiple scenes demonstrate that our method outperforms existing state-of-the-art RGB-Infrared object detection methods.