MDM: Modality decoupling for visible and infrared Mamba-based object detection.

Abstract

Object detection methods that fuse visible and infrared modalities significantly enhance detection accuracy and robustness by leveraging complementary information from both modalities. However, existing methods suffer from the following shortcomings: 1) They are limited to specific scenarios (e.g., conventional or remote sensing small targets), making it difficult to meet the detection needs of multi-scale targets; 2) CNN-based fusion methods, constrained by static convolutional kernels, struggle to handle fusion tasks involving significant modality differences; 3) Transformer-based fusion methods, limited by quadratic computational complexity, hindering practical application. To enhance the model's adaptability to both conventional and remote sensing scenarios, we design the Global and Local Mamba modules, which extract global contextual information from a global perspective and construct local receptive fields from a local perspective, respectively. Secondly, to reduce the computational overhead of long-sequence modeling across modalities during modality fusion, we introduce the Modality Decouple module to decouple features into modality-agnostic and modality-specific features. Based on the degree of modality difference among the decoupled features, differentiated fusion strategies are implemented, effectively reducing the computational resource consumption associated with redundant modality-agnostic features during modality fusion. For modality-agnostic features with small modality differences, we employ a lightweight Spatial Attention module for simple modeling fusion. For modality-specific feature with significant modality differences, we utilize efficient the Spatial Mamba and Channel Mamba modules to perform complex modeling fusion from spatial and channel dimensions, respectively. Simultaneously, to address the potential memory challenges of Mamba in cross modal long sequence modeling, we propose a cross modal interaction paradigm for Mamba within the Spatial and Channel Mamba modules. This paradigm achieves efficient cross modal fusion by interleaving cross modal features at corresponding positions. Extensive experiments on three conventional scene object datasets (FLIR, LLVIP, M3FD) and two remote sensing small object datasets (VEDAI, DroneVehicle) demonstrate that our MDM not only achieves SOTA performance efficiently but also addresses the challenges of small object detection, exhibiting strong robustness. To our knowledge, this study explores the interaction paradigm of Mamba for cross modal fusion. Compared to the simple interaction schemes in existing work, we further unlock the potential of Mamba in cross modal fusion tasks. The code is available at: https://github.com/SEUZYC/MDM.