Cross-modal transformer fusion via local sampling for drone RGB-infrared object detection.

Abstract

Visible and infrared feature fusion plays a critical role in drone-based RGB-IR object detection. Previous studies have shown that insufficient feature fusion limits its performance, particularly in detecting objects with confusing structures or small objects. However, most transformer-based cross-modal fusion methods rely on downsampled, low-resolution intermediate features for interaction, which can lead to the loss of local details and contextual information, ultimately reducing detection accuracy. To address this issue, we propose a novel Cross-Modal Transformer Fusion via Local Sampling (CTFLS) network based on the two-stream strategy, which effectively captures both intra-modal and inter-modal information from high-resolution feature maps. Specifically, we introduce a Local Cascade Transformer (LCT) module, consisting of a cascade of local intra-modal and cross-modal transformer blocks. This module enables the network to extract richer modality-specific and modality-invariant information from high-resolution intermediate features. Within the local cross-modal transformer block, local fusion is achieved by computing cross-attention on sampled points of local features, exploring deeper complementary relationships between different modalities. Additionally, we propose a Detail-Enhanced Mixed-Convolution Attention (DMA) module to enhance the representation capabilities of the fused features, particularly in capturing subtle textures and global contextual dependencies. We perform experiments on the DroneVehicle and FLIR datasets. Experimental results show that our method outperforms other state-of-the-art methods in the drone-based object detection task, especially in effectively detecting small objects and distinguishing objects with similar structures.