Collect-Then-Select: Multispectral Detection Transformer With Collaborative Aggregation and Progressive Refinement.

Abstract

Red-green-blue (RGB)-infrared (IR) object detection predominantly builds upon two architectural paradigms: convolutional neural network (CNN)-based and detection transformer (DETR)-based, both seeking to effectively integrate the infrared modality into well-established RGB-based frameworks. While CNN-based RGB-IR (CNRI) methods typically adopt a "Collect-while-Select" (CWS) strategy to perform simultaneous feature fusion and refinement, existing DETR-based RGB-IR (DTRI) detectors predominantly follow this CWS strategy. This design overlooks the critical potential of query selection and the decoder in refining multimodal features, rendering the model susceptible to interference from irrelevant modality information. Inspired by the hierarchical human visual system, we propose the customized multispectral detection transformer (CS-MDETR), which instantiates a "Collect-then-Select" (CTS) strategy. CS-MDETR comprehensively integrates cross-modal complementary information and progressively refines task-specific multimodal cues through two sequential phases: Collect and Select. In the Collect phase, the dynamic differential compensation (DDC) module adaptively compensates cross-modal complementary information, while the spatial-channel consistency constraint (S3C) module enforces consistency across multiscale features. In the Select phase, the salience-CIoU-aware query selection (SCQS) strategy incorporates position priors and aspect-ratio cues to ensure that the selected queries are better aligned with target regions. Furthermore, the proposed geometric relation encoding (GRE) module is embedded within the decoder layers to model interlayer geometric relations between query predictions, thereby mitigating prediction inconsistencies under the bipartite matching scheme. Extensive experiments on six representative and challenging benchmarks demonstrate the effectiveness of CS-MDETR with the CTS strategy, consistently outperforming state-of-the-art CNRI and DTRI methods.