HFFNet: Hierarchical feature fusion network for lightweight RGB-T semantic segmentation.

Abstract

In recent years, the integration of RGB and thermal infrared (TIR) images has shown great potential in advancing semantic segmentation, particularly under challenging conditions such as low-light environments and cluttered backgrounds. However, existing models often suffer from inefficient multi-modal feature fusion and high computational cost. To address these issues and achieve high segmentation accuracy with low complexity, we propose a hierarchical feature fusion network (i.e., HFFNet) for lightweight RGB-T semantic segmentation. Specifically, HFFNet introduces a multi-scale feature fusion (MFF) module that operates on mid-level and high-level encoder features from both modalities. The MFF module employs a parallel-cascade structure with depthwise separable convolutions of varying dilation rates (dilated DSConv) to extract multi-scale features efficiently. Furthermore, a two-stage fusion strategy is adopted: first to merge RGB and TIR features at each individual scale, and then to combine all scales for enhanced cross-modal representation. To further enhance object localization, we design a hierarchical integration (HI) module to align and aggregate multi-level fused features, thereby combining high-level semantic information with low-level spatial details. Extensive experiments on two public RGB-T datasets clearly demonstrate that our HFFNet not only achieves superior segmentation accuracy compared to state-of-the-art methods but also maintains low computational complexity, making it suitable for real-time GPU inference and resource-constrained RGB-T segmentation scenarios.