Mosaic Pattern Excavation Transformer for Spectral Imaging.

Abstract

Single spectral image demosaicing for multispectral filter array (MSFA) is an essential task in spectral imaging, aiming to recover a mosaic-free spectral image from its mosaic raw counterpart. Existing deep learning-based methods typically improve the reconstruction performance by indiscriminately stacking CNN-based blocks, failing to effectively handle the intertwined spatio-spectral correlations caused by spatial sub-sampling and spectral aliasing. In this paper, we propose Mosaic Pattern Excavation Transformer (MPEFormer) to achieve better reconstruction by effectively modelling the intertwined spatio-spectral correlations. Specifically, the proposed three-branch model integrates low-frequency information, edge information, and fine high-frequency details essential for spectral image reconstruction, with the third branch serving as the core component. In this branch, we design the Dual Fusion Self-attention Block (DFSAB) and the Mosaic Pattern-guided Spectral Modulation Module (MPSM). DFSAB incorporates the Mosaic Pattern Excavation Self-attention (MPESA) mechanism, which effectively captures non-local spatio-spectral correlations induced by the MSFA pattern distributed across the whole image, thereby enhancing the expressive capability of the model. By dynamically integrating various MSFA pattern-related dependencies, MPSM enables adaptive recalibration of spectral information. Extensive experimental results demonstrate the effectiveness of our MPEFormer, highlighting its greater potential over the state-of-the-art MSFA demosaicing methods. The code will be uploaded at https://github.com/Matsuri247/MPEFormer.