Physics-Guided Cross-Modal Decoupling with Test-Time Adaptation for Hyperspectral Image Restoration.

Abstract

Hyperspectral image (HSI) restoration tasks including super-resolution, denoising, and inpainting, present significant challenges due to intrinsic spectral-spatial coupling and limited training data availability. Recent advances in RGB image restoration demonstrate that models pretrained on large-scale datasets acquire exceptional generalization capabilities, suggesting potential cross-modal knowledge transfer solutions for HSI recovery. However, existing approaches exhibit two critical limitations: (i) prohibitive computational costs from mandatory fine-tuning procedures, and (ii) inadequate cross-modal adaptation causing spectral distortions. To address these challenges, we propose a Two-Stage Cross-Modal Decoupling Network (CMDN) achieves spectral-faithful HSI restoration without fine-tuning the pretrained RGB prior; instead, we perform unsupervised test-time learning only on a lightweight spectral rectifier for sample-specific spectral calibration. Our methodology introduces two fundamental innovations: First, we develop a theoretically grounded framework using Singular Value Decomposition (SVD) to decouple HSIs into orthogonal spatial coefficients and spectral bases. This decomposition enables strategic reconfiguration of spatial coefficients into pseudo-RGB formats through band reorganization, facilitating direct deployment of frozen RGB-pretrained models for spatial textures recovery while preserving spectral integrity. Second, we propose a Physics-Motivated Spectral Rectifier (PMSR) that dynamically adjusts spectral reconstruction weights using spatial gradient priors, correcting spectral deviations through physics-consistent optimization rather than explicit error modeling, thereby achieving superior spectral fidelity. Comprehensive experiments confirm our method's superiority in both spatial reconstruction accuracy and spectral consistency over state-of-the-art techniques. Code is available at: https://github.com/QYo-Liu/CMDN.