BiGSTF-Net: inter-modal mutual guidance and intra-modal spatio-temporal fusion for EEG-fNIRS cognitive classification.

Abstract

Electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) provide complementary temporal and spatial information for brain-computer interfaces (BCIs). However, effectively exploiting this complementarity remains challenging due to the heterogeneous characteristics of electrophysiological and hemodynamic signals. In this study, we propose BiGSTF-Net, a multimodal architecture designed to improve cognitive state decoding by leveraging the complementary properties of EEG and fNIRS signals. The proposed framework first employs heterogeneous spatio-temporal feature extractors to capture temporal-oriented and spatial-oriented representations from each modality. To facilitate cross-modal interaction, a Modal Residual Interaction Unit (MRIU) is introduced to enable bidirectional inter-modal guidance while preserving modalityspecific characteristics. Subsequently, a Spatio-Temporal Gated Unit (STGU) performs intra-modal feature integration to produce compact and discriminative representations. Experiments conducted under cross-session evaluation on multiple BCI datasets demonstrate that BiGSTF-Net consistently outperforms representative multimodal fusion baselines. Ablation studies further verify the effectiveness of the proposed architectural components, while visualization analyses reveal activation patterns that align with known neurophysiological characteristics of EEG and fNIRS signals. These results indicate that the proposed framework provides an effective approach for multimodal neural signal decoding.