EMBC Special Issue: CogniFuse and Multimodal Deformers: An Extended Study on Benchmarking and Modeling Biosignal Fusion.

Abstract

Human physiological signals reflect complex biological processes and provide important insights into physical and mental states. Extracting such information from biosignal data collected during everyday activities holds great potential for real-time monitoring of physical and mental states, but is challenging due to noise and artifacts. To address this, we introduce CogniFuse, the first publicly available multi-task benchmark for multimodal biosignal fusion in such unconstrained environments. In addition, we develop a comprehensive benchmarking pipeline that emphasizes comparability, reproducibility, accessibility, and usability, while demonstrating robustness across architectures, tasks, and model sizes. For many biosignals, particularly electrophysiological signals, information in different frequency bands is critical for assessing physiological states. Motivated by this, we propose a family of Multimodal Deformer models that capture multi-level power features along with both long- and short-term temporal dependencies across multiple biosignal modalities. In particular, our Multi-Channel Deformer achieves the highest average benchmark score, outperforming all models of comparison. By advancing multimodal biosignal fusion in everyday settings, this work supports real-time monitoring of physical and mental states outside highly controlled clinical conditions. To ensure full transparency and reproducibility, and to facilitate future research, all code and data are made publicly available.