Dynamic Multi-Biosignal Fusion for Detecting the Mental States of Drivers and Passengers in Vehicles.

Abstract

As transportation systems grow in complexity and autonomous vehicle technologies advance, detecting the mental states of drivers and passengers is crucial for enhancing traffic safety and user experience. Recent studies have explored multi-biosignal fusion to monitor these states. Despite these efforts, existing methods fail to reflect the complexity and usefulness of various modalities, and do not consider the informativeness of biosignals based on signal quality or the contribution of each modality as biomarkers. To address this issue, we introduce dynamic multi-biosignal fusion (DMBF) to detect the mental states of drivers and passengers. DMBF employs a dynamic gate mechanism that estimates reliability based on data quality through confidence-aware learning, integrating this information into the learning process. Moreover, a spatial-temporal attention module is utilized to capture and combine key biosignal patterns across channels and time. Extensive evaluations were conducted on five datasets related to motion sickness, drowsiness, distraction, emotions, and sustained attention. Each dataset yielded F1 scores of 0.5569, 0.7187, 0.6647, 0.9378, and 0.8092, respectively, outperforming existing baseline models. The experimental results and ablation studies have demonstrated that DMBF is a more robust and versatile method for detecting the mental states of drivers and passengers. DMBF provides a foundation for future enhancements in traffic safety and passenger experience and has the potential for broader applications in multi-biosignal monitoring systems.