A Multimodal Sensor Fusion Approach for Mental Health Detection Using Mobile, Wearable, and IoT Sensors.

Abstract

Depression and anxiety are among the most prevalent mental health disorders, yet many cases remain undetected due to the lack of continuous and context-aware monitoring in everyday life. Prior work has demonstrated the potential of mobile and wearable devices for passive mental health sensing; however, their inconsistent usage limits coverage of in-home routines and context-rich behaviors. To address these limitations, we propose a multimodal sensing approach that integrates data from mobile phones, wearable devices, home IoT sensors, and smart speakers to capture daily behavioral patterns in real-world settings. We conducted a 30-day in-the-wild study with 20 participants aged 20 to 30 living in single-person households, a group at elevated risk for depression and anxiety. We developed a multimodal pipeline that extracts and aligns features across all modalities and applies both machine learning and deep learning models. Our results show that multimodal integration generally improves performance over single-modality baselines for both depression and anxiety detection. Deep learning models achieved AUROCs of 0.690 for depression and 0.634 for anxiety in generalized settings, while personalized tree-based models performed best, achieving up to 0.736 and 0.728, respectively. We publicly release our code and feature sets to support reproducibility and further research in pervasive mental health.