Multimodal LLM for Patient Activity Recognition: Integrating Video, Audio, and Text in Clinical Environments.

Abstract

Accurate recognition of patient activities within hospital environments is essential to improve safety and quality of care. Existing approaches often struggle to correctly recognize complex patient activities and clinical variability. To address these issues, we propose ClinActNet, a multimodal framework based on large language models that combines video, audio, and clinical documentation through a context-aware Consultation Transformer. To enhance patient specific monitoring, we propose three key components: first a patient profile encoder to learn EHR informed context vectors, second a Clinical Knowledge Graph Reasoner for medical logic inference, and third a personalization layer to adapt predictions per patient. ClinActNet is evaluated using more than 672 hours of hospital data covering nine clinically significant activity classes, achieving an accuracy of 89.7% and a precision of 98. 2% in critical safety events. To ensure reproducibility, we also benchmarked ClinActNet on the public dataset (VAST), where our model performed 84.1% accuracy, confirming its robustness beyond private clinical data. Implementation in real world environments reduced overlooked critical incidents by 48%. The output demonstrated the system's capability to facilitate interpretable patient centered AI within healthcare environment.