A Feature-Augmented Transformer Model to Recognize Functional Activities From in-the-Wild Smartwatch Data.

Abstract

Human activity recognition (HAR) from wearable sensor data traditionally identifies atomic movements (e.g., sit, stand, walk). However, many medical fields require recognizing functional activities-higher-level, goal-directed behaviors (e.g., errands, socialize, work). Functional activity recognition is critical for cognitive health assessment, rehabilitation, post-surgical recovery, and chronic disease management, yet remains largely unexplored due to its inherent complexity and variability for in-the-wild settings. This work addresses these challenges by investigating methods for functional HAR and introducing a novel approach that augments feature representations with feature token-transformer embeddings to improve classification performance. We compare a range of machine learning and deep learning methods, analyzing their ability to generalize across a diverse population. Additionally, we present ArWISE, a large-scale functional activity dataset collected longitudinally from n = 503 participants, consisting of over 32 million labeled points. Our experiments demonstrate the advantages of incorporating feature embeddings into functional HAR models, particularly in handling real-world variability and data sparsity. By bridging the gap between atomic movement recognition and functional behavior modeling, this work lays the foundation for more advanced, behavior-aware applications in digital health and human-centered AI.