Generalized Cross-Domain Framework for Gesture Recognition via Wrist-Worn Sensing.

Abstract

Wearable sensing technology offers a natural and convenient means of human-computer interaction, particularly for gesture recognition, yet domain shifts in wrist-worn single-site sensing pose significant challenges for cross-domain gesture recognition. To address this, we proposed a generalized cross-domain framework for fine-grained gesture recognition using wrist-worn single-site sensing. Concretely, we presented a Multi-Branch Network, which combines feature-level multimodal fusion with enhanced inter-modal interaction to effectively capture fine-grained gestures. To this end, we constructed a multimodal dataset, which comprises fifteen static and eighteen dynamic gestures. Furthermore, we developed five fine-tuning strategies and evaluated them across the paradigms of cross-session, cross-subject, cross-gesture, and cross-modality. Through comprehensive analyses, this study provides valuable insights into the selection of optimal fine-tuning strategies and elucidates the internal mechanisms underlying multiple cross-domain paradigms. To investigate the intricate trade-off between recognition accuracy and computational cost, we applied nonlinear least squares to construct the Accuracy-Cost trade-off functions. Experimental findings indicated that the optimal transfer learning ratios for these cross-domain paradigms ranged from 6.1% to 9.0%, with most clustering around 9.0%, offering a valuable reference for determining optimal transfer learning ratios within diverse cross-domain scenarios. Additionally, we implemented a real-time online gesture recognition system, validating the feasibility of our approach through preliminary tests in real-world scenarios. In conclusion, this study serves as a preliminary investigation into the application of wrist-worn single-site sensing for fine-grained gesture recognition.