Group-wise relation mining for weakly-supervised fine-grained multimodal emotion recognition.

Abstract

With the advancement of multimedia technologies, fine-grained emotion annotation has made real-time, dynamic, and precise recognition possible, posing new challenges for traditional emotion recognition methods. On the one hand, fully-supervised learning methods rely on costly and labor-intensive annotations. Self-supervised learning methods face a natural gap between auxiliary tasks and emotion classification tasks, making it difficult to achieve outstanding performance. On the other hand, existing weakly-supervised paradigms struggle to achieve ideal fine-grained emotional state recognition solely based on trial-wise coarse annotations, resulting in ambiguous boundaries between segments lacking detailed supervision. To address this issue, this paper proposes a group-wise relation mining method (GRM4WFER) designed to enhance fine-grained emotion recognition under weak supervision. GRM4WFER aims to bolster the discriminability of boundaries within challenging segment samples by broadening the information receptive field. It achieves this by jointly exploring relationships between segments across multiple trials of the same subject. Through the integration of a relative distance constraint based on Bayesian probabilistic encoding, GRM4WFER enhances boundary discriminability, particularly for segments that are difficult to activate. Additionally, we employ relation reasoning strategies to facilitate an adaptive exploration of valuable co-occurring semantic contexts with an extended receptive field. Extensive experiments on two fine-grained multimodal emotion datasets demonstrate that our proposed model outperforms existing state-of-the-art baselines. The code is publicly available at https://github.com/jinxyBJTU/GRM4WFER_2024.