Bridging Subjectivity in Affective Explanation Captioning via Consensus-Prompted Emotion Reasoning.

Abstract

Affective Explanation Captioning (AEC) aims to perform viewer-centered visual emotion analysis by not only identifying the emotions evoked by an image but also explaining their underlying causes. Prior efforts have achieved promising results by fine-tuning LLMs on affective data; however, two key challenges remain: 1) the inherent subjectivity of human emotion leads to diverse interpretations of the same image, making it difficult for models to catch dominant emotions; and 2) the affective gap between abstract emotions and concrete visual content hinders models from capturing both semantic and emotional aspects effectively. To tackle these challenges, we propose Consensus-Prompted Emotion Reasoning (CPER), a new framework that explicitly models emotional diversity and enforces emotional-semantic alignment. Inspired by psychological studies, we observe that common emotional patterns often emerge within certain groups, which we refer to as affective consensus. Capturing this consensus across varying levels is helpful for bridging the subjectivity in AEC. Specifically, we introduce a consensus-based bucket prompt, which depicts the consensus level of each emotional perspective, serving as a control signal to adjust the emotion reasoning. To reconcile abstract emotion understanding and concrete visual grounding, we design a dual-space representation, where a CLIP encoder extracts objective semantic evidence and an emotion encoder captures abstract affective cues for AEC. Furthermore, an emotion consistency learning strategy is devised, which explicitly aligns the generated explanation with the input image and the emotion label, ensuring both emotionally and semantically grounded explanations. Extensive experiments on three benchmark datasets, ranging from visual arts (ArtEmis v1.0 and ArtEmis v2.0) and real-world images (Affection), demonstrate the effectiveness of our CPER in terms of emotional diversity and semantic coherence compared to state-of-the-art methods. Our code is publicly available at https://github.com/songpipi/CPER.