Synergy of Sight and Semantics: Holistic Visual Understanding With CLIP.

Abstract

Holistic Visual Understanding (HVU), encompassing tasks like intention recognition, emotion analysis, scene understanding, and content moderation, necessitates integrating low-level visual perception ('sight') with high-level semantic reasoning ('semantics'). While large Vision-Language Models (VLMs) like CLIP offer powerful representations, their inherent 'sight' bias limits their direct application to these semantically rich tasks. Our prior work, IntCLIP, addressed Multi-label Intention Understanding (MIU) using a dual-branch architecture but faced challenges with label generation instability (Hierarchical Class Integration - HCI) and limited feature interaction (unidirectional Sight-assisted Aggregation). This paper introduces an enhanced framework that significantly extends IntCLIP to tackle the broader HVU challenge. We propose Semantic Label Refinement (SLR), an iterative, metric-guided process leveraging Large Language Models (LLMs) and quantitative evaluation within the CLIP embedding space to generate stable, optimized semantic labels. We also introduce a novel bidirectional attention mechanism (Symmetric Aggregation) that enables balanced, mutual refinement between sight and semantic feature maps. By evaluating on a comprehensive benchmark spanning MIU, Image Emotion Recognition, Indoor Scene Recognition, and Visual Content Moderation, we demonstrate that our framework not only advances the state-of-the-art in MIU but also achieves superior performance across diverse HVU tasks. This framework provides a unified and robust solution for synergizing sight and semantics, pushing towards more human-like visual intelligence. Code is available at https://github.com/yan9qu/PAMI25-HVU.