Visual dialog with semantic consistency: An external knowledge-driven approach.

Abstract

As a crucial subfield of intelligent human-machine interaction, visual dialog involves answering multi-turn questions based on visual content and history dialog, presenting significant technical challenges. Although recent works have made steady progress in visual dialog, several issues remain to be addressed. First, there are bias issues in fine-grained multimodal modeling, including information asymmetry and representation inconsistency, which lead to incomplete information understanding and decision-making biases during question answering. Second, previous visual dialog models relying on external knowledge suffer from poor knowledge quality and insufficient knowledge diversity, which introduce noise into the model and undermine the accuracy and coherence of the question responses. In this work, a novel semantic consistency visual dialog model enhanced by external knowledge (SCVD+) is proposed to cope with these challenges. Specifically, fine-grained structured visual and textual scene graphs are constructed to mitigate the issue of information asymmetry, which equally prioritize both linguistic and visual elements, ensuring a comprehensive capture of object relationships in images and word associations in dialog history. Furthermore, beneficial external knowledge sourced from a commonsense knowledge base is integrated to alleviate the representation inconsistency in multimodal scene graphs and to promote the model's interpretability. Finally, implicit clues are derived from pre-trained large models and integrated with explicit information from scene graphs using a proposed dual-level knowledge fusion and reasoning strategy, which ensures the diversity of external knowledge and enhances the model's reasoning capability in complex scenarios. Experimental results demonstrate the effectiveness of our method on the public datasets VisDial v0.9, VisDial v1.0, and OpenVisDial 2.0.