FIT: Enhancing multimodal knowledge graph completion via fine-grained interaction and TriConvTransformer.

Abstract

Multi-modal Knowledge Graphs (MMKGs) contain features in various forms, such as text, audio, and images. MMKGs frequently encounter the challenge of incomplete data. To overcome its inherent data incompleteness, the goal of multi-modal knowledge graph completion (MMKGC) is to predict missing triples in multi-modal knowledge graphs by integrating multi-modal information of entities into the model. Existing MMKGC methods suffer from insufficient fine-grained interactions between modalities and insufficient interactions between entities and relations. To address these problems, we propose a new framework, enhancing MMKGC via Fine-Grained Interaction and TriConvTransformer (FIT). The framework fully utilizes fine-grained information between modalities by obtaining and fusing the embedding representations of various modalities through a fine-grained modal hierarchical interaction (FMHI). Meanwhile, we propose a new decoder, TriConvTransformer, which can effectively capture deep interactions between entities and relations. In addition, we introduce the cross-modal self-attention contrastive learning (CM-SACL) and adaptive loss interaction (ALI) mechanisms, which can effectively fuse information from different modalities. Experiments on the standard MMKGC baseline show that our method surpasses the latest model, highlighting its excellent performance.