DMutDE: Dual-View Mutual Distillation Framework for Knowledge Graph Embeddings.

Abstract

Knowledge graphs (KGs) have caught more and more attention in recent years. Currently, in some practical scenarios, KG embedding (KGE) models are expected to reduce their spatial complexity without losing much performance to address the challenges of storage limitations and knowledge reasoning efficiency. To achieve this, existing works use one or more large and high-performance teacher models to improve the performance of a lightweight student model via knowledge distillation (KD), thus meeting the requirements of some practical complicated applications. However, in resource-constrained scenarios, obtaining high-performance teacher models is challenging due to high training costs and significant storage requirements. Thus, enhancing the student model's performance without large teacher models is crucial. To address this issue, we propose Dual-View Mutual Distillation Framework for Knowledge Graph Embeddings (DMutDE), a distillation framework leveraging mutual learning for peer-to-peer distillation between two KGE models with different architectures. In KGE models, we notice that the way of modeling relational directed edges determines the model view of KGE model for learning KG data. Thus, integrating the model views from two different KGE models by KD into a student KGE model can improve its generalization, so as to increase its performance. To identify an effective dual-view fusion method, we design two modules in the DMutDE framework. Specifically, we design a novel soft-label fusion (SLF) module for noise filtering and response knowledge transfer. Then, we propose an entity embedding distillation (EED) module to distill structural features from each other. Finally, we conduct several comprehensive experiments on the standard open-source benchmarks to demonstrate that our framework achieves the state-of-the-art results. The code is available at https://github.com/RuizhouLiu/DMutDE.