Graph knowledge distillation with high frequency in homophily and heterophily graphs.

Abstract

Graph neural networks (GNNs) have found extensive applications across various domains, yet they face significant challenges, including high computational costs and deployment difficulties. Graph knowledge distillation has effectively alleviated these challenges; however, existing methods largely overlook the issue of heterophilous graphs. In heterophilous graphs, the high likelihood of edges connecting nodes of different types highlights the significance of high-frequency information within the graph structure. However, existing graph knowledge distillation methods fail to effectively leverage this high-frequency information. To solve this problem, we propose a graph knowledge distillation with high frequency in homophilous and heterophilous graphs, which can distill both high-frequency and low-frequency information, thereby achieving superior performance on both homophilous and heterophilous graphs. Specifically, our method proposes a dual-teacher and dual-student framework, where one pair leverages existing GNN methods to learn low-frequency information, while the other pair utilizes high-frequency information to meet the requirements of heterophilous graphs. In addition, to fully exploit the effective frequency information, our method incorporates contrastive learning. Finally, our method introduces an advanced fusion module to effectively integrate the low- and high-frequency information derived from the student models. Extensive experiments demonstrate that our model achieves satisfactory performance on both homophilous and heterophilous graphs. Moreover, the incorporation of high-frequency information significantly enhances the robustness of the model against noise.