Enhancing graph neural networks through universal self-knowledge distillation.

Abstract

Compared to graph distillation, graph self-distillation has gained increasing attention due to its lower memory and time requirements. In recent years, various methods have explored the use of handcrafted soft labels to improve student performance in less time than graph knowledge distillation (KD). These approaches generally acquire labels through auxiliary branches or contrastive learning. While they are faster than graph knowledge distillation, they still involve considerable overhead when compared to directly training models. To address the limitations of these methods, we propose a general and effective soft label acquisition method called Universal Graph Self-Knowledge Distillation(UGKD). Unlike traditional knowledge distillation, UGKD enables the model to distill knowledge from its own intermediate outputs by using the student's target logit as soft target labels and generates soft non-target labels based on the ranks of intermediate features according to Zipf's law. The UGKD method is the first graph self-knowledge distillation method that works well with both MLP and GNN models with very little extra time and memory usage, leading to state-of-the-art results. The code is available at https://www.github.com/2251821381/UGKD.