Heterogeneous Feature Knowledge Distillation based on Enhanced Feature Projector Correlation.

Abstract

Knowledge Distillation (KD) is a widely used technique to enhance model performance. However, most existing methods are designed under the assumption that the teacher and student models belong to the homogeneous architecture -particularly those relying on intermediate feature KD. In practice, mainstream model architectures exhibit significant differences in feature structure and representation, making feature alignment challenging. To address this issue, we propose a heterogeneous feature knowledge distillation based on enhanced feature projector correlation. Specifically, we first project both teacher and student features into the structurally consistent latent space to measure their semantic distribution. To mitigate the potential loss of semantic relevance caused by feature decorrelation during projection, we introduce a cross-space fusion mechanism to model the correlation between the original and latent features. Furthermore, to better leverage the rich representations from deep teacher layers, we design a multi-level feature knowledge distillation loss that guides the student by regressing features from multiple semantic levels. Finally, to reduce the noise introduced by the inherent limitations of student features and the projection process, we incorporate a denoising mechanism based on diffusion models to enhance class-wise discriminability in the student feature space. We conducted extensive experiments on the CIFAR-100 and ImageNet datasets to validate the effectiveness of the proposed distillation method across various mainstream architectures, including Convolutional Neural Networks (CNNs), Transformers, and Multilayer Perceptrons (MLPs) for image classification. In addition, we performed semantic segmentation experiments on the Cityscapes dataset to further verify the performance of our method. Code is available at https://github.com/chenKP/HFKD-Diff.