Non-target divergence hypothesis: Toward understanding modality differences in cross-modal knowledge distillation.

Abstract

Compared with unimodal knowledge distillation (KD), cross-modal KD is more challenging due to modality differences. However, how such differences affect cross-modal KD remains insufficiently understood. In this paper, we propose the Non-Target Divergence Hypothesis (NTDH), which states that modality differences mainly affect cross-modal KD through divergences in non-target class predictions, and that smaller non-target divergence leads to better student performance. We further provide a theoretical analysis based on Vapnik-Chervonenkis (VC) theory, deriving an upper bound on the cross-modal KD error that supports the proposed hypothesis. Extensive experiments on five cross-modal datasets validate the effectiveness, generality, and practical relevance of NTDH.