TCPFMC: Trustworthy Cyclic Progressive Fusion for Multimodal Classification.

Abstract

The exponential growth of multimodal data has facilitated significant advances in multimodal classification techniques. Most multimodal classification algorithms aggregate information from different modalities into a unified representation to capture the holistic information. However, existing fusion mechanisms overly rely on high-quality data, which compromises the robustness of the model. Meanwhile, the fusion process can cause the loss of modality-specific information due to inherent modality differences. In this work, we propose a trustworthy cyclic progressive fusion method for multimodal classification, named TCPFMC. To enhance robustness and reduce reliance on high-quality data, we designed a modality energy score to evaluate the confidence of each modality, quantifying the informativeness of the modalities. In addition, for efficient integration of modality-specific information, we propose a novel cyclic progressive fusion approach that combines modality information at a fine-grained level, enhancing the overall performance of the model. Extensive experiments on six multimodal datasets show that TCPFMC outperforms state-of-the-art methods. The original code is available at https://github.com/aoli-hrbust/TCPFMC.