MMFormer: Multi-Modality semi-Supervised vision transformer in remote sensing imagery classification.

Abstract

Significant progress has been made in the application of transformer architectures for multimodal tasks. However, current methods such as the self-attention mechanism rarely consider the benefits that feature complementarity and consistency between different modalities bring to fusion, leading to obstacles such as redundant fusion or incomplete representation. Inspired by topological homology groups, we introduce MMFormer, a novel semi-supervised algorithm for high-dimensional multimodal fusion. This method is engineered to capture comprehensive representations by enhancing the interactivity between modal mappings. Specifically, we advocate for the representational consistency between these heterogeneous representations through a complete dictionary lookup and homology space in the encoder, and establish an exclusivity-aware mapping of the two modalities to emphasize their complementary information, serving as a powerful supplement for multimodal feature interpretation. Moreover, the model attempts to alleviate the challenge of sparse annotations in high-dimensional multimodal data by introducing a consistency joint regularization term. We have formulated these focuses into a unified end-to-end optimization framework and are the first to explore and derive the application of semi-supervised visual transformers in high-dimensional multimodal data fusion. Extensive experiments across three benchmarks demonstrate the superiority of MMFormer. Specifically, the model improves overall accuracy by 3.12% on Houston2013, 1.86% on Augsburg, and 1.66% on MUUFL compared with the strongest existing methods, confirming its robustness and effectiveness under sparse annotation conditions. The code is available at https://github.com/LDXDU/MMFormer.