Multi-Branch Tree-based Fusion Neural Architecture Search with Zero-Cost Screen for Multi-Modal Classification.

Abstract

Multi-modal classification (MMC) leverages effective fusion of information from diverse modalities to achieve superior classification performance. Existing fusion methods, however, rely on expert-designed architectures that demand substantial domain expertise and computational resources, with fixed topologies that offer limited structural flexibility across tasks and datasets. Although neural architecture search (NAS)-based fusion methods have been proposed to automatically discover high-performing architectures, these approaches are computationally expensive and predominantly rely on pairwise modality combinations, which fail to capture complex multi-variable correlations, limiting the architectures' expressiveness and flexibility. Consequently, there remains a lack of multi-modal fusion frameworks that can simultaneously achieve high efficiency and high accuracy. To break through these bottlenecks, we propose a multi-branch tree-based fusion neural architecture search framework (MBTF-NAS). For performance enhancement, MBTF-NAS employs a multi-branch tree-structured encoding strategy that enables dynamic and computationally efficient exploration of fusion topologies and substantially strengthens cross-modal interaction. A learnable model-level attention weighting mechanism further emphasizes informative modalities, improving the overall quality of multi-modal feature fusion. For efficiency improvement, MBTF-NAS leverages zero-cost proxy metrics for architecture evaluation, enabling rapid identification of high-potential candidates while dramatically reducing computational overhead. We conducted a comprehensive evaluation of MBTF-NAS on seven representative multi-modal benchmarks. The experimental results demonstrate that MBTF-NAS consistently outperforms state-of-the-art approaches, highlighting its effectiveness and generalizability.