Spike-hammer: An efficient spike-driven hybrid architecture for multi-modal emotion recognition with physiological signals.

Abstract

Multi-modal emotion recognition has attracted increasing attention due to its superior performance. However, Artificial Neural Network (ANN)-based multi-modal approaches often incur substantial computational and energy costs, hindering their deployment, especially on resource-constrained edge instrumentation platforms. While Spiking Neural Networks (SNNs) offer energy-efficient potential, existing SNN architectures struggle to effectively handle the heterogeneity and interactivity between modalities. To tackle these challenges, we propose Spike-Hammer, a novel spike-driven hybrid architecture integrates Conv-based blocks for spatial-temporal feature extraction with Transformer-based blocks for facilitating multi-modal fusion. By leveraging a spike-driven paradigm, our approach takes advantage of sparse additive operations that eliminate energy-intensive multiplications. In Conv-based blocks, we develop modules to accommodate modal heterogeneity, including a spatial-temporal fusion patch embedding module for complex electroencephalogram (EEG) signals and a lightweight variant for other modalities. These heterogeneous modules mitigate underfitting and homogenization, thereby enhancing subsequent multi-modal fusion. In Transformer-based blocks, we propose a token-channel fusion self attention mechanism to capture fine-grained global features across two dimensions. These features are then seamlessly integrated via an energy-negligible cross attention mechanism, uncovering interactivity without incurring additional energy costs. Extensive experiments on the DEAP and MAHNOB-HCI datasets demonstrate state-of-the-art performance on the arousal dimension under leave-one-subject-out cross-validation, while achieving an average 16 ×  reduction in energy consumption compared to ANN-based methods. Code is available at https://github.com/zhusiyu-rick/Spike-Hammer.git.