Temporal local attention with adaptive decoding: Enhancing spiking neural networks for temporal computing applications.

Abstract

The brain-inspired spiking neural networks (SNNs) are considered to have great potential in complex learning due to their rich neural dynamics and high energy efficiency. Their unique mechanisms are naturally suited for performing temporal computing tasks. However, whether SNNs can effectively capture sufficient temporal features solely based on the neural dynamics remains to be explored, as they usually encounter difficulties in long-sequence gradient propagation during training. In this work, we introduce temporal local attention (TLA), which helps SNNs effectively reduce the sequence length, thereby enhancing the model performance and accelerating the training process. Furthermore, we optimize the output process of SNNs by incorporating adaptive decoding (AD) based on finding the correlation between the decoding strategy and model performance. Finally, we combine these two mechanisms (TLA-AD) into complete SNN modeling with training and inference. We use LSTM and a recent SNN model with dendritic heterogeneity (DH-SNN) as baselines on Electroencephalogram (EEG) and natural language processing (NLP) datasets. The experimental results demonstrate that the proposed TLA-AD method significantly enhances the performance of SNNs and accelerates the training without a significant increase of the number of parameters. It achieves state-of-the-art accuracy scores of 93.52% and 93.41% on the DEAP dataset (valence and arousal), as well as competitive accuracy scores of 87.41% and 86.31% on SEED and IMDB datasets, respectively, compared to other SNN methods. This work provides effective optimization approaches for enhancing SNNs with advanced performance in temporal computing tasks.