MSFI: Multi-timescale spatio-temporal features integration in spiking neural networks.

Abstract

Dynamic vision sensors (DVS) asynchronously encode the polarity of brightness changes with high temporal resolution and a wide dynamic range, making them ideal for capturing temporal information. Spiking neural networks (SNNs) are well-suited for handling such event streams due to their inherent temporal information processing capability. However, existing SNNs only transmit membrane potential across timesteps, neglecting spatial dependencies and failing to extract complex temporal features. To overcome this limitation, we propose a novel plug-and-play module, the Multi-timescale Spatio-temporal Features Integration (MSFI) module. MSFI is specifically designed to extract various spatiotemporal features and fuse them with the original spiking features to enhance the representative capabilities of SNNs. MSFI comprises the Short-term Spatio-temporal Module (SSM) and the Long-term Spatio-temporal Module (LSM), which extract short-term and long-term spatio-temporal features. Our proposed MSFI improves the performance of SNNs on several neuromorphic and static datasets, including CIFAR10-DVS, DVS128 Gesture, DVS128 Gait, CIFAR10/100, and ImageNet datasets. Experimental results on these datasets show that our MSFI significantly outperforms the baselines. Our codes are available at https://github.com/dfxue/MSFI.