Assisting Training of Deep Spiking Neural Networks With Parameter Initialization.

Abstract

Spiking neural networks (SNNs) exhibit significant advantages in terms of information encoding, computational capabilities, and power usage. We regard initializing weight distribution as a key problem for effective SNN training. When backpropagation (BP) through time is used in the initial training phase, it has a significant impact on gradient generation. We first derive an asymptotic formula for the response curve of spiking neurons, which approximates the real neuron response distribution. To avoid gradient vanishing, we then provide an initialization technique based on the slant asymptote. Finally, validations on classification tasks on the MNIST and CIFAR10 datasets demonstrate that our strategy can significantly speed up training and improve the model accuracy compared with other initialization methods. Further testing on various neuron configurations and training hyperparameters demonstrates comparable versatility and superiority to other methods. Based on the analyses, some recommendations for SNN training are made.