RoSwish: A novel Rotating Swish activation function with adaptive rotation around zero.

Abstract

This paper innovatively proposes the Rotating Swish (RoSwish) activation function to address the limitations of existing activation functions in terms of nonlinear expressive capabilities, which is mathematically defined as f(x)=(x+α)·sigmoid(β·x)-0.5·α. RoSwish integrates the advantageous features of Rectified Linear Unit (ReLU), Gaussian Error Linear Unit (GELU), Exponential Linear Unit (ELU), Parametric ReLU (PReLU), and Swish activation functions, achieving a unification of smoothness, gradient stability, self-gating effects, adaptability, etc. The core design is based on the Sigmoid gating mechanism, which introduces learnable parameters α and β to dynamically adjust the rotation degree of the Swish function, thereby obtaining more flexible and powerful nonlinear expressive capabilities. Experimental results demonstrate that RoSwish outperforms existing activation functions across multiple application domains. In MNIST classification, it achieves at least a 16.02 % improvement in performance; in MNIST autoencoding, it reduces the MSE by at least 5 % under low learning rate; in tweet tagging, it improves performance by at least 3.65 % under medium to high learning rate, and when combined with batch normalization under low learning rate, it achieves a 30.19 % improvement; in ETTm2 time series prediction, it improves performance by at least 6.06 % under medium to low learning rate. Additionally, inspired by non-stationary stochastic process theory, this study designs a series of activation functions based on non-stationary sine waves, aiming to enhance the neural network's ability to represent non-stationary signals. In the MNIST Classification task, this series of activation functions improves performance by at least 6.55 %. Code is available in https://github.com/weirdoZHANG/RoSwish.