Weird-Net: Weighted Relative Distance Attention for Efficient and Robust Sequence Processing.

Abstract

Sequence processing is a fundamental research area in artificial intelligence (AI) that encompasses various tasks and applications. Existing models-such as recurrent neural networks (RNNs) and transformers-have drawbacks such as slow computation, high complexity, and overfitting. In this article, we propose Weird-Net, a novel sequence processing model that leverages the weighted relative distance (Weird)-attention mechanism. Weird-Net can capture positional inductive relationships more robustly and efficiently than transformers. Moreover, it enables parallel computation and can handle extremely long sequences with near-linear complexity. We conduct extensive experiments on multiple datasets and tasks and evaluate Weird-Net using various metrics. The results demonstrate that Weird-Net achieves state-of-the-art (SOTA) performance on several language modeling benchmarks and surpasses other models in terms of accuracy, speed, and memory usage.