PolaFormer++: Polarity-Aware Linear Attention with Channel-wise Spikiness.

Abstract

Linear attention has recently emerged as an efficient alternative to Softmax-based self-attention by reducing quadratic computational complexity to linear. However, existing linear attention mechanisms still suffer from substantial performance degradation, primarily due to two limitations: the loss of negative elements in query and key vectors caused by non-negative feature maps, and the failure to preserve the spikiness property as in Softmax attention, which weakens token discriminability. To address these issues, we propose PolaFormer++, an enhanced polarity-aware linear attention mechanism with Channel-wise Spikiness. Specifically, we first introduce the polarity-aware linear attention to recover both same- and opposite-polarity interactions via polarity decomposition, preserving complete similarity information. Secondly, we propose a Polarity-aware Channel-wise Spiky (PaCS) feature map, which incorporates two learnable temperature vectors into the dual polarity flows to endow each channel with a distinct degree of spikiness, thereby enabling adaptive control of channel-wise attention across polarity flows. Moreover, we provide the first theoretical analysis of how channel-wise feature maps influence the entropy of attention weight, revealing conditions under which a feature map preserves entropy-reducing behavior. Comprehensive experiments on image classification, object detection, semantic segmentation, image super-resolution, diffusion-based generation, and 3D novel view synthesis demonstrate that PolaFormer++ achieves a superior balance between efficiency and accuracy, establishing a strong theoretical and practical foundation for linear attention in vision Transformers. Code is available at this https URL.