Neuronal attention circuit (NAC) for representation learning.

Abstract

Attention improves representation learning over RNNs, but its discrete nature limits continuous-time (CT) modeling. We introduce Neuronal Attention Circuit (NAC), a novel, biologically inspired CT-attention mechanism that reformulates attention logit computation as the solution to a linear first-order ODE with nonlinear interlinked gates derived from repurposing the wiring of C. elegans Neuronal Circuit Policies (NCPs). NAC replaces dense projections with sparse sensory gates for query-key projections and introduces a sparse backbone network with two heads for computing content-target and learnable time-constant gates, enabling efficient adaptive dynamics. To improve efficiency and memory consumption, we implement an adaptable, sparse, subquadratic Top-K pairwise concatenation mechanism that selectively curates query-key interactions. We provide rigorous theoretical guarantees, including state stability and bounded approximation errors. Empirically, we implement NAC in diverse domains, including irregular time-series, long-range forecasting, lane-keeping for autonomous vehicles, and industrial prognostics. We observe that NAC matches or outperforms in accuracy against several CT state-of-the-art baselines, while being interpretable at the neuron cell level.