Dendritic nonlinearities mitigate communication costs.

Abstract

Why have modern artificial neural networks not adopted the nonlinear dendritic structures found in biological brain cells, and what is the core advantage of such active dendritic units? While early studies suggested that dendritic nonlinearities can enhance learning capabilities by boosting capacity, we provide empirical evidence reassessing this. Using extensive machine learning experiments, we show that dendritic nonlinearities in neural networks offer comparable learning capacity to standard point-neuron models when controlled for parametric complexity. Instead, we believe that their key advantage lies in enabling network scaling while substantially reducing communication costs via localized feature aggregation. Our experiments and analysis suggest that incorporating nonlinear dendritic architectures can significantly lower memory access or data transfer overhead during neural network inference-the primary sources of energy consumption in modern AI systems-and potentially during training as well. We argue that these insights motivate further theoretical and architectural exploration of dendritic-like structures in artificial neural networks.