Fundamental performance bounds on reservoir computing.

Abstract

Reservoir computing (RC) exploits the intrinsic dynamics of a chaotic system, called the reservoir, to perform various time-varying functions. An important use-case of RC is the generation of temporal sequences via a trainable output-to-reservoir feedback loop while leaving the reservoir intact, providing a hopeful premise where biological neural networks could harness unstructured groups of neurones for functionality. Despite the promise of RC in various domains, an understanding of why RCs fail in certain scenarios is lacking. Here we formulate an existence condition for a feedback loop that produces the target sequence. We next demonstrate that, given a sufficiently large network reservoir, two separate factors are needed for successful training: global network stability of the target orbit, and the ability of the training algorithm to drive the system close enough to the target, which we term "reach." By training a reservoir over a range of target output amplitudes and periods, we verify that reach-limited failures depend on the training algorithm while stability-limited failures are invariant across different algorithms. We leverage dynamical mean-field theory to provide an amplitude-period scaling bound on achievable outputs by RC networks and propose a way of enhancing algorithm reach via forgetting. Using the new understanding of RC failure modes, we explore the effect of changing reservoir properties and find that simply scaling up reservoir size results in a stability-reach trade-off across regimes, while introducing distinct neurone types ameliorates the problem. Taken together, the mechanistic understanding of RC performance bounds can guide the future design and deployment of reservoir networks, while also shedding light on how biology might acquire established network characteristics for functional competence.