Sparsely Connected, Hebbian Networks with Strikingly Large Storage Capacities.

Abstract

Conspicuous problems confront the use of fully-connected networks (F-nets) in the construction of realistic partial models of biological memory. These problems include the high synaptic densities of F-nets, and the low information storage capacities of F-nets with simple, biologically realistic learning rules. Most auto-associative networks constructed with low connectivities have employed random projections of path length 1. Projective networks (P-nets) are nonrandom, multilayer networks which achieve extremely low connectivities by linking all neurons in the same layer through paths of length 2. In this paper we derive a lower bound on the storage capacities of a class of simple, two-layer P-nets with binary Hebbian synapses. Given a 1% tolerance for spurious neurons, we find that the P-net with 1000 synapses per neuron (2 x 10(6) neurons) will store more than 1.5 x 10(6) training vectors with 20 active neurons per vector (0.25 bits per synapse). Copyright 1997 Elsevier Science Ltd.