Eigen value based loss function for training attractors in iterated autoencoders.

Abstract

The way that the human brain handles the input variations has been one of the most interesting areas of research for neuroscientists. There are some evidences that the human brain acts like an attractor when trying to memorize or retrieve some information. Based on this fact, in this research, a new method is presented for creating attractors during training of an iterated autoencoder. In this method a new loss function is presented which decreases the absolute real of Eigen values while preserving the reconstruction error during training. A fully connected structure is chosen for constructing the iterated autoencoder in this research which mostly faces with local minima especially when they are deep. For getting through this issue, a layer-by-layer pre-training approach is taken to train the network. Using the evaluation on MNIST dataset, it is shown that the proposed model can retrieve 59.98% of test samples which shows a considerable improvement over Dense Associative Memory (DAM) when trained on 100 similar MNIST test samples. The performance of the proposed model is compared to overparameterized autoencoder (OAE) model which was recently presented and showed promising results in constructing associative memories. The results show that the proposed model outperforms OAE in terms of the number of attractors learned by the network in a similar number of network parameters. Finally, the performance of the proposed model is evaluated with corrupted version of training samples, revealing significant robustness when compared to the baseline autoencoder.