Contrastive Forward-Forward: A training algorithm of vision transformer.

Abstract

Although backpropagation is widely accepted as a training algorithm for artificial neural networks, researchers are always looking for brain-inspired strategies to improve performance. Forward-Forward is a novel training algorithm that is claimed to be more biologically plausible than backpropagation, although there is a significant performance gap between them. In the Forward-Forward algorithm, the loss functions are placed after each layer, and the updating of a layer is done using two local forward passes and one local backward pass. Forward-Forward is in its early stages and has been designed and evaluated on simple multi-layer perceptron networks to solve image classification tasks. In this work, we have extended the use of this algorithm to a more complex and modern network, namely the Vision Transformer. From a machine learning viewpoint and inspired by insights from contrastive learning, we have modified this algorithm by introducing Contrastive Forward-Forward, while preserving its biologically plausible properties. Experimental results demonstrate that our proposed algorithm significantly outperforms the baseline Forward-Forward, achieving up to a 10 % improvement in accuracy and accelerating convergence speed by 5 to 20 times. Furthermore, if we take cross-entropy as the baseline loss function in backpropagation, our modifications to Forward-Forward reduce the performance gap. In certain scenarios - such as under inaccurate supervision - our approach even outperforms backpropagation. Code is available at: https://github.com/HosseinAghagol/ContrastiveFF.