Less Is More: Minimal Components for Contrastive Learning Delivering Superior Accuracy and Efficiency.

Abstract

Recent advances in contrastive self-supervised learning (SSL) have pursued increasingly complex architectures, such as momentum encoders and deep projection heads, adding significant computational overhead. This paradigm of growing complexity motivates a systematic deconstruction of modern frameworks to identify their truly essential components. Our analysis reveals that a precisely defined "Cross-View Symmetric InfoNCE" loss, which purifies the contrastive objective by exclusively using interview negative samples, is a primary driver of both superior performance and accelerated convergence. We encapsulate this principle within contrastive representation with essential components (CORE), a minimalist framework built upon a shared encoder and an optimized three-layer projection head. On ImageNet, CORE achieves a competitive 73.7% top-1 accuracy with just 100 training epochs and demonstrates remarkable efficiency, outperforming key baselines by up to 1.5% on ImageNet-100. Furthermore, this efficient framework reveals a counter-intuitive finding: the simpler shared encoder consistently outperforms the more complex momentum encoder under large-batch settings (e.g., 1024), challenging its conventional necessity for stable training. Our results, supported by extensive ablations and theoretical insights from an information bottleneck perspective, provide a clear, validated design paradigm for building highly efficient and powerful SSL systems. Code and training logs are available at https://github.com/Jaywalk18/CORE.