TC3Net: Transformer and Convolution Coupled Contrastive Network for Single Image Super-Resolution.

Abstract

The convolutional neural network (CNN) and transformer have gained significant attention in the field of single image super-resolution (SISR), owing to their powerful capacity in nonlinear feature extraction. Nonetheless, these two types of approaches hold their own limitations. For instance, the interaction between convolutional kernels and image content is agnostic in CNN, while the computational complexity increases quadratically along with the spatial resolution in the transformer. To address these concerns, in this article, we propose a novel unified framework named transformer and convolution coupled contrastive network (TC3Net) for SISR, which holds a triple-branch structure to integrate the merits of both CNN and transformer. The proposed TC3Net is mainly composed of several stacked CNN feature extraction (CFE) blocks, transformer feature extraction (TFE) blocks, and coupled contrastive blocks (CCBs) for diverse feature extraction. Particularly, the CCB that consists of the coupled attention block (CAB) and the local-global feature extraction (LGFE) block is designed to fuse feature maps and extract coupled information for better image reconstruction. Moreover, a contrastive loss between the transformer and CNN feature maps is further introduced to enhance their discriminative characteristics and complement the fused features. Experimental results demonstrate that TC3Net outperforms several state-of-the-art (SOTA) methods in the aspect of achieving a better balance between model size and performance.