EfficientLoRA: Rethinking the efficiency of low-rank adaptation in pre-trained language models.

Abstract

Fine-tuning pre-trained language models to adapt to downstream tasks has become a key paradigm in natural language processing. Low-Rank Adaptation (LoRA) reduces computational and storage overhead by introducing low-rank matrices to adapt to specific tasks. However, LoRA often overlooks more efficient ranks, leading to suboptimal fine-tuning performance. Thus, we propose EfficientLoRA, inspired by model pruning, to extract more effective ranks. We introduce the Sparse-Refined Straight-Through Estimator (SR-STE), which ensures that the discreteness arising from the pruning process remains trainable. Meanwhile, it rewards the updates of significant ranks. We conduct extensive experiments on multiple pre-trained models for natural language understanding and natural language generation tasks to validate the effectiveness of EfficientLoRA. Experimental results show that EfficientLoRA significantly improves over baseline models and achieves state-of-the-art performance levels on three benchmarks. Our source code is publicly available¹.