RaLo: Rank-aware low-rank adaptation for pre-trained foundation models.

Abstract

In the era of large language models (LLMs), low-rank adaptation (LoRA) has emerged as an essential technique for parameter-efficient fine-tuning, significantly reducing the computational and memory overhead during model adaptation. However, the potential of LoRA is limited by its reliance on fixed-rank incremental matrices, which restricts its ability to fully optimize the fine-tuning process. This paper devises a novel rank-aware low-rank adaptation (RaLo) approach, consisting of norm-constrained and rank-aware modules, to achieve better rank allocation and more efficient compression of task-specific trainable parameters. The norm-constrained module induces low-rank structures for the incremental matrices by constraints on the loss function. In contrast, the sparsity promoting rank-aware module is applied to prune redundant parameters. Hence, the modules can complement each other to capture crucial data features with fewer parameters. Extensive experiments demonstrate that RaLo effectively compresses incremental matrices and achieves superior rank allocation, showing outstanding performance in fine-tuning natural language understanding and generation tasks. It surpasses all baselines in average performance with only a minimal number of trainable parameters.