Triangular Adaptive Low-Rank Adaptation for Parameter-Efficient Fine-Tuning.

Abstract

Parameter efficiency and adaptability are key challenges in fine-tuning large language models (LLMs). Existing parameter-efficient fine-tuning (PEFT) methods such as low-rank adaptation (LoRA) reduce training cost but rely on fixed low-rank adapters, limiting flexibility and underutilizing low-rank structure. We propose triangular adaptive LoRA (TriAdapt-LoRA), a PEFT method that couples a triangular split-based low-rank parameterization with a lightweight importance-driven rank-growth scheme. TriAdapt-LoRA comprises three components: 1) a triangular split-based low-rank structure that enables more flexible modeling within the low-rank subspace; 2) an efficient importance estimator based on changes in the Frobenius norm of transformation matrices, serving as a gradient-informed proxy for module contribution without storing per-step gradients; and 3) a dynamic-rank-growth mechanism that allocates a fixed global rank budget toward more important modules. The design is motivated by the need to allocate limited adaptation capacity to the most useful components during optimization, and is instantiated as a practical, fully gradient-based algorithm. Extensive experiments on natural language understanding, question answering, multimodal reasoning, and multilingual benchmarks show that TriAdapt-LoRA is competitive with strong PEFT baselines and often outperforms AdaLoRA, IncreLoRA, and related methods under matched or reduced adaptation budgets, while reducing the cost of importance estimation, making it a scalable and resource-efficient solution for fine-tuning LLMs.