LoRASculpt: Harmonious Low-Rank Adaptation for Multimodal Large Language Models.

Abstract

Although Multimodal Large Language Models (MLLMs) demonstrate strong generalization across modalities and tasks, effectively adapting them to downstream tasks while maintaining general knowledge remains a critical challenge. While Low-Rank Adaptation (LoRA) enables parameter-efficient task-specific fine-tuning, it may still introduce redundant updates during training, which exacerbates forgetting and hinders downstream adaptation. We propose LoRASculpt+, a framework for harmonizing general and specialized knowledge in MLLMs. To eliminate harmful redundancy, we introduce sparse LoRA updates with theoretical guarantees to enable compact and precise knowledge injection. We further analyze the structural heterogeneity of MLLMs, and accordingly propose an asymmetric adaptation strategy which assigns distinct sparsity and capacity based on the roles of the LLM and connector modules. To better calibrate the optimization trajectory of LoRA, we introduce regularization terms that steer updates away from critical pretrained regions while enhancing subspace expressiveness, thereby alleviating upstream generalization conflicts and improving downstream representational capacity. Extensive experiments across diverse downstream tasks, model scales, and backbones demonstrate that, even at a high degree of sparsity, our method simultaneously enhances generalization and downstream performance. This confirms that our approach enables efficient and precise knowledge injection, mitigates catastrophic forgetting, and thus facilitates harmonious downstream low-rank adaptation in multimodal large language models.