LS-PRISM: A layer-selective pruning method via low-rank approximation and sparsification for efficient large language model compression.

Abstract

Large Language Models (LLMs) have significantly advanced natural language processing (NLP), establishing new benchmarks across a wide range of tasks. However, their large parameter sizes present challenges for deployment on resource-constrained devices. Current compression techniques often treat all layers uniformly, disregarding the operational differences across layers, which can lead to varying effects on performance. In this paper, we introduce a novel method named Layer-Selective Pruning via low-Rank Approximation and Sparsification Method (LS-PRISM), efficiently compressing LLMs while preserving their performance on key NLP benchmarks, such as BoolQ, RTE, and ARC-Challenge. LS-PRISM dynamically applies low-rank approximation to selected matrices within each model layer based on their impact on accuracy and loss, with ranks adaptively determined using Dynamic Rank Selection method, retaining approximations that improve performance and leaving others unaltered. Additionally, we employ unstructured pruning on the remaining matrices to further sparsify the model, followed by optional fine-tuning using LoRA to recover performance loss. Experimental results demonstrate that LS-PRISM achieves significant reductions in both parameter count and storage, with minimal degradation in accuracy. Specifically, for a 2.5B parameter LLM, we achieve up to a 12 % reduction in parameters, while maintaining performance comparable to the original model. We also explore the method's applicability to even smaller models, and discuss the observed performance differences. LS-PRISM offers a scalable and effective solution for compressing LLMs in resource-constrained environments.