Generalized and group spherical linear interpolation for token-level context compression.

Abstract

The rapid development of language models has facilitated the solution of various language-related problems. However, current approaches often have certain resource requirements. To address this issue, we propose GSlerp-CC, which incorporates two Slerp-based (Spherical Linear Interpolation) methods designed to reduce the sequence length of attention computations for different model architectures. For Encoder-Only models, we use Generalized Slerp to merge the context of an explanatory prompt corresponding to a token with a special token in the sequence, thus compressing the contextual information of the explanatory prompt into the corresponding token. For Decoder-Only models, we apply Group Slerp to merge the Key or Value of specific tokens in the KV-Cache, achieving context compression. We conduct extensive experiments including UIE, NLU and Long Text benchmarks to validate the effectiveness of our proposed method.