LiteSwitchCodec: Neural speech coding with token space quantization and causal U-Net for personalized real-time communications.

Abstract

With the rapid development of online conferences and live streaming, personalized real-time communication (PRTC) has emerged as a critical capability for next-generation communication systems, placing demands on latency, complexity, and security. This paper presents LiteSwitchCodec, a lightweight neural speech codec specifically designed for PRTC services on operator-managed platforms. It aims to achieve high-quality speech compression while facilitating PRTC by integrating a voice adaptation (VA) module, thereby avoiding potential risks to voice copyright and security. For speech compression, we first design LiteSpeechCodec, which employs fully causal convolutional layers as the encoder and decoder and reduces the complexity of the decoder through a mirrored structure. We introduce scalar quantization (SQ) as an alternative to residual vector quantization (RVQ), reducing model complexity while maintaining high speech quality. This approach facilitates the learning of a high-quality compression domain, which in turn simplifies the generation of the quantized tokens. A lightweight causal U-Net model is introduced in the token space to extract global information for personalized VA, supporting dynamic switching of target speakers. Specifically, we propose a two-stage training strategy. First, we train the LiteSpeechCodec on public datasets for speech compression. We then construct a voice conversion (VC) dataset to train the token-level causal U-Net VA network. Experiments demonstrate that LiteSwitchCodec maintains RTC quality while reducing model parameters by 38 ×  compared to the state-of-the-art codec, achieving an objective quality of ViSQOL: 4.32 at 7.2 kbps. Moreover, LiteSwitchCodec achieves real-time VA with a low latency of 40 ms. Compared with VC models in RTC transmission, our method shows superior performance in both subjective and objective metrics, achieving an objective Resemblyzer similarity of 92.61% and a subjective speaker-similarity score (S-MOS: 4.67 vs. 3.59), highlighting its effectiveness in PRTC. Critically, LiteSwitchCodec inherently safeguards voice copyrights and prevents unauthorized impersonation, satisfying the security demands of commercial RTC deployments.