NEURAL-VOX: NEURal auditory language decoding for voice and text reconstruction.

Abstract

Neural decoding of perceived linguistic content from non-invasive brain recordings remains a profound scientific challenge with transformative implications for assistive technologies. Existing approaches often struggle to generate intermediate representations, such as mel spectrograms or phonemes, and seldom integrate multi-modal information to enhance text decoding. This study presents a framework for decoding non-invasive brain activity into text, phoneme sequences, and mel-spectrogram-based acoustic representations, termed NEURAL-VOX. Leveraging a three-stage training strategy, NEURAL-VOX not only improves the accuracy of brain-to-text decoding, but also enables text generation to benefit from joint optimization with speech synthesis. By incorporating multi-scale frequency-domain analysis, our model more effectively captures the hierarchical structure of language processing in neural activity. Experiments across multiple datasets demonstrate that NEURAL-VOX achieves substantial gains over existing methods. The learned phoneme representations encode rich linguistic information and further strengthen text decoding, while model interpretability analysis reveals strong alignment with neurobiological patterns.