EEG fine-grained visual semantic decoding via a multimodal framework.

Abstract

In recent years, research on decoding visual-induced brain activity from electroencephalograms (EEG) and reconstructing corresponding images has garnered significant attention. The core challenge lies in achieving precise decoding of brain activity. However, current research primarily focuses on extracting coarse-grained semantic information such as object categories from EEG signals, with limited ability to decode fine-grained attribute visual semantics, such as colors, object states, and object relationships. Therefore, how to accurately extract fine-grained semantic information from EEG signals remains a critical issue. In this study, we propose a multimodal EEG decoding and visualization framework, where captions generated by Qwen3-VL provide detailed semantic supervision. An EEG-Text Encoder (ET-Encoder) maps EEG signals into the text embedding space, enabling the decoding of attribute-level visual semantics, and a pre-trained text-to-image diffusion model, Stable Diffusion, is introduced to reconstruct semantically aligned images. Experimental results on the ImageNet-EEG-40 and ImageNet-EEG-4 datasets demonstrate that our method achieves better performance in terms of image quality, diversity, and semantic fidelity, while also showing its ability to capture attribute-level semantics beyond class labels and generate semantically consistent images. The code is available at https://github.com/tran601/EEG-Fine-grained-Visual-Semantic-Decoding-via-a-Multimodal-Framework.