A Dual-Modal Fusion Framework for Detection of Mild Cognitive Impairment Based on Autobiographical Memory.

Abstract

This paper introduces a dual-modal early cognitive impairment detection system based on autobiographical memory (AM) tests, and our approach is to automatically extract pre-defined acoustic features and self-designed embeddings to enhance linguistic representation of the spontaneous speech data. By integrating dual-modal data, we effectively enrich the features that aid in model learning, especially addressing the subtle symptoms exhibited by individuals with mild cognitive impairment (MCI), an intermediate stage between healthy individuals and those with Alzheimer's disease (AD). To account for spontaneous speech's unstructured and implicit nature, two additional embeddings, namely, speaker embedding and conversation embedding, are introduced to augment the information available for model learning, thus enriching the feature set for improving the model accuracy. The proposed dual-modal approach is tested on a self-collected Chinese spontaneous speech dataset due to the limited unstructured speech open-access dataset for MCI detection. The system's effectiveness is evaluated through a series of experiments, including ablation studies, to determine the impact of each module on overall performance. The proposed system achieved an average accuracy of 78% in detecting MCI, demonstrating its comparative effectiveness. Enhancements in our system are achieved by integrating a directional encoder tailored to capture temporal information across sequential visits. This addition leads to a 3% increase in detection accuracy within a subset of participants who have undergone multiple AM test sessions. Implementing such a longitudinal approach in analyzing unstructured speech data for MCI detection taps into a relatively underexplored area of research, offering new insights.