RDguru: A Conversational Intelligent Agent for Rare Diseases.

Abstract

Large language models (LLMs) hold significant promise in clinical practice, yet their real-world adoption is constrained by their propensity to produce erroneous and occasionally harmful outputs, particularly in the intricate domain of rare diseases (RDs). This study introduces RDguru, a conversational intelligent agent leveraging the LangChain framework and powered by GPT-3.5-turbo. RDguru offers a comprehensive suite of functionalities, encompassing evidence-traceable knowledge Q&A and professional medical consultations for differential diagnosis (DDX), integrating authoritative knowledge sources and reliable tools. A novel multi-source fusion diagnostic model, rooted in deep Q-network, amalgamates three diagnostic recommendation strategies (GPT-4, PheLR, and phenotype matching) to enhance diagnostic recall during medical consultations. Through tailored tools and advanced algorithms for retrieval-augmented generation, RDguru excels in knowledge Q&A, automated phenotype annotation, and RD DDX. A multi-aspect Q&A analysis demonstrates RDguru outperforms ChatGPT in generating descriptions aligned with authoritative knowledge, quantified by ROUGE scores, GPT-4-based automatic rating, and RAGAs evaluation metrics. Testing on 238 published RD cases reveals that RDguru's top 5 multi-source fusion diagnoses recapture 63.87% of actual diagnoses, marking a 5.47% improvement over the state-of-the-art diagnostic method PheLR. Furthermore, RDguru's consultation strategy proves effective in eliciting diagnostically beneficial phenotypes and refining the prioritization of genuine diagnoses through multi-round phenotype-orient questioning. Evaluations against established benchmarks and real-world patient data demonstrate RDguru's efficacy and reliability, highlighting its potential to enhance clinical decision-making in the realm of RDs.