Specialized AI and neurosurgeons in niche expertise: a proof-of-concept in neuromodulation with vagus nerve stimulation.

Abstract

Applying large language models (LLM) in specialized medical disciplines presents unique challenges requiring precision, reliability, and domain-specific relevance. We evaluated a specialized LLM-driven system against neurosurgeons in vagus nerve stimulation (VNS) for drug-resistant epilepsy knowledge assessment-a complex neuromodulation therapy requiring transdisciplinary expertise in neural anatomy, epileptic disorders, and device technology. Thirty-six European neurosurgeons who completed a 2-day VNS masterclass were assessed using a multiple-choice questionnaire comprising 14 items with 67 binary propositions. We deployed open-source models-LLaMa 2 70B and MXBAI embedding model-using Neura, an AI infrastructure enabling transparent grounding through advanced retrieval augmented generation. The knowledge base consisted of 125 VNS-related publications curated by multidisciplinary faculty. Scoring ranged from -1 to + 1 per question. Performance was analyzed using Wilcoxon signed-rank tests, confusion matrices, and metrics including accuracy, precision, recall, and specificity. The AI achieved a score of 0.75, exceeding the highest individual clinician score (0.68; median: 0.50), with statistical significance (p < 0.001). AI performed better in questions involving anatomical and technical information, while clinicians excelled in scenarios requiring practical judgment. Confusion matrices revealed higher true correct and true incorrect rates for AI, demonstrating perfect precision and specificity scores with no hallucinations detected. Specialized LLM performance in this VNS knowledge assessment, coupled with its verifiability, points to promising applications across neurosurgical subspecialties for clinical decision support and education. The complementary strengths observed suggest that valuable implementations will emerge from synergistic approaches combining human experiential knowledge with AI's information processing capabilities across the broader field of neurosurgery.