Evaluating the Accuracy of Large Language Models in Answering Asthma Multiple Choice and Objective Structured Clinical Examination Questions.

Abstract

Large language models (LLMs) are demonstrating increasing promise across clinical applications, but their domain-specific knowledge in asthma has not been thoroughly explored. Additionally, state-of-the-art models released in 2025 (ChatGPT-5, ChatGPT-o3, Claude-3.7, DeepSeek-V3, and Grok-3) have yet to be studied in asthma. How accurate are LLMs at answering adult asthma multiple choice questions (MCQs) and Objective Structured Clinical Examinations (OSCEs)? Fourteen LLM chatbots (Asthma-GPT, ChatGPT-4o, ChatGPT-o1, ChatGPT-o3-mini, ChatGPT-o3-mini-high, ChatGPT-5, Claude-3.7-Sonnet, DeepSeek-V3, Gemini-2.0-Flash, Grok-3, Le-Chat, Llama-3, NICE-Asthma-Bot, and Biomistral) were tested in 5 iterations on 116 adult asthma MCQs and 3 asthma OSCEs. Using generalized linear mixed-effects models, we compared accuracy between the LLMs using log-odds scale and pairwise comparisons with Tukey adjustment, and assessed difference in MCQs vs OSCEs, generic vs medicine-specific models, open-source vs proprietary models, and patient- vs clinician-oriented questions. MCQ accuracy was excellent, with 13 of 14 models exceeding 85%, and ChatGPT-o1, Asthma-GPT, NICE-Asthma-Bot, ChatGPT-5, and ChatGPT-4o surpassing 95% accuracy. OSCE accuracy varied more significantly, but most models still achieved 70% to 86% accuracy. ChatGPT-o1 ranked highest in MCQs, and ChatGPT-5 ranked highest in OSCEs. Generalized linear mixed-effects model revealed that LLMs performed better at MCQs than OSCEs (92.8% vs 71.1%, P < .001), and that LLMs performed better at patient-oriented MCQs than clinician-oriented MCQs (96.3% vs 85.6%, P < .01). No significant differences were found between open-source and proprietary models in MCQs, or between generic and medicine-specific LLMs. Our results indicate that current LLMs demonstrate high accuracy across asthma-related clinical and patient-oriented questions, particularly the ChatGPT family of models. These models may be useful tools to integrate into future patient and clinician education interventions.