Real-world evaluation of large language model for patients medical and administrative queries in nuclear medicine.

Abstract

Large language models (LLMs) may support patients by addressing their queries; however, their real-world clinical use remains uncertain. Patients' queries prospectively collected and answered by nuclear medicine physicians, administrative staffs, and ChatGPT v4.1. Responses evaluated and Likert-scored by medical and administrative experts and two independent non-experts using 15 out of 17 dimensions of the proposed QUEST framework. LLM-generated responses classified as "better," "equivalent," or "worse" relative to human-generated responses on question-by-question basis; binomial tests assessed whether LLM performance exceeded 50%. Inter-rater agreement was assessed using Prevalence-Adjusted Bias-Adjusted Kappa (PABAK); statistical significance set at p < 0.05. 339 drug interaction, 42 medical, and 76 administrative queries were analysed. For medical queries, in 8 of 10 dimensions, 76-98% of LLM-generated responses rated equivalent or better than human-generated responses by medical expert (p < 0.001). For administrative queries, non-expert raters judged LLM-generated responses more informative (97%) and preferred (86%). For medical queries, LLM-generated responses rated more informative (67%), human-generated responses judged easier to understand (62%), with 60% disagreement on overall preference. PABAK showed higher agreement for LLM- than human-generated responses across medical (0.14-0.90 vs -0.90-0.52) and administrative (0.92-1.00 vs -0.63- - 0.13) queries. LLM-generated responses were consistently rated favourably, particularly for administrative queries, though further validation is required before clinical use.