From algorithms to answers: a comparative analysis of popular search engines and large language models on clubfoot patient education.

Abstract

This study evaluates Chat Generative Pre-Trained Transformer 4o's (ChatGPT-4o's) utility in clinical relevance and accuracy compared with Google for pediatric clubfoot treatment questions. Both were queried for the 15 most frequently asked questions related to pediatric clubfoot treatment, with Google as control. Questions were classified using the modified Rothwell criteria for online sources. Questions and answers were independently graded for clinical relevance (0 = not clinically relevance, 1 = some clinical relevance, 2 = very clinically relevant) and clinical accuracy (0 = inaccurate, 1 = somewhat accurate, 2 = accurate), respectively (D.E.P. and N.G.). Questions and answers were validated by an expert, board-certified pediatric orthopedic surgeon (P.H.), who also resolved any discrepancies in grading. Per modified Rothwell criteria, Google responses were most frequently classified as either 'notion' or 'indications/management' while ChatGPT-4o responses were most likely addressed as 'notion' or 'longevity'. Google sources were primarily from academic and government platforms, while ChatGPT-4o exclusively used academic sources. ChatGPT-4o questions scored higher for clinical relevance (P = 0.006); however, clinical accuracy of answers was equivalent (P = 0.570). ChatGPT-4o provides clinically relevant questions, more so than Google with regard to pediatric clubfoot treatment. Furthermore, ChatGPT-4o uses a greater proportion of academic sources compared with Google. While both sources provided clinically accurate answers, large language models appeared to provide information that was more relevant and scholarly to patients' concerns regarding clubfoot; however, further validation and extensive testing are required to prevent the unnecessary spread of misinformation and its utilization in a clinical setting.