ChatGPT-4.0 and DeepSeek-R1 does not yet provide clinically supported answers for knee osteoarthritis.

Abstract

Large Language Models (LLMs) such as ChatGPT-4.0 and DeepSeek-R1 provide advanced natural language capabilities, but they also raise concerns regarding accuracy in medical applications. There is a lack of systematic evaluation of their performance against orthopedic guidelines, particularly for knee osteoarthritis (KOA). This study assessed the accuracy and consistency of these LLMs in relation to the most recent Chinese clinical practice guidelines for KOA. Queries regarding 17 guideline-recommended KOA therapeutic strategies were posed to ChatGPT-4.0 and DeepSeek-R1. Two independent reviewers evaluated response concordance (Concordance, Discordance, or No Concordance) with guidelines. Inter-rater reliability was assessed using Cohen's kappa coefficient. A chi-square test was employed to compare the response patterns between the two models. ChatGPT-4.0 showed 59 % concordance; DeepSeek-R1 achieved 71 %. Both models gave inconsistent recommendations for ozone therapy and arthroscopy. ChatGPT-4.0 had five inconsistent responses; DeepSeek-R1 had three. Inter-rater agreement was high (κ = 0.90 and 0.86). No significant difference was found in concordance rates (P = 0.7; P = 1). Only DeepSeek-R1 provided references (38 in total), but just 8 were fully verifiable. Neither ChatGPT-4.0 nor DeepSeek-R1 consistently produced responses aligned with evidence-based clinical guidelines. These findings highlight the need for cautious interpretation of medical advice generated by current AI platforms, both by clinicians and patients.