Evaluating generative AI responses at the 3-month timepoint following total knee arthroplasty.

Abstract

Patients increasingly seek health information prior to and following total knee arthroplasty (TKA), often using generative AI chatbots integrated into search engines. As these tools grow in popularity, their accuracy and utility in postoperative education for complex medical topics like TKA remain unclear. This study assessed four leading AI models (ChatGPT, Gemini, Copilot, and Grok), on their ability to answer common patient questions three months following TKA. Four frequently asked questions (FAQs) were chosen based on surgeon experience in a high-volume total joint practice, covering return to sport, range of motion, persistent symptoms, and scar healing. Questions were input into each AI platform using a cleared browser. Ten orthopedic surgeons rated responses on a 1-4 scale (1 = Excellent, no clarification needed; 4 = Unsatisfactory, needing substantial clarification) and selected preferred answers. Mean scores, standard deviation (SD), and vote counts were analyzed using ANOVA for consistency and consensus. Significant score variations were noted for three questions (1, 2, and 4). Gemini consistently scored lowest (best), with mean scores of 1.4-1.6 and low variability (SD: 0.48-0.70), earning the most first place votes (7/10) for three questions. ChatGPT ranked second, followed by Grok and Copilot, showing higher (worse) scores and variability. Subjective questions like symptom normalization showed greater rater disagreement. Generative AI shows potential for postoperative education, but response quality differs across platforms. Gemini had highest consistency and clinical alignment supporting cautious integration of AI in patient communication with ongoing surgeon oversight for safety and accuracy.