Assessing information provided via artificial intelligence regarding distal biceps tendon repair surgery.

Abstract

The purpose of this study was to analyze the quality, accuracy, reliability and readability of information provided by an Artificial Intelligence (AI) model ChatGPT (Open AI, San Francisco) regarding Distal Biceps Tendon repair surgery. ChatGPT 3.5 was used to answer 27 commonly asked questions regarding 'distal biceps repair surgery' by patients. These questions were categorized using the Rothwell criteria of <i>Fact, Policy</i> and <i>Value</i>. The answers generated by ChatGPT were analyzed using the DISCERN scale, <i>Journal of the American Medical Association</i> (JAMA) benchmark criteria, Flesch-Kincaid Reading Ease Score (FRES) and grade Level (FKGL). The DISCERN score for <i>Fact</i>-based questions was 59, <i>Policy</i> was 61 and <i>Value</i> was 59 (all considered 'good scores'). The JAMA benchmark criteria were 0, representing the lowest score, for all three categories of <i>Fact, Policy</i> and <i>Value</i>. The FRES score for the <i>Fact</i> questions was 24.49, <i>Policy</i> was 22.82, <i>Value</i> was 21.77 and the FKGL score for <i>Fact</i> was 14.96, <i>Policy</i> was 14.78 and <i>Value</i> was 15.00. The answers provided by ChatGPT were a 'good' source in terms of quality assessment, compared to other online resources that do not have citations as an option. The accuracy and reliability of these answers were shown to be low, with nearly a college-graduate level of readability. This indicates that physicians should caution patients when searching ChatGPT for information regarding distal biceps repairs. ChatGPT serves as a promising source for patients to learn about their procedure, although its reliability and readability are disadvantages for the average patient when utilizing the software.