Reference Hallucination in AI-Assisted Academic Writing: A Comparative Analysis of ChatGPT, Gemini, and Perplexity in Rotator Cuff Literature.

Abstract

This study aimed to compare the reliability of references generated by three different AI-based conversational agents (ChatGPT, Gemini, Perplexity) within the rotator cuff literature. Designed as a cross-sectional study, 30 subtopics were posed to each bot in two formats (letter to the editor and original article), resulting in a total of 3150 references. Reference reliability was assessed using the Reference Hallucination Score (RHS), which included the following criteria: existence/verifiability, bibliographic accuracy, PMID validity, and topical relevance. When all formats were analyzed together, ChatGPT had the lowest mean RHS (1.81 ± 3.40) and thus emerged as the most reliable model. Gemini scored 4.01 ± 4.89, while Perplexity had the highest score at 6.51 ± 4.89 (<i>p</i> < 0.001). In the letter format, ChatGPT, Gemini, and Perplexity scored 1.81 ± 3.20, 3.81 ± 4.83, and 6.43 ± 4.78, respectively. In the article format, ChatGPT scored 4.02 ± 1.70, Gemini 4.13 ± 1.57, and Perplexity 6.31 ± 1.68. ChatGPT was more reliable than Gemini and significantly superior to Perplexity. AI-based conversational agents may play a supportive role in academic writing; however, they exhibit critical limitations in reference accuracy. While ChatGPT was relatively the most reliable model, Perplexity had the highest hallucination rate. Researchers must refrain from using references generated by these tools without verification, as doing so poses significant concerns for scientific integrity and ethics. The online version contains supplementary material available at 10.1007/s43465-026-01807-0.