Evaluating the quality of online health information: A comparative study of COPD patient information in Facebook forums versus ChatGPT.
cross_sectional · Level IV
Where this comes from
- Record sourced from PubMed, PMID 41934770.
- Also identified by DOI 10.1016/j.ijmedinf.2026.106421.
- No licence information is recorded for this record.
- Because redistribution is not established, this page shows the abstract only. Follow the links below for the full text.
Abstract
Patients with chronic obstructive pulmonary disease (COPD) increasingly turn to online platforms for health information. However, the quality of information shared in social media forums remains uncertain. This study compared the accuracy of user-generated Facebook responses to AI-generated replies from ChatGPT. Posts and comments were extracted from two COPD-related Facebook groups in October 2024. A total of 48 posts from each group were selected, yielding 2,761 comments. Each post was also submitted to ChatGPT. Responses were categorized as 'useful', 'misleading', or 'neither' and rated on a 5-point Likert scale by three independent reviewers. Interrater reliability was assessed using Fleiss' kappa. Differences in quality scores were analyzed using the Wilcoxon signed-rank test and Hodges-Lehmann 95% CI. In the Danish group, 47% (667/1,433) of comments were relevant, with 34% deemed useful and 9% misleading. Results were similar in the English group (734/1,315 relevant; 38% useful, 9% misleading). Critically, 16% and 15% of comments encouraging behavioral changes contained misleading information in the Danish and English group, respectively. Likert ratings showed significantly higher accuracy for ChatGPT compared to Facebook. For the Danish group, the median score for Facebook was 4.0 (IQR 3.0-4.5) versus 5.0 (IQR 4.0-5.0) for ChatGPT (p < 0.001; Hodges-Lehmann difference: -0.75, 95% CI -1.0 to -0.5). For the English group, Facebook scored a median of 4.0 (IQR 3.0-4.0) versus 5.0 (IQR 5.0-5.0) for ChatGPT (p < 0.001; Hodges-Lehmann difference: -1.0, 95% CI -1.25 to -0.75). No significant performance differences were found between the Danish and English groups for either platform. ChatGPT delivered more accurate responses than Facebook users, highlighting its potential as a reliable educational tool. However, Facebook remains valuable for peer support, suggesting that combining AI and social platforms may enhance digital care strategies for COPD.