Trial Files: Leveraging large language models to summarize practice-changing clinical trials for clinicians.

Abstract

Each day, over 100 randomized controlled trials (RCTs) are published, making it impossible for clinicians to stay up-to-date with medical literature. Large language models (LLMs) can identify and summarize emerging clinical evidence and support medical education. We created and prospectively evaluated a newsletter, Trial Files, which leverages an LLM to summarize RCT abstracts relevant to general internal medicine. We created a software tool, called PaperScrape, which leverages the Medline application programming interface (API) to identify trials published in five high-impact journals. Information from each RCT's abstract was extracted, and plain-language summaries were generated using OpenAI's LLM API. We analyzed the accuracy of summaries generated by an LLM (compared to manual review), results of a subscriber survey, and effectiveness of marketing strategies on user growth. From June 2023 to March 2025, 50 newsletters with 3 RCTs each were distributed to 648 subscribers. A subset of 96 RCTs was randomly selected to evaluate reporting accuracy with prompt engineering. The accuracy for reporting study information with prompt engineering, compared to manual review, was 97.1% for study phase, 92.2% for blinding, 85.4% for sample size, 97.9% for patient population, 94.7% for comparison groups, and 92.7% for primary outcome. Forty-three subscribers completed a survey about Trial Files. The mean overall rating was 4.7 out of 5 (5 representing "very good"), and all respondents agreed the newsletter made it easier to keep up-to-date with emerging clinical trials in internal medicine. The most effective strategy for user growth was promotion at a meeting, conference, or education session (6.8 subscribers per day, compared to 0.7 subscribers gained per day on days without promotion, p < 0.0001). LLMs can provide concise, accurate summaries of RCTs, which can help general internists stay up-to-date on recently published trials.