A Large Language Model Assistant for Summarizing Hepatology Referral Documents.

Abstract

To use a large language model (LLM) to create accurate and useable summaries of patient records for clinicians triaging new hepatology referrals. We developed a comprehensive list of data elements required to triage a new hepatology referral and engaged in an iterative prompt engineering process to instruct an LLM to extract relevant data from patient referral documents. The final prompt was used on 50 original patient records from June to July 2025 to generate corresponding artificial intelligence (AI) summaries, which were assigned to 2 providers to review and triage according to their usual process. We assessed time to triage original vs AI files and accuracy of the AI files. A linear mixed-effects model was used to determine an adjusted time ratio comparing the time to triage AI files vs original files. AI-generated summaries were significantly shorter than original files (median [interquartile range] 2 [2-3] vs 23 [10.2-38.8] pages, P < 0.001). AI summaries had high accuracy (median [interquartile range]: 94.6% [86.5%-97.3%]) with a low hallucination rate. Use of the AI summaries led to a 60% reduction in triage time (adjusted mean triage time of 37.2 seconds for AI files vs 94.2 seconds for original files, P < 0.001). The use of an LLM led to significantly reduced document length, maintained an appropriate level of accuracy, and led to a significant decrease in clinician time to review the patient record. Future steps involve creating a fully automated workflow that is integrated into the electronic health record for widespread use.