Leveraging Fine-Tuned Large Language Models for Interpretable Pancreatic Cystic Lesion Feature Extraction and Risk Categorization.

Abstract

<b>BACKGROUND</b>. Manual extraction of pancreatic cystic lesion (PCL) features from radiology reports is labor-intensive, limiting large-scale studies needed to advance PCL research. <b>OBJECTIVE</b>. The purpose of this study was to evaluate GPT-4o (closed source [ OpenAI]), Llama (Llama-3.1-8B-Instruct, open source), and DeepSeek (DeepSeek-R1-Distill-Llama-8B, open source) large language models (LLMs) for PCL feature extraction, without and with chain-of-thought (CoT) reasoning. <b>METHODS</b>. We curated a dataset of 6469 abdominal MRI or CT reports (2005-2024) that described PCLs from 5615 patients. Llama and DeepSeek were fine-tuned using quantized low-rank adaptation on GPT-4o-generated CoT labels for extracting PCL and main pancreatic duct features. Features were mapped to risk categories per institutional policy. Evaluation was performed on 285 held-out human-annotated reports from 281 patients. Model outputs for 100 cases were independently reviewed by three radiologists. Feature extraction was evaluated using exact-match accuracy, risk categorization with a macro-averaged F1 score, and radiologist-model agreement with Fleiss kappa values. Error analyses were performed to assess how and why models made mistakes. <b>RESULTS</b>. CoT fine-tuned LLMs showed a feature extraction accuracy of 97% (95% CI, 97-98%) for Llama, 98% (95% CI, 97-98%) for DeepSeek, and 97% (95% CI, 97-98%) for GPT-4o. Risk categorization F1 scores were 0.93 (95% CI, 0.89-0.97) for Llama, 0.94 (95% CI, 0.90-0.98) for DeepSeek, and 0.97 (95% CI, 0.93-0.99) for GPT-4o. Radiologist interreader agreement was high (κ = 0.888) and showed no significant difference with the addition of Llama (κ = 0.882; <i>p</i> > .99), DeepSeek (κ = 0.893, <i>p</i> > .99), or GPT-4o (κ = 0.897, <i>p</i> > .99). Across all models, object identification and clinical reasoning were the most frequent error types, accounting for 29.3-37.3% and 18.1-21.1% of total errors, respectively. <b>CONCLUSION</b>. LLMs show feasibility for automatically extracting PCL features from radiology reports. Fine-tuned open-source LLMs achieved performance comparable to that of GPT-4o. CoT reasoning improved accuracy and enabled interpretable error analysis. Model-assigned risk categories showed high agreement with abdominal radiologists. <b>CLINICAL IMPACT</b>. LLMs have the potential to enable creation of large structured registries from existing radiology reports to support population-level research on PCLs.