Fine-tuning a compact large language model on real-world cases yields diagnostic performance comparable to flagship models in rheumatology.

Abstract

Large language models (LLMs) show strong potential in medical applications. However, most leading models are closed-source and accessible only via cloud-based APIs (application programming interfaces), raising privacy and compliance concerns when handling sensitive clinical data, while high-performing open-source models are often prohibitively large and costly to deploy and smaller models show limited diagnostic capability. Although fine-tuning with domain-specific data may improve diagnostic performance, systematic evaluations on real-world clinical cases remain lacking, leaving real-world diagnostic performance unclear. We fine-tuned a general-purpose LLM (Qwen3) using 19,682 real-world rheumatology cases, examining the effects of training sample size and model scale on performance. The best-performing setting (Qwen3-8B with full training dataset) was further fine-tuned and compared with other models (GPT-5.6 Sol, GPT-5.6 Luna DeepSeek-V4-Pro, DeepSeek-V4-Flash, Baichuan-M2-32B) on diagnostic performance (hit1), with a separate assessment of local deployment requirements and serving capacity for Qwen3-8B and the larger open-source models. Diagnostic performance was evaluated on a held-out internal test set of real-world cases, with hit1 defined as the percentage of cases where the model's top prediction matched the physician-recorded primary diagnosis, adjudicated by GPT-Judge. Blinded physician adjudication was performed in randomly sampled test cases to assess model performance, while manual review characterized the patterns underlying the improvement from fine-tuning. Fine-tuning increased Qwen3-8B's hit1 from 68.41% to 79.84%, the highest among the evaluated models on the internal test set. In the 1000-case physician-adjudicated comparison, its Adjudicated Hit1 was 91.6% versus 93.5% for GPT-5.6 Sol. Validation loss consistently decreased with larger training sets and model scales. The fine-tuned model substantially lowered the upfront hardware barrier to local deployment compared with the assessed configurations for larger open-source models. Fine-tuning Qwen3 on real-world rheumatology cases substantially improved diagnostic performance, yielding a compact model that lowers the upfront hardware barrier to local deployment. This approach offers a practical route to locally deployed diagnostic support, allowing clinical data and model weights to remain within the institution.