Evaluating Large Reasoning Models Versus Human Multidisciplinary Teams in Lung Cancer Decision-Making: Real-World Study.

Abstract

Large language models (LLMs) and large reasoning models (LRMs) have shown excellent performance on medical benchmarks, although evaluations concerning real-world medical workflows are still lacking. Lung cancer care is particularly dependent on the multidisciplinary team (MDT) integration of radiology, pathology, staging, and treatment planning, making it a high-bar setting for evaluating LRMs. This study aimed to compare the quality of recommendations generated by 2 LRMs (GPT-5-Thinking and Deepseek-v3-r1) with each other and with human MDT decisions in real-world lung cancer cases, as well as to assess whether MDT awareness of AI comparison influences the quality of MDT decisions. This was a single-center real-world comparative study of 100 consecutive lung cancer MDT cases (50 retrograde and 50 anterograde) from the University Hospital of Split, Croatia. For each case, deidentified structured reports (containing all necessary patient or case data, while excluding MDT conclusions) were submitted once to GPT-5-Thinking and Deepseek-v3-r1 to generate recommendations for radiologic diagnostics, pathologic diagnostics, oncologic therapy, and overall usefulness. Two independent lung oncologists graded MDT decisions and model outputs on 1-5 Likert scales. An average recommendation score (avg_rec) was calculated as the mean of radiology, pathology, and therapy scores. Analyses used Wilcoxon tests for paired model comparisons, Mann-Whitney tests for between-phase comparisons, and Spearman correlations (2-sided α=.05). Ratings showed ceiling effects. In the retrograde phase (N=50), the mean (95% CI) GPT-5-Thinking scores were higher than Deepseek-v3-r1 scores for radiologic diagnostics (4.89, 4.78-4.99 vs 4.76, 4.62-4.89; P<.001), oncologic therapy (4.82, 4.69-4.94 vs 4.18, 3.82-4.54; P<.001), and usefulness (4.82, 4.69-4.94 vs 4.18, 3.84-4.53; P<.001); pathologic diagnostics were similar (4.88, 4.78-4.97 vs 4.73, 4.57-4.90; P=.15). In the anterograde phase (n=50), the mean (95% CI) GPT-5-Thinking scores remained higher for radiology (4.94, 4.85-5.03 vs 4.64, 4.47-4.81; P<.001) and pathology (4.96, 4.90-5.02 vs 4.78, 4.65-4.91; P=.008), with smaller differences for therapy (4.46, 4.18-4.74 vs 4.20, 3.86-4.54; P=.24) and usefulness (4.50, 4.24-4.76 vs 4.16, 3.83-4.49; P=.12). The mean (95% CI) GPT-5-Thinking avg_rec exceeded MDT grade in both phases (retrograde: 4.90, 4.84-4.95 vs 4.14, 3.96-4.33; P<.001; anterograde: 4.79, 4.69-4.89 vs 4.34, 4.16-4.52; P<.001); Deepseek-v3-r1 exceeded MDT in the retrograde phase (4.56, 4.40-4.72 vs 4.14, 3.96-4.33; P<.001) but not the anterograde phase (4.54, 4.41-4.67 vs 4.34, 4.16-4.52; P=.15). MDT grades did not differ between phases (P=.13). In 100 real-world lung cancer MDT cases, both LRMs produced high-quality recommendations, with GPT-5-Thinking consistently outperforming Deepseek-v3-r1 and exceeding expert-graded MDT decision quality in both phases. MDT decision quality was unchanged by awareness of AI benchmarking. LRMs can thus generate recommendations comparable to or exceeding expert MDT decisions, though the single-center design and ceiling effects limit generalizability. Whether integrating such tools into MDT workflows improves clinical decisions warrants prospective study.