Leveraging computerized adaptive testing for cost-effective evaluation of large language models in medical benchmarking.

Abstract

Large language models (LLMs) are increasingly applied in healthcare, yet their evaluation relies predominantly on static benchmarks that are costly, contamination-prone, and lack calibrated measurement properties for fine-grained performance tracking. We developed and validated a computerized adaptive testing (CAT) framework grounded in item response theory to enable scalable, psychometrically rigorous assessment of standardized medical knowledge in LLMs. A two-phase study comprising Monte Carlo simulations and empirical evaluation of 38 LLMs was conducted between July and September 2025. The CAT protocol achieved a near-perfect correlation with full-bank results (r = 0.988) using only 1.3% of items. Evaluation time decreased from 6.85 hours to 8.4 min per model, and token usage dropped from 1.77 million to 0.03 million. Model rankings were fully preserved (Spearman's ρ = 1.0). At current API pricing, per-model evaluation costs fell from approximately $1,475 to under $5. This adaptive methodology serves as an essential pre- screening and continuous monitoring tool under a standardized testing protocol. Crucially, it is not a substitute for real-world clinical validation or safety-oriented prospective studies; rather, it enables developers and healthcare institutions to implement rigorous, high-frequency, evidence-based evaluation of foundational knowledge prior to more resource-intensive downstream testing.