Performance Drift in a Nationally Deployed Population Health Risk Algorithm in the US Veterans Health Administration.

Abstract

Clinical risk algorithms inform clinical decision support and system-level quality metrics. However, algorithm performance can drift over time and possibly promote misinformed decision-making and resource allocation. The Veterans Health Administration (VA) Care Assessment Needs (CAN) algorithm is a nationally deployed population risk algorithm used to predict risk of 90-day hospitalization and/or mortality and to allocate resources for more than 5 million veterans annually. However, drift affecting the VA CAN has not been assessed. To evaluate the impact of drift in the VA CAN algorithm and the extent, mechanisms, and clinical consequences of performance changes. This was a retrospective cohort study using electronic health records (EHRs) and administrative data from the VA Corporate Data Warehouse, which contains observations from more than 5 million veterans per study year. The data comprised 27 787 152 observations among 7 215 711 unique veterans receiving VA primary care from 2016 to 2021. Data analysis was performed from January 2023 and December 2024. Two primary outcomes were change in model performance (true positive rate [TPR], false positive rate [FPR], positive predictive value [PPV], negative predictive value [NPV], F1 score, and accuracy); and a national quality metric (% of veterans with CAN ≥90th percentile with a palliative care visit). The study population included 7 215 711 eligible veterans, with a mean (SD) age of 62.1 (16.5); 91.2% were male and 18.2% were Black, 6.6% Hispanic, and 76.2% White individuals. From 2016 to 2021, PPV decreased by 4.0% (95% CI, -2.8% to -5.1%); F1 score decreased by 4.6% (95% CI, -6.1% to 3.0%); NPV increased by 0.43% (95% CI, 0.30% to 0.57%); and FPR increased by 0.34% (95% CI, 0.10% to 0.58%), which corresponds with 18 288 increased false positive results. TPR and accuracy remained stable. The 90-day hospitalization and/or death rates decreased from 3.8% in 2017 to 3.0% in 2021. Covariate shifts were observed in 19 covariates, with demographic characteristics, health care utilization, and laboratory covariates exhibiting the largest shifts. The palliative care quality metric was 2.9% (95% CI, 2.8% to 2.9%) in 2018, 2.6% (95% CI, 2.6% to 2.7%) in 2019, and 2.8% (95% CI, 2.7% to 2.8%) in 2020, with FPRs among metric-eligible veterans increasing from 81.6% (95% CI, 81.5% to 81.7%) in 2018 to 85.7% (95% CI, 85.6% to 85.8%) in 2020. This cohort study found that CAN algorithm performance declined from 2016 to 2021 due to shifts in outcome prevalence and distributions of health care utilization and demographic covariates. Close surveillance of clinical risk algorithms and quality metrics derived from algorithm-generated risk scores could mitigate suboptimal resource allocation or decision-making.