Prospective validation and real-time implementation of an automated machine learning postoperative mortality prediction model.

Abstract

Machine learning prediction models require prospective validation to ensure implementation fidelity and feasibility. Our primary objective was to prospectively validate a previously reported postoperative mortality prediction model in inpatients undergoing surgery. Our secondary objective was to evaluate feasibility of a pilot clinical decision support tool. We prospectively validated and implemented a random forest machine learning model trained to predict in-hospital mortality using data from a single academic medical centre. A reduced 32-feature model was implemented into the electronic health record (EHR) using a real-time data mart at the same institution. To assess model performance, the area under the receiver operating characteristic curve (AUROC), area under the curve precision-recall (AUCPR), and other performance measures were calculated. To assess feasibility, implementation workflow metrics were evaluated and a survey was administered to anaesthesiologists trained to use the pilot clinical decision support tool. The AUROC for the prospectively implemented model was 0.874 (95% confidence interval [CI] 0.860-0.887), and the AUCPR was 0.111. By comparison, the AUROC for the 58-feature model was 0.925 (95% CI 0.900-0.947), and for ASA physical status the AUROC was 0.814 (95% CI 0.802-0.827) and the AUCPR was 0.103. The implementation demonstrated feasibility through real-time data updates, automated transfer of model outputs to the EHR, and provider survey entries. This prospective validation and EHR implementation of a previously published random forest machine learning model predicting postoperative in-hospital mortality demonstrated acceptable real-world performance of the implemented model and feasibility of integrating such a system into clinical practice.