Evaluating the impact of explainable AI on clinicians' decision-making: A study on ICU length of stay prediction.

Abstract

Explainable Artificial Intelligence (XAI) is increasingly vital in healthcare, where clinicians need to understand and trust AI-generated recommendations. However, the impact of AI model explanations on clinical decision-making remains insufficiently explored. To evaluate how AI model explanations influence clinicians' mental models, trust, and satisfaction regarding machine learning-based predictions of Intensive Care Unit (ICU) Length of Stay (LOS). This retrospective mixed-methods study analyzed electronic health record data from 8,579 patients admitted to a surgical ICU in South Korea between 2019 and 2022. Seven machine learning models were developed and evaluated to predict ICU LOS at 2-hour intervals during the initial 12 hours post-admission. The Random Forest (RF) model in the 10- to 12-hour window, with an AUROC of 0.903, was selected for explanation using SHapley Additive exPlanations. Fifteen ICU clinicians assessed four distinct types of explanations ('Why', 'Why not', 'How to', and 'What if') via web-based experiments, surveys, and interviews. Clinicians' feature selections aligned more closely with the RF model after explanations, as demonstrated by an increase in Spearman correlation from -0.147 (p = 0.275) to 0.868 (p < 0.001). The average trust score improved from 2.8 to 3.9. The average satisfaction scores for the 'Why', 'Why not', 'How to', and 'What if' explanations were 3.3, 3.8, 3.6, and 4.1, respectively. AI model explanations notably enhanced clinicians' understanding and trust in AI-generated ICU LOS predictions, although complete alignment with their mental models was not achieved. Further refinement of AI model explanations is needed to support better clinician-AI collaboration and its integration into clinical practice.