Applying Conformal Prediction to a Deep Learning Model for Intracranial Hemorrhage Detection to Improve Trustworthiness.

Abstract

Purpose To apply conformal prediction to a deep learning (DL) model for intracranial hemorrhage (ICH) detection and evaluate model performance in detection as well as model accuracy in identifying challenging cases. Materials and Methods This was a retrospective (November-December 2017) study of 491 noncontrast head CT volumes from the CQ500 dataset, in which three senior radiologists annotated sections containing ICH. The dataset was split into definite and challenging (uncertain) subsets, in which challenging images were defined as those in which there was disagreement among readers. A DL model was trained on patients from the definite data (training dataset) to perform ICH localization and classification into five classes. To develop an uncertainty-aware DL model, 1546 sections of the definite data (calibration dataset) were used for Mondrian conformal prediction (MCP). The uncertainty-aware DL model was tested on 8401 definite and challenging sections to assess its ability to identify challenging sections. The difference in predictive performance (<i>P</i> value) and ability to identify challenging sections (accuracy) were reported. Results The study included 146 patients (mean age, 45.7 years ± 9.9 [SD]; 76 [52.1%] men, 70 [47.9%] women). After the MCP procedure, the model achieved an F1 score of 0.919 for localization and classification. Additionally, it correctly identified patients with challenging cases with 95.3% (143 of 150) accuracy. It did not incorrectly label any definite sections as challenging. Conclusion The uncertainty-aware MCP-augmented DL model achieved high performance in ICH detection and high accuracy in identifying challenging sections, suggesting its usefulness in automated ICH detection and potential to increase trustworthiness of DL models in radiology. <b>Keywords:</b> CT, Head and Neck, Brain, Brain Stem, Hemorrhage, Feature Detection, Diagnosis, Supervised Learning <i>Supplemental material is available for this article.</i> © RSNA, 2025 See also commentary by Ngum and Filippi in this issue.