Interrater Agreement and Diagnostic Accuracy of a Novel Computer-Aided Detection Process for the Detection and Prevention of Retained Surgical Instruments.

Abstract

The purpose of this study is to evaluate the diagnostic accuracy of a process incorporating computer-aided detection (CAD) for the detection and prevention of retained surgical instruments using a novel nondeformable radiopaque μTag. A high-specificity CAD system was developed iteratively from a training set (n = 540 radiographs) and a validation set (n = 560 radiographs). A novel test set composed of 700 thoracoabdominal radiographs (410 with a randomly placed μTag and 290 without a μTag) was obtained from 10 cadavers embedded with confounding iatrogenic objects. Data were analyzed first by the blinded CAD system; radiographs coded as negative (n = 373) were then independently reviewed by five blinded radiologists. The reference standard was the presence of a μTag. Sensitivity and specificity were calculated. Interrater agreement was assessed with Cohen kappa values. Mean (± SD) image analysis times were calculated. The high-specificity CAD system had one false-positive (sensitivity, 79.5% [326/410]; specificity, 99.7% [289/290]). A combination of the CAD system and one failsafe radiologist had superior sensitivity (98.5% [404/410] to 100% [410/410]) and specificity (99.0% [287/290] to 99.7% [289/290]), with 327 (47%) radiographs not requiring immediate radiologist review. Interrater agreement was almost perfect for all radiologist pairwise comparisons (κ = 0.921-0.992). Cumulative mean image analysis time was less than one minute (CAD, 29 ± 2 seconds; radiologists, 26 ± 16 seconds). The combination of a high-specificity CAD system with a failsafe radiologist had excellent diagnostic accuracy in the rapid detection of a nondeformable radiopaque μTag.