Examination of Pathologist-Artificial Intelligence Interactions and Their Impact on Pathologist Accuracy Using Artificial Intelligence-Assisted Scoring of Immunohistochemistry for Human Epidermal Growth Factor Receptor 2.

Abstract

Advances in computer vision have fueled the development of artificial intelligence (AI)-based algorithms for pathology. AI-assisted approaches may streamline the diagnostic workflow and reduce variability. To assess the impact of an AI-assist model for human epidermal growth factor receptor 2 (HER2) scoring on pathologist reproducibility and accuracy and to understand pathologist-model interactions. An AI-Assist algorithm for HER2 scoring, AI-Measurement of HER2 (AIM-HER2), was developed to generate slide-level scores of HER2 immunohistochemistry (IHC) aligned with guidelines from the American Society of Clinical Oncology/College of American Pathologists. AIM-HER2 was assessed in a retrospective reader study wherein HER2-trained pathologists (n = 20) scored breast cancer cases (n = 200) with and without model assistance using a 2-cohort crossover design with a 3-week washout. A separate panel of expert pathologists (n = 5) provided manual reference scores. As an AI-assist tool, AIM-HER2 improved interrater agreement both overall and at the 0/1+ and 1+/2+ cutoffs and significantly increased positive percentage agreement at the 0/1+ and 1+/2+ cutoffs. Pathologists displayed a wide range of model override rates, and the quality of these overrides was correlated with each pathologist's manual accuracy. Measurements of AIM-HER2 accuracy were highly dependent on reference panel composition. The use of AI-assist tools, such as AIM-HER2, for scoring HER2 IHC in breast cancer may improve pathologist reproducibility and accuracy, particularly at the 0/1+ and 1+/2+ cutoffs. However, improved consistency of pathologist interpretation of AI-assisted IHC scoring guidance may be necessary for AI-assist tools to reach their full potential.