Use of statistical analysis of cytologic interpretation to determine the causes of interobserver disagreement and in quality improvement.

Abstract

Disagreements in cytologic interpretation can have several causes, including differences in diagnostic threshold and diagnostic accuracy. These can be distinguished by a combination of statistical analyses. For demonstration purposes, a nonrandom collection of 80 cervicovaginal smears, the majority of which (74) were originally diagnosed as atypical cells of undetermined significance (ASCUS), were reviewed by 3 separate observers and classified as either negative, negative and reactive, ASCUS favor reactive, ASCUS not otherwise specified, ASCUS suggestive of a squamous intraepithelial lesion (SIL), low grade SIL, or high grade SIL. The results were compared with corresponding biopsies and analyzed with distribution analysis, the kappa statistic, threshold analysis, and receiver operating characteristic (ROC) curve analysis. Distribution analysis of diagnoses from the three observers demonstrated statistically significant differences in how cases were classified and a low level of agreement. Kappa analysis confirmed a very poor interobserver agreement. Threshold analysis revealed that one observer used a threshold between negative and ASCUS that was statistically more specific but less sensitive than the other observers. ROC curve analysis showed that another observer was more accurate than this observer. Variation in cytologic interpretation may have several causes. Distribution, threshold, and ROC analysis allow distinction between differences in diagnostic accuracy and diagnostic thresholds. This approach to analyzing cytologic interpretation may be useful for quality improvement efforts.