Clinical evaluation of MiADE: a natural language processing system for assisting structured diagnosis recording at the point of care.

Abstract

To evaluate the usability, usefulness and impact of a novel point of care natural language processing (NLP) system, Medical information AI Data Extractor (MiADE), to assist structured diagnosis recording in electronic health records. Mixed methods evaluation of the implementation of MiADE in a major National Health Service hospital, with surveys, interviews and observed outpatient consultations. The number of structured diagnoses recorded per outpatient encounter was compared before and after MiADE, and completeness of inpatient problem lists was evaluated using billing diagnoses as a gold standard. 85 clinicians consented to the study and were provided access to MiADE and 24 used MiADE to receive structured data suggestions during the study period. Baseline survey data and observations showed wide variation in structured data recording despite clinicians considering it to be important. Half of postimplementation survey respondents considered MiADE to be 'very' or 'moderately' useful. Multilevel quasi-Poisson regression of 12 309 outpatient encounters (accounting for time and clustering by clinician) estimated that the post-MiADE period was associated with 23.7% more diagnoses recorded per encounter. No improvement was seen in the inpatient setting. Structured recording of key information such as diagnoses using a clinical terminology is essential for safe, efficient patient care, but is currently done incompletely because it is time-consuming for clinicians. MiADE was associated with an increase in outpatient structured diagnosis recording despite low uptake of the tool. Point of care NLP using MiADE can potentially improve structured data recording, but further development and better clinician engagement are needed to maximise its impact. ISRCTN58300671.