Early clinical implementation and evaluation of an NLP-Based AI system for thrombophilia assessment using electronic health records.

Abstract

Thrombophilia evaluation requires integration of biochemical findings with clinical history, much of which is embedded in unstructured electronic health record (EHR) text. Manual chart review is labour-intensive and prone to omissions. Natural language processing (NLP) offers a potential alternative by automatically identifying and highlighting relevant information to support more efficient and accurate assessments. We developed a transformer-based NLP model for thrombosis and integrated it with previously validated bleeding (transformer-based), anticoagulant, and antithrombotics (rule-based) models in an AI system that highlights key phrases in unstructured Danish EHRs to expedite clinician chart review. The system was implemented in routine clinical care to support thrombophilia evaluations. We retrospectively evaluated its performance based on 50 real-world EHRs reviewed by clinicians using the system. Eye-tracking was used to assess information-seeking behaviour during simulated reviews, and semi-structured interviews explored adoption potential post-implementation. The thrombosis model achieved high sentence-level performance (sensitivity 98.8%, specificity 99.8%). In retrospective review, AI-assisted clinicians identified all previously documented findings, and identified additional relevant information in 68% of cases. Eye-tracking showed a 33% reduction in review time and improved identification of relevant content. Users adopted two distinct navigation styles: AI-guided scanning and manual scrolling, reflecting varied trust and cognitive strategies. Interviews revealed strong support for system accuracy and efficiency, with integration and training identified as key challenges. This evaluation highlights the potential of NLP tools to support clinical decision-making. Clear model design, sentence-level transparency, and user-centred evaluation are essential for safe and effective integration into clinical workflows.