Human in the loop in AI-enabled clinical decision support: a systematic scoping review and reporting checklist for lifecycle governance.

Abstract

To examine the characteristics, implementation strategies, and reported impacts of Human-in-the-Loop (HITL) processes across the lifecycle of AI-enabled Clinical Decision Support Systems (CDSS), and to propose a reporting checklist for HITL in clinical AI research. HITL is a conceptual and technical approach that integrates human expertise into AI systems, widely recognised for its potential to improve transparency, safety, and contextual relevance in clinical settings. While AI-CDSS tools are increasingly embedded in healthcare, HITL practices remain poorly defined and unevenly implemented. This review aimed to systematically map how HITL is conceptualised and deployed across the lifecycle of AI-CDSS. A comprehensive search was conducted across MEDLINE, Embase, Web of Science, PsycINFO, Google Scholar, and Scopus in August 2024, followed by a manual identification phase concluding in mid-2025. We included primary studies involving healthcare professionals interacting with AI-enabled CDSS in any clinical setting, where HITL processes were described at any stage of the model lifecycle (development, review, oversight, or maintenance). Studies were excluded if they lacked real-world clinical relevance or did not involve clinician interaction with AI systems. Study selection and data extraction were performed by two independent reviewers using Covidence. Extracted data were mapped to four idealised phases of the AI-CDSS lifecycle. Twelve studies met the inclusion criteria. All included clinician involvement during development-primarily through expert annotation and rule-based model design. Fewer studies reported HITL processes during review (n = 11), oversight (n = 0), or maintenance (n = 2) phases. HITL was frequently limited to static or retrospective contributions, with few examples of iterative or real-time engagement. Most studies involved small-scale datasets and a limited number of annotators. Conceptual ambiguity and inconsistent terminology further limited clarity in how HITL was defined or justified. HITL utilisation and reporting predominantly focused on early-stage model development, with limited attention to oversight or iterative refinement. The small scale and narrow scope of clinician involvement may constrain generalisability and impact. To address these gaps, we propose a pragmatic HITL reporting checklist structured around six domains of the AI-CDSS life cycle. Adoption of standardised HITL reporting can support the governance, regulatory compliance, and safe translation of clinical AI systems into routine care.