Who's really in the loop? Rethinking oversight in AI-assisted health care.

Abstract

Human-in-the-loop oversight is widely invoked as a safeguard against potential harm from artificial intelligence (AI) used in health care, yet it functions more as symbolic reassurance than substantive protection. We argue that human-in-the-loop fails for three interconnected reasons: AI used in health care can amplify existing structural inequities at unprecedented scale, intersectional harms elude detection by oversight models premised on neutral singular reviewers, and clinicians operate under constraints that preclude meaningful interrogation of algorithmic outputs. Drawing on actor-network theory, feminist epistemology, and political philosopher Iris Marion Young's social connection model of justice, we show that current governance individualises responsibility while obscuring institutional complicity. We propose three pathways towards more substantive accountability: co-reasoning frameworks that position AI as one voice in clinical deliberation, community-owned governance with authority to suspend harmful systems, and institutional liability structures that redistribute responsibility from clinicians to the organisations that design and deploy these tools.