Enabling interoperability across disparate health data sources using Large Language Models.

Abstract

The heterogeneity of health data systems remains a major barrier to interoperability and data integration, and a challenge for clinical decision support and AI-driven research. The coexistence of multiple Common Data Models (CDMs) complicates integration, requiring complex manual or rule-based schema mapping, which is labor-intensive, error-prone, and difficult to scale. We evaluated three leading Large Language Models (LLMs): ChatGPT, Gemini, and DeepSeek, for their ability to match data elements from four widely used CDMs (PCORnet, OMOP, Sentinel, and i2b2) to FHIR R4 via web-based chat interfaces. Reference mappings from the HL7 Common Data Models Harmonization Implementation Guide were used as the benchmark. Standardized prompts were submitted across 36 total runs (3 LLMs × 4 schemas × 3 repetitions), and accuracy was measured using F1-scores with 95% confidence intervals, alongside strict (AND) and relaxed (OR) match criteria. DeepSeek obtained the highest numerical mean F1-scores (0.78-0.99) with strong consistency (AND Match = 0.87). Gemini showed good coverage (OR Match = 0.96) but greater variability (AND Match = 0.64), while ChatGPT achieved moderate accuracy with lower consistency (AND Match = 0.51). A two-way ANOVA indicated no statistically significant differences by LLM type (p = 0.138), CDM schema (p = 0.194), or their interaction (p = 0.451). Performance varied by domain, with Demographics and Condition mapped most accurately, while Observation showed the highest error rates (44% of total errors). Out of 2,583 total matchings, 314 errors were observed and categorized into four failure patterns: semantic confusion (42%), structural misalignment (28%), ambiguous terminology (18%), and terminology variance (12%). This study demonstrates the feasibility of using LLMs to automate health data integration as a complementary approach to traditional ontology-based methods, particularly for well-structured, context-rich CDMs. However, safe deployment requires human-in-the-loop validation, governance, prompt optimization, metadata integration, and validation to ensure clinical and research reliability.