Multimodal integrated knowledge transfer to large language models through preference optimization with biomedical applications.

Abstract

The scarcity of high-quality multimodal biomedical data limits the effective fine-tuning of large language models (LLMs) for specialized tasks. We introduce MINT (multimodal integrated knowledge transfer), a framework that aligns unimodal decoder-only models with domain-specific patterns from multimodal biomedical data through preference optimization and that is primarily implemented using the odds ratio preference optimization (ORPO) framework. MINT leverages upstream multimodal machine-learning models to transfer domain expertise to downstream text-only or image-only LLMs, as demonstrated in two applications: (1) rare genetic disease prediction from texts and (2) tissue-type classification using cell nucleus images. In both cases, MINT-based models outperform those enhanced with alternative approaches such as supervised fine-tuning (SFT) and retrieval-augmented generation (RAG), even surpassing much larger foundation models in some scenarios. Our study highlights how MINT effectively grafts the classification strengths of encoder-only models into large decoder-only models, thereby enhancing reasoning abilities and reducing hallucination in biomedical applications.