Enhancing adverse drug event extraction and summarization for cancer drugs through large language models.

Abstract

Cancer treatments such as chemotherapy, targeted therapy, and immunotherapy can effectively combat malignant cells but frequently cause serious side effects by damaging healthy tissues. This underscores the need for robust Adverse Drug Event (ADE) reporting and summarization frameworks to improve patient safety, support clinical decision-making, and optimize treatment outcomes. In this work, we propose the Multitasking Multimodal Pharmacovigilance Summarization Framework (MMPSF), where we project image embeddings into the textual latent space using a learned projection matrix. The image and text embeddings are then concatenated to generate comprehensive multimodal summaries of ADEs reported by cancer patients. The framework incorporates direct preference optimization (DPO) using a preference-aware loss function to align the LLM-generated summaries with human-written summaries. To support this task, we introduce the MMCADRS dataset, comprising 1800 ADE-focused narratives and 500 medically relevant images capturing visible symptoms. The proposed MMPSF framework surpassed all summarization baselines. Performance improvements were driven by visual cue integration (0.20↑), entity extraction (0.190↑), and direct preference optimization (0.267↑). Human evaluations confirmed clinical accuracy (4.18), factual recall (0.92), and omission rate (0.19), demonstrating its effectiveness in producing clinically reliable summaries. Quantitative metrics and human evaluations confirm the effectiveness of MMPSF in generating accurate, cancer-aware multimodal summaries. This framework holds significant promise for real-world deployment in pharmacovigilance for cancer care.