Collaborate large and small language models for multi-modal emergency rumor detection.

Abstract

Multi-modal emergency rumors are spreading in the current digital era, causing significant disruptions and negative impacts. Most existing methods focus on exploring rumor detection using individual small language models (SLMs) or large language models (LLMs), achieving a certain degree of success but with underlying issues. Approaches based on SLMs have reached a bottleneck due to their limited knowledge and capacity. In contrast, LLMs have unique strengths in deep analysis that compensate for the weaknesses of SLMs; however, they struggle to select and integrate analyses to draw appropriate conclusions. Furthermore, recent works on multi-modal feature fusion remain superficial, limiting the ability of these models to fully comprehend and identify rumors. In this work, we propose Collaborate Large and Small Language Models for Multi-Modal Emergency Rumor Detection (M2ERD). Specifically, it consists of two main components. First, LLMs generate multi-dimensional rationales based on multi-perspective prompts, from which SLMs selectively derive insights for rumor detection. Second, a multi-source cross-modal penetration fusion network not only accomplishes unidirectional fusion of auxiliary information such as multi-dimensional rationales but also achieves complete mutual complementation between text and the image. Comprehensive experiments demonstrate the effectiveness of M2ERD for rumor detection on Weibo, RumorEval, and Pheme datasets, achieving a 2.6% improvement in accuracy and a 1.9% improvement in F1-score compared to all baselines. We release the code and data at https://github.com/youchengyan/M2ERD.