Cross-modal recipe retrieval via multi-granularity alignment.

Abstract

With the increasing global emphasis on food safety and nutrition management, the cross-modal recipe retrieval technology has become increasingly prominent in the realm of food computing. However, previous methods generally focus on aligning the recipe texts and images with coarse embeddings for learning a sharing space, which overlooks the alignment of main food ingredients between image and text modalities, making them hard to distinguish those foods with similar ingredients and consequent inaccurate searching results. To overcome the challenge, we propose a Cross-modal Multi-Granularity Alignment (CMGA) method for recipe retrieval task. Specifically, a Key Information Enhancement (KIE) module, which incorporates the cross-attention mechanism and Large Language Model (LLM) is first designed to make the text encoder focus on key explicit food ingredient. And then an Image Detail Enhancement (IDE) module is proposed to capture food components and cooking details in images by combining the sparse attention and dynamic feature fusion module. Finally, a bidirectional contrastive loss function, which can make the module focus on hard samples learning is employed to further distinguish similar recipes. Experiments on the Recipe1M dataset show that CMGA outperforms baseline models in the cross-modal retrieval task for recipes.