A vision-language model for multitask classification of memes.

Abstract

The emergence of social media and online memes has led to an increasing demand for automated systems that can analyse and classify multimodal data, particularly in online forums. Memes blend text and graphics to express complicated ideas, sometimes containing emotions, satire, or inappropriate material. Memes often represent cultural prejudices such as objectification, sexism, and bigotry, making it difficult for artificial intelligence to classify these components. Our solution is the vision-language model ViT-BERT CAMT (cross-attention multitask), which is intended for multitask meme categorization. Our model uses a linear self-attentive fusion mechanism to combine vision transformer (ViT) features for image analysis and bidirectional encoder representations from transformers (BERT) for text interpretation. In this way, we can see how text and images relate to space and meaning. We tested the ViT-BERT CAMT on two difficult datasets: the SemEval 2020 Memotion dataset, which contains a multilabel classification of sentiment, sarcasm, and offensiveness in memes, and the MIMIC dataset, which focuses on detecting sexism, objectification, and prejudice. The findings show that the ViT-BERT CAMT achieves good accuracy on both datasets and outperforms many current baselines in multitask settings. These results highlight the importance of combined image-text modelling for correctly deciphering nuanced meanings in memes, particularly when spotting abusive and discriminatory content. By improving multimodal categorization algorithms, this study helps better monitor and comprehend online conversation.