Uncovering Large Language Model Weaknesses in Character and Word Understanding and Manipulating.

Abstract

Recently, large language models (LLMs) have showcased remarkable capabilities across a diverse range of applications, including general natural language processing (NLP) and domain-specific tasks. Empirical evidence indicates that LLMs have matched or even surpassed human performance in various areas, such as language translation, reading comprehension, and logical reasoning. However, preliminary research reveals that LLMs struggle with basic character and word editing, which is crucial for practical tasks such as creating 1000-word articles or modifying specific text information. To comprehensively assess the capabilities of LLMs in character and word understanding and manipulation (CWUM), we introduce the CWUM benchmark in Chinese and English. CWUM comprises 23 tasks focusing on text editions, including counting, identification, insertion, and reversal. A comprehensive evaluation of nine advanced LLMs on CWUM is conducted, which highlights significant failures of existing LLMs on CWUM tasks that humans can solve perfectly with 100% accuracy. Meanwhile, specific deficiencies of LLMs in basic language understanding and manipulation are revealed by performing quality and quantity analysis. Furthermore, in the experiment part, various methods are investigated to improve model performance, demonstrating the effectiveness of supervised fine-tuning (SFT) in enhancing model performance on CWUM while maintaining generalization abilities on unseen tasks.