Are LLMs qualified evaluators and generators of one-liner jokes.

Abstract

Humor evaluation and generation remain challenging problems in computational humor due to their subjective nature. Although large language models (LLMs) have demonstrated strong performance in various creative tasks, prior research has primarily focused on humor generation-particularly puns-while their ability to reliably evaluate humor and generate structured one-liner jokes remains largely unexplored. To address this gap, this paper introduces OneLinerJoke (OLJ), the first benchmark dataset designed to assess LLMs' capability to evaluate one-liner jokes, and proposes a capability-aware prompting framework that integrates the structured Joke Prospector methodology with probing-informed prompt adaptation. This approach first probes the model's conceptual understanding and then reformulates prompts to align with its knowledge, improving generation reliability and reducing hallucinations. Experimental results show that LLMs can consistently distinguish funny and unfunny jokes and that capability-aware prompting significantly improves the quality of generated jokes compared to conventional prompting methods. These findings demonstrate the feasibility of using LLMs as both humor evaluators and generators, advancing research in computational humor and prompt engineering.