SecReEvalBench: A real-world scenario-based security resilience benchmark for large language models.

Abstract

We study the security resilience of large language models to prompt-chain attacks that exploit context retention and sequencing. Existing evaluations emphasize single-turn or fixed-domain settings and thus under-measure vulnerabilities to sequence-aware adversaries. We introduce SecReEvalBench, a scenario-based benchmark with six attack sequences (one-off, successive, successive-reverse, alternative, sequential ascending, sequential descending) and four metrics tailored to multi-turn security: Prompt Attack Resilience Score, Prompt Attack Refusal-Logic Score, Chain Attack Resilience Score, and Chain Attack Refusal Timing Score. The accompanying dataset spans seven security domains, sixteen attack techniques, and four maliciousness levels with dual-LLM adjudication for intent labels. Our evaluation protocol combines an unsafe-content detector (Llama-Guard 3) with a refusal classifier to disentangle eventual unsafe cooperation from refusal timing, yielding bounded, auditable, and sequence-sensitive measurements. Experiments on five representative open-weight models (Llama 3.1, Gemma 2, Mistral v0.3, DeepSeek-R1, Qwen 3) and two proprietary models (gpt-3.5-turbo and gpt-4o-mini) show that sequence-aware metrics surface vulnerabilities that single-turn tests miss, including delayed or absent refusals under escalation and leakage through visible intermediate reasoning despite surface-level refusals. SecReEvalBench standardizes multi-turn safety assessment and provides a reproducible basis for developing domain- and sequence-aware defenses. Our dataset is made publicly available at https://kaggle.com/datasets/5a7ee22cf9dab6c93b55a73f630f6c9b42e936351b0ae98fbae6ddaca7fe248d.