Researchers from Tsinghua University and Ant Group introduce IFEval++, a benchmark designed to evaluate "nuance-oriented reliability" in large language models, revealing that current models struggle significantly with subtle prompt variations despite high standard benchmark scores.
- The study defines "cousin prompts" as inputs conveying similar intents but differing in phrasing or context, and proposes the reliable@k metric to quantify consistency across them.
- IFEval++ consists of 541 test cases, each containing 10 cousin prompts generated via an automated pipeline using rephrasing, distractor addition, and task reconfiguration.
- Evaluations of 20 proprietary and 26 open-source models show performance drops of up to 61.8% for Qwen3-0.6B and 54.7% for GPT-3.5-turbo-1106 when moving from standard IFEval accuracy to reliable@10 on IFEval++.
- Even the most reliable model, GPT-5, experienced an 18.3% decrease in reliability under nuanced conditions.
- The authors identify parallel test-time scaling via rejection sampling as the most effective improvement method, allowing weaker models like Qwen3-4B to outperform stronger ones like LLaMA-3.3-70B-Instruct.
The findings highlight nuance-oriented reliability as a critical gap in current LLM capabilities and a necessary step toward building more dependable and trustworthy AI systems.