Benchmark · general
IFEval
IFEval (Instruction-Following Eval) measures how reliably a language model obeys explicit, machine-checkable formatting and constraint instructions in a prompt, such as length or format rules. The score is the percentage of these verifiable instructions the model actually satisfies.
Read more
- Example
- A prompt like "Write a summary in exactly 3 bullet points and do not use any commas" — the model must produce an answer that meets every stated constraint.
- Scoring
- Each response is automatically checked against its verifiable constraints (bullet count, word count, forbidden characters, required keywords, casing, etc.), and the score is the percentage satisfied, reported at both the instruction level and the whole-prompt level (every constraint in a prompt met).
- Verification
- Verification is fully automatic: a small deterministic program tests each constraint (for example, counting the bullets or scanning for commas), so there is no human judging or grading by another LLM.
- Why it matters
- It isolates whether a model can follow precise instructions from whether its answer is factually correct, which matters because dependable formatting and constraint-following is what makes models usable in real apps and automated pipelines.
Worked example
Task
Write a short tip on staying productive while working from home, in exactly two paragraphs. Write your entire response in all lowercase letters (no capital letters allowed) and do not use any commas anywhere. End your response with the exact phrase: 'that is all.'
Solution
working from home rewards a bit of structure so pick three goals each morning and give every goal its own block of time on your calendar. put your phone in another room and take a short walk between blocks to reset your attention.
a clear desk and a fixed start time help your brain shift into work mode fast. stop at a set hour and celebrate one finished task so you never drift toward burnout. that is all.
Walkthrough
The response obeys every verifiable constraint: exactly two paragraphs, no uppercase characters, zero commas, and the exact closing phrase. IFEval grades each instruction with a deterministic Python verifier and reports strict/loose accuracy at both the instruction level and the whole-prompt level (a prompt counts only if every instruction passes).
| Date | Model | Score | Source |
|---|---|---|---|
| 2026-06-30 | Agents-A1 | 80.6pts | Scaling the Horizon, Not the Parameters: Reaching Trillion-Parameter Performance with a 35B Agent |
| 2025-12-15 | GPT-5 | 95.9% | Tsinghua and Ant Group find LLM instruction-following reliability drops up to 61.8% on nuanced prompts |
| 2025-04-14 | GPT-4.1 | 87.4% | OpenAI releases GPT-4.1 family with 1M token context and coding improvements |
| 2025-02-24 | Claude 3.7 Sonnet | 93.2% | Anthropic releases Claude 3.7 Sonnet with dynamic reasoning control |
| 2024-12-17 | Falcon3-1B-Instruct | 54.4% | Falcon3 family releases five open models with improved science, math, and code capabilities |
| 2024-12-17 | Falcon3-10B-Instruct | 78.0% | Falcon3 family releases five open models with improved science, math, and code capabilities |