Benchmark · reasoning
ZebraLogic
ZebraLogic is a reasoning benchmark of programmatically generated logic-grid ("Einstein"/"Zebra") puzzles that tests an LLM's constraint-satisfaction and deductive reasoning; the headline metric is puzzle-level accuracy — the share of puzzles solved entirely correctly (with cell-level accuracy as a finer-grained secondary metric).
Read more
- Example
- A typical item gives N houses in a row and several attribute categories (e.g. pet, drink, color), each with N distinct values, plus a list of natural-language clues; the model must deduce the one assignment of every value to every house that satisfies all the clues.
- Scoring
- Two metrics. Puzzle-level (exact-match) accuracy = the fraction of puzzles whose entire grid the model fills in correctly; cell-level accuracy = the fraction of individual house–attribute cells assigned correctly. Puzzles are bucketed by grid size / search-space size to expose how accuracy scales with complexity.
- Verification
- Every puzzle is generated with a constraint solver so it has exactly one valid solution; the model's structured output (the full grid) is compared cell-by-cell to that ground truth. A puzzle counts as solved only if all cells match — there is no partial credit at the puzzle level.
- Why it matters
- Logic-grid puzzles require multi-step, systematic deduction that resists memorization and pattern-matching, and are procedurally generated so they are hard to leak into training data. ZebraLogic exposes the "curse of complexity" — accuracy falls sharply as the search space grows — making it a clean probe of genuine reasoning and of how much inference-time reasoning (chain-of-thought / more compute) actually helps.
Worked example
Task
There are 3 houses in a row, numbered 1 (left) to 3 (right). Each house has a different pet (Cat, Dog, Bird) and a different drink (Tea, Water, Milk).
Clues:
1. House 1 owns the Cat.
2. The Cat owner drinks Tea.
3. The Dog owner lives immediately to the left of the Bird owner.
4. The Bird owner drinks Milk.
5. Water is drunk in house 2.
Determine each house's pet and drink.
Solution
- Clue 1: house 1 = Cat. Clue 3: Dog is immediately left of Bird, so (Dog, Bird) occupies houses (2, 3) — house 1 is already Cat. → house 2 = Dog, house 3 = Bird.
- Clue 2: the Cat owner (house 1) drinks Tea. Clue 4: the Bird owner (house 3) drinks Milk. Clue 5: house 2 drinks Water.
- Final: House 1 = Cat, Tea; House 2 = Dog, Water; House 3 = Bird, Milk.
Walkthrough
The clues chain into a single forced assignment, so the solution is unique — exactly the property the generator guarantees. It is graded by exact match of the whole grid against the solver's ground truth: all six house–attribute cells here are correct, so the puzzle scores as solved.
No verified scores reported yet for this benchmark.