Benchmark · reasoning
BIG-Bench Hard
BIG-Bench Hard (BBH) is a suite of 23 challenging tasks drawn from BIG-Bench on which earlier language models failed to beat the average human rater; it measures multi-step reasoning and is scored by exact-match accuracy.
Read more
- Example
- A task type such as Boolean Expressions — evaluating a nested True/False expression — alongside others like multi-step arithmetic, date understanding, logical deduction, tracking shuffled objects, and navigation, each needing several reasoning steps to reach one answer.
- Scoring
- Per-item exact-match accuracy: the model's extracted final answer must exactly equal the gold label (a multiple-choice option or short string). The headline number is the macro-average accuracy across the 23 tasks, usually reported for both direct answer-only and few-shot chain-of-thought prompting.
- Verification
- A prediction counts as correct only when the extracted final answer string exactly matches the reference answer; results are typically compared against the average and maximum human-rater baselines from BIG-Bench, with chain-of-thought and answer-only prompting reported separately.
- Why it matters
- It isolates the hard reasoning subset where models historically trailed humans, and it was the setting that showed chain-of-thought prompting can push large models past the average human rater — making it a standard probe of multi-step reasoning.
Worked example
Task
Boolean Expressions task — Evaluate the result of the following Boolean expression: not ( True and False )
Solution
Step 1: True and False = False. Step 2: not False = True. Final answer: True
Walkthrough
Boolean 'and' is False unless both operands are True, so (True and False) is False; negating False gives True. Grading is exact string match of the final answer against the gold label.
| Date | Model | Score | Source |
|---|---|---|---|
| 2024-12-17 | Falcon3-7B-Base | 51.0% | Falcon3 family releases five open models with improved science, math, and code capabilities |
| 2024-12-17 | Falcon3-10B-Base | 59.7% | Falcon3 family releases five open models with improved science, math, and code capabilities |
| 2024-07-15 | Qwen2-72B | 82.4% | Qwen2 Technical Report introduces dense and MoE models up to 72B |
| 2024-07-15 | Qwen2-72B | 82.4% | Qwen Team releases Qwen2 series with 72B dense and MoE models |
| 2024-06-20 | Claude 3.5 Sonnet | 93.1% | Anthropic releases Claude 3.5 Sonnet addendum with improved coding and vision benchmarks |