Benchmark · reasoning
Humanity's Last Exam
Humanity's Last Exam (HLE) is a benchmark of frontier, expert-level questions spanning many academic domains (mathematics, sciences, humanities, and more), deliberately built to be extremely hard for AI. It reports performance as percent accuracy.
Read more
- Example
- Each item is a closed-ended expert question with a single correct answer — for instance, an advanced mathematics problem or a graduate-level science question (some with an accompanying image), posed as multiple choice or an exact short answer.
- Scoring
- The metric is accuracy: the percentage of questions the model answers correctly. Some versions also report a calibration (confidence) measure alongside accuracy.
- Verification
- Each question has a known correct answer, and a model's response is graded automatically against that reference (the correct option for multiple choice, a matching short answer) — objective automated grading, not human votes.
- Why it matters
- Standard benchmarks have become saturated (top models score near the ceiling), so HLE aims to be a hard, discriminating test at the frontier of human expert knowledge — a meaningful gauge of how far AI is from expert-level reasoning.
Worked example
Task
Hummingbirds (order Apodiformes) uniquely possess a bilaterally paired, oval sesamoid bone embedded in the caudolateral part of the expanded, cruciate aponeurosis of insertion of the m. depressor caudae. How many paired tendons does this sesamoid bone support? Answer with a single integer.
Solution
4
Walkthrough
The oval sesamoid forms within the cruciate aponeurosis of the tail-depressor muscle and supports four paired tendons of that insertion — a specialized ossification unique to hummingbird tail anatomy. HLE grades the response by exact match of the submitted integer against the reference answer (4) and separately elicits a stated confidence for calibration scoring.