Benchmark · math
FrontierMath
FrontierMath is a benchmark from Epoch AI that measures how well an AI model can solve extremely hard, original research-level mathematics problems. The score is the percentage of problems the model answers correctly, and today's models solve only a small fraction — far from saturation.
Read more
- Example
- A single, previously unpublished problem from an advanced field such as number theory, algebraic geometry, or combinatorics, whose solution demands deep expertise and reduces to one definite final answer (for example a specific integer or an exact mathematical object).
- Scoring
- The metric is accuracy: the fraction of problems whose final answer exactly matches the reference answer, reported as a percentage.
- Verification
- Each problem has a single definite, automatically checkable answer, so a solution is accepted only when it matches the reference exactly; the problems are crafted by expert mathematicians to be almost impossible to get right by guessing.
- Why it matters
- Because the problems are novel it resists memorization, and it remains far from being solved, making it one of the few math benchmarks that still cleanly separates genuine advanced mathematical reasoning from pattern-matching.
Worked example
Task
Determine the number of integers $n$ with $0 \le n < 3^{13}$ for which the central binomial coefficient $\binom{2n}{n}$ is not divisible by $3$.
Solution
By Kummer's theorem, $3 \nmid \binom{2n}{n}$ exactly when adding $n+n$ in base 3 produces no carries, i.e. every base-3 digit of $n$ is 0 or 1. Over $0 \le n < 3^{13}$ this gives 2 choices for each of the 13 digits, so the count is $2^{13} = 8192$.
Walkthrough
Kummer's theorem gives $v_3\binom{2n}{n}$ as the number of carries when adding $n$ to itself in base 3; zero carries forces each base-3 digit to be 0 or 1, i.e. 2 choices per digit over 13 digits. Grading: an automated checker exact-matches the single final integer (8192).