Benchmark · math
FrontierMath
FrontierMath is a benchmark from Epoch AI that measures how well an AI model can solve extremely hard, original research-level mathematics problems. The score is the percentage of problems the model answers correctly, and today's models solve only a small fraction — far from saturation.
Read more
- Example
- A single, previously unpublished problem from an advanced field such as number theory, algebraic geometry, or combinatorics, whose solution demands deep expertise and reduces to one definite final answer (for example a specific integer or an exact mathematical object).
- Scoring
- The metric is accuracy: the fraction of problems whose final answer exactly matches the reference answer, reported as a percentage.
- Verification
- Each problem has a single definite, automatically checkable answer, so a solution is accepted only when it matches the reference exactly; the problems are crafted by expert mathematicians to be almost impossible to get right by guessing.
- Why it matters
- Because the problems are novel it resists memorization, and it remains far from being solved, making it one of the few math benchmarks that still cleanly separates genuine advanced mathematical reasoning from pattern-matching.
Worked example
FrontierMath items are self-contained, research-level math problems whose answer is a single exact value that an automatic checker (often SymPy) can verify, and which is large or specific enough to be effectively unguessable. A representative item in that style: "For each prime p, let ord_p(2) be the multiplicative order of 2 modulo p — the least positive integer k with 2^k ≡ 1 (mod p). Compute Σ ord_p(2) over all primes p with 100 < p < 200, and give the exact integer." Answer: 2140. Reasoning: there are 21 such primes, and for each one ord_p(2) must divide p−1 (Fermat's little theorem), so you test the divisors of p−1 in increasing order for the smallest k with 2^k ≡ 1 — e.g. modulo 127 you get 2^7 = 128 ≡ 1 so ord = 7, while modulo 101 the number 2 is a primitive root so ord = 100. Adding the 21 orders yields 2140, which the grader checks by exact integer match. (Real FrontierMath problems are far harder and often take an expert mathematician hours, but the format — one precise, machine-checkable number — is exactly this.)
No verified scores reported yet for this benchmark.