Benchmark · math
AIME 2024
AIME 2024 measures a model's advanced mathematical reasoning using the 15 problems from the 2024 American Invitational Mathematics Examination, reporting the percentage solved correctly.
Read more
- Example
- A typical item is a hard high-school competition problem — for example a number-theory or combinatorics question whose answer is a single integer (each AIME answer is an integer, conventionally from 0 to 999).
- Scoring
- The metric is accuracy: the share of the 15 problems answered correctly, so each solved problem is worth 1/15 of the score.
- Verification
- A result is accepted by exact match — the model's final integer answer must equal the official answer, checked automatically with no partial credit and no human judging.
- Why it matters
- Because it is a compact set of hard, recent competition problems that demand multi-step reasoning, it is widely used to compare frontier models, and its recency limits training-data contamination.
Worked example
Task
How many positive integers n ≤ 600 are divisible by 4 or by 6 but not by 5? (AIME answers are integers from 0 to 999.)
Solution
Multiples of 4 or 6 that are ≤ 600: 150 + 100 − 50 = 200. Of these, the ones also divisible by 5 (multiples of 20 or 30, minus 60): 30 + 20 − 10 = 40. Answer: 200 − 40 = 160.
Walkthrough
By inclusion–exclusion, 200 integers are divisible by 4 or 6, and 40 of those are also divisible by 5, leaving 160. AIME auto-grades only the final integer (0–999), with no partial credit for the work.