Benchmark · reasoning
GPQA Diamond
GPQA Diamond is the hardest split of GPQA, a set of graduate-level, "Google-proof" multiple-choice science questions in biology, physics, and chemistry written by domain experts. A model is scored by its percentage accuracy at picking the correct answer.
Read more
- Example
- A typical item is a hard multiple-choice question that needs graduate-level reasoning — for instance, a chemistry question describing a reaction mechanism and asking which of four options gives the correct product, so tough that a non-expert can't solve it even with web search.
- Scoring
- The score is the percentage of questions answered correctly (accuracy): the number of items where the model picks the right option divided by the total number of questions.
- Verification
- Each answer is graded automatically by exact match against the single known-correct option, so no human judging is needed; the "Diamond" label itself was assigned during dataset creation, when qualified experts agreed on the answer and non-experts still got it wrong.
- Why it matters
- It is one of the toughest tests of genuine expert-level scientific reasoning (not facts you can look up), so a strong GPQA Diamond score is a headline signal that a frontier model can reason through hard, specialized problems.
Worked example
Task
An electron is in the (unnormalized) spin state (3i, 4)ᵀ. Find the expectation value of its spin along the y-axis, S_y. Options: (A) 12ħ/25 (B) −12ħ/25 (C) 25ħ/2 (D) −25ħ/2.
Solution
Norm: |3i|²+|4|²=25. σ_y=[[0,−i],[i,0]]. ⟨S_y⟩=(ħ/2)·(ψ†σ_yψ)/(ψ†ψ)=(ħ/2)·(−24/25)=−12ħ/25 → (B).
Walkthrough
Using ⟨S_y⟩ = ψ†S_yψ / ψ†ψ with S_y = (ħ/2)σ_y, the numerator ψ†σ_yψ = −24 and the norm ψ†ψ = 25, giving −12ħ/25 (option B). GPQA grades by exact match of the selected letter against the gold key.