The GPQA-Diamond benchmark, introduced in late 2023 by researchers at New York University and Anthropic, consists of 198 graduate-level multiple-choice questions in biology, chemistry, and physics designed to be Google-proof. It serves as a rigorous evaluation tool for scientific reasoning and scalable oversight, requiring deep domain expertise rather than simple information retrieval.
- The benchmark features 198 questions crafted by PhD-level experts, with non-experts achieving ~34% accuracy while experts score ~65–70%.
- Early models like GPT-4 scored ~39%, but state-of-the-art systems have since surpassed human performance.
- OpenAI’s O1 achieved roughly 77% accuracy in September 2024, exceeding expert benchmarks.
- Autopoiesis Sciences’ Aristotle-X1 reached 92.4% in mid-2025, followed by Google’s Gemini 3.1 Pro Preview at ~94.1% in February 2026.
- Other top performers include GPT-5.2 (~92.4%), Gemini 3 Pro (~91.9%), and Claude Opus 4.6 (~91.3%).
GPQA-Diamond’s rapid saturation highlights its role as a measuring stick for scientific reasoning and underscores the need for more complex evaluations as AI capabilities advance.