Benchmark · agentic
GAIA
GAIA (General AI Assistant) tests whether AI assistants can answer real-world, multi-step questions that require using tools, browsing the web, and reasoning across several sources. The score is the percentage of questions answered correctly.
Read more
- Example
- A typical item is an everyday-sounding question whose answer is buried across several steps — e.g. "look up a fact in a linked document, then combine it with data from a website to reach one short answer" — requiring browsing and tool use rather than a single lookup.
- Scoring
- The metric is accuracy: the share of questions whose final answer matches the expected answer. Each question has one unambiguous reference answer, so a response is either right or wrong.
- Verification
- Answers are checked automatically by exact (quasi-exact) string match against the reference answer, and test-set results are submitted to a public leaderboard where the correct answers are kept private.
- Why it matters
- It measures practical assistant ability — combining reasoning, web browsing, and tools on tasks that are easy for people but hard for AI — making it a common yardstick for autonomous agents.
Worked example
Task
Using the current English Wikipedia, how many studio albums did the band Radiohead release from 1993 through 2007, inclusive? Give a single integer as your final answer.
Solution
Albums in range: Pablo Honey (1993), The Bends (1995), OK Computer (1997), Kid A (2000), Amnesiac (2001), Hail to the Thief (2003), In Rainbows (2007) → 7. Final answer: 7
Walkthrough
GAIA items have one unambiguous ground-truth answer reached by web browsing plus light aggregation — here, counting discography entries within a date window (compilations and live albums excluded). It is graded by quasi-exact match: the normalized final-answer string must exactly equal the reference '7'.