Benchmark · general
LMSYS Arena (Elo)
LMSYS Chatbot Arena measures how good AI chatbots are at open-ended conversation by having real people compare two anonymous models side by side. Results are aggregated into an Elo rating, the same relative-skill number used in chess (roughly 1000-1500+).
Read more
- Example
- A visitor types a free-form prompt — say, 'Explain quantum entanglement to a 10-year-old' or 'Write a Python function to reverse a linked list' — sees two hidden models answer, then picks the better reply.
- Scoring
- Each blind vote is a pairwise win/loss/tie; these votes are pooled across many users and converted into an Elo score, where a higher number means the model wins more of its matchups.
- Verification
- There are no automated tests or reference answers — a result is decided purely by crowdsourced human preference votes, cast blind so brand names can't bias the choice.
- Why it matters
- Because it reflects real users' subjective judgment on everyday prompts rather than a fixed test set, it's a widely watched, hard-to-game leaderboard for overall chatbot quality.
Worked example
Task
Chatbot Arena prompt (open-ended, two models compared side-by-side): 'Explain the difference between TCP and UDP to a curious 10-year-old using a simple analogy, then name one everyday app that relies on each.' Two anonymous models answer the same prompt and you vote for the better reply.
Solution
Strong reply: 'TCP = numbered letters the post office confirms and resends if lost, so everything arrives in order — web browsing, email. UDP = shouting across a playground: fast, but missed words are never resent — video calls, online games.' No single gold answer exists; the winner is whichever reply a human rates more helpful, accurate, and clear.
Walkthrough
The reply is judged good because it is technically accurate (TCP = reliable and ordered, UDP = fast, best-effort), uses one age-appropriate analogy, and covers both requested parts (an app for each). Grading: there is no gold label — two anonymous models answer the same prompt, a human picks the better response, and pairwise votes aggregate into an Elo/Bradley-Terry leaderboard.