Benchmark · coding
Aider Polyglot
Aider Polyglot is a code-editing benchmark from the Aider coding tool that tests how well an LLM edits existing code across many programming languages. The score is the percentage of tasks whose edits produce correct code that passes the task's tests.
Read more
- Example
- A typical item is an Exercism-style coding exercise in one of several languages (Python, Rust, Go, Java, JavaScript, or C++), where the model must edit the provided source files to implement the required function so its tests pass.
- Scoring
- The metric is the percentage of tasks solved: a task counts only when the edited code passes all of its automated tests. Aider also tracks how often edits come back in the correct, applicable format.
- Verification
- Results are checked automatically: the edited files are run against each exercise's unit-test suite, and a task passes only if every test passes — no human votes or exact-match comparison.
- Why it matters
- It mirrors a real developer workflow — editing existing files across many languages rather than writing snippets from scratch — so it is a practical signal of how useful a model is as a coding assistant.
Worked example
Task
Aider Polyglot draws exercises from Exercism across 6 languages: the model gets a problem spec plus a stub file (e.g.
acronym.py containing def abbreviate(words): ...) and must edit it so the hidden pytest suite passes. Example spec: convert a phrase to its acronym — 'Portable Network Graphics' → 'PNG', treating spaces, hyphens and underscores as separators and ignoring other punctuation.Solution
import re
def abbreviate(words):
return "".join(w[0].upper() for w in re.findall(r"[A-Za-z']+", words))Walkthrough
The regex pulls out word tokens (letters plus intra-word apostrophes), takes each token's first letter, uppercases it and joins them, so 'Portable Network Graphics' → 'PNG'. Grading runs the exercise's hidden unit tests; the item scores only if every test passes (aider reports pass@2 over the suite).