Benchmark · coding
Aider Polyglot
Aider Polyglot is a code-editing benchmark from the Aider coding tool that tests how well an LLM edits existing code across many programming languages. The score is the percentage of tasks whose edits produce correct code that passes the task's tests.
Read more
- Example
- A typical item is an Exercism-style coding exercise in one of several languages (Python, Rust, Go, Java, JavaScript, or C++), where the model must edit the provided source files to implement the required function so its tests pass.
- Scoring
- The metric is the percentage of tasks solved: a task counts only when the edited code passes all of its automated tests. Aider also tracks how often edits come back in the correct, applicable format.
- Verification
- Results are checked automatically: the edited files are run against each exercise's unit-test suite, and a task passes only if every test passes — no human votes or exact-match comparison.
- Why it matters
- It mirrors a real developer workflow — editing existing files across many languages rather than writing snippets from scratch — so it is a practical signal of how useful a model is as a coding assistant.
Worked example
Aider Polyglot draws 225 of the hardest Exercism coding exercises spread across C++, Go, Java, JavaScript, Python, and Rust; the model is given the instructions plus a stub file and must return an edit (in aider's SEARCH/REPLACE block format) that makes the hidden unit tests pass, with one automatic retry if the first attempt fails. A representative Python item, "Acronym," asks: convert a phrase to its acronym by taking the uppercased first letter of each word, treating spaces and hyphens as separators and ignoring other punctuation — given `acronym.py` containing `def abbreviate(text): pass`, make tests like `abbreviate("Portable Network Graphics") == "PNG"` and `abbreviate("Ruby on Rails") == "ROR"` pass. A correct solution replaces the stub with a SEARCH/REPLACE block:
```
acronym.py
<<<<<<< SEARCH
def abbreviate(text):
pass
=======
import re
def abbreviate(text):
words = re.split(r"[^A-Za-z']+", text)
return "".join(word[0].upper() for word in words if word)
>>>>>>> REPLACE
```
Reasoning: split the text on any run of non-letter characters so that both spaces and hyphens delimit words, discard the empty fragments, then concatenate each remaining word's uppercased first letter — giving "ROR" for "Ruby on Rails." The item counts as solved only when every unit test passes, and the benchmark reports the percentage of the 225 exercises solved this way.
No verified scores reported yet for this benchmark.