Benchmark · coding
Commit-0
Commit-0 measures whether a coding agent can implement an entire Python library from scratch, given only its specification — function and class signatures with docstrings plus a hidden unit-test suite. The primary metric is the fraction of unit tests that pass.
Read more
- Example
- Given the skeleton of a real Python library — every module, class, and function present as a typed stub with a docstring but an empty body — implement all of them so the library's own test suite runs green (for example, filling in a small library's parsing, serialization, and utility functions from their docstrings).
- Scoring
- Score is the unit-test pass rate: each library's pytest suite is run in an isolated environment and the library scores the fraction of tests that pass, aggregated across the benchmark's 54 libraries. Some configurations also require the code to pass lint and type checks.
- Verification
- A result is accepted by executing the hidden test suite inside a reproducible Docker container — pytest is the oracle, so a test counts only when it actually passes, with no reference-output matching. Agentic runs may iterate on test and lint feedback, but final scoring re-runs the suite from a clean checkout.
- Why it matters
- It pushes past function- or single-file generation (HumanEval, SWE-bench) to whole-library synthesis from a spec, probing long-horizon planning, cross-module consistency, and the ability to satisfy a real API contract — much closer to shipping real software.
Worked example
Task
One function stub inside a library skeleton, to be implemented so its hidden tests pass:
```python
def parse_version(text: str) -> tuple[int, int, int]:
"""Parse a semantic version 'MAJOR.MINOR.PATCH' into a tuple of ints.
Raises ValueError if text is not three dot-separated integers.
"""
raise NotImplementedError
```
Solution
```python
def parse_version(text: str) -> tuple[int, int, int]:
parts = text.split(".")
if len(parts) != 3:
raise ValueError(f"invalid version: {text!r}")
major, minor, patch = (int(p) for p in parts)
return (major, minor, patch)
```Walkthrough
The body honors the docstring contract: it splits on '.', rejects anything that isn't exactly three parts with a ValueError, and converts each part to int. Grading runs the library's hidden pytest suite in the sandbox, and this item passes only when every assertion about parse_version holds.
No verified scores reported yet for this benchmark.