Benchmark · coding
RepoBench
RepoBench is a repository-level code auto-completion benchmark for Python and Java that measures whether a system can predict the next line of code using both in-file context and cross-file context from other files in the same repository. It has three tasks — retrieval (RepoBench-R), next-line completion (RepoBench-C), and the end-to-end pipeline (RepoBench-P) — scored mainly by Exact Match and Edit Similarity, with retrieval accuracy for the retrieval task.
Read more
- Example
- Given the preceding lines of a source file plus code snippets pulled from other files in the same repository (for instance a module that defines a helper function or class), predict the single next line of code — typically a line that calls or references something defined cross-file.
- Scoring
- Completion (RepoBench-C and -P): the predicted next line is compared to the ground-truth line by Exact Match (percentage of exactly identical lines) and Edit Similarity (a 0–100 score = 1 − normalized Levenshtein edit distance); CodeBLEU is also reported. Retrieval (RepoBench-R): accuracy@k — the fraction of items whose gold cross-file snippet is among the top-k retrieved candidates.
- Verification
- Grading is automatic and reference-based (no code execution): the model's output line is normalized and string-matched against the real next line held out from the original repository. Under Exact Match a completion counts as correct only if it is string-identical to the gold line; Edit Similarity gives graded partial credit; a retrieval is accepted when the gold snippet appears in the top-k.
- Why it matters
- Real coding assistants must reason across a whole repository, not just the open file. RepoBench isolates cross-file context modeling — retrieving the right context and predicting the next line that depends on it — so it became a standard testbed for repo-aware and long-context code models.
Worked example
Task
Cross-file context (helpers/formatting.py):
def to_title(text):
return text.strip().title()
In-file context (report.py):
from helpers.formatting import to_title
def make_heading(raw):
Predict the next line.
Solution
return to_title(raw)Walkthrough
The held-out next line calls to_title — the helper defined in the cross-file snippet and imported at the top — on the function's argument, which is exactly what the original repository code did here; a model that ignored the other file could not know that to_title is the right call. Grading: Exact Match against the gold line, with Edit Similarity giving partial credit for near-matches.
No verified scores reported yet for this benchmark.