Benchmark · coding

RepoCoder

niche 0 results 0 models

RepoCoder evaluates repository-level code completion: given a position in a source file, a model must predict the missing code using context drawn from across the whole repository. Its RepoEval benchmark reports Exact Match and Edit Similarity for line- and API-invocation completions, and a unit-test pass rate for whole function bodies.

Example: Finishing a partially written line inside a repository file that calls a function or class defined in a different file of the same project — the model has to use cross-file repository context to produce the correct completion.
Scoring: Line and API-invocation completions are scored with Exact Match (1 if the prediction equals the held-out ground-truth code, else 0) and Edit Similarity (1 − normalized Levenshtein distance), averaged over all samples as percentages. Function-body completions are scored by execution: the generated body replaces the original and must pass the repository's existing unit tests.
Verification: For line/API items, a prediction is accepted per-sample by comparing it to the reference code held out from the source repository — exactly (EM) or by edit-distance similarity (ES). A function completion is accepted only when the repository's own unit tests pass against the generated body. Repositories were chosen recent and high-quality to reduce training-data leakage.
Why it matters: Real coding happens inside large existing codebases, not isolated snippets. RepoCoder/RepoEval tests whether a model can use project-specific APIs, symbols, and conventions scattered across many files — a more realistic measure of coding assistants than standalone function synthesis.

Worked example

Task

Line-level completion (RepoEval). In the repository, utils/config.py defines a helper load_config(path), and trainer.py begins with from .utils.config import load_config. Given the code below plus retrieved cross-file repository context, predict the single held-out next line inside __init__: ```python class Trainer: def __init__(self, path): ```

Solution

self.config = load_config(path)

Walkthrough

Correct because load_config is the repository's own helper, imported at the top of trainer.py; only cross-file repository context reveals it, so a generic guess (such as inlining a JSON read) would miss. The line is graded by Exact Match against the held-out ground truth, with Edit Similarity giving partial credit.

No verified scores reported yet for this benchmark.