Benchmark · coding
EvalPlus
EvalPlus rigorously tests LLM-generated code by extending the HumanEval and MBPP benchmarks (as HumanEval+ and MBPP+) with far more test cases; the metric is pass@k (usually pass@1).
Read more
- Example
- A Python function-completion task: given a function signature plus a natural-language docstring with a few examples, the model writes the body, which is then run against a greatly enlarged test suite that adds edge cases (empty inputs, boundaries, large values) the original benchmark missed.
- Scoring
- The metric is pass@k, almost always pass@1. A problem counts as solved only if the generated code passes every test (the original 'base' tests plus the new 'plus' tests). pass@k is the unbiased estimate: over n≥k samples with c correct it is 1 − C(n−c, k)/C(n, k); the reported score is the fraction of problems solved across the set.
- Verification
- Each solution is executed in a sandbox against the full base+plus suite under a per-test timeout and is accepted only if all assertions pass. The extra test inputs are produced by type-aware mutation plus LLM seeding and filtered against validated ground-truth reference solutions, so a 'passed' verdict means agreement with the reference on every input.
- Why it matters
- The original HumanEval/MBPP test suites are too thin and let plausible-but-wrong code pass, overstating model correctness; EvalPlus's ~80× (HumanEval+) and ~35× (MBPP+) larger suites catch those errors and typically drop pass@1 by several points, making it the more trustworthy standard for evaluating code LLMs.
Worked example
Task
Representative EvalPlus (HumanEval+)-style item — complete the function body from its signature and docstring:
```python
from typing import List
def sum_even_at_odd_index(numbers: List[int]) -> int:
"""Return the sum of all even values located at odd indices.
>>> sum_even_at_odd_index([3, 8, 7, 4, 5])
12
>>> sum_even_at_odd_index([1, 3, 5])
0
"""
```
Solution
```python
def sum_even_at_odd_index(numbers: List[int]) -> int:
return sum(n for i, n in enumerate(numbers) if i % 2 == 1 and n % 2 == 0)
```Walkthrough
The solution iterates with enumerate, keeps values at odd indices (i % 2 == 1) that are even (n % 2 == 0), and sums them, matching both docstring examples and edge cases like an empty or all-odd list. Grading is pass/fail: the function is run against the full base+plus suite and the item counts only if every test passes.
No verified scores reported yet for this benchmark.