Benchmark · coding
HumanEval+
HumanEval+ is a rigorous, extended version of OpenAI's HumanEval code-generation benchmark: it keeps the original 164 hand-written Python programming problems but adds roughly 80x more test cases (via the EvalPlus framework), and it is scored with the pass@k metric (usually pass@1).
Read more
- Example
- Each item gives a Python function signature plus a natural-language docstring — often with a doctest example — describing the wanted behavior (e.g., 'return the list elements strictly greater than a threshold, in order'), and the model must generate the function body that implements it.
- Scoring
- The metric is pass@k: for each problem n >= k completions are sampled, c pass all tests, and the unbiased estimator pass@k = 1 - C(n-c, k) / C(n, k) is averaged over the 164 problems (pass@1 is most reported). A completion counts only if it passes the full augmented test suite, so HumanEval+ scores run lower than plain HumanEval.
- Verification
- Each completion is executed in an isolated sandbox against the expanded test suite under time and memory limits, and is accepted only if every test case passes — the original tests plus the EvalPlus-generated inputs. There is no partial credit: one failing or timing-out test fails the whole sample.
- Why it matters
- HumanEval's original tests are sparse, letting subtly incorrect solutions slip through and inflate reported pass rates; HumanEval+'s ~80x denser tests expose these false positives and reshuffle model rankings, making it a standard, tougher correctness check for code-generation models.
Worked example
Task
A representative HumanEval+-style item — a typed Python signature with a docstring and a doctest, where the model must complete the function body:
```python
from typing import List
def above_threshold(numbers: List[float], threshold: float) -> List[float]:
"""Return the numbers from the input list that are strictly
greater than the given threshold, preserving their original order.
>>> above_threshold([1.0, 5.5, 2.3, 8.0], 3.0)
[5.5, 8.0]
"""
```
Solution
```python
from typing import List
def above_threshold(numbers: List[float], threshold: float) -> List[float]:
return [n for n in numbers if n > threshold]
```Walkthrough
The list comprehension keeps exactly the elements strictly greater than the threshold and preserves input order, satisfying the spec and the doctest ([5.5, 8.0]). Grading runs this against the full HumanEval+ suite — many auto-generated edge cases such as empty lists, negatives, and all-equal values — and scores 1 only if every test passes.
| Date | Model | Score | Source |
|---|---|---|---|
| 2026-07-13 | Qwen3.5-4B | 13.0% | Flint compresses reasoning traces to reduce token usage while maintaining accuracy |
| 2026-06-25 | iLLaDA-Instruct | 16.5pts | iLLaDA: An 8B Masked Diffusion Language Model with Fully Bidirectional Attention |