Benchmark · coding
MBPP+
MBPP+ is a rigorously test-augmented version of the MBPP (Mostly Basic Python Problems) coding benchmark, built with the EvalPlus framework. It measures the functional correctness of model-generated Python functions, reported as pass@1.
Read more
- Example
- A typical item gives a short natural-language description of a basic Python function to write (for example, 'find the shared elements of two lists') together with a few example assertions that pin down the expected behavior.
- Scoring
- The metric is pass@k, usually pass@1: a task counts as solved only if the generated function passes every test; the score is the percentage of tasks solved. pass@k is computed with the standard unbiased estimator over n sampled completions.
- Verification
- Each candidate function is executed in a sandbox against the full suite — the original MBPP assertions plus EvalPlus's automatically generated inputs (type-aware mutation and LLM seeding, roughly 35x more tests than MBPP). It is accepted for a task only if it passes all tests within the time limit.
- Why it matters
- The original MBPP ships only about three tests per problem, so subtly incorrect code can slip through as a false positive. MBPP+ hardens the evaluation, so reported pass rates and model rankings reflect real correctness rather than weak testing.
Worked example
Task
Write a function to find the shared elements of two given lists, returned as a sorted list of unique values. Your code should pass tests such as: assert shared_elements([1, 2, 3, 4], [3, 4, 5, 6]) == [3, 4]
Solution
```python
def shared_elements(list1, list2):
return sorted(set(list1) & set(list2))
```Walkthrough
Intersecting the two lists as sets and sorting yields the unique common values in order, satisfying the assertions. MBPP+ also runs many extra inputs (empty lists, duplicates, larger cases); graded pass@1, the solution counts only if it passes them all.
No verified scores reported yet for this benchmark.