Benchmark · coding

MBPP

saturated 5 results 4 models

MBPP (Mostly Basic Python Problems) is a code-generation benchmark of about 1,000 short, entry-level Python programming tasks described in plain English, each paired with automated test cases. It measures how often a model writes a correct function on its first try, reported as pass@1.

Example: A typical item asks the model to write a small Python function from a one-line description — for example, "write a function that returns the nth Fibonacci number" or "check whether a string is a palindrome" — which is then run against a few provided assert-based tests.
Scoring: The metric is pass@1: for each task the model generates one solution, and the score is the fraction of tasks whose generated code passes all of that task's test cases.
Verification: A solution is accepted automatically — the generated function is executed against the task's assert-based unit tests and counts as correct only if every test passes (no human judging or exact-string matching).
Why it matters: MBPP is a long-standing, easy-to-run baseline for basic Python code generation; because today's strong models score near the ceiling, it is now considered saturated and mainly useful as a sanity check rather than a way to separate top systems.

Worked example

Task

MBPP-style item: 'Write a python function to count the number of vowels (a, e, i, o, u) in a given lowercase string.' It ships with three hidden assert tests, e.g. assert count_vowels('hello') == 2, assert count_vowels('world') == 1, assert count_vowels('rhythm') == 0; the function must match the name the tests call.

Solution

def count_vowels(s):
    return sum(1 for ch in s if ch in 'aeiou')

Walkthrough

Scanning the string and counting characters that belong to the vowel set 'aeiou' yields 2, 1, and 0 for the three inputs, so every assert passes. MBPP grades a solution pass/fail by executing the generated function against all three provided assert test cases (functional correctness, pass@k).

Falcon3-10B-Base LLaDA-8B Qwen3.5-4B Granite 4.0 3B

Timeline

Date	Model	Score	Source
2026-07-16	Granite 4.0 3B	63.2%	IBM open-sources CodeAlchemy, a massive synthetic dataset of high-quality code
2026-07-13	Qwen3.5-4B	13.0%	Flint compresses reasoning traces to reduce token usage while maintaining accuracy
2026-06-29	LLaDA-8B	41.0%	Scheduling Thoughts: Learning the Order of Thought in Diffusion Language Models
2026-06-25	LLaDA-8B	41.0%	Self-Aware Scheduling Learns Token Unmasking Order in Diffusion Language Models
2024-12-17	Falcon3-10B-Base	73.8%	Falcon3 family releases five open models with improved science, math, and code capabilities