Benchmark · coding

HumanEval

saturated 8 results 6 models

HumanEval measures a model's ability to write correct Python code from a natural-language description, scored by pass@1 — the share of problems it solves on the first try.

Example: A typical item gives a function signature and its docstring — for example, "return the list of all prime numbers below n" — and the model must fill in the function body.
Scoring: The metric is pass@1: each generated solution is run against hidden unit tests, and the score is the percentage of the 164 problems whose completion passes all of its tests.
Verification: Fully automatic: a solution counts as correct only if it passes every hidden unit test for that problem — no human judgment and no exact-match string comparison.
Why it matters: It was one of the first widely used code-generation benchmarks and became a standard reference for coding ability, though top models now score so high that it is effectively saturated and barely distinguishes them.

Worked example

Task

Complete the body of this Python function so it passes the hidden unit tests: ```python from typing import List def below_zero(operations: List[int]) -> bool: """You're given a list of deposit and withdrawal operations on a bank account starting at zero balance. Detect whether the balance ever drops below zero and return True at that point; otherwise return False. >>> below_zero([1, 2, 3]) False >>> below_zero([1, 2, -4, 5]) True """ ```

Solution

from typing import List


def below_zero(operations: List[int]) -> bool:
    balance = 0
    for op in operations:
        balance += op
        if balance < 0:
            return True
    return False

Walkthrough

Keep a running balance, add each operation in order, and return True the moment it drops below zero; if the loop ends the balance never went negative, so return False. HumanEval grades by appending the model's completion to the prompt and executing it against hidden unit tests (pass@k) — the item counts as solved only if every assert passes.

Grok-1.5 Claude 3.5 Sonnet Qwen2-72B Granite 4.0 3B GPT-4 GPT-4o

Timeline

Date	Model	Score	Source
2026-07-16	Granite 4.0 3B	83.5%	IBM open-sources CodeAlchemy, a massive synthetic dataset of high-quality code
2024-07-15	Qwen2-72B	64.6%	Qwen2 Technical Report introduces dense and MoE models up to 72B
2024-07-15	Qwen2-72B	64.6%	Qwen Team releases Qwen2 series with 72B dense and MoE models
2024-06-20	Claude 3.5 Sonnet	92.0%	Anthropic releases Claude 3.5 Sonnet addendum with improved coding and vision benchmarks
2024-06-20	Claude 3.5 Sonnet	92.0%	Anthropic
2024-05-13	GPT-4o	90.2%	OpenAI
2024-03-28	Grok-1.5	74.1%	xAI releases Grok-1.5 with improved reasoning and 128K context
2023-03-14	GPT-4	67.0%	OpenAI