Benchmark · coding
BigCodeBench
BigCodeBench is a Python code-generation benchmark of 1,140 real-world programming tasks that each require composing calls to multiple libraries, scored by execution-based Pass@1 — the fraction of tasks whose generated solution passes every unit test.
Read more
- Example
- A typical task gives a function signature plus a detailed docstring — for instance, 'load a CSV, compute per-group summary statistics with pandas, then draw a bar chart with matplotlib' — and asks the model to write a body that correctly wires several library calls together.
- Scoring
- The metric is calibrated Pass@1, usually with greedy decoding: each generated program is executed against the task's unit tests in a sandbox, a task counts as solved only if all tests pass, and the score is the percentage of tasks solved. Two splits are reported — Complete (finish a docstring-scaffolded function) and Instruct (only a natural-language instruction) — plus a 148-task Hard subset.
- Verification
- Acceptance is purely by execution: the solution must pass all of a task's hand-written unit tests (about 5.6 per task, with ~99% average branch coverage) when run in an isolated environment. There is no partial credit and no reference-text matching.
- Why it matters
- Unlike short single-function benchmarks such as HumanEval, BigCodeBench tests whether a model can follow complex instructions and correctly use diverse real-world APIs across 139 libraries and 7 domains, which better reflects practical software work; the Instruct split and Hard subset further stress instruction-following and tool composition.
Worked example
Task
```python
import pandas as pd
import numpy as np
def task_func(df, column):
"""
Standardize the given numeric column of a DataFrame to zero mean
and unit variance, and return the modified DataFrame.
Requirements:
- pandas
- numpy
Parameters:
df (pd.DataFrame): The input data.
column (str): Name of the numeric column to standardize.
Returns:
pd.DataFrame: The DataFrame with the column standardized.
Example:
>>> df = pd.DataFrame({'a': [1, 2, 3, 4, 5]})
>>> round(task_func(df, 'a')['a'].mean(), 5)
0.0
"""
```
Solution
```python
df[column] = (df[column] - np.mean(df[column])) / np.std(df[column])
return df
```Walkthrough
Subtracting the column mean and dividing by the population standard deviation (np.std, ddof=0) rescales the values to mean 0 and unit variance, satisfying the docstring and its doctest (the mean rounds to 0.0), and it uses both required libraries. Grading is execution-based: BigCodeBench runs the completed function against the task's hidden unit tests and awards Pass@1 only if every test passes.
No verified scores reported yet for this benchmark.