Benchmark · coding
DS-1000
DS-1000 is a benchmark of 1,000 real data-science coding problems collected from Stack Overflow and spanning seven Python libraries (NumPy, Pandas, SciPy, scikit-learn, PyTorch, TensorFlow, Matplotlib); it measures whether a model generates code that correctly solves each problem, reported as pass@1 execution accuracy.
Read more
- Example
- A representative item is a natural-language question about doing something with a specific library — e.g., filtering or reshaping a Pandas DataFrame, a NumPy array computation, or a Matplotlib plot — paired with a starter code context the model must complete (offered in Completion and Insertion formats).
- Scoring
- The metric is pass@1 execution accuracy. Each generated solution is run against hidden test cases and counts as solved only if it passes all tests AND meets the surface-form constraints (regex checks that required APIs are used and forbidden shortcuts are not). The score is the fraction of the 1,000 problems solved.
- Verification
- Acceptance is fully automatic: the completed program is executed in a sandbox and its output is checked for functional correctness against the test cases, combined with the surface-form constraints. This multi-criteria check is designed to keep false-accept and false-reject rates low, and problems were perturbed from their Stack Overflow originals to reduce memorization.
- Why it matters
- It captures realistic, everyday data-science coding needs rather than synthetic puzzles, covers seven widely used libraries, and its reliable execution-based grading makes it a common yardstick for evaluating code-generation models on practical DS work.
Worked example
Task
Problem: Given a Pandas DataFrame df with a numeric column 'value', produce a DataFrame containing only the rows whose 'value' is greater than the column's mean.
<code>
import pandas as pd
df = pd.DataFrame({'value': [1, 5, 3, 9, 2]})
</code>
BEGIN SOLUTION — assign the answer to
result. (Illustrative, not a verbatim official item.)Solution
result = df[df['value'] > df['value'].mean()]Walkthrough
Boolean masking keeps rows where 'value' exceeds df['value'].mean(); the harness runs the completed script and compares result to the expected DataFrame (e.g., with pandas testing helpers), while surface-form constraints confirm a genuine Pandas solution. Graded pass/fail by execution.
No verified scores reported yet for this benchmark.