Benchmark · coding
CRUXEval
CRUXEval (Code Reasoning, Understanding, and eXecution Evaluation) measures how well a model reasons about code execution across 800 short Python functions, via two tasks — predicting an input that yields a given output, and predicting the output for a given input — scored by pass@1 functional correctness.
Read more
- Example
- A short, self-contained Python function is shown. In output prediction the model is given a specific input and must state the exact value the function returns; in input prediction it is given the output and must supply any input that reproduces it.
- Scoring
- The metric is pass@1 (pass@k is also reported using the standard unbiased estimator): the model samples one or more answers per problem, each is checked by running the function, and the score is the fraction of the 800 problems solved — reported separately for CRUXEval-I (input) and CRUXEval-O (output).
- Verification
- Every prediction is verified by execution, not string match. An output answer is accepted only if it equals the function's true return value; an input answer is accepted only if feeding it to the function actually produces the target output, so any valid input counts, not just the original one.
- Why it matters
- Predicting execution behavior isolates code reasoning from code writing — a model can generate plausible code without tracking what it truly does. The functions are simple and self-contained, making CRUXEval a clean probe of that skill, and the two directions test both forward (output) and backward (input) reasoning.
Worked example
Task
def f(nums):
result = []
for n in nums:
result.append(n * 2)
return result
assert f([1, 2, 3]) == ??
CRUXEval-O (output prediction): replace ?? with the value f returns.
Solution
[2, 4, 6]Walkthrough
The loop doubles each element of [1, 2, 3], producing [2, 4, 6]. Grading executes assert f([1, 2, 3]) == [2, 4, 6] and accepts the answer only if that assertion passes.
No verified scores reported yet for this benchmark.