Benchmark · coding
LiveCodeBench
LiveCodeBench evaluates how well large language models solve competitive-programming problems, using coding challenges collected continuously from recent contests to avoid training-data contamination. The headline metric is pass@1 — the share of problems the model solves correctly on its first attempt.
Read more
- Example
- A typical item is a contest-style coding problem — for example, given an array of integers, write a function that returns the length of the longest strictly increasing subsequence — which the model must solve by generating working code.
- Scoring
- Each generated solution is run against the problem's test cases; pass@1 is the fraction of problems for which the model's first submission passes all the tests.
- Verification
- A solution is accepted automatically when the generated code passes every hidden and public test case for that problem — no human judging or exact-text matching is involved.
- Why it matters
- Because its problems come from contests released after a model's training cutoff, LiveCodeBench measures genuine reasoning and coding ability rather than memorized answers, making it a trusted, contamination-resistant gauge of coding skill.
Worked example
Task
Competitive-programming style (stdin/stdout). Given n integers, count the pairs (i, j) with i < j such that nums[i] + nums[j] is even. Input: first line is n, second line is n space-separated integers; print the count.
Solution
import sys
def main():
data = sys.stdin.read().split()
n = int(data[0])
nums = list(map(int, data[1:1 + n]))
odd = sum(x % 2 for x in nums)
even = n - odd
print(odd * (odd - 1) // 2 + even * (even - 1) // 2)
main()Walkthrough
A sum is even exactly when both addends have the same parity, so the answer is C(evens, 2) + C(odds, 2); counting parities is O(n). LiveCodeBench runs the model's program against hidden unit tests and scores it pass@1 — correct only if every test passes.