Researchers introduce LLM-as-a-Verifier, a general-purpose verification framework that provides fine-grained feedback for agentic tasks without requiring additional training. Unlike standard LM judges that produce discrete scores, this method computes the expectation over scoring token logits to generate continuous scores.
The probabilistic formulation enables scaling along three dimensions: score granularity, repeated evaluation, and criteria decomposition. Scaling these factors leads to better separation between positive and negative solutions and improved verification accuracy through variance and complexity reduction. The framework also includes a cost-efficient ranking algorithm for selecting the best solution among candidates.
LLM-as-a-Verifier achieves state-of-the-art performance on Terminal-Bench V2 (86.5%), SWE-Bench Verified (78.2%), RoboRewardBench (87.4%), and MedAgentBench (73.3%). The fine-grained signals can also serve as a proxy for estimating task progress, with an extension built for Claude Code to help developers monitor agentic systems.