The authors present Speculative Uncertainty (SU), a technique that recovers a predictive failure signal for black-box LLM agents by analyzing their output tokens alone, without requiring access to logits, weights, or activations. By inverting speculative decoding, a small open-weight draft model scores the agent's trajectory in a single forward pass to extract phase-aware features and calibrate them against a verifiable objective.
- SU generates a failure-likelihood score that can be consumed by downstream policies such as routing or human intervention.
- The method was instantiated as a pre-execution veto gate for Qwen3-Coder-480B and Claude 3.5 Sonnet agents.
- Deployment results show a reduction in execution error rate by 6-8 percentage points and token cost by 14-19%.
- The approach transfers to out-of-distribution benchmarks without retraining and generalizes across different agent models.
This method allows for the detection of confident errors before costly execution, enabling more efficient and reliable agentic coding workflows.