SPANUQ introduces span-level uncertainty quantification for LLMs

Researchers introduce SPANUQ, a lightweight probe that performs Span-Level Uncertainty Estimation (SLUE) by distilling uncertainty knowledge from multi-sample inference into a single forward pass over LLM hidden states. The method employs a DETR-style span decoder to simultaneously detect spans and estimate their uncertainty via a Mixture of Beta distribution.

• SPANUQ-BENCH is the first span-level uncertainty benchmark, comprising 20K prompts, 293K annotated spans, and continuous soft labels derived from multi-sample claim verification.
• Experiments on five LLM backbones show SPANUQ consistently achieves the best span-level uncertainty quality, outperforming the strongest probe baseline and all sampling-based methods while being 10-20x faster.
• Its DETR-based span detector attains 0.910 F1, surpassing the best heuristic by 39.4%, enabling precise error localization that sequence-level methods cannot provide.

The framework generalizes across five LLMs spanning two model families, addressing the limitations of token-level and sequence-level uncertainty scores.