Researchers introduce GRAPHEVAL, a graph-based framework that reframes uncertainty quantification (UQ) as a holistic reasoning fidelity problem to address flaws in standard decoding strategies like Self-Consistency. The study proposes the Graph Reasoning Coherence Score (GRCS), a metric that quantifies semantic-structural consensus and captures pathological mode collapse and confident hallucinations.
- GRCS is identified as the only metric consistently negatively correlated with reasoning faithfulness across both more capable and smaller models.
- The framework introduces Graph Self-Consistency (GSC), a medoid-based decoding strategy that trades nominal accuracy for reasoning fidelity.
- GSC exposes how Self-Consistency is inflated by unfaithful lucky guesses in smaller models while preserving or improving accuracy in more capable ones.
- Adversarial medoid ablation demonstrates that the GSC-selected path acts as a "load-bearing path," where forcing models away from it degrades reasoning faithfulness and causes accuracy drops.
The authors consider this important because it provides a method to select more faithful reasoning compared to naïve majority voting, revealing the degree of unfaithful reasoning in standard evaluation methods.