Researchers have introduced "consistency guidelines" within the ALTK-Evolve system, a new mechanism designed to address the variability in LLM agent performance that standard average accuracy metrics often hide. By identifying and stabilizing decision points prone to flipping, this approach significantly improves task success consistency without sacrificing overall capability.
- The Consistency Analyzer detects "flip-prone" steps by resampling a single recorded trajectory with k=5 completions, requiring no ground truth or end-to-end re-runs.
- On AppWorld test_normal with a ReAct agent using GPT-4.1, consistency guidelines reduced the consistency gap (Mean@5 minus Pass^5) from 24.4 percentage points to 12.0 percentage points.
- Aggregate Pass^5 increased from 53.0% to 69.0%, while Mean@5 rose slightly from 77.4% to 81.0%, preserving average accuracy.
- The method generalizes to similar tasks, lifting Pass^5 by 13.0 percentage points, and shows transferable gains on weaker models like gpt-oss-120b.
This work highlights that consistency is an orthogonal axis to capability, urging developers to report Pass^k alongside Mean@k to better assess agent reliability in production environments.