The article introduces EvalAudit, an executable artifact designed to determine whether a benchmark conclusion—such as a winner or complete ranking—is identified across a declared family of evaluator specifications. By fixing the evaluated systems and data while varying metrics and aggregation rules, the tool computes whether the set of possible claims remains a singleton.
The authors applied EvalAudit to three heterogeneous benchmarks: AgentDojo v0.1.35, AutoML Benchmark, and BEIR SciFact. Results showed that while winners were often stable across admitted and disputed specifications, complete rankings were not always identified within the sensitivity envelope. The tool requires users to declare semantic targets, varying choices, and evidence for each candidate specification.
EvalAudit enables low-cost reproduction of deterministic audits from frozen score declarations without regenerating upstream scores. It separates semantic uncertainty from sampling and stochastic uncertainty, providing normalized JSON reports and checksum-closed manifests for verification.