The creator of Whetstone, an open-source promotion gate for models and agents, is seeking users to test the tool against real model updates. Whetstone compares paired item-level outcomes to detect hidden regressions that aggregate scores might miss.
- It applies exact significance tests to produce reproducible PASS, HOLD, or BLOCK receipts.
- In a local test, it identified three gains and three regressions in a tied baseline, blocking the candidate.
- The hosted version is stateless for disposable inputs, while real tests run locally to keep data private.
- The author invites builders with base models, fine-tunes, or agent updates to challenge its assumptions.
The goal is to find awkward cases where the gate gives wrong answers or becomes impractical.