This work formulates budget-aware test-time model selection for large language models, treating resampling the committed model and rerouting to an alternative as competing uses of a single per-query cost budget. The authors propose an online resample-or-reroute (RoR) allocation policy driven by estimated marginal correctness per unit cost to maximize expected correctness given an imperfect verifier.
- Replay experiments use regenerated multi-draw correctness tensors from an eleven-model open-weight pool over four benchmarks.
- The RoR policy attains a favorable cost-quality Pareto front relative to single-route, one-commit-router, budget-aware best-of-K, cascade, and random-allocation baselines.
- Gains are largest on the most heterogeneous benchmark and are verifier-gated, shrinking as verifier quality degrades.
- Robustness replays under a provider price vector and a label-free agreement verifier delineate where conclusions carry over.
The approach addresses the gap between deployed routers and per-instance oracles by optimizing budget allocation without requiring idealized correctness labels.