Benchmark · agentic

Vending-Bench

0 results 0 models

Vending-Bench is an agentic benchmark from Andon Labs that tests an LLM agent's long-term coherence by having it autonomously run a simulated vending-machine business; the headline metric is net worth — cash balance plus the value of held inventory — accumulated over the run.

Example: A representative task type: across a long-running simulation (each run spans far more tokens than the context window can hold), decide when to search for wholesalers, email suppliers to place orders, set retail prices above cost, restock the machine, collect cash, and cover a recurring daily fee — each action is trivial, but they must stay consistent over thousands of steps.
Scoring: The primary score is the agent's net worth (bank balance plus the value of held inventory) at the end of the game; the environment also tracks money balance, units sold, and tool use over time. Because performance is high-variance, each model is run multiple times and the distribution/mean is reported rather than a single run.
Verification: Fully automated: the simulated economy computes sales from prices, stock, and demand and keeps the ledger, so net worth is read straight from the environment with no human grading; results are accepted as the aggregate over repeated independent runs, since any single run can derail.
Why it matters: The per-step tasks are easy, so the benchmark isolates the hard part of autonomous agents — staying coherent over very long horizons. In practice even strong models still 'melt down' (misread deliveries, forget orders, wrongly conclude the business failed, or spiral into off-task loops), and these breakdowns aren't cleanly explained by running out of context — making it a pointed test for real long-horizon deployments.

Worked example

Task

A representative decision the agent faces mid-simulation: the bank balance is low, the vending machine is empty, the recurring daily fee is still being charged, and a supplier's email reply quotes a wholesale price per unit. The agent must choose its next tool actions to keep the business solvent and grow net worth.

Solution

1. Read the supplier email and note the wholesale unit cost. 2. Check that cash covers the order plus upcoming daily fees, keeping a buffer. 3. Email the supplier to place an order sized to expected demand and storage capacity. 4. Advance days until the delivery arrives (track that an order is pending). 5. Restock the machine slots from the delivered stock. 6. Set a retail price above wholesale cost at a margin the demand still buys. 7. Each day, collect accumulated cash and ensure the daily fee is paid. Result: net worth (cash + inventory value) trends upward over the following days.

Walkthrough

This cycle is sound because it never lets the balance go negative under the daily fee, prices above cost to capture margin, sizes orders to demand and capacity to avoid dead stock, and — crucially — keeps track of the pending order so the agent doesn't re-order or declare failure. Grading is not by matching this exact sequence but by the net worth the environment reports at the end of the run.

No verified scores reported yet for this benchmark.