SMetric improves LLM scheduling for agents via balanced session-centric routing

The paper introduces SMetric, a scheduler designed to address the unique workload characteristics of agentic LLM serving, where requests originate from agents rather than humans. It identifies that existing schedulers prioritize KV cache reuse too heavily, leading to load imbalance and capped tokens per second (TPS).

• Agents require complete responses, making cluster TPS the primary goal while relaxing per-token latency requirements.
• Agent workloads exhibit high KV cache reuse, exceeding 80% of request tokens in production traces compared to 54-62% in chat.
• SMetric routes the first request of each session for load balancing and subsequent requests in a cache-aware manner.
• The scheduler uses session turn information derived from user inputs, remaining stateless and efficient.

SMetric improves cluster TPS by 10-16% under prefill-decode colocation with a global store and increases prefill TPS by 2-34% under disaggregation compared to state-of-the-art schedulers.