Together AI has added autoscaling capabilities to its Dedicated Model Inference platform, allowing deployments to scale based on metrics the inference engine understands, such as in-flight requests, time-to-first-token (TTFT), and GPU utilization.
- The system uses a proportional control loop with configurable replica bounds and timing windows to manage scaling up and down.
- Users can choose from eight metrics, categorized as concurrency-driven (e.g., inflight_requests), SLO-driven (e.g., ttft), or efficiency-driven (e.g., gpu_utilization).
- Concurrency-driven metrics are recommended as the default because they serve as leading indicators of load before latency degrades.
- The platform does not support automatic scale-to-zero; setting min and max replicas to 0 explicitly stops the deployment.
- An experiment with a Qwen3.5-9B model showed that only the inflight_requests metric successfully scaled during traffic spikes, while TTFT and GPU utilization policies failed to react.
Autoscaling helps balance cost and performance by preventing over-provisioning of idle GPUs and under-provisioning that leads to severe latency degradation during peak traffic.