Together AI details the architecture of its Dedicated Model Inference platform, which links endpoints, deployments, and immutable configs through a capacity-aware traffic split. This system routes requests based on effective capacity (weight multiplied by ready replicas) rather than fixed percentages, enabling features like zero-downtime rollouts and A/B testing.
- Configs specify engine, GPU type, and optimization profiles (latency, throughput, or balanced) and are immutable to prevent behavior drift.
- Deployments bind a specific model revision to a config and manage autoscaling policies for replicas.
- Endpoints provide a stable identity for clients, using traffic splits to distribute load across deployments proportionally to their capacity.
- The platform supports A/B cohorts and canary rollouts that compose with the base routing logic.
The article demonstrates how this model prevents saturation issues during scaling and highlights the importance of testing profiles like latency versus throughput at varying concurrency levels to avoid production stalls.