An AI developer is asking for architectural advice to build a real-time voice AI system that achieves approximately 500 ms or less first-token latency while remaining cost-effective and scalable.
The request covers component choices across the entire pipeline, including audio transport protocols like WebRTC, Voice Activity Detection, Speech-to-Text providers such as Deepgram and AssemblyAI, LLMs like Gemini Live and OpenAI Realtime, Text-to-Speech services, and orchestration frameworks. The developer specifically asks for production-grade insights on which components contribute most to latency, whether speech-to-speech models outperform traditional STT-LLM-TTS pipelines, and which providers offer the best balance of latency, quality, and cost.
The author requests real-world numbers, benchmarks, and lessons learned from production deployments to help identify common architectural mistakes that increase latency.