Developer explores orchestration patterns for production voice AI agents

A developer on the Hugging Face forums seeks to understand how production platforms like Bland.ai, Retell, and Vapi manage prompt orchestration without relying on massive, hand-written system prompts. The author describes their current implementation using FastAPI, Sarvam STT/TTS, and Pipecat SmartTurn V3, noting that manual state-wise injection leads to growing complexity and edge-case failures.

• The developer uses a base system prompt with state-wise injection for tasks like booking or Q&A.
• Current architecture includes Sarvam Speech-to-Text, Streaming LLM, Sarvam Text-to-Speech, Pipecat SmartTurn V3, and Silero VAD.
• Challenges include handling interruptions, topic changes, and partial information without a monolithic prompt.
• The goal is an orchestration layer that maintains deterministic workflows while allowing natural conversation with sub-second latency.

The post asks for architectural patterns used in production systems to balance model autonomy with deterministic business logic.