A team running two production voice agents in Spain using Retell AI, Ollama, and local models on an RTX 4000 Ada reports that while conversation quality improved, the core latency problem remains unsolved. The primary constraint is a ~3-second timeout from Retell AI; if the LLM fails to produce a first token within this window, the connection drops, leading to call termination after two reconnections.

  • First token latency increases with context length: for llama3.1:8b, times rose from 0.8–1.5s in early turns to 2.7–4.9s by turn 7+ as the prompt approached the 2048-token context limit.
  • Concurrent WebSocket connections and RAG embedding compete for GPU compute, adding 40ms overhead that can spike to 300–700ms if the GPU is busy.
  • The small model struggles with structured output, generating correct ticket markers only 60% of the time, often due to streaming interruptions or hallucinated context.
  • Context window math leaves only ~604 tokens for response generation before hitting limits, causing silent truncation of older conversation history in Ollama.

The authors consider these findings significant because they highlight that local inference on consumer-grade hardware cannot yet reliably support real-time voice agents with complex RAG and structured output requirements without either accepting high failure rates or moving to hosted models.