NVIDIA has released NemotronLabs VoiceChat 11B, an open 11B parameter end-to-end speech-to-speech model designed for real-time, full-duplex conversation. Unlike cascaded stacks that chain ASR, LLM, and TTS, this unified network performs streaming speech understanding and generation simultaneously, achieving a measured smooth turn-taking latency of 448 ms on Full-Duplex-Bench 1.0.

  • The architecture combines a Fast Conformer encoder, the Nemotron Nano v2 LLM backbone, and an NVIDIA TTS decoder, trained on approximately 550k hours of audio.
  • It supports live tool calling via a separate output channel for <TOOLCALL> scripts, allowing operators to define "on-hold" messages to prevent dead air during API calls.
  • The model allows users to barge in mid-turn with a take-over rate of 1.00 at 480 ms, and ranks #2 among open full-duplex models on VoiceBench.
  • Weights are available under the permissive OpenMDW-1.1 license, but NVIDIA labels the checkpoint as ready for research purposes only due to known failure modes like context ceiling limits.
  • Deployment requires a single GPU with at least 80 GB of VRAM, such as an A100 or H100, and no hosted API is currently provided.

The release provides a deployable option for pilots in contact centers, automotive assistants, and accessibility tooling, though it is not yet recommended for production use.