Open-Source Speech-to-Speech Models: Can They Take Phone Calls?

The main open speech-to-speech models are Kyutai's Moshi, NVIDIA's PersonaPlex 7B and NemotronLabs VoiceChat 11B, and Alibaba's Qwen3-Omni. Each runs on one data-centre GPU, but each reasons far below hosted APIs. Dograh, the open-source AI voice agent platform, runs fully local calls today as a cascade of open models on your own servers.

Key Takeaways

• PersonaPlex weights use NVIDIA's own licence. That makes them open weights.
• Only NemotronLabs VoiceChat calls tools, and its values are right 42.2% of the time.
• For fully local phone agents today, run a cascade of open models.

This post is part of our guide to speech-to-speech voice AI for phone calls, where open models are the one route that keeps call audio on your own servers. Below, we compare their licences, GPU needs and readiness for phone calls, then cover what to run locally today.

Open speech-to-speech models compared

Four open models can hear and speak in one network, and three of them are full duplex. The table also lists the open cascade you can run in Dograh today, since that is the local alternative a phone team weighs them against.

Kyutai's Moshi repository is the root of this family. PersonaPlex is a fine-tune of Moshi, and NemotronLabs VoiceChat cites both. Ultravox also publishes open weights under MIT, but its model hears speech and writes text, so it needs a separate voice model to speak.

Open source or open weights

The licences differ more than the news coverage suggests, and the difference matters if you ship a product. Moshi is the most open: its code is MIT for Python and Apache for Rust, and its weights are CC-BY 4.0, which asks only for credit.

PersonaPlex is open weights. Its code is MIT, but the PersonaPlex model card puts the weights under the NVIDIA Open Model License, NVIDIA's own agreement, and it is a gated download that asks you to accept that licence first. NVIDIA calls the model ready for commercial use, so you can build on it. What you accept, though, is NVIDIA's own terms rather than an open source licence.

NemotronLabs VoiceChat uses the OpenMDW License 1.1, which lets you use the model materials without restriction as long as you keep the licence and notices when you redistribute them. Your rights end if you sue over patents or copyright in the model. Qwen3-Omni is plain Apache-2.0, the simplest of the four for a commercial product.

Running them on your own GPU

Every model here runs on one data-centre GPU, but none fits a gaming card comfortably. Moshi in PyTorch needs a GPU with about 24 GB of memory, and Kyutai does not support quantization for the PyTorch version. Kyutai reports latency as low as 200 ms on an L4.

PersonaPlex lists A100 and H100 GPUs on Linux and was tested on an 80 GB A100. If your card is short on memory, a CPU offload flag moves part of the model off the GPU, at the cost of speed. Size the card with a test on your own hardware.

NemotronLabs VoiceChat has the most useful numbers. The NemotronLabs VoiceChat model card names vLLM as its serving engine, and NVIDIA's paper reports a 95th-percentile time of 118 ms for each 160 ms chunk of audio with four streams at once on one 80 GB H100, which it puts at 1.36 times real-time throughput. By our reading that points to about four calls per H100, a starting point for your own load test.

Qwen3-Omni is the heaviest. Its model card puts the theoretical BF16 minimum at about 79 GB for a 15-second video input. Turning off its speech output saves about 10 GB, but then it no longer speaks. Our post on RunPod pricing for voice agents turns GPU hours like these into a cost per call.

Open Source Alternative to Vapi / Retell

Self-hosted voice agent platform — no per-minute fees

dograh-hq/dograh

Star on GitHub

How ready they are for phone calls

Open full-duplex models solve turn-taking before they solve understanding. On Artificial Analysis's speech-to-speech leaderboard, PersonaPlex scores 91.0% on conversational dynamics, a measure of pauses, interruptions and backchannels, close to the 96% to 97% of hosted leaders. On Big Bench Audio, a set of spoken reasoning questions, it scores 19.1%, Nemotron Voicechat 27.0% and Moshi 4.4%, while GPT-Live-1 (Astra, medium) scores 90.1% and Gemini 3.8 Live 91.7%.

Tools are the next gap. NemotronLabs VoiceChat is the only one with tool calling, and on Full-Duplex-Bench v3 it picks the right tool 82.5% of the time but passes the right values 42.2% of the time, for 33.0% of tasks done right first time. A booking line cannot run on those numbers yet.

The phone line adds two more limits. PersonaPlex answers only in English and NemotronLabs VoiceChat was tested only in English. None of these models ships a phone path, so you would build the bridge that turns 8 kHz phone audio into the 16 kHz or 24 kHz they expect. Our explainer on full-duplex voice AI covers where duplex helps a call and where it does not.

What to run on your own servers today

If call audio must stay on your own servers, run a cascade of open models now and keep testing the full-duplex ones. Dograh runs the call flow, tools and telephony as the orchestration layer, while the speech and language models are a separate choice you make. With Dograh self-hosted and open models on your own GPU, neither the platform nor the audio leaves your infrastructure.

A stack that fits one 24 GB card is Parakeet TDT 0.6B v2 for speech-to-text and a 4-bit Qwen3.8 27B for the language model. For the voice, Chatterbox (MIT) can clone a voice, which suits pre-recorded clips with a cloned fallback, and Magpie TTS offers fixed voices under NVIDIA's own open-weights licence. Our guide to running voice AI on local, self-hosted models covers sizing and setup.

Self-hosted Dograh has no per-minute platform fee, but servers, model keys and phone minutes still cost money. Dograh Cloud charges a 1¢-a-minute platform fee. If you want full duplex today and your data rules allow a hosted model, run GPT-Live or Gemini 3.8 Live on your own keys through the same platform and move to an open model when one reasons well enough.