Speech-to-Speech Voice AI for Phone Calls: The Complete Guide

Speech-to-speech voice AI uses one model that hears the caller's audio and answers in audio, with no text step in between. On a phone line it cuts turn delay and copes well with interruptions, but it is weaker than a cascade on complex tool calls and exact wording, so the right choice is made per agent.

Key Takeaways

• OpenAI and Ultravox document SIP paths. Other models reach phones through a bridge.
• Tool calls break first when a caller corrects a detail. Test those calls.
• Open full-duplex models reason far below hosted APIs.

We put every speech-to-speech API and open model on one map, then cover phone lines, tool calls, open models and when a cascade fits better. We show how each part works in Dograh, the open-source AI voice agent platform we build, which you can self-host or run in our cloud.

What speech-to-speech means on a phone call

A speech-to-speech model takes the caller's audio in and sends audio back, with no transcript and no separate voice model in the middle. A cascade does the same job in steps. Speech-to-text writes down what the caller said, a language model decides the reply, and text-to-speech reads it out. Each handoff adds delay, but each step can also be swapped or checked on its own.

Speech-to-speech models come in two kinds. Most take turns: they listen until the caller stops, then answer, and a voice activity detector decides when the turn has ended. A few are full duplex, meaning they listen and speak at the same time, so they can hear a caller say "mm-hm" or "wait" in the middle of a sentence and react to it. OpenAI's GPT-Live is the main hosted full-duplex model, and our post on full-duplex voice AI explains what that changes on a call.

On a phone line the gap shows up in small moments. Callers talk over the agent and pause to find a policy number, and both are normal. Native audio models handle those moments more naturally than a cascade. A cascade, in turn, makes it easier to see what the agent heard when something goes wrong, because the transcript sits between hearing and answering.

Every speech-to-speech model on one map

The table puts the hosted APIs and the open models side by side on the four things that decide a phone agent: how the call reaches the model, how it calls tools, the list price and whether it runs in Dograh. Prices are list rates. The real cost per call depends on call length, thinking and the backend model, which our posts on OpenAI Realtime API pricing and Gemini Live API pricing work out line by line.

Two notes on the table. Gemini 3.1 Flash Live is scheduled to shut down on 17 November 2026, with Gemini 3.8 Live as Google's replacement, so new agents should start on 3.8. And the same OpenAI realtime models also run on Azure OpenAI, which matters if your contracts already sit with Microsoft.

Independent numbers help more than vendor claims here. On Artificial Analysis's speech-to-speech leaderboard, tau-Voice measures how often a model finishes customer-service tasks that need tools. GPT-Live-1 (Astra, medium) scores 74.5%, Grok Voice 3.0 71.6% and Gemini 3.8 Live Extended Thinking (High) 68.6%. Eleven Realtime v1 Max scores 47.6%, GPT-Realtime-2.1 (High) 45.7% and standard Gemini 3.8 Live 30.1%.

Speed follows a different order. On the same board's time to first audio, Eleven Realtime v1 Max answers in 0.57 seconds and Grok Voice 3.0 in 0.81 seconds. Gemini 3.8 Live takes 1.18 seconds and GPT-Live-1 (Astra) 1.34 seconds. The models that think or hand off before speaking score best on tasks, so pick by the job the call has to finish.

The board is live and moves as vendors ship, so treat these figures as a snapshot. For a full head-to-head of the three leading APIs on interaction quality, telephony and voice cloning, read GPT-Live-1 vs Gemini Live vs Grok Voice.

Open Source Alternative to Vapi / Retell

Self-hosted voice agent platform — no per-minute fees

dograh-hq/dograh

Star on GitHub

Getting a model onto a phone line

OpenAI and Ultravox are the vendors here that document phone paths of their own. Ultravox lists SIP beside native Twilio, Telnyx, Plivo and Exotel connections. OpenAI's GPT-Live guide lists telephony and SIP beside WebRTC and WebSockets, and calls arrive at a SIP address of the form sip:PROJECT_ID@sip.api.openai.com. Your carrier sends the call there, your server accepts it, and OpenAI handles the audio. Our post on the OpenAI Realtime API on phone calls walks through each step.

Every other hosted model takes a WebSocket stream, so something has to carry the phone audio between the carrier and the model. That piece is a media bridge. xAI publishes a Twilio example. Dograh is the bridge for the rest: speech-to-speech agents run on Twilio, Telnyx, Plivo, Vonage, Vobiz, Exotel, Cloudonix and Asterisk ARI lines, so Gemini, Grok and Nova reach a phone number without SIP support of their own.

The audio rate changes on the way. Most phone calls still reach the agent as 8 kHz audio, while a browser demo runs at 16 kHz. The Gemini Live API takes 16 kHz audio in and always answers at 24 kHz, and OpenAI's realtime models take 24 kHz audio or the phone codec itself. Resampling changes the format, but it cannot add back what the phone line removed, and our post on phone audio quality covers why that hurts recognition.

Long calls hit session limits. An OpenAI Realtime session lasts at most 60 minutes, and a GPT-Live call over SIP up to 2 hours. A Gemini Live audio session runs 15 minutes without context compression, and a single connection about 10 minutes. Amazon Nova 2 Sonic connections stop at 8 minutes and continue on a new one. Our guide to the Gemini Live API on a phone line shows what that means mid-call.

Tool calls are the weak spot

Our view is that cascades are stronger at reasoning and tool calling, and that speech-to-speech models can fail on calls that need complex tool calls. A booking that looks up a slot, checks a policy and writes a record is the kind of call where that shows.

The research supports a narrower version of that claim. Full-Duplex-Bench v3, from National Taiwan University and NVIDIA, tested chained tool calls with real speech full of pauses and self-corrections. GPT-Realtime passed 60% of tasks on the first try and beat the benchmark's cascade (Whisper, GPT-4o and OpenAI's voice), which passed 45%. So the cascade does not always win, and the paper is most useful for showing how speech-to-speech models fail.

The main failure is timing. The authors found models "commit intermediate parameters before the correction arrives." In one published example, Gemini Live 3.1 fired its flight search early and locked in Rome after the speaker had switched to Milan. The same model said nothing at all in 22% of examples, yet still ran tools in 86% of those silent cases. The caller hears nothing while the booking changes behind the scenes.

Models that hand tools to a text model do better. GPT-Live passes each workflow step to a backend model, and its tau-Voice score above is the highest on the board. Our post on speech-to-speech tool calling covers each failure mode and how to test for it before go-live.

Open models you can run on your own GPU

Open full-duplex models exist, but none is ready to run a business call on its own. Kyutai's Moshi was the first widely used one, with code under MIT and Apache licences and weights under CC-BY 4.0. NVIDIA built PersonaPlex 7B on Moshi and released its weights under the NVIDIA Open Model License, which makes it open weights rather than open source. NVIDIA's NemotronLabs VoiceChat 11B, under the OpenMDW licence, is the first open full-duplex model with tool calling. Alibaba's Qwen3-Omni is Apache-2.0 and speaks in 10 languages, though it is not described as full duplex.

Their weak point is reasoning. On Artificial Analysis's Big Bench Audio, a set of spoken reasoning questions, Moshi scores 4.4%, PersonaPlex 19.1% and Nemotron Voicechat 27.0%, where the hosted models in the map above score 76% to 99%. PersonaPlex reaches 91.0% on the same board's conversational dynamics score, which shows turn-taking is solved before understanding is. PersonaPlex answers only in English and NemotronLabs VoiceChat was tested only in English, and none of the open models ships a phone path.

That is why Dograh's fully local route is a cascade of open speech-to-text, language and voice models today. Dograh is the orchestration layer that runs the call flow, tools and telephony, and the speech and language models are a separate layer you choose. Our post on open-source speech-to-speech models compares each model's licence, size and GPU needs.

When a cascade is the better choice

A cascade is the safer pick when the call depends on exact words or many tool calls. Names, account numbers, spelled email addresses and amounts all pass through a transcript you can check, correct with a dictionary of brand words and log. Cascades also suit cloned voices, since you choose the voice model, and fixed lines such as disclosures, which a voice model reads as written.

Speech-to-speech fits calls that are mostly conversation with a few simple lookups. Think of an appointment reminder, a short survey or a front-desk call where the agent answers from a small set of facts. There the gain in natural turn-taking is worth more than the loss in control.

You make this choice per agent. A team can run a reminder line on Gemini 3.8 Live and a claims intake line on a cascade from the same account. Our comparison of Cascade vs Speech-to-Speech covers tone, latency and cost per minute for the two designs in detail.

Which speech-to-speech models run in Dograh

In Dograh, a speech-to-speech agent is a named model configuration in Realtime mode. You open Models in the sidebar, click Add Configuration, choose Realtime and pick a provider connection that holds your own key. Realtime mode also takes a separate language model for jobs such as extracting variables and running QA after the call, which keeps those steps on text even when the call is audio only.

The providers built in are OpenAI (gpt-live-1, gpt-realtime-2.1, gpt-realtime-2.1-mini and gpt-realtime-2), Gemini Live on Google AI Studio and on Vertex AI, Azure OpenAI Realtime, Grok Voice, the hosted Ultravox API and Amazon Nova 2 Sonic. Gemini 3.8 Live works with no extra setup, on your own Google AI Studio key. On Vertex AI you can pick the eu or us location to keep processing inside that geography. With gpt-live-1, voice turns go to a backend text model, gpt-5.4-mini by default, which OpenAI bills apart from the voice minutes.

Each agent picks its configuration in its own Settings, so one account can mix speech-to-speech and cascade agents. A few things still need a cascade agent. A speech-to-speech agent can’t hand the call to another AI agent, so it keeps helping the caller instead. When it transfers to a person on Twilio, it skips the spoken line that introduces the caller. Typed lines in tool messages and transitions don’t play either, so any line that must be said word for word should be a pre-recorded clip, which plays exactly as recorded whichever model runs the call.

Transfer Call to a person still works on speech-to-speech agents, on Twilio, Telnyx, Plivo, Cloudonix and Asterisk ARI. While the caller hears hold music, Dograh dials the destination and connects the two once someone answers. If nobody answers, the line is busy, the dial fails or it times out, the agent gets transfer_failed and the call goes on.

Self-hosted Dograh has no per-minute platform fee, but servers, model keys and phone minutes still cost money. Dograh Cloud charges a 1¢-a-minute platform fee. The speech-to-speech provider bills its own minutes or tokens to your account on top.

How to pick one for your calls

Start from the call, then the model. If the call needs several lookups, exact numbers or a cloned voice, start on a cascade and move to speech-to-speech only once the flow is simple enough. If the call is short, chatty and has one or two tools, a speech-to-speech model will sound more natural.

Among the hosted models, GPT-Live leads on tool tasks and has its own SIP path, at a higher price per minute than Gemini 3.8 Live. Gemini 3.8 Live has the lowest list rate in the table and a Vertex AI option for regional processing. Grok Voice 3.0 scores close to GPT-Live on tasks and answers faster. If your audio cannot leave your own servers, run an open cascade today and watch the open full-duplex models.

Whichever you pick, test it on a real phone number rather than in a browser. Place calls where the caller corrects a detail or cuts in, include a tool that fails, read the traces and keep the agent on a cascade until those calls pass.

Each part of speech-to-speech above has its own deep dive:

• OpenAI Realtime API on Phone Calls: SIP, Tools and 8 kHz Audio
• Gemini Live API on a Phone Line: What Works and What Breaks
• Speech-to-Speech Tool Calling: Where It Breaks on Phone Calls
• Open-Source Speech-to-Speech Models: Can They Take Phone Calls?
• Full-Duplex Voice AI Explained: Talking and Listening at Once