Dograh

The Ultimate Guide to Real-Time Voice AI on Phone Calls

The Ultimate Guide to Real-Time Voice AI on Phone Calls
TechnicalNovember 19, 2026· Updated November 20, 2026·9 min read

The Ultimate Guide to Real-Time Voice AI on Phone Calls

Abhishek Kumar
Abhishek Kumar·Co-founder, Dograh AI

Co-founder of Dograh, building the future of open-source voice AI agents. Own your voice AI stack.

Real-time voice AI answers a caller fast enough to feel like a conversation, about 800 ms per turn, by streaming speech-to-text, the language model and the voice so each stage starts before the last ends. For teams that want every hop on their own servers, Dograh, the open-source AI voice agent platform, runs self-hosted beside open-weight models.

Key Takeaways

  • Real time on a phone call is four clocks, from transcript to first audio.
  • People leave about 300 ms between turns, so every hop in the chain counts.
  • Self-hosted Dograh with open-weight models puts every hop on servers you control.

On a phone call, real time is a budget of milliseconds shared by the network, three models and turn-taking. We define the four clocks, then we talk about cascade versus speech-to-speech, where the budget goes, how open models on your servers shorten it, and how Dograh measures it.

Real time starts with where every part runs

Every hop between services costs time, and a voice agent has many of them. Twilio's latency guide, published on 17 November 2025, counts at least ten network traversals for each response of a cascaded agent, two voice legs over the phone network plus eight handoffs between services. Each one you remove or shorten comes straight off the reply time.

Where you can move a hop depends on who owns the parts. With open-source orchestration and open-weight models, the speech-to-text, the language model and the voice can run on the same server or in the same zone as the call itself, which is what colocation means. With closed APIs you cannot run the model yourself, so the most you can do is pick providers in the same cloud region as your servers.

That is why the open-source and on-prem route matters for latency as well as for data control. Dograh is the orchestration layer that runs the call flow, tools and telephony, and the speech and language models are a separate layer you choose. Run both on your own hardware and every hop is yours to shorten. Our post on how to reduce voice-agent latency with server colocation covers the setup in detail.

The four clocks of a real-time call

"Real time" on a call is easier to fix once you split it into four clocks. The first is the streaming transcript. Speech-to-text sends partial words while the caller is still talking, so the system is never waiting for a whole recording before it starts work.

One caller's turn along the call's path, with four stopwatches. The first runs while a microphone feeds out a tape of partial words, the second stops when the caller pauses, the third when the language model writes its first word on a page, and the last, highlighted, when the reply from a loudspeaker reaches the caller's ear.
First audio is the clock the caller feels and the one to report, since the other three only matter as parts of it. Set end of turn too early and the agent talks over people who paused to think, and too late and every reply drags.

The second clock is end of turn, the moment the system decides the caller has finished. Too early and the agent talks over people who paused to think, and too late and every reply drags. Deepgram's Flux documentation quotes about 260 ms for its end-of-turn detection, with a confidence threshold of 0.7 by default and a 5,000 ms silence timeout that forces the turn to end.

The third clock is first token, the time from the end of the turn to the language model's first word. The model keeps writing while the voice starts speaking, so only that first piece is on the critical path, and a short system prompt and a small, fast model both help here.

The fourth clock is first audio, when the caller actually hears the reply begin. Text-to-speech streams audio as soon as it has a few words, and those bytes still have to cross the network and the phone line. This is the clock the caller feels, and it is the one to report, since the other three only matter as parts of it.

How fast is fast enough

Human conversation sets the bar, and it is a tight one. A 2025 study of the Fisher corpus of US English telephone conversations found that the average gap between speakers was 0.300 seconds. A cascade of three models plus a phone network will not match that, and callers do not expect it to, but long gaps quickly read as a machine that has stalled.

Our working bar is about 800 ms from the end of the caller's turn to the start of the reply. Published targets sit close to it. In the same November 2025 guide, Twilio sets a median target of 885 ms for the platform turn gap and 1,115 ms mouth to ear, which also counts the phone legs, and it calls these starting benchmarks rather than best-in-class figures.

Treat any single number with care. A claim of 300 ms may cover only one stage, and a demo in a browser hides the phone network entirely. Our post on how the voice AI pipeline works sets out the full table of stage targets and what breaks between the stages.

Dograh

Open Source Alternative to Vapi / Retell

Self-hosted voice agent platform — no per-minute fees

dograh-hq/dograh

Star on GitHub

Where the latency budget goes

The phone network takes the first slice, and no model choice removes it. Carriers, codecs and the distance between the caller, your servers and each model provider set a floor before any AI runs. Most phone calls still reach the agent as 8 kHz audio, while a browser demo runs at 16 kHz, which also makes the transcript harder to get right.

Endpointing usually takes the next big slice. A fixed silence wait after every turn is time the caller spends listening to nothing, which is why smarter end-of-turn detection pays off so quickly. After that, speech-to-text is often the bottleneck, more than the language model, especially on noisy phone audio where the model needs more context before it commits to words.

Then come first token and first audio. Both depend on the provider, the model size and how far away it runs. Tool calls add their own time mid-turn. A lookup against a database next to the agent fits inside the turn, while a slow outside API does not, so the agent needs a short line to say while it waits.

In Dograh, the HTTP API tool's Messages setting plays that line, such as "Let me check that for you", so the caller never hears silence. Our speech latency playbook walks through cutting each stage, from provider choice to measuring in production.

Turn-taking decides when the agent speaks

Two decisions run on every turn, when the caller has finished and whether the caller may cut in. Voice activity detection (VAD) listens for speech in the audio, and an end-of-turn model or rule decides that the speech is a finished thought. Getting them right matters as much as any model speed.

In Dograh, Silero VAD runs by default, and some speech-to-speech providers bring their own. Under Turn Detection in the agent's Settings tab, the Detection Strategy is either Transcription-based, for short answers, or Smart Turn Analyzer, a model for longer answers with natural pauses that comes with an Incomplete Turn Timeout in seconds.

Interruptions are set separately. The Interruption Strategy chooses between Voice activity and Minimum words, so a cough or a quick "yeah" does not stop the agent mid-sentence. Allow Interruption is set per node, so the agent can finish a read-back in full and still be easy to cut in on everywhere else. Our post on barge-in, endpointing and turn-taking goes deeper on tuning these.

Join the Dograh Community

Dograh is an OSS alternative to Vapi. Join our Slack community for queries, releases, best practices & community interactions.

Cascade or speech-to-speech for real time

A speech-to-speech model takes audio in and gives audio out, which removes the handoffs between three models. That sounds like the obvious real-time choice, and for some calls it is. The trade shows up in tool calling and in where the model can run.

A March 2026 paper on building realtime voice agents shows the gap is smaller than it looks. Its cascaded streaming pipeline, with Deepgram speech-to-text, a language model served on vLLM and ElevenLabs voice, reached 755 ms to first audio, 729 ms at best, with full function calling. Qwen3-Omni through a cloud realtime API reached about 702 ms audio to audio but cannot be self-hosted, and its local run produced text only. The authors conclude that the cascaded streaming pipeline remains the practical architecture for self-hosted realtime agents.

That matches what we see. Cascades are strong at reasoning and tool calling, so they suit complex calls, while speech-to-speech models can fail on calls with complex tool use. Dograh runs both, with speech-to-speech options including OpenAI Realtime, Gemini Live, AWS Nova Sonic, Grok, Ultravox and Azure Realtime. Text transition lines do not play on speech-to-speech models, so use a recording there. Our guide to speech-to-speech voice AI for phone calls covers each model and when it fits.

Measuring and running real time in Dograh

You cannot tune what you do not measure per turn. Each call's View Trace opens its trace in Langfuse, showing the speech-to-text, AI and voice time for every turn, so you can see which clock is slow on which turn. The Reasoning Delay shown in the call transcript is only part of a turn, so never compare it with the 800 ms bar.

Measure on real phone calls, not only browser tests. The 8 kHz audio, the carrier legs and the caller's network all show up only there. If your language model provider has a slow moment, LLM fallback can switch to a backup model when the main one errors or gives no output within a set time, 1.5 seconds by default. It is off until you turn it on, and it works with your own LLM keys.

Running it yourself is where the latency wedge pays off. You can self-host Dograh with Docker next to open-weight speech and language models, and Enterprise adds on-prem deployment inside your own data perimeter. A self-hosted install runs 10 concurrent calls by default, and you can raise the limit to what your servers and phone provider can carry.

Running Dograh yourself means no per-minute platform fee. You still pay for servers, model keys and phone minutes, and your team looks after the database, the upgrades and the GPUs your models need. Support comes from the open-source community, and Enterprise adds custom SLAs. If you would rather not host it, Dograh Cloud runs it for a 1 cent a minute platform fee.

Where to start

Measure first. Run twenty real phone calls through your current setup, read the per-turn trace and find the clock that takes the most time. Fix that one stage, measure again, and only then think about changing the whole architecture.

Glossary

First audio
The moment the caller hears the reply begin, measured from the end of their turn. It includes end-of-turn detection, the model, the voice and the network.
Endpointing
Deciding that the caller has finished a turn. A fixed silence wait is simple but slow, and model-based end-of-turn detection reacts faster to finished thoughts.
Mouth to ear
The full gap a caller experiences, phone legs included. Twilio's November 2025 median target is 1,115 ms, against 885 ms inside the platform.
Colocation
Running speech-to-text, the language model and the voice on the same server or zone as the call. Only self-hosted open models can share a box this way.

Frequently Asked Questions

Get started with Dograh

Build, deploy, and scale AI agents with Dograh. Join the community of developers building the future.