How to Reduce Voice-Agent Latency With Server Colocation

To cut voice-agent latency with server colocation, run speech-to-text, the language model and the voice on the same machine or availability zone as the platform. Only open models you host can share a box. For hosted models, pick their region nearest your server. Dograh, an open-source voice agent platform, can run on the same box.

Key Takeaways

• A cascade reply makes at least ten network trips, per Twilio.
• Only open models you run yourself can share a box with the platform.
• Read each turn in the trace before and after you move anything.

This post is part of our guide to running voice AI on local, self-hosted models, where colocation is open only to teams that run their own servers. Below, we count the hops in a turn, decide where each part of the call sits and measure the result in Dograh.

Count the network hops in one turn

A cascade agent sends audio and text across a network many times for every reply, and each trip adds a little delay. Twilio's latency guide counts at least ten network traversals for one reply: two voice legs over the public network and eight handoffs between services. In its breakdown, a hop between services in one place costs about 10 ms.

So distance matters more than the number of steps. Eight handoffs at about 10 ms each come to roughly 80 ms inside one location, which a turn can absorb. Spread the same handoffs across distant regions and every one of them grows. You can't remove the steps of a cascade, but you can keep each one short.

Placement is also only part of the budget. Our bar for a natural reply is about 800 ms, and in our experience speech-to-text usually takes the biggest share, not the network. Twilio's own starting targets give speech-to-text 350 ms, the language model's first token 375 ms and the voice's first byte 100 ms. Network delay is the part you cut with placement alone, without touching the models.

What can share a box, and what only a region

Colocation means running the speech, language and voice models on the same machine or in the same availability zone as the platform, and only models whose weights you hold can go there. Open models qualify: faster-whisper or Parakeet for speech-to-text, Qwen3.8-27B or Gemma 4 through vLLM or Ollama, and Kokoro or Chatterbox for the voice. Daily measured open models on one RTX 5090 at 508 ms P50 voice-to-voice in its own test, measured at the server, so a phone call adds the carrier's time on top, which shows what one box can do.

Hosted models are different. GPT, Claude and any speech API you reach by key run on their provider's servers, and the weights never leave them. With a hosted model, colocation means choosing the provider region closest to your server, which shortens the hop without removing it. On AWS, a geographic inference profile keeps requests inside one geography, as our post on AWS Bedrock pricing for voice agents explains.

Some model hosts now run speech and language models together in one cluster of their own. That shortens the hops between their models, but your platform, your database and the phone line still reach that cluster over the internet. Colocation you control starts with models you can place yourself.

Open Source Alternative to Vapi / Retell

Self-hosted voice agent platform — no per-minute fees

dograh-hq/dograh

Star on GitHub

Where each part of the call should sit

Every turn touches the platform running the call, the models, any system the agent looks up and the carrier bringing the audio in. Put the platform and open models on one machine first. Our tutorial on building a voice agent on fully local open-source models wires each stage to a loopback address, and our post on Hetzner GPU servers for self-hosted voice agents shows one box that holds them all.

Your own data comes next. A lookup against a database in the same data center fits inside the turn, while a slow outside API often does not. An HTTP API tool waits 5 seconds by default before it returns an error to the agent. For an outside system that may be slow, set the tool's Messages, a line that plays when the tool starts and another while it is still running, so the caller never sits in silence.

Then the phone line. Put the server near where your carrier hands the call's audio over, and near your callers. Twilio advises placing your servers at a Twilio Edge Location, and with Asterisk ARI your own PBX can sit in the same data center as the platform.

One more lever skips a hop entirely. Pre-recorded clips play the agent's fixed lines straight from a recording, so those lines never wait on the voice model.

Box-level colocation needs self-hosted Dograh

Putting the models on the same machine as Dograh is a self-hosted setup. Dograh is the orchestration layer, holding the call flow, tools and telephony, and the speech and language models sit in a layer you choose beneath it. Self-hosted, it accepts localhost and private network addresses for those models through a Local Models (Speaches) provider.

Dograh Cloud accepts only public addresses for model servers, so on Cloud every model is reached across the internet. There, the region choice above is the lever that applies. On a self-hosted Docker install, remember that localhost means the Dograh container itself, so enter each model server's network address.

Self-hosted Dograh has no per-minute platform fee, but servers, model keys and phone minutes still cost money. Dograh Cloud charges a 1¢-a-minute platform fee. Either way, the models and the carrier bill at their own rates. For the full self-hosted setup, see self-hosted voice AI on open-weight models.

Measure the turn before and after you move

Change one placement at a time and read the same kind of turn again. Each call's trace shows the speech-to-text, AI and voice time for every turn, so you can see which stage moved. Our guide to reading call traces to debug a voice agent walks through finding the slow layer.

Keep one caution in mind. The Reasoning Delay shown in the call transcript covers only part of a turn, so never set it against the 800 ms bar. The QA step can also read each call's average and maximum latency and the language model's average time to first token when its prompt asks for them, which lets it flag slow calls after the fact.

Finish on a real phone call. A server-side figure leaves out the phone leg, and the phone leg is what your caller hears.