Self-hosted voice AI means running the speech-to-text, language and text-to-speech stages on hardware you own, using open-weight models rather than a cloud API. Measured server-side voice-to-voice latency on one consumer GPU sits near 508 milliseconds, but telephony adds 300 to 600ms on a real call. Teams choose it for control and data residency, not lower cost.
Key Takeaways
- Local self-hosting buys control and data residency, not a lower unit price.
- One GPU does the server side in about 500ms. Telephony eats the rest.
- Open-weight speech models trail on accuracy while staying comfortably usable.
This guide covers what local means in a voice stack, why to host it yourself, latency, and where open-weight models still lose. We end with the stack we recommend running today in Dograh, the open-source AI voice agent platform we build, and how it compares with a hosted platform.
What local actually means in a voice stack
Local means the models run on machines you control, so everything except the phone call itself can stay inside your network, and that is only possible with open weights.
A voice agent has two layers, and they get muddled constantly. The first is the orchestration layer, which holds the workflow, the turn-taking, the tool calls, the telephony and the whole call lifecycle. Dograh sits in that layer. It is an orchestrator and not a model. The second layer is inference, meaning the speech-to-text stage, the model that decides what to say, and the text-to-speech stage that says it out loud. Some teams collapse all of that into a single speech-to-speech model instead.
You can fill the inference layer with the models Dograh provides, with any third-party provider through a key you bring, or with open-weight models you host yourself. Only the last one gives you full data sovereignty. Bring-your-own-key moves the commercial contract and leaves the data path exactly where it was, because the audio still reaches the provider. New accounts start on Dograh's managed models. To keep call audio off them, go to the Models page in the dashboard sidebar, set the Organization default to BYOK with your own providers (or local models), and check that no agent has its own model override. If you bring several keys, our post on STT and TTS concurrency limits explains when extra API keys actually add capacity.
This is also why colocation is available only to self-hosters. Colocation means putting every stage on the same machine or in the same availability zone so no network hop sits between them. Closed providers do not ship weights, so with a hosted model the best you can manage is picking a region that happens to match theirs. That shortens the hop without removing it.
That hop is not just a tidiness preference. In a typical hosted setup, speech to text runs on one provider's servers, the language model on another, and text to speech on a third. Every hand-off between them means sending audio out over the network and waiting for it to come back. Put every stage on one self-hosted machine using open-weight models and that hand-off delay is not reduced. It is gone, because there is nothing left to hand off between.
What actually justifies a self-hosted voice stack
Cost is the weakest reason to go local. Control over where the audio goes is the strongest.
Cloud pricing keeps falling. Vercel's AI Gateway Production Index recorded a 23.2 percent fall in price per token during August 2026, the third straight monthly drop. Betting your whole case on being cheaper than the cloud is a bet against a number that keeps shrinking. Self-hosting also has a real cost that never shows up on a GPU invoice. Someone has to keep the model running and roll the stack forward when a better checkpoint lands, and that person's time is usually the biggest expense, not the hardware.
What self-hosting genuinely gives you on cost is predictability, not savings. A GPU you own costs the same at 200 calls a day or 20,000. Per-minute cloud pricing does not, and at high volume that flat cost is what finance actually wants. That is a different claim from cheaper, and it is the one that survives falling cloud prices.
The argument that does hold is jurisdiction. NTT DATA surveyed 2,567 senior leaders across 34 markets for its 2026 Global AI Report. More than 95 percent said private and sovereign AI matter to them, only 29 percent were doing anything concrete about it, and nearly 60 percent named cross-border data rules as a major challenge. That gap between wanting control and having it is the real opportunity for a local stack.
The real test for whether a workload needs full self-hosting is not how sensitive the data feels. It is whether the obligation can be handed to a vendor by contract. HIPAA has exactly that mechanism, a business associate agreement. Attorney-client privilege does not, and once privilege is waived it cannot be undone. That second kind of obligation forces full self-hosting, and no vendor contract fixes it.
Bring-your-own-key does not solve this either. It gives you billing control, not residency. The audio still reaches the provider either way. We wrote the commercial version of this argument up separately in the case for keeping enterprise voice AI on-prem. How BYOK and managed models compare on spend is covered in BYOK vs managed models for voice AI.
This is the short version of the legal case. The full argument, why voice counts as biometric data and which obligations can and cannot be handed to a vendor, is in Why Voice Data Needs to Stay Local.
Open Source Alternative to Vapi / Retell
Self-hosted voice agent platform — no per-minute fees
dograh-hq/dograh
Star on GitHub
Latency is answered, and architecture decides the rest
Against the 800ms end-to-end bar for natural conversation, the phone network leaves a real server-side budget closer to 200 to 500ms, not 800ms.
Voice AI has almost no room for delay. A slow reply is what breaks the feeling of a real conversation, and that is the strongest practical reason to keep the work on hardware you control. When the models and the orchestrator sit on the same machine, Dograh in this case, the audio never travels out to another company's server and back. The hop does not get shorter. It stops existing. Our post on live vs recorded speech-to-text explains why the call needs a fast live model while recordings get a more accurate pass afterwards.
Daily measured this directly. On a voice agent built on NVIDIA's open-weight models, running on one ordinary RTX 5090, server-side voice-to-voice latency came in at 508ms P50 and 544ms P90, broken down as 19ms for speech to text, 171ms for the language model and 108ms for text to speech. That figure covers processing only. The phone network is paid out of the same 800ms budget, so treat 508ms as a good sign rather than the finish line. We went through that budget in detail in the sub-800ms playbook.
Hardware still matters a lot. Run the identical stack on a DGX Spark and the same measurement came back at 1,180ms P50, past a full second and outside the conversational bar, on exactly the same software. Quote the hardware every time you quote a number like this.
Which speech-to-text architecture you pick matters just as much. The Open ASR Leaderboard paper evaluated 86 systems on identical A100 hardware in a snapshot dated 27 March 2026. Among models with transformer or LLM decoders, Conformer-based encoders averaged 758 RTFx against 201 RTFx for Whisper-based ones, close to four times the throughput on the same silicon. RTFx counts seconds of audio processed per second of compute, so higher is faster. Models with lighter CTC decoders go further still: NVIDIA FastConformer CTC Large reached 6,400 RTFx on short-form English.
The takeaway is that local self-hosted voice AI is fast enough today. A slow deployment is usually not a hardware problem. It is the wrong architecture for the job, and teams reach for a bigger card when they should be reaching for a different encoder.
Where open weights still lose
Open-weight speech recognition does not lead on accuracy. On Artificial Analysis's AA-AgentTalk dataset, which is conversational agent speech rather than clean dictation, only two of the twenty systems ranked were open-weight, both from Mistral's Voxtral line. As measured on 23 September 2026, the better of the two, Voxtral Small, scored a 2.27 percent word error rate against 1.44 percent for the top-ranked StepAudio 3 ASR.
That gap is small enough not to matter for a voice agent. A 2.27 percent word error rate is about one word in forty-four, and most of the remaining errors land on proper nouns and digits, which a custom dictionary and a quick confirmation step already handle. You choose local because the audio stays where you put it, not because it wins every accuracy leaderboard. For the after-call pass on saved recordings, Nvidia Nemotron 3.5 Transcribe is an open-weight model built for recordings rather than live calls.
One caution on reading any of this. The Artificial Analysis numbers and the Open ASR Leaderboard numbers quoted earlier come from different test sets on different scales. Never place one beside the other, and never subtract one from the other. Name the benchmark every single time you quote a word error rate, or the number means nothing.
Join the Dograh Community
Dograh is an OSS alternative to Vapi. Join our Slack community for queries, releases, best practices & community interactions.
The stack worth running today
A working local stack comes down to four choices: speech to text, the language model, text to speech, and how they fit together on one GPU. Two of the models below require attribution under their licence, which is worth knowing before you ship.
How to Build a Voice Agent on Fully Local Open-Source Models walks through the whole thing: sizing the card, quantizing the model, wiring the stages together, and what to check if your numbers come in worse than published. For the voice stage on its own, how to run Kokoro TTS locally runs it as a GPU service in Docker.
If you want the deployment shape without the build detail, we keep a standing page on self-hosted voice AI on open-weight models. For a first hands-on step, see how to run faster-whisper locally with Speaches as speech-to-text in Dograh's Docker setup.
Speech to text
For speech to text, the right pick depends on your languages. Parakeet TDT 0.6B v2 has the best accuracy at 6.05 percent average WER on the Open ASR Leaderboard and is English only. Its sibling v3 covers 25 European languages for a small trade-off at 6.32 percent. Both require attribution under CC-BY-4.0. Canary-Qwen 2.5B is a strong alternative that can also answer questions about what it just transcribed. For the widest coverage at 52 languages and dialects, Qwen3-ASR 1.7B carries the more permissive Apache 2.0 licence.
Sizing is where these builds usually go wrong. Voxtral Mini 3B is a good model, but it needs roughly 9.5GB of GPU memory, and a 24GB card has only about 4.2 GiB left once the recommended 27B language model is resident at a 64K context. The two do not fit together. Parakeet TDT 0.6B v2 loads in roughly 2GB and does fit, which is why it is the safer default pairing. Voxtral Small, a much bigger 24B model, does not fit at all, so do not substitute it in. Our comparison of Whisper vs Parakeet vs Voxtral covers how each is served, GPU memory, languages and which to start with.
The language model
For the language model, Dograh recommends Qwen3.8-27B. At 16-bit precision a 27-billion-parameter model needs roughly 54GB just for weights, more than double what a single 24GB card has. Quantized to 4-bit it comes down to 15.93 GiB of weights and peaks at about 20,266 MiB in use with a 64K context, which is what actually makes it fit alongside a small speech-to-text model on one ordinary GPU. That compression costs a little accuracy. Voice work absorbs it well, because the task is narrow: follow a workflow and call a tool correctly. Gemma 4 31B is a strong second choice on the same approach, and both are Apache 2.0, so no attribution is required. For how to weigh one model against another on tool calls, see our guide to choosing an LLM for voice agents.
Text to speech
For text to speech, Dograh recommends Magpie TTS multilingual at 357M parameters, which covers twelve languages and is small enough that the voice stage stops being a latency problem. Two limits are worth knowing. It ships open weights, which is not the same thing as an open source licence, so read the agreement on the model card before you ship. It also ships fixed speaker voices and cannot clone one, so it will not work if your design needs a cloned-voice fallback. Licence terms on TTS models vary far more than on speech-to-text models, and a non-commercial clause on the voice is the most common way a local stack becomes unshippable after the build is finished. If you want a voice you can clone, Kokoro vs Chatterbox vs Coqui XTTS compares which open-source voices you can ship and which ones clone.
Roughly two thirds of what a scripted agent says can come from a pre-recorded human clip library instead of live synthesis, which is where the synthesis bill actually falls. How to Cut Voice AI Costs by Mixing Pre-Recorded Audio With TTS works through those numbers. To keep those clips and live speech in one voice, see voice cloning for pre-recorded and TTS playback.
How this compares with a hosted voice platform
The comparison that matters is where the audio goes, and the three options split cleanly on exactly that.
| Option | Where the audio goes | Cost shape | What you own |
|---|---|---|---|
| Hosted platform | The platform and its model vendors | Per minute, plus a platform fee | Nothing, you are renting the platform |
| Bring your own key | Your chosen model provider | Per token or per minute, billed to you | The contract, not the infrastructure |
| Self-hosted, open weights | Nowhere outside your network, apart from the phone call itself | Fixed, mostly GPU and engineer time | The whole stack, including the audio |
Most closed platforms charge roughly 5 to 7 cents a minute for the platform layer alone, with the model usage billed separately on top. Vendors' own published all-in ranges run from about 7 cents to well over 30 cents a minute depending on which models you select. There is no single industry number to quote here, and a figure presented as one is a price list rather than a market.
The point we would make is that the orchestration layer should not change when you move between those three columns. Dograh is the same open-source orchestrator whether the inference sits on its own managed models, on a provider you brought a key for, or on open weights running on your own GPU. Teams routinely start on hosted models to get a workflow live quickly, then move the inference layer local once compliance catches up with the pilot. If the orchestrator is locked to a vendor, that second step becomes a rewrite. If it is yours, it is a configuration change. Our open-source alternative to Vapi guide covers what you own once you self-host.
Picking models is the easy half. Running them as a production phone agent, with turn-taking, telephony, tool calls and call traces you can actually audit, is the half that takes real plumbing.
Glossary
- Colocation
- Running every stage of a voice pipeline on the same machine or in the same availability zone so there is no network hop between them. It requires open weights, because closed providers do not ship models you can place.
- RTFx
- How many seconds of audio a model transcribes per second of compute. A model at 100 RTFx gets through 100 seconds of audio in one second of wall clock, so higher values mean more concurrent calls per card.
- Quantization
- Storing model weights at lower numeric precision so a large model fits on a smaller GPU. It trades a small amount of accuracy for a large reduction in memory, which is what puts a 27B model on a 24GB card.
- Word error rate
- The share of words a transcript gets wrong, counting insertions, deletions and substitutions. It is comparable only within a single benchmark, because different test sets produce different absolute numbers.
The decision here has stopped being a technical one. Latency was the last honest objection to running voice AI locally, and it has been measured rather than argued since January. What remains is a question about where your audio is allowed to live, and that answer comes from your compliance team rather than your benchmark suite. So pick the architecture before you pick the GPU, and keep the orchestrator yours so the inference layer stays a choice you can change later.
