Building a voice agent on fully local open-source models means running speech-to-text, a quantized language model and text-to-speech on one GPU you own. A 24GB card fits a 4-bit 27B model beside a 600M speech model. Measured voice-to-voice latency on that shape of hardware is around 508 milliseconds.
Key Takeaways
- A 24GB card fits a 4-bit 27B model plus a small speech model.
- Time to first token matters more than decode speed for voice.
- Small models handle short tool calls well and long planning badly.
This post is part of our guide to Running Voice AI on Local, Self-Hosted Models. That guide argues why the audio should stay on your hardware. This one is the build, with Dograh as the open-source orchestration layer wrapped around models you host yourself.
What actually has to fit on one card
Start with the memory arithmetic, because that one number settles the rest of your stack. An independent RTX 4090 benchmark run in August 2026, with fixed seeds and a published evidence bundle, measured both candidate language models at Q4_K_M on a single 24,564 MiB card. Qwen3.8-27B decoded at 49.09 tokens per second and peaked at 20,266 MiB with a 64K context window. Gemma 4 31B-it decoded at 45.00 and peaked at 21,906 MiB, though only at 32K. At 64K with an F16 key/value cache it ran out of memory.
| Model at Q4_K_M | Weights on disk | Peak VRAM | Context tested | Decode | Warm first token |
|---|---|---|---|---|---|
| Qwen3.8-27B | 15.93 GiB | 20,266 MiB | 64K | 49.09 tok/s | 0.107s |
| Gemma 4 31B-it | 17.07 GiB | 21,906 MiB | 32K | 45.00 tok/s | 0.421s |
The figure that matters for a voice build is the leftover. Qwen3.8-27B at 64K leaves about 4.20 GiB free, and that remainder is your entire speech budget. It also rules out the model most people reach for first. Voxtral Mini 3B wants roughly 9.5GB in bf16 under vLLM, so it does not fit here. Parakeet TDT 0.6B v2 loads in about 2GB, and Magpie TTS multilingual at 357M parameters barely registers.
The language model is seated first, and the flag marks what it leaves behind. Both candidate speech models are drawn at the same scale as that slot, so whether each one fits is something you can see rather than something you have to work out.
Drop the context window to 32K and either language model leaves more room for the speech stages, and phone calls rarely get near 32K tokens anyway. It does not change the Voxtral Mini answer, though. The leftover at 32K is still well short of the 9.5GB it wants. Check the Qwen3.8-27B model card for the tensor layout before you pick a quantization, since the vision projector adds another 0.864 GiB if you load it.
Check the leftover on your own card rather than trusting anyone's table, including this one.
# what the card is actually holding while a call is running
nvidia-smi --query-gpu=memory.used,memory.total --format=csv
# memory.used [MiB], memory.total [MiB]
# 20266 MiB, 24564 MiB
Pick a runtime per stage, then tune the cache
Nothing forces you to serve every stage with the same runtime, and one runtime for everything is usually the wrong call. For the quantized language model, llama.cpp is the shortest path. It is MIT licensed and serves GGUF weights anywhere from 1.5-bit to 8-bit. The server it exposes speaks the OpenAI API, so your agent code does not care what sits behind it. Ollama wraps the same engine in a single binary if you want a model answering in one command, and vLLM is the one to reach for when you need throughput across concurrent calls, since paged attention and continuous batching are built for exactly that load shape.
The knob almost nobody touches is key/value cache precision. That August benchmark ran an F16 cache by default, which is where Gemma 4 31B fell over at 64K. Switching the cache to Q8_0 brought the same model back inside the card at 21,956 MiB. When your context is long and your card is full, change the cache before you change the model.
Those two cache flags are the whole trick, and they are the first thing to reach for when a context window will not fit.
llama-server \
--model qwen3.8-27b-instruct-Q4_K_M.gguf \
--ctx-size 65536 \
--cache-type-k q8_0 \
--cache-type-v q8_0 \
--n-gpu-layers 99 \
--host 127.0.0.1 --port 8080
That exposes an OpenAI-compatible endpoint on loopback, which is what lets the agent code stay the same whichever runtime you end up with.
Run the speech stages as separate processes on the same machine. Both NVIDIA speech models load through NeMo, and the text-to-speech stage needs a streaming server instead of a batch call.
import nemo.collections.asr as nemo_asr
asr = nemo_asr.models.ASRModel.from_pretrained("nvidia/parakeet-tdt-0.6b-v2")
asr = asr.cuda().eval()
Open Source Alternative to Vapi / Retell
Self-hosted voice agent platform — no per-minute fees
dograh-hq/dograh
Star on GitHub
Tool calling decides whether a small model holds
A voice agent does not need a frontier model, provided the tool calls come back correctly every time. The AgentFloor benchmark puts real numbers on that. It scored 16,542 runs across 16 open-weight models from 0.27B to 32B, plus GPT-5, and on overall task completion gemma4:26b landed at 59.9 percent against GPT-5 at 59.9 percent, at roughly fifteen times lower cost when self-hosted.
State the scope honestly, because the paper does. Its tier breakdown shows small models are strong on short structured tool use and close to useless on long-horizon planning, where several score zero. The authors recommend routing, meaning small models carry the routine base while a frontier model stays in reserve for rare deep planning.
Voice work is the short structured case. Look up an order, check a slot, write a record, read the answer back. That is the tier small models already pass, which is why the argument holds here and would not hold for an autonomous coding agent.
Time to first token is the other number to watch. On that RTX 4090 run, Qwen3.8-27B returned a warm first token in 0.107 seconds against 0.421 seconds for Gemma 4 31B on the same 512-token prompt. Decode speed separated them by eight percent. First-token latency separated them by four times, and a caller hears the first token rather than the hundredth. On raw reliability the same suite scored both at 90 of 90 single tool calls, with Gemma taking 30 of 30 multi-step runs against 28 of 30 for Qwen3.8.
Wire the stages so nothing crosses a network
Colocation is the whole point of this build, and it only pays off if the orchestration layer sits on the same machine as the models. Daily measured a fully open-model agent at 508ms P50 and 544ms P90 server-side voice-to-voice on one RTX 5090, splitting into 19ms for speech to text, 171ms to the language model's first byte and 108ms for the voice. Those figures came from a build where every stage ran on one box, with no network card sitting between one stage and the next.
In practice colocation is something you can read off the configuration. Every stage points at loopback, and the moment one of these becomes a host you do not own, the audio has left the machine.
LLM_BASE_URL=http://127.0.0.1:8080/v1
STT_URL=http://127.0.0.1:8081
TTS_URL=http://127.0.0.1:8082
Two build details do most of that work. Text to speech has to stream instead of synthesizing a whole sentence before it speaks. Daily's own testing found this moved mean first-audio latency by roughly 3x on the same hardware. Turn detection has to run in parallel with transcription instead of waiting for a final transcript.
Above all of it sits the orchestrator, which is a separate layer from every model named so far. Dograh handles the workflow, the turn-taking, the tool calls, the telephony and the call lifecycle. It is not a model. There are three ways to fill the inference layer. Use Dograh's own models, bring your own key for any third-party provider, or host open weights yourself. Only the last one keeps the audio inside your boundary, since bring-your-own-key moves the contract and leaves the data path where it was. The commercial version of that argument is in our case for on-prem enterprise voice AI.
Join the Dograh Community
Dograh is an OSS alternative to Vapi. Join our Slack community for queries, releases, best practices & community interactions.
What a published latency number will not tell you
Every figure above came from someone else's hardware, and the spread between two local machines is wider than most people expect. The same open-model stack that hit 508ms P50 on an RTX 5090 measured 1,180ms P50 and 1,359ms P90 on a DGX Spark, with the language model alone taking 750ms. One stack, two local boxes, more than double the latency. Quote the hardware every time you quote a number like this.
Then add the leg a server-side measurement leaves out. A WebRTC agent carries roughly 250ms of extra client-side latency, and a telephony agent carries 300ms to 600ms. So a 508ms server-side P50 lands nearer 800ms to 1,100ms on a real phone call, which is right at the 800ms mark where a conversation stops feeling natural to the person on the line. Our sub-800ms latency playbook covers how to claw that back, and the cascade versus speech-to-speech tradeoff moves the budget again.
The voice crosses the machine, then carries on down a phone line that the server-side measurement never saw. The slower box leaves its caller waiting further along, and passes the 800ms mark before the phone leg has even started.
Read every licence before you ship. Qwen3.8-27B and Gemma 4 31B are Apache 2.0. Parakeet TDT 0.6B v2 and Canary-Qwen 2.5B are CC-BY-4.0, so attribution is required. Magpie TTS ships under NVIDIA's own open model licence, which makes it open weights rather than open source. Do not group it with the Apache 2.0 models as though the terms match. It also ships fixed speaker voices and cannot clone one, so if your design needs a cloned-voice fallback behind a library of human recordings, you need a different engine. Chatterbox is MIT and does clone. The pattern that makes a cloned fallback worth having in the first place, a library of real recordings with synthesis only for the lines that vary, is the subject of cutting voice AI costs with pre-recorded audio.
Build it in the order the constraints appear. Size the card first, then pick the quantization and cache precision that leave room for the speech stages. Prove tool calling against your real workflow before you tune anything else, and benchmark your own box on day one, because every published figure was measured on hardware you do not own. Keep Dograh, the orchestrator, yours, and the inference layer stays a choice you can change later.
Glossary
- Q4_K_M
- A 4-bit GGUF quantization that stores most weights at four bits and keeps a few sensitive tensors higher. It is what turns a 27B model into a file of roughly 16 GiB that loads on a 24GB card.
- Key/value cache
- The stored attention state for every token already in the conversation. Its precision, F16 or Q8_0, is often the difference between a context window that fits on your card and one that crashes the server.
- Time to first token
- How long a language model takes to emit its first output token. In a phone call it sets perceived responsiveness, so it matters more than the decode rate that follows it.
- Tool calling
- A model emitting a structured function call instead of prose, so the agent can query a database or book a slot mid-call. Reliability here, rather than raw reasoning ability, decides whether a small local model can run a voice workflow.
