One GPU has to hold three models plus a slice of cache for every live call. A 24 GB card fits a 4-bit 27B language model with Parakeet and Kokoro, while Voxtral Mini needs a 32 GB card or larger. Dograh, the open-source AI voice agent platform we build, runs the calls beside that card when you self-host it.
Key Takeaways
- Each live call needs its own cache, about 250 MiB at 4,000 tokens.
- Cap vLLM's memory share, or the speech models get nothing.
- Keep speech models loaded on a shared card instead of unloading them.
This post is part of our guide to Running Voice AI on Local, Self-Hosted Models, where sizing is the step most local builds get wrong. Here we fit speech-to-text, the language model and the voice on one card, size the cache each call takes, and wire it into Dograh.
What has to share the card
Three things compete for one card's memory, and only one of them is easy to see. The first is model weights, for the language model, the speech-to-text model and the voice model. The second is runtime overhead, the working memory each serving process keeps for itself. The third is the cache each live call needs while the language model reads its conversation, and that one grows with every caller you add.

Weights are what most sizing tables list, and they are fixed. The cache is what decides how many calls one card carries, and a table that says a card "fits the stack" without it is only sizing the first call.
Dograh itself needs no GPU. It is the orchestration layer that runs the call flow, tools and telephony, and the speech and language models are a separate layer on the card. For sizing a whole server, its CPU, memory and monthly bill, see running voice AI on your own GPUs. This post stays on the card.
Every live call takes its own slice
The language model keeps keys and values for every token of a conversation, the KV cache, so the model need not reread the whole call on each turn. Its size per token comes straight from the model's config. Qwen3.8-27B, the model we recommend, has 64 layers, and only 16 of them use full attention, with 4 key-value heads of 256 dimensions each. The other 48 layers keep a small fixed state per call instead.
Our arithmetic from those numbers, at 16-bit, worked through for one call:
- Per token, 2 (keys and values) x 16 layers x 4 heads x 256 dimensions x 2 bytes = 65,536 bytes, or 64 KiB.
- A call holding 4,000 tokens of prompt and history, 4,000 x 64 KiB = 256,000 KiB, about 250 MiB.
- A call that fills a 32K context, 32,768 x 64 KiB = 2 GiB.
That fits what our one-card build measured. A 4-bit build at a 64K context peaked at 20,266 MiB on a 24,564 MiB card, about 3.9 GiB above its 15.93 GiB of weights, close to the 4 GiB a 64K cache works out to. It left about 4.2 GiB free, and every concurrent call has to come out of a budget like that one.
Open Source Alternative to Vapi / Retell
Self-hosted voice agent platform — no per-minute fees
dograh-hq/dograh
Star on GitHub
What fits on 16, 24, 32 and 48 GB
The table below adds published model sizes to our cache arithmetic. Treat it as a set of examples, and measure your own card before you commit.
| Card | Language model | Speech-to-text | Voice | Left for calls and overhead |
|---|---|---|---|---|
| 16 GB | A 4-bit model of 8B to 14B | Parakeet TDT 0.6B v2 | Kokoro-82M | Measure. A 4-bit 27B model alone is over 17 GB |
| 24 GB | Qwen3.8-27B 4-bit, 64K context | Parakeet TDT 0.6B v2 | Kokoro-82M | About 4.2 GiB before speech, so little room for calls |
| 32 GB | Qwen3.8-27B 4-bit | Voxtral Mini 3B | Kokoro-82M | About 5 GB |
| 48 GB | Qwen3.8-27B 4-bit | Voxtral Mini 3B | Kokoro-82M | About 21 GB |
The speech model is usually what tips the budget. Voxtral Mini's model card puts it at about 9.5 GB of GPU memory in bf16, so it does not fit beside a 27B model on 24 GB. Parakeet's model card asks for at least 2 GB of memory to load it, and longer audio needs more, so check its real footprint on your own calls. Kokoro has 82 million parameters, about 160 MB at 16-bit by our arithmetic.
The 32 and 48 GB rows add up 15.93 GiB of language model weights, about 17.1 GB, plus 9.5 GB for Voxtral and 0.16 GB for Kokoro, which is close to 27 GB. On a 48 GB card, the 21 GB left holds dozens of 4,000-token calls by memory alone, so speed runs out before memory does. Each model also needs a server that speaks the route Dograh calls, and our comparison of Whisper vs Parakeet vs Voxtral shows which server fits each one.
Keep the runtimes from fighting over memory
On one card, the serving processes need explicit limits, or the first one to start takes the room. vLLM's engine arguments set --gpu-memory-utilization, the share of the card it may use, to 0.92 when you leave it unset. On a 24 GB card that leaves under 2 GB for everything else, too little for Parakeet.
Work the share out from what the speech side needs. A labeled example for a 48 GB card running Voxtral and Kokoro beside the language model:
- Speech side, 9.5 GB for Voxtral + 0.2 GB for Kokoro + 2 GB of margin = 11.7 GB.
- Language model share, (48 - 11.7) / 48 = 0.756, so set 0.75.
If Voxtral also runs in vLLM, that second process takes 0.92 by default too, so give each vLLM process its own share. --kv-cache-memory-bytes sets the cache size directly when you would rather fix the call budget than a fraction.
Speech servers can give memory back at the wrong time. In the Speaches configuration, the two model TTL settings, stt_model_ttl and tts_model_ttl, unload an idle model after 300 seconds by default. On a quiet line, the next caller waits for a reload. Set both to -1 on a one-card build, so the models stay loaded and the memory you planned for them stays theirs.
Join the Dograh Community
Dograh is an OSS alternative to Vapi. Join our Slack community for queries, releases, best practices & community interactions.
When it does not fit, and the load test
When the sum comes out over the card, trade in this order. Shorten the context first, since phone calls rarely need 64K tokens and a 32K limit halves the largest cache. Then pick a smaller speech-to-text model, such as Parakeet or a distilled Whisper in place of Voxtral. Then a smaller language model, and only then a second card.
Once it fits on paper, check it on the card. Run nvidia-smi with every model loaded and a test call running, and compare the memory used with your table. Then raise the number of calls step by step, open View Trace on calls at each level to see the speech-to-text, AI and voice time per turn, and stop where a turn crosses your latency budget. Self-hosted Dograh runs 10 calls at once by default, which is a sensible first ceiling for one card.
Connecting and running the card in Dograh
Dograh reaches the card through the Local Models (Speaches) provider. For speech-to-text it starts at Systran/faster-distil-whisper-small.en with a base URL of http://localhost:8000/v1. The language model uses the same provider pointed at any OpenAI-compatible server, such as vLLM.
The voice starts at Kokoro with the af_heart voice on the same base URL. If Dograh runs in Docker, localhost means Dograh's own container, so point each base URL at the model server's address on your network. Our tutorial on how to run faster-whisper locally with Speaches walks through the compose file and the first test call.
Pre-recorded clips take load off the voice model. Lines the agent says most play as recordings, and Kokoro only speaks the rest, which matters most at the busy hour you sized the card for.
The card is the main cost of this build. Self-hosted Dograh has no per-minute platform fee, but servers, model keys and phone minutes still cost money. Dograh Cloud, if you later move the orchestration there, charges a 1¢-a-minute platform fee.
A one-card build also makes the server, the database and the upgrades your own job. Help on the open-source version comes from the community, and the Enterprise plan adds custom SLAs and on-premise deployment support. In return, with all three models on your own card, no call audio goes to an outside model provider.
Glossary
- KV cache
- The keys and values a language model keeps for every token of a conversation, so it does not reread the whole call on each turn. It grows with call length and with every concurrent call, which is why it decides how many calls one card carries.
- Full attention
- A model layer type that attends to every earlier token and so stores a KV cache entry per token. Qwen3.8-27B has 16 of them out of 64 layers, which keeps its cache at about 64 KiB per token.
- gpu-memory-utilization
- A vLLM setting for the share of the GPU's memory the server may use. It resolves to 0.92 when unset, which leaves too little for speech models on a shared card, so set it from your speech budget.
- Model TTL
- How long Speaches keeps an idle speech model loaded, 300 seconds by default. Setting it to -1 keeps the model loaded, so a quiet line never pays a reload on the next call.

