Gemma 4 31B and Qwen3.8 27B are the strongest open-weight models for voice agents today. On the public aiewf-eval benchmark, Gemma passed 96.4% of turns and Qwen 96.0%, both hosted on Baseten, and Qwen at 4-bit on a local RTX 5090 still passed 95.1%. Dograh, the open-source AI voice agent platform, runs either model on your own servers or on Dograh Cloud.
Key Takeaways
- Gemma 4 31B and Qwen3.8 27B lead the open-weight rows on aiewf-eval.
- Only one Qwen3.8 27B row ran on a local GPU, with no network time.
- Llama and Mistral aren’t in the benchmark, so test them on your own calls before choosing.
This post is part of our guide to running voice AI on local, self-hosted models, where Qwen3.8 27B and Gemma 4 31B lead the language model pick. Below, we read a public benchmark's open-weight rows, then run the same test on one GPU and on a real call in Dograh.
What the benchmark measures
aiewf-eval scores a model across a 30-turn scripted voice conversation, judged turn by turn. Kwindla Hultman Kramer of Daily, the company behind Pipecat, publishes it under the MIT licence on GitHub, with every row's settings next to its score.
Each turn is graded on three things a phone agent gets wrong in practice: calling the right tool with the right values, following the instructions in the system prompt, and answering from the knowledge base it was given. A Claude model acts as the judge. The pass rate counts completed conversations only, so a model that stalls mid-call is not scored on the turns it never reached.
Speed is reported as TTFAT P50, the median time from the caller's turn to the first word of the answer or the first tool call. That is what a caller feels as the pause before the agent speaks. It is not the time to finish the whole reply, and it does not include speech-to-text or the voice.
Open-weight here means the model's weights are published so you can run it on your own hardware. For what that changes about where call data goes, see our page on self-hosted voice AI on open-weight models.
The open-weight rows, and where each ran
Two dense models in the 27B to 31B range lead the open-weight rows, and the fastest one ran on a different setup from the rest. These are the rows with thinking switched off, the setting a phone agent needs.
| Model and setting | Pass rate | Tool errors | Instruction errors | First answer, median | Served on |
|---|---|---|---|---|---|
| Gemma 4 31B it | 96.4% | 3.3% | 3.6% | 497 ms | Baseten |
| Qwen3.8 27B, FP8 | 96.0% | 1.6% | 3.7% | 649 ms | Baseten |
| Qwen3.8 27B, NVFP4 | 95.1% | 1.9% | 4.7% | 101 ms | Local RTX 5090 |
| Qwen3.6 27B | 92.1% | 1.8% | 6.0% | 665 ms | Baseten |
| Gemma 4 26B A4B it | 83.4% | 13.4% | 16.6% | 578 ms | Baseten |
| Qwen3.6 35B A3B, FP8 | 82.6% | 5.6% | 15.7% | 762 ms | Baseten |
Read the speed column with care. Only the Qwen3.8 27B NVFP4 row ran on a local RTX 5090, and the README says local rows use localhost and leave out network time. The Gemma and Qwen FP8 rows ran on Baseten's hosted servers, which include the trip across the internet. So 101 ms against 649 ms shows what removing the network does, along with 4-bit weights on a fast card. It is not one model beating another.
The README also says capture dates, SDK versions, hardware, caching and sampling settings differ across rows. Treat close numbers as a tie. The clear signal is in the bottom two rows. Both are mixture-of-experts models that run only a few billion parameters per token, and both fall to about 83%. Gemma 4 26B A4B made a tool error on 13.4% of turns, four times the rate of its 31B sibling.
For scale, gpt-4.1 scored 95.9% at 743 ms on the same script through OpenAI's API. The two leading open models sit level with it on this test. Qwen3.8 27B and Gemma 4 31B both carry the Apache 2.0 licence on their model cards.
Other open models worth testing
Llama, Mistral and gpt-oss aren’t in the aiewf-eval table yet, so test them on your own script before choosing.
Mistral Small 3.2 24B and gpt-oss-20b are both Apache 2.0, and both fit on one 32 GB card once quantized. Llama 3.3 70B is open weights under Meta’s own licence, so read the terms before you ship. At 4-bit its weights alone need about 35 GB, more than an RTX 5090 holds, so plan for a bigger card or two.
PhoneLLM alpha 1, from the Pipecat team, is also worth a run. It’s a fine-tune of Nemotron 3 Nano 30B-A3B, trained to call tools without thinking first. Check its speed on your own calls. In Dograh, each model is a configuration you switch in the agent’s Settings, so you can run the same test calls against several models and compare the traces.
Open Source Alternative to Vapi / Retell
Self-hosted voice agent platform — no per-minute fees
dograh-hq/dograh
Star on GitHub
Run the same test on one rented GPU
The benchmark runs against any OpenAI-compatible server, so one rented card is enough to fill the gaps. An RTX 5090 matches the card behind the local row, and RunPod lists it at $1.19 an hour on Secure Cloud. Our post on what a RunPod GPU costs per call covers the billing rules.
Start the model on the pod with vLLM. The flags that matter are --served-model-name, which the benchmark asks for by name, and --tool-call-parser, which must match the model family on vLLM's tool calling page:
vllm serve mistralai/Mistral-Small-3.2-24B-Instruct-2506 \
--served-model-name mistral-small-3.2 \
--max-model-len 32768 \
--enable-auto-tool-choice \
--tool-call-parser YOUR_PARSER \
--port 8000
What this does: It serves Mistral Small 3.2 on port 8000 under the name
mistral-small-3.2, with tool calling on. Thirty turns fit easily inside the 32K context.
Then point the benchmark at it. The repo's vllm-openai service reads the address from VLLM_BASE_URL and keeps thinking off unless you turn it on, which matches the rows above:
git clone https://github.com/kwindla/aiewf-eval && cd aiewf-eval
uv sync
export VLLM_BASE_URL=http://YOUR_POD_ADDRESS:8000/v1
uv run multi-turn-eval run aiwf_medium_context --model mistral-small-3.2 --service vllm-openai --pipeline text
uv run multi-turn-eval judge runs/aiwf_medium_context/<timestamp>_mistral-small-3.2
What this does: The first command plays the 30 scripted turns against your server and records each answer and its timing. The second grades every turn, so set up an Anthropic key for the judge first.
As an example, if one model takes two hours to load, run and grade, it costs 2 x $1.19 = $2.38 of GPU time. Your number is on the same footing as the local row, because both skip the network.
Then test it on a phone call in Dograh
The benchmark types its turns, and a phone call adds misheard numbers and names, which is where tool calls break. The models are their own layer. Dograh orchestrates the call flow, tools and telephony, and you choose the speech and language models.
Point Local Models at the same vLLM address and type the served model name, as the LLM settings describe. Then place real calls that hit every tool, with order numbers and names spoken the way your callers say them.
Our guide to choosing an LLM for voice agents sets out that test, and building a voice agent on fully local open-source models shows the whole stack on one box.
Each call's trace shows speech-to-text, model and voice time for every turn, and the tool arguments the model sent. That is where a misheard digit shows up as a wrong value.
The QA node works out the model's average time to first token per call, so you can compare two models across a batch of calls. Small open models sometimes write a tool call out as plain text. Dograh catches those and runs them as tool calls, so they don't get spoken to the caller.
Self-hosted Dograh has no per-minute platform fee, but servers, model keys and phone minutes still cost money. Dograh Cloud charges a 1¢-a-minute platform fee. Keep the model that passes both tests at the speed your calls need.
Join the Dograh Community
Dograh is an OSS alternative to Vapi. Join our Slack community for queries, releases, best practices & community interactions.
Glossary
- TTFAT
- Time to first answer token, aiewf-eval's speed measure: how long a model takes to produce the first word of its answer or its first tool call. The voice can start speaking at that point, so it is the pause a caller hears from the model, though not from speech-to-text or the voice.
- NVFP4
- NVIDIA's 4-bit number format for model weights on its newest cards. It shrinks a 27B model enough to fit one RTX 5090, which is how the fastest Qwen3.8 27B row in the benchmark ran on a single local GPU.
- Mixture of experts
- A model design that runs only some of its parameters for each token. Gemma 4 26B A4B runs about 4B per token, which makes it fast, but on the voice benchmark it made far more tool errors than the dense 31B Gemma.
- Tool-call parser
- The vLLM setting that turns a model's raw output into a structured tool call. Each model family writes tool calls differently, so a wrong parser makes a good model look like it cannot call tools.
