Dograh

Best LLMs for Voice Agents by Latency in 2026: Model and Host

Best LLMs for Voice Agents by Latency in 2026: Model and Host
ComparisonNovember 17, 2026·9 min read

Best LLMs for Voice Agents by Latency in 2026: Model and Host

Vemu Sandeep
Vemu Sandeep·GTM Engineer, Dograh AI

GTM engineer at Dograh working across voice AI and enterprise AI. IIT (BHU) and IIM Visakhapatnam alumnus.

On short prompts, gpt-oss models on Groq and Cerebras return a first token fastest in Artificial Analysis's November 2026 measurements, and Llama 3.3 70B on Amazon Bedrock leads the non-reasoning picks. Teams wanting to own the platform can self-host Dograh, the open-source AI voice agent platform, or run it on Dograh Cloud.

Key Takeaways

  • GPT-4.1 returned a first token in 0.92 seconds on Azure and 1.42 on OpenAI.
  • Reasoning models think before speaking, so compare them at the setting you will ship.
  • At low reasoning, gpt-oss-120b on Cerebras gave the quickest first answer on our list.

Picking a language model for a voice agent comes down to how fast the caller hears the first word, and the host matters as much as the model. We set the criteria, rank seven picks from published benchmarks, show how to measure them yourself and where open-source Dograh fits.

How we ranked the picks

A voice agent starts speaking as soon as the first sentence of the reply streams in, so the number that matters most is how long the model takes to send its first useful token. We ranked on four things, in this order.

First, time to first token on a short prompt, close to a real voice turn. Second, the first answer token for reasoning models, with the reasoning setting named, because thinking time counts before the caller hears anything. Third, the host, since the same model can answer at very different speeds on different clouds. Fourth, whether a self-serve account can actually use that model on that host today.

Every figure comes from Artificial Analysis's November 2026 measurements, an independent benchmark that tests each host on the same workloads. We left out vendor speed claims and small hosts, and we did not reprint the aiewf-eval medians, which our model benchmark post already covers.

What Artificial Analysis measures

Artificial Analysis averages each host over the past five days and re-runs its tests every six hours. Its per-model provider pages include a 1k-input workload, about a thousand tokens of prompt, which is the closest match to a voice turn with a system prompt and a short conversation.

Its providers leaderboard reports the first answer token on a headline mix of 1k, 10k and 100k prompts. The longer prompts make every model look slower than it would on a phone call, so we use the mix only to rank reasoning models against each other.

The two measures answer different questions, so we never compare a 1k figure with a mixed figure. Time to first token on a reasoning model can be a thinking token. The first answer token includes the thinking, which is what a caller actually waits for.

A caller waits with a stopwatch while a reasoning model drops crumpled scraps of thinking into a wastebasket before its first answer token comes out and goes back to the agent, who speaks the first word.
Thinking time counts before the caller hears anything, and at high reasoning settings a caller hears that thinking as silence. Judge a reasoning model by its first answer token, not its first token.

1. gpt-oss-120b on Cerebras or Groq

Artificial Analysis's gpt-oss-120b page puts both hosts at about 0.2 seconds to first token on the 1k workload at high reasoning, 0.208 seconds on Cerebras and 0.205 on Groq. That is the fastest first token of any pick here, though at high reasoning it may be a thought rather than a word.

The answer token tells the real story. At low reasoning on the headline mix, Cerebras delivered the first answer in 0.65 seconds and Groq in 1.26. gpt-oss-120b is open-weight, so you can also host it yourself. Our Groq vs Cerebras comparison covers tool-call strictness, rate limits and which tier each host allows for production.

Dograh

Open Source Alternative to Vapi / Retell

Self-hosted voice agent platform — no per-minute fees

dograh-hq/dograh

Star on GitHub

2. Llama 3.3 70B on Amazon Bedrock

Llama 3.3 70B does not reason before answering, so its first token is the first word of the reply. On the 1k workload, Artificial Analysis's Llama 3.3 70B page measured Amazon Bedrock at 0.567 seconds and Azure at 0.949.

Groq measured faster, at 0.311 seconds, but Groq retired Llama 3.3 70B on its free and developer tiers on 16 August 2026 and now sells it to Enterprise customers only. For a self-serve team, Bedrock is the quickest host for this model. It is an older model, so check how well it follows your prompt and calls your tools before you commit.

3. GPT-4.1 mini on Azure OpenAI

GPT-4.1 mini is a non-reasoning model, so it answers without a thinking step. On the 1k workload, Artificial Analysis measured it at 0.799 seconds to first token on Azure and 1.344 seconds on OpenAI's own API.

That gap of more than half a second comes from the host alone, with the same model behind both. GPT-4.1 mini suits flows with simple tools and short replies, where a smaller model keeps the bill and the wait both small. If your account already runs on Azure, it is the quickest way into this model.

4. GPT-4.1 on Azure OpenAI

GPT-4.1 shows the same host split. Artificial Analysis's GPT-4.1 page measured 0.916 seconds to first token on Azure and 1.415 seconds on OpenAI for the 1k workload, again with no reasoning step.

GPT-4.1 is the larger sibling of GPT-4.1 mini, for flows where the smaller model misses instructions or tool calls. If your agent needs it, Azure gives you the same model about half a second sooner per turn. Over a ten-turn call, those half seconds add up to a noticeably livelier conversation.

Join the Dograh Community

Dograh is an OSS alternative to Vapi. Join our Slack community for queries, releases, best practices & community interactions.

5. gpt-oss-20b on Groq

gpt-oss-20b is the smaller open-weight sibling of gpt-oss-120b. On the 1k workload at high reasoning, Artificial Analysis measured Groq at 0.193 seconds to first token. At low reasoning on the headline mix, Groq's first answer came in 1.36 seconds, close to its 1.26 for the larger model.

Its strength is size. It fits on a single large GPU once quantized, which makes it a candidate for teams that want to run the model on their own servers. Our guide to open-weight LLMs for voice agents covers how to test it that way.

6. Claude Sonnet 5.5 at low reasoning on Azure

Claude Sonnet 5.5 is a frontier model with a reasoning setting, so the host and the setting both matter. At low reasoning on the headline mix, Artificial Analysis measured the first answer at 1.18 seconds on Microsoft Azure, 1.90 on Google Vertex, 2.16 on Amazon Bedrock and 2.42 on Anthropic's own API.

That is the widest host spread on this list, more than a second between the quickest and slowest for the same model and setting. Teams that need Claude's instruction following can win back much of the wait by choosing the host carefully.

7. GPT-5.5 Instant on OpenAI

GPT-5.5 Instant, the June 2026 version, is listed by Artificial Analysis without a reasoning level. On the headline mix its first answer arrived in 1.81 seconds on OpenAI's API.

It sits last here on speed, but it is worth testing where accuracy on complex calls matters more than a fraction of a second. Our GPT vs Claude vs Gemini benchmark shows how the top models of each family compare on a 30-turn phone script.

All seven picks in one table

The table keeps the two measures apart. Never compare a figure in one column with a figure in the other.

PickReasoning settingFirst token, 1k promptFirst answer, headline mix
gpt-oss-120b on CerebrasHigh for 1k, low for the mix0.208 s0.65 s
gpt-oss-120b on GroqHigh for 1k, low for the mix0.205 s1.26 s
Llama 3.3 70B on Amazon BedrockNone0.567 sNot used
GPT-4.1 mini on Azure OpenAINone0.799 sNot used
GPT-4.1 on Azure OpenAINone0.916 sNot used
gpt-oss-20b on GroqHigh for 1k, low for the mix0.193 s1.36 s
Claude Sonnet 5.5 on Microsoft AzureLowNot used1.18 s
GPT-5.5 Instant on OpenAINo level listedNot used1.81 s

Two fast models are missing on purpose. On the headline mix at low reasoning, Gemini 3.8 Flash measured 4.79 seconds and Claude Haiku 5.5 4.92 seconds to the first answer, slowed by the long prompts in the mix. On a short phone script they behave very differently, so test them on your own calls before ruling them out.

How to measure first-token time on your calls

Published numbers rank the options, but your prompt, tools and region decide what callers hear. Each agent reply in a Dograh test call's Call Transcript carries a Reasoning Delay, which is the language model's time to first token for that turn. Switch only the model, run 20 or more calls on each, and compare. Reasoning Delay covers one part of the turn, so never set it against an end-to-end latency budget.

Look at the slow turns as well as the typical one. Coval, a voice testing vendor, reports the 95th percentile rather than the median for this reason, since the slowest turns are the ones callers remember.

For the whole turn, View Trace opens the call with speech-to-text, AI and voice time on each turn, and our guide to reading call traces shows how to find the slow layer.

Outside any platform, stream your real system prompt and a short user turn to each endpoint, set the reasoning level you will ship, and time the first content token rather than the first thinking token.

Where Dograh fits

Orchestration is Dograh's job, covering the call flow, tools and telephony, and the speech and language models are a separate layer you choose. For every agent you choose the speech-to-text, LLM and voice provider, using your own keys or open models running on your hardware. Providers include OpenAI, Azure OpenAI, AWS Bedrock, Google, Groq, OpenRouter and Hugging Face, and you can type in a model ID that is not on the list.

For GPT-5 models the platform asks for minimal reasoning, which keeps the first answer quick. On your own keys there is also an LLM fallback to switch on, a backup model that steps in when the main one errors or stays silent past a set time, 1.5 seconds by default.

Self-hosted Dograh has no per-minute platform fee, but servers, model keys and phone minutes still cost money. Dograh Cloud charges a 1¢-a-minute platform fee. A self-hosted setup leaves the server, the database and upgrades to your team, with community help on the open-source edition and custom SLAs on the Enterprise plan.

Glossary

Time to first token
How long a host takes to send back the first token of a reply. On a reasoning model that token can be a hidden thought, not a word.
First answer token
The first token of the reply itself, measured after any thinking. It is the closer guide to what a caller waits for.
1k workload
Artificial Analysis's test with about a thousand tokens of prompt. It is the closest of its workloads to a single voice turn.
Headline mix
Artificial Analysis's average across 1k, 10k and 100k prompts. The long prompts make every model look slower than on a phone call.

Frequently Asked Questions

Get started with Dograh

Build, deploy, and scale AI agents with Dograh. Join the community of developers building the future.