GPT vs Claude vs Gemini for Phone Agents: The Benchmark

GPT, Claude and Gemini are tied at the top for phone agents, so pick on speed and price. On the public aiewf-eval benchmark, Claude Sonnet 5.5, Gemini 3.7 Flash and gpt-5.6-terra each passed 99.9% of turns in a 30-turn script, with median first answers of 838, 821 and 1,004 ms. Smaller models split widely, from 97.2% for Claude Haiku 5.5 to 80.9% for gpt-5-mini. Dograh, the open-source AI voice agent platform, runs all three on your own keys, self-hosted or on Dograh Cloud, so you can test them on the same calls.

Key Takeaways

• The top Claude, Gemini and GPT rows all pass 99.9% of turns.
• Speed and the reasoning setting are what set them apart.
• Smaller models split widely, from Claude Haiku 5.5 to gpt-5-mini.

This post is part of our guide to LLM API pricing for voice agents, which prices each model family on one 5-minute call. Here we read one public benchmark's instruction-following and delay results for GPT, Claude and Gemini, then show how to run it on your own script.

One script, three model families

A fair comparison needs one script run against every model, and aiewf-eval is the public one that covers all three families. Kwindla Hultman Kramer of Daily, the company behind Pipecat, publishes it on GitHub under the MIT licence.

It plays the same 30-turn scripted conversation to each model and grades every turn on calling the right tool with the right values, following the system prompt and answering from a given knowledge base. A Claude model is the judge. Daily's write-up of the benchmark explains why each turn is graded on its own: one bad turn on a phone call is enough to lose the caller.

Each family ran on its own provider's API, Claude on Anthropic's, Gemini on Google AI Studio and GPT on OpenAI's. The README says capture dates, SDK versions, caching and sampling settings differ across rows, so a few points apart is a tie. Pass rates count completed conversations only, and the table shows how many completed out of up to 30.

The results, family by family

At the top the three families tie, and what separates them is delay and the reasoning setting each needed to get there. These rows are text-mode results on the benchmark's script.

Gemini 3.7 Flash and Claude Sonnet 5.5 are the fastest of the top rows, about 20 ms apart. gpt-5.6-terra reaches the same score at medium reasoning and takes about 170 ms longer. Gemini 3.8 Flash also scores 99.9%, but its median first answer is 1,411 ms, slow enough for a caller to notice.

Below the top, instruction errors decide more than tool errors. gpt-4.1, Dograh's default OpenAI model, is quick at 743 ms but broke an instruction on 4.1% of turns, about one turn in 25. Gemini 3.5 Flash made few tool errors at 0.4% but 5.2% instruction errors. Among the smaller models, Claude Haiku 5.5 at low reasoning held up best, at 97.2% and 651 ms. gpt-5-mini fell to 80.9% and completed only 11 conversations, so read its row as a warning, not a measurement.

Model delay is only part of a call's delay

The benchmark's speed column is the time to the model's first word or tool call, and the pause a caller hears is longer than that. Each turn also includes speech-to-text finishing the caller's words before the model starts, and the voice starting to speak after it. Our rule of thumb is that the whole turn should stay under 800 ms, and speech-to-text is usually the slower part, not the model.

Daily's write-up treats roughly 700 ms to the first token as a working guide for text models in this kind of pipeline. The README calls it a guideline rather than a measured threshold. On that guide, all the top rows except Gemini 3.8 Flash sit within a few hundred milliseconds of it.

The route to the model adds its own time. In Dograh, Claude runs through OpenRouter or AWS Bedrock, which puts one more service between your agent and Anthropic than the benchmark had. On OpenRouter, a provider order setting in the LLM settings lists which providers to try first, so turns don't land on a slow one. Our post on Claude API pricing covers which Claude model is fast enough per call.

Open Source Alternative to Vapi / Retell

Self-hosted voice agent platform — no per-minute fees

dograh-hq/dograh

Star on GitHub

The settings Dograh sends each model

A model's score depends on its settings, and Dograh sets some for you. For any GPT-5 model, Dograh sends minimal reasoning and low verbosity, and no temperature. None of the benchmark's GPT-5 rows ran at minimal, so the 99.9% for gpt-5.6-terra at medium does not tell you how it does at that setting.

For gpt-4.1, Gemini on AI Studio and models through OpenRouter, Dograh sets the temperature to 0.1, which keeps replies steady from call to call. Bedrock uses the provider's own default. On AI Studio, Dograh's default model is gemini-3.5-flash, and you can type any other model ID.

A backup model covers the turns where the main one fails. With your own keys, turn on the LLM fallback and a second model takes over if the first errors or sends nothing within 1.5 seconds by default. It is off until you switch it on. For models that hear and speak audio directly, see our guide to speech-to-speech voice AI. This comparison stays on text models in a speech-to-text, LLM and voice pipeline.

Run the benchmark, then your own calls

The benchmark is free to run on your own keys, and each family needs only its own service name. The commands below run one model through each provider, with --service picking the provider and --model taking the provider's model ID:

git clone https://github.com/kwindla/aiewf-eval && cd aiewf-eval
uv sync
uv run multi-turn-eval run aiwf_medium_context --model YOUR_CLAUDE_MODEL_ID --service anthropic
uv run multi-turn-eval run aiwf_medium_context --model YOUR_GEMINI_MODEL_ID --service google
uv run multi-turn-eval run aiwf_medium_context --model YOUR_GPT_MODEL_ID --service openai
uv run multi-turn-eval judge runs/aiwf_medium_context/<timestamp>_<model>

What this does: Each run plays the 30 scripted turns to one model and records its answers and timing in a folder under runs/. The judge step then grades every turn, giving you the same columns as the table above for your own settings.

Then run the check that counts, on real phone calls. In Dograh, build one agent with your real tools and save one model configuration per candidate, so only the model changes between runs. Place about 10 calls per model with the same caller script, including a spoken order number and a rule the agent must keep, such as one question per turn. Our guide to choosing an LLM for voice agents sets out that test step by step.

Read the results from each call. The QA node works out the model’s average time to first token per call, and the transcript shows each broken instruction.

Pick the fastest model that keeps every rule on your calls, then rerun the check when a provider ships a new version. With your own keys, every provider bills you at its own rate. Self-hosted Dograh has no per-minute platform fee, but servers, model keys and phone minutes still cost money. Dograh Cloud charges a 1¢-a-minute platform fee.