Speech-to-Speech Tool Calling: Where It Breaks on Phone Calls

Speech-to-speech tool calling breaks in repeatable ways. The model fires a tool before the caller finishes correcting a detail, picks the right tool with the wrong value, or runs tools while saying nothing. Test those cases on real calls, confirm values before any tool changes a record, and move the agent to a cascade when calls chain several tools.

Key Takeaways

• The tool choice is usually right. The value passed to it is what breaks.
• The top model on tool tasks hands them to a text model.
• Switch an agent to a cascade when a call chains several tools.

This post is part of our guide to speech-to-speech voice AI for phone calls, where tool calls are the weak spot. Below, we cover how those calls fail, what benchmarks show and when to move an agent to a cascade in Dograh, the open-source AI voice agent platform we build.

How speech-to-speech tool calls fail

Speech-to-speech tool calls fail in four ways, and each one can be tested. The first is acting too early. In Full-Duplex-Bench v3, a research test of chained tool calls with hesitant, self-correcting speech, the authors found that models "commit intermediate parameters before the correction arrives." In one published example, Gemini Live 3.1 started a flight search while the speaker was still talking and kept Rome after the speaker changed it to Milan.

The second is the right tool with the wrong value. NVIDIA's NemotronLabs VoiceChat paper reports that its open model picks the right tool 82.5% of the time on the same benchmark, but gets the values right only 42.2% of the time. FDBv3 shows the same pattern on a hosted model: Ultravox chose the right tools for a currency task but converted dollars to Nigerian naira instead of euros.

The third is silence. In the FDBv3 examples, Gemini Live 3.1 said nothing at all in 22% of cases, yet in 86% of those silent cases it still ran tool calls. On a phone call, the caller hears dead air while a record changes behind the scenes.

The fourth is claiming success. A model can say "done" for an action that never ran, and our comparison of Cascade vs Speech-to-Speech shows a published case of an agent confirming an address change it never made. These come from benchmarks rather than our own calls, and each one points at a test you can run.

What the benchmarks show

The numbers back a narrower claim than "cascades always win." FDBv3 scored each system on Pass@1, the share of tasks done right on the first try, with real speech full of pauses and corrections.

GPT-Realtime beat the paper's cascade, so a speech-to-speech model can win on tools. But every system dropped on harder scenarios. GPT-Realtime went from 0.750 on easy tasks to 0.433 on hard ones, and hard tasks are what a claims or billing line looks like.

Task benchmarks show a wider spread. On Artificial Analysis's tau-Voice, a set of customer-service tasks that need tools, GPT-Live-1 (Astra, medium) scores 74.5% and Gemini 3.8 Live Extended Thinking (High) 68.6%. GPT-Realtime-2.1 (High) scores 45.7% and standard Gemini 3.8 Live 30.1%. Our view holds: speech-to-speech models can fail on calls with complex tool calling, and the gap depends heavily on the model.

Open Source Alternative to Vapi / Retell

Self-hosted voice agent platform — no per-minute fees

dograh-hq/dograh

Star on GitHub

Why handing tools to a text model helps

The highest-scoring model, GPT-Live-1, does not call tools from the voice model directly. OpenAI's GPT-Live guide describes a full-duplex voice model that hands reasoning and tool use to a backend agent, and leaves confirmations to your application. The voice keeps the conversation going while a text model does the careful part.

When you run gpt-live-1 in Dograh, the voice model is told to pass every substantive request and every workflow decision to that backend, gpt-5.4-mini by default. It is also told to use the caller's latest correction and never to claim an action succeeded before the backend confirms it. Those two rules target the first and fourth failures above.

Gemini takes a different route. The Gemini Live tool guide shows Gemini 3.8 Live calling tools asynchronously, so it can keep talking while a lookup runs, where Gemini 3.1 Flash Live pauses on every call. The Extended Thinking variant reasons before it acts, which is where its higher tau-Voice score comes from.

How to test tool calls before go-live

Build a short test script from the failure modes, then run it on real phone calls. Include a booking where the caller corrects the date mid-sentence, an amount with a currency, a spelled name and a request that needs two tools in a row. Run each one several times, because the same model can answer differently on the same input.

Write the confirmation into the prompt. Before any tool that changes a record, the agent reads the values back and waits for a yes. That costs a few seconds and catches the wrong-value failure before it reaches your system. Our guide to adding tool calls to a voice agent shows the tool setup step by step.

Set up the tool so a bad run is easy to spot and undo. An HTTP API tool times out after 5 seconds by default and does not retry, so the agent gets an error it can explain. Typed tool messages do not play on speech-to-speech agents, so use a recording for the line the caller hears during a lookup. A Preset Parameter named workflow_run_id lets your system ignore a duplicate booking.

Then read the evidence. View Trace opens each call's trace in Langfuse, where you can see which tool ran, with which values and when. QA scores each node separately, so a node where the agent claimed a booking without a tool call stands out.

When to switch the agent to a cascade

Base the switch on how complex the call is, rather than on one failed test. If a call chains more than one tool that changes a record, or one wrong value costs money or trust, run that agent on a cascade. Keep speech-to-speech for calls that are mostly conversation with a lookup or two.

In Dograh the switch is per agent. Each agent uses a named model configuration, so you move it from a Realtime configuration to a cascade one in its Settings and publish. An API trigger can pick a different configuration for one call, but Pre-Call Data Fetch cannot change it once the call has connected.

Some flows need the cascade anyway. Handing the call to another agent refuses on a speech-to-speech call and tells the agent to keep helping, as the Transfer Agent page describes. Transfer Call to a person works on both. Running the two kinds side by side in one account lets you move one line at a time and compare their traces.