Cascade vs Speech-to-Speech: Tradeoffs Explained

A cascade chains speech-to-text, a language model and text-to-speech, while a speech-to-speech model takes the caller's audio in and answers in audio. Speech-to-speech handles spoken reasoning well and hears tone, but fails far more often on multi-step tasks that call tools. Use a cascade for tool-heavy calls and speech-to-speech for short, simple ones.

Key Takeaways

• Realtime models ace spoken reasoning yet fail most multi-step tool tasks.
• Speech-to-speech is not automatically faster; more thinking adds seconds.
• Choose per agent: tool-heavy flows on a cascade, short calls on speech-to-speech.

This post is part of our guide to Voice AI Explained: The Complete Glossary and Beginner's Guide. The guide sets out the two ways to build the model layer; this post looks at the measurements behind that choice and where each one breaks. Dograh runs both kinds of agent, so the last section treats it as a choice you make per agent.

How each architecture handles one turn

The real difference is whether the caller's words ever become text the system can read before the agent replies.

In a cascade, speech-to-text writes a transcript. The language model reads it with the instructions for the current step, decides whether to call a tool, and writes the tool's arguments as text. Text-to-speech then speaks the reply. Each hand-off is a checkpoint: you can read what the agent heard and swap any one model on its own. How the voice AI pipeline works follows one sentence through those stages.

A speech-to-speech model takes the caller's audio in and gives audio out. Full-duplex models go further and listen while they talk. Nothing in the middle can be inspected during the turn, and the transcript most providers return is a side output that can differ from what the model acted on.

Almost every tradeoff below follows from that missing text step.

Tool calling is where speech-to-speech falls behind

Our view is that cascades are good with reasoning, tool calling and voice cloning, which makes them the fit for complex use cases, while speech-to-speech models can fail on calls that involve complex tool calling.

The word "reasoning" needs care here. Speech-to-speech models are good at answering spoken questions that need thinking, and much weaker at tasks that need tools. On Artificial Analysis's speech-to-speech leaderboard in October 2026, Gemini 3.1 Flash Live (High) scored 97% on Big Bench Audio, a set of spoken reasoning questions. On tau-Voice, a set of customer-service tasks that need tools, the same model scored 37.7%, and GPT-Realtime-2.1 (High) scored 45.7%.

The tau-Voice benchmark paper shows the same split. Across 278 tasks, three audio-native realtime models (gpt-realtime-1.5, gemini-live-2.5-flash-native-audio and grok-voice-agent) passed 31-51% in clean audio and 26-38% with background noise and accents. GPT-5 with reasoning passed 85% of the same tasks in text. In one transcript, an agent told the caller "I've updated your shipping address" without ever making the tool call.

A cascade keeps its reasoning in a text model, the side that scored 85% here, so it starts from the stronger position, minus whatever speech-to-text mishears. The paper measured realtime models only, so this is our reading rather than a head-to-head result.

Gemini 3.1 Flash Live also makes tool calls one at a time, according to Google's Live API docs. The newer speech models that score best on tool tasks work like a cascade inside: they pass the thinking to a text model behind the voice. On the same leaderboard, GPT-Live-1 (Astra, medium) reached 74.5% on a single trial and Gemini 3.8 Live Extended Thinking (High) 68.6%. Newer alone does not mean better, though: Gemini 3.8 Live without extended thinking scored 30.1%. Our comparison of GPT-Live-1 vs Gemini Live vs Grok Voice API tests those models on phone calls.

Open Source Alternative to Vapi / Retell

Self-hosted voice agent platform — no per-minute fees

dograh-hq/dograh

Star on GitHub

Speed, tone and cost

Speech-to-speech wins back ground on how a call feels, though less on speed than its reputation suggests.

Latency

In our experience, a reply feels natural when the full round trip stays under about 800 milliseconds. Speech-to-speech does not clear that bar by default. Artificial Analysis measured time to first audio for Gemini 3.1 Flash Live at 0.96 seconds on Minimal thinking and 2.99 seconds on High. GPT-Realtime-2.1 took 0.97 seconds on Minimal and 1.21 seconds on High. The thinking that lifts tool-task scores costs time.

A tuned cascade can sit inside the bar. Salesforce AI Research measured a streaming cascade at 755 milliseconds to first audio. Its setup used commercial speech-to-text and text-to-speech around an open LLM served on vLLM, with function calling on. The two tests used different setups, so neither number can be subtracted from the other.

Full duplex does help turn-taking, since the model keeps listening while it speaks.

Voice and tone

This is the strongest case for speech-to-speech. The model hears how something was said, including hesitation or frustration, which a transcript drops. It is not solved yet: on the Audio MultiChallenge benchmark (ACL 2026), the best end-to-end spoken dialogue model passed 54.65% of 452 multi-turn conversations, and the authors concluded that models are not yet robust to natural speech.

Cascades win on the voice itself. You can use any text-to-speech model, including ones that clone a voice, such as the open-source Chatterbox. The pattern in How to Cut Voice AI Costs by Mixing Pre-Recorded Audio With TTS, real recordings with the same voice cloned as the fallback, only works on a cascade. Speech-to-speech models offer their provider's voices. GPT-Live-1 does support custom voices, but access goes through OpenAI's sales team.

Cost per minute

Speech-to-speech comes as one price for one model. Gemini 3.1 Flash Live costs $0.005 a minute for audio in and $0.018 a minute for audio out, with any thinking billed as text, on Google's Gemini API pricing page. On OpenAI's pricing page, GPT-Live-1 costs $0.05 a minute, with the text model behind it and its tool use billed on top. gpt-realtime-2.1 is priced per token instead: $32 per million audio tokens in and $64 out.

A cascade has three bills, and you can cut each one on its own, down to open-weight models on your own servers with no per-token fee. Servers still cost money. For cost-sensitive use cases, we choose a cascade.

Names, numbers and fixed wording

When a caller reads out an account number or the agent must say an exact sentence, the text step in a cascade is what lets you check it.

In a cascade, a tool's arguments arrive as text, so you can check an order number's format before the lookup runs. In a speech-to-speech model, a misheard digit shows up only when the agent says something wrong.

Exact wording works the same way. A language model generates its words, so a prompt cannot guarantee a sentence. Exact wording only holds where the platform speaks fixed text, such as the greeting, a transition line between steps or a pre-recorded clip. In Dograh, text transition lines do not play on speech-to-speech models, so a required disclosure between steps belongs on a cascade agent.

Choose per agent, on one orchestration layer

The choice belongs to each agent, so the orchestration layer you build on should run both.

Sources: speech-to-speech figures from the Artificial Analysis leaderboard, October 2026; the cascade's 755 ms from Salesforce AI Research.

Dograh, an open-source, self-hostable voice agent platform, runs speech-to-speech and full-duplex models out of the box: Gemini 3.1 Flash Live, OpenAI Realtime, Azure Realtime, Grok Voice Agent and GPT-Live. You pick the mode for each agent on the Models page. A speech-to-speech agent still uses a separate LLM for variable extraction and QA. Agent handoff, the spoken introduction before a transfer and text transition lines need a cascade agent.

Dograh is the orchestration layer: it runs the call flow, tools and telephony, and the speech and language models are a separate layer you choose. That split is covered in what an orchestration layer in voice AI does.

If you need an open-source speech-to-speech voice agent that runs entirely on your own servers, the cascade is the practical route today. In the Salesforce AI Research paper, Qwen3-Omni's full pipeline took about 146 seconds under plain local Transformers, while its cloud realtime API answered in about 702 milliseconds but cannot be self-hosted. The authors chose a cascade for self-hosted realtime agents.

A hybrid voice agent architecture, in practice, means separate agents for separate calls: a short reminder on speech-to-speech, and the agent that books or updates records on a cascade.

Start with a cascade for anything that touches your systems of record, and try speech-to-speech on the short calls where tone matters more than tools. Judge both on a real phone call.