Dograh

Language Switching Mid-Call: How Voice Agents Handle It

Language Switching Mid-Call: How Voice Agents Handle It
TechnicalNovember 19, 2026· Updated November 20, 2026·9 min read

Language Switching Mid-Call: How Voice Agents Handle It

Abhishek Kumar
Abhishek Kumar·Co-founder, Dograh AI

Co-founder of Dograh, building the future of open-source voice AI agents. Own your voice AI stack.

A voice agent follows a language switch in three layers. Speech-to-text must recognize the new language, the language model must answer in it, and the voice must speak it. When the switch is unclear, ask a short question and confirm first. Teams can build this in Dograh, the open-source AI voice agent platform, self-hosted or on Dograh Cloud.

Key Takeaways

  • Confirm the new language with one short question before switching the whole call.
  • Speech-to-speech models switch on their own, while a cascade needs auto-detecting speech-to-text.
  • One voice rarely sounds native in two languages, so plan a voice per language.

A caller who starts in English and moves to Spanish halfway through tests every layer of a voice agent at once. We cover what each layer has to do, how speech-to-speech models differ and why we confirm before switching. Then we talk about how to set the language in Dograh.

What changes when a caller switches language

Three things have to change for the agent to follow a caller into a new language, and in a cascade agent each one is a separate model. Speech-to-text has to recognize words in the new language. The language model has to reply in that language. The voice has to pronounce the reply like a native speaker would.

A caller starts in English and switches to Spanish. Inside the agent, drawn as a cascade of three separate models, an ear for speech to text and a page where the language model writes the reply each have their own dial turned from English to Spanish, and at the voice an English voice is set aside for a Spanish one.
A recognizer locked to English turns Spanish into English-sounding nonsense, and the model then answers nonsense. A voice built for English reads a correct Spanish reply with an English accent, which sounds worse than a plain English answer.

If any one of them stays behind, the call breaks in a way the caller notices at once. A recognizer locked to English turns Spanish into English-sounding nonsense, and the model then answers nonsense. A model that reads Spanish but was told to answer in English replies in the wrong language. A voice built for English reads a correct Spanish reply with an English accent, which sounds worse than a plain English answer.

A speech-to-speech model folds all three layers into one, which changes the picture, and we come to it below. Most phone agents today are cascades, so we start with the ear.

Speech-to-text has to hear the new language

Speech-to-text models handle language in one of three ways, and the choice decides how fast an agent can follow a switch. Some take one fixed language per session and never move. Some detect the language for each utterance, so a switch shows up on the next sentence. A few tag the language word by word, which is what mixed speech needs.

Soniox's language identification tags every token with a language and may revise early tags as more audio arrives, and Soniox recommends passing language hints when you know the likely set. Deepgram's Flux multilingual model handles a switch inside a sentence across its ten supported languages. Sarvam's saaras models detect the language when it is set to unknown and have a codemix mode for Indian languages mixed with English.

None of this is solved. The WildASR study from Boson AI notes that recognizers often force mixed speech into one script or drop the second language. On English mixed with Korean, the mixed error rate was 9.4 for Gemini 3 Pro and 56.4 for Deepgram's older Nova 2 model. The authors also warn that tests on synthetic audio can underestimate failures, so test with recordings of your own callers.

The language model answers in whatever it reads

The language model has no ear, so it answers in whatever language the transcript shows, unless the prompt says otherwise. That makes the prompt the place where the switching rule lives. Tell the model which languages it may use, when to change and what to do when it is unsure.

Single words are the common trap. A Spanish speaker who says "okay" or names an English product has not switched, and a model told to "reply in the caller's language" can flip on one word. Write the rule so the model follows the language of the whole sentence and keeps names, product terms and account numbers as they are.

Numbers need their own line in the prompt. A date or an amount said in one language and read back in another is easy to get wrong, so the agent should read it back in the language the caller used and wait for a yes.

Dograh

Open Source Alternative to Vapi / Retell

Self-hosted voice agent platform — no per-minute fees

dograh-hq/dograh

Star on GitHub

One voice rarely sounds native in two languages

The voice is the layer most often left behind. Many text-to-speech voices are built for one language, and a voice that can read several often keeps the accent of the one it was trained on. A reply that is perfect on paper then sounds foreign on the line.

The safe pattern is one voice per language, picked by listening to each on a real phone call. Switching voices mid-call is noticeable, which is why some teams prefer a single multilingual voice. That works only when you have heard the voice in every language the agent will use. Do not count on one voice to speak two languages inside a single sentence, and test that case on a call before you rely on it. Our post on multilingual TTS for voice agents compares voices for Hindi, Spanish and Arabic.

Speech-to-speech models switch on their own

A speech-to-speech model listens, reasons and speaks in one model, so it can follow a caller into a new language without any setting changing. Google's Gemini Live models with native audio pick the spoken language on their own and switch when the caller does. The language code cannot be set for them, and Google says to restrict languages through the system instructions instead.

That freedom has a cost. The model may switch on a guess, so the same confirm-first rule belongs in its instructions. Speech-to-speech models also tend to be weaker at complex tool calls than a cascade with a strong text model, so an agent that books, looks up and writes records may still be better built as a cascade.

Ask, confirm, then switch

Our standing view is that an agent should never carry on confidently in a language it may have guessed wrong. When the language is unclear or mixed, it asks one short question that offers the likely choices, such as "Would you like to continue in Spanish?", and switches only after the caller says yes. A default language can carry that question, but it is not a substitute for real support in the caller's language.

If the agent cannot serve the language reliably, the right next step is a person. Transfer the call to a qualified person where your phone setup allows it. Where it does not, collect only the minimum needed and arrange a callback in the caller's preferred language.

Mixing and switching are different problems. Many Hindi speakers move between Hindi and English inside one sentence all day, and an agent that asks "Hindi or English?" every few seconds is worse than one that just follows. For those lines, choose a recognizer built for mixed speech and keep one reply style. Our posts on Sarvam AI for Indian-language voice agents and Hindi AI voice agents for collections cover the script and wording choices.

Join the Dograh Community

Dograh is an OSS alternative to Vapi. Join our Slack community for queries, releases, best practices & community interactions.

Three ways to build it in a cascade

The first design is one multilingual stack, which pairs an auto-detecting recognizer and a prompt that names the allowed languages with a voice per language or a tested multilingual voice. It is the simplest to run and suits lines where most callers stay in one language.

The second is a router. A short first agent finds the caller's language, confirms it, then hands the call to an agent built for that language, with its own prompt and voice. Each language agent can be tested on its own, which matters when one language carries most of the calls. Our guide to multilingual voice agents covers routing and staffing for this design.

The third is fixed agents per number, where each published number or menu choice leads to one language. It never switches mid-call, so it cannot guess wrong. It suits teams that already split callers by line.

Setting the language in Dograh

In Dograh the language is a speech-to-text setting on each agent's model configuration, and what it does depends on the provider. Choosing Multilingual (Auto-detect) with Dograh's own model or Deepgram lets the recognizer follow the caller. Soniox auto-detects when no language hint is set, and for Sarvam pick saaras:v3 in the Model field and set the language to unknown. Gladia runs one fixed language per agent. For the router design, the Transfer To Agent tool hands the live call to a language agent, and when both agents use the same speech-to-text provider and model, the new agent's language setting is applied to the call. The Greeting Text plays before the agent can hear the caller, so write it in both languages, for example "Hola, hello".

Where the platform runs decides where the call audio and transcripts in every language are stored. Running Dograh yourself means no per-minute platform fee. You still pay for servers, model keys and phone minutes, and your team looks after the database and upgrades. Support comes from the open-source community, and Enterprise adds custom SLAs. If you would rather not host it, Dograh Cloud runs it for a 1 cent a minute platform fee.

Start with the one extra language most of your callers use, record a week of real calls and read where the agent switched before you add a third.

Glossary

Mixed speech
Speech that moves between two languages inside one conversation or one sentence, as many bilingual callers do. A recognizer has to tag each word's language to follow it.
Language identification
Working out which language each part of the speech is in. Soniox tags every token with a language and may revise early tags as more audio arrives.
Mixed error rate
An error rate for speech that mixes two languages, counted over the combined stream of words from both. Lower is better.
Language hint
A short list of likely languages passed to a speech-to-text model. It narrows the guess without locking the model to one language.

Frequently Asked Questions

Get started with Dograh

Build, deploy, and scale AI agents with Dograh. Join the community of developers building the future.