Barge-In, Endpointing, and Turn-Taking: Voice UX Fundamentals

Turn-taking is how a voice agent decides who speaks and when. Endpointing judges that the caller has finished, so the agent can reply. Barge-in is the caller cutting in while the agent talks, and the agent stopping to listen. Good voice UX replies quickly once the caller is really done and lets a short "mm-hm" pass.

Key Takeaways

• VAD silence (0.2 s) and the end-of-turn wait (2 s) are separate settings.
• A two-word minimum keeps a lone "yeah" from stopping the agent.
• Turn interruptions off for disclosures and read-backs, on where callers push back.

This post is part of our guide to Voice AI Explained. The guide covers turn-taking in a few paragraphs. Here we take each decision behind it in turn, what goes wrong when it misfires, and the settings that control it in Dograh, an open-source, self-hostable voice agent platform.

What the agent decides on every turn

Every time a caller speaks, the agent makes two quick decisions, and callers notice when either goes wrong. The first is endpointing. Voice activity detection (VAD) tells the agent whether sound that looks like speech is happening right now. Endpointing goes further and judges whether the caller has finished their thought or has only paused. The second comes up while the agent is talking: if the caller makes a sound, it has to tell a real interruption from a backchannel, the short "mm-hm" a listener says to show they are following.

Both decisions sit in the orchestration layer, between speech-to-text and the language model. How the voice AI pipeline works shows where they fall inside a single turn.

People set the timing bar. In a 2025 study of US English phone calls from the Fisher corpus, Cavalcanti and Skantze measured an average floor-transfer offset of 0.300 seconds. That is the gap between one speaker stopping and the other starting. An agent running three models cannot match it end to end, and it does not need to. In our experience, a reply under about 800 ms feels natural, and our post on speech latency breaks that budget down. The turn decisions just must not eat most of it.

In our view, good turn-taking starts with low latency. On top of that, the agent has to handle backchannels well and cope with callers whose emotions change the way they speak.

Endpointing: knowing the caller is done

Getting endpointing wrong costs you in either direction. End the turn too early and the agent talks over someone who was mid-sentence. In AssemblyAI's 2026 Voice Agent Report, a vendor survey of 455 builders, 47.5% said their agent interrupts users mid-sentence. End it too late and the caller hears dead air and wonders if the line dropped.

Two timers are easy to mix up here. VAD silence is how long the speech detector waits before it marks speech as stopped. In Dograh, voice activity detection does this with a stop time of 0.2 seconds. The end-of-turn wait is a separate and longer decision about the whole thought. Dograh's Smart Turn option waits up to 2 seconds by default when a turn sounds unfinished, and some speech-to-text services allow 5 seconds of silence before they force a turn to end.

A plain silence timer only waits for a pause, so "My account number is..." followed by a breath looks like a finished turn. Semantic turn detection also reads the words, so a sentence that stops on "is" counts as unfinished. Audio-based models listen to how something is said, such as pitch and timing.

Listening can beat reading here. In a September 2026 study of end-of-turn detection, a detector that used only the sound of the voice scored 0.93 out of 1 on F1, a standard accuracy score. It wrongly called a turn finished in 7.8% of cases, and half of its decisions came within 400 ms. When the researchers added the transcript, it cut callers off more often, with no gain in accuracy.

Dograh offers two turn detection strategies in the workflow settings. Transcription-based, the default, ends the turn from the speech-to-text transcript and suits short answers of one or two words, like "yes" or "Tuesday". Smart Turn Analyzer runs the open-source Smart Turn model on the audio itself and suits longer answers with natural pauses. Daily's Smart Turn v3 is an 8 MB model that runs in about 12.6 ms on a server CPU and covers 23 languages. When it judges a turn unfinished, Dograh waits up to the Incomplete Turn Timeout, 2 seconds by default, before replying anyway.

Open Source Alternative to Vapi / Retell

Self-hosted voice agent platform — no per-minute fees

dograh-hq/dograh

Star on GitHub

Barge-in, and the sounds that should not stop the agent

Barge-in should stop the agent for a real interruption and let everything else pass. A missed interruption is the caller saying "no, that's the wrong address" while the agent keeps reading. A false barge-in is the agent stopping dead for a cough in the room or a quiet "mm-hm". The second is harder, because backchannels are normal listening. An agent that halts on each one sounds jumpy and keeps restarting its sentences.

Speech-to-speech models show how hard this tradeoff is. Full-Duplex-Bench v1.5 tested earlier versions of Amazon Nova Sonic, Gemini and GPT-4o (amazon.nova-sonic-v1:0, gemini-2.0-flash-live-001 and gpt-4o-realtime-preview-2024-12-17). When the user only backchanneled, the Nova Sonic version carried on talking 98% of the time and the Gemini version 93% of the time, against 70% for the GPT-4o version. When the user really interrupted, the GPT-4o version stopped in 0.23 seconds, while the Gemini and Nova Sonic versions took 2.20 and 2.25 seconds. The models that ignored backchannels best were the slowest to give way. On a speech-to-speech agent, the model provider makes much of this decision, one of the differences covered in cascade vs speech-to-speech.

On a cascade agent, Dograh splits barge-in into two settings. Allow Interruption is set on each step of the workflow and is on by default. When you turn it off, the agent does not listen while that step's line plays, so nothing the caller says over it gets through. Interruption Strategy, set once for the workflow, decides what counts as an interruption on steps where it is on. "Voice activity" stops the agent as soon as speech is detected. "Minimum words", the default, waits for a set number of transcribed words, 2 unless you change it. That is the backchannel guard. A lone "yeah" does not stop the agent, while "wait, no" does.

Choose per step. Turn interruptions off where the caller must hear every word, such as a required disclosure or a read-back of details to confirm them. Leave them on where callers push back or ask things, such as objection handling and open questions. Keep the lines on non-interruptible steps short, since the caller cannot cut in.

Backchannels also run the other way. During a long caller turn, like a complaint, a short "I see" from the agent tells the caller the line is still alive. Keep it brief and put it in a natural pause, never over the caller.

Callers who are upset or unsure

Emotion changes how people take turns, so timing rules that suit a calm caller can fail a stressed one. An upset caller talks faster and interrupts more, often repeating the same point. A hesitant caller does the opposite, pausing mid-sentence while they find a policy number or work out how to describe what happened. One silence threshold cannot serve both. Set it short and it cuts off the hesitant caller. Set it long and the frustrated caller waits after every sentence.

Good voice UX plans for both. If callers often describe a problem in their own words, such as an incident or a complaint, an audio-based end-of-turn model with a longer incomplete-turn timeout gives them room to pause. It can also hear a trailing, unsure tone that a transcript misses. Where a frustrated caller is likely to push back, allow interruptions so the agent stops when they cut in, and keep the agent's lines short so there is less to talk over.

Then check what happened. Dograh's QA scores each step of a call separately, with its own sentiment and summary, so steps where callers got cut off or grew frustrated show up across many calls.

Why turn-taking belongs in the orchestration layer

Turn rules are a call-flow decision as much as a model decision, so they belong in the layer that runs the call. Many speech-to-text services and realtime models now build end-of-turn logic into the model. That ties your agent's turn behaviour to one vendor, and the settings move every time you change models. A step that must not be interrupted is a fact about your call flow. It stays the same when you swap the speech model.

Dograh is the orchestration layer: it runs the call flow, tools and telephony, and the speech and language models are a separate layer you choose. Turn detection and interruption rules live in the workflow, so you can change the speech-to-text model without retuning them and set a different interruption rule on each step. Because Dograh is open source, the turn logic is code you can read when a call goes wrong. Our post on what an orchestration layer does covers its other jobs.

A sensible tuning order starts from the defaults, where interruptions are off and the Minimum words guard sits at 2. Turn interruptions on for the steps where callers push back. If callers often give long answers, switch the workflow to Smart Turn. Then listen to real phone calls, since phone audio is harder on these decisions than a browser test.