Voice AI Explained: The Complete Glossary and Beginner's Guide

At its simplest, voice AI is software that holds a spoken conversation with a person, usually over a phone or web call. Most voice agents chain three models: speech-to-text hears the caller, a language model writes the reply, and text-to-speech speaks it. An orchestration layer, such as open-source Dograh, runs the call: turn-taking, tools, transfers and the phone line.

Key Takeaways

• Phone calls mostly arrive as 8 kHz audio, while browser demos run at 16 kHz.
• A cascade chains separate speech and language models, while speech-to-speech uses one.
• Callers judge an agent by when it replies and when it stops.

Voice AI comes with a lot of new vocabulary. This guide follows one phone call from the first ring to the reply, then collects the key terms in a glossary at the end.

What voice AI is and what a voice agent does

Voice AI is software that understands speech and answers out loud. A voice agent is voice AI with a job: it holds a full conversation on a phone or web call and can act during it, such as booking an appointment or looking up an order.

Every voice agent is built from two groups of parts. The AI models turn the caller's speech into text, write a reply and turn that reply back into speech. The orchestration layer runs the call around them. It decides when the agent should talk and when it should stop, which tool to use and when to hand the caller to a person. In Dograh, you set up these jobs in a visual workflow builder, one step of the call at a time.

How a voice agent turns speech into a reply

Most voice agents work as a relay of three models, each passing its output to the next.

Speech-to-text (STT), also called automatic speech recognition (ASR), turns the caller's words into text as they speak. A large language model (LLM) reads that text, with its instructions for the current step, and writes the reply. Text-to-speech (TTS) turns the reply into audio.

This chain is called a cascade. Its biggest strength is that you can swap each model on its own without rebuilding the rest. The weakness is time. In our experience, a reply feels natural when the full round trip stays under about 800 milliseconds, and speech-to-text, more often than the language model, is the slowest stage. Our post on speech latency breaks that budget down. To save time, the stages stream, so the caller hears the first words of a reply before the rest is written.

How the voice AI pipeline works follows one caller's sentence through all three stages and shows where the time goes.

Cascade or speech-to-speech: two ways to build the model layer

The alternative to a cascade is a single speech-to-speech model that takes audio in and gives audio out.

Speech-to-speech (S2S) models, also called realtime models, skip the text steps. Some run full duplex, which means they can listen and talk at the same moment. With no hand-offs between models, replies can come back faster, and the model can hear how the caller sounds as well as what they said.

Our view is that cascades are better at reasoning and tool calling, and they support voice cloning, so they fit complex use cases better. Speech-to-speech models can fail on calls with complex tool calling, such as checking an order in one system and then updating a booking in another.

Dograh, an open-source, self-hostable voice agent platform, runs speech-to-speech and full-duplex models out of the box, including Gemini 3.1 Flash Live, OpenAI Realtime, Azure Realtime, Grok Voice Agent and GPT-Live. A few features still need a cascade agent, such as handing the call to another agent.

A sensible way to decide is per agent: a short, simple conversation can run on speech-to-speech, while an agent that looks things up and acts on them stays on a cascade. Cascade vs speech-to-speech compares the two in depth, and GPT-Live-1 vs Gemini Live vs Grok Voice API looks at the realtime models on phone calls.

The orchestration layer that runs the call

Models produce words and audio, and the orchestration layer runs the call around them.

Picture a voice agent as two layers. The inference layer is the AI models. The orchestration layer is everything else, from the phone line and the call flow to tool calls and call records. On a real call, it answers the phone and keeps track of which step the conversation is on. It decides when the caller has finished speaking and when the agent should look something up, such as an order. If the caller asks for a person, it hands the call over, and when the call ends, it keeps a record of what was said.

Dograh is the orchestration layer, and you choose the models for each agent on the Models page. You can use Dograh's own models, any provider with your own API keys, or open-weight models you host yourself. Only that last option keeps all call audio on your own servers. Bring your own key (BYOK) moves the contract to you, while the audio still travels to the provider. Our guide to running voice AI on local, self-hosted models covers the fully local setup.

A good orchestration layer also does jobs no model does on its own. With your own LLM keys, Dograh can switch to a backup language model if the main one returns an error or stays silent past a time limit, 1.5 seconds by default. What is an orchestration layer in voice AI goes through these jobs and how to judge a platform on them.

Open Source Alternative to Vapi / Retell

Self-hosted voice agent platform — no per-minute fees

dograh-hq/dograh

Star on GitHub

Turn-taking: when the agent talks and when it stops

Much of what makes a voice agent feel natural or awkward comes down to turn-taking, the rules for who speaks when.

Voice activity detection (VAD) is a small model that decides, many times a second, whether someone is speaking. VAD cannot tell whether the caller has finished a thought. That decision is endpointing. Reply too early and the agent cuts people off. Wait too long and the silence feels like a dropped call. Dograh judges the end of a turn from the transcript by default, or with a Smart Turn model that listens to the audio.

Barge-in is the opposite case: the caller talks while the agent is speaking, and the agent stops to listen. The hard part is telling a real interruption from a backchannel, the short "mm-hm" a listener says to show they are following. In Dograh, interruptions are set for each step and are on by default. Upset callers interrupt more and pause in odd places, so good voice design gives them room to finish.

Barge-in, endpointing and turn-taking covers each of these decisions and the settings that control them.

How a call reaches the agent

A caller reaches a voice agent in one of two ways: through the phone network, or through a browser.

Phone calls travel over the public switched telephone network (PSTN). A telephony provider, such as Twilio or Vonage, connects that network to your agent and gives it phone numbers. For production, a SIP trunk is usually the most reliable route. SIP (Session Initiation Protocol) sets up internet phone calls, and a SIP trunk links your system to the phone network. Browser calls use WebRTC, the standard for live audio in a web page. Most demos and builder test calls run this way.

The route changes what the agent can do. Transferring a live call to a person needs a real phone call, and in Dograh it works on Twilio. In Gartner's 2026 customer survey, 87% of customers said companies using generative AI for service must offer a way to reach a human agent.

Why a voice agent sounds different on a phone call

Most phone calls still reach the agent as 8 kHz audio, while a browser demo runs at 16 kHz. That gap changes both what the agent hears and how its voice sounds to the caller.

What 8 kHz audio leaves out

Sample rate is how many times per second the audio is measured. The standard phone codec, G.711, samples speech 8,000 times a second, which carries sound from roughly 300 to 3,400 Hz. Wideband audio sampled at 16 kHz carries roughly 50 to 7,000 Hz. The high frequencies that narrowband cuts off carry sounds like "s" and "f", which is why they are easy to mix up on a phone line.

Dograh's own phone connections show how common narrowband still is. Twilio, Plivo and Vobiz send the agent 8 kHz audio, while Vonage sends 16 kHz, the same rate as a web call. Mobile calls over VoLTE (voice over 4G LTE) or 5G can carry wideband audio, but fall back to 8 kHz on a weak signal.

What it does to recognition and the voice

Narrowband audio makes speech-to-text work harder. In our experience, the word error rate on a phone call can be up to three times higher than on a web call. The gap depends on the model and the phone path, so measure it on your own line.

Word error rate is the share of words a speech-to-text model hears wrong. In WildASR's 2026 tests, GPT-4o Transcribe barely changed when the audio was turned into G.711 phone audio. Its word error rate went from 2.8% to 2.9%. Qwen2-Audio jumped from 5.8% to 48.3%, which means it got about half of all words wrong once the audio sounded like a phone call. Some speech models handle phone audio well and some fall apart, and a browser demo will not tell you which one you have. Test your chosen model on a real phone line. A 2026 study from the Indian Institute of Science found that compression in older GSM (2G) mobile calls did most of the damage. So how the call is squeezed, and the route it takes through the phone network, matter as much as the sample rate.

The agent's voice suffers too, often coming out flat over a phone line. In a 2026 benchmark by Caller Digital, a voice AI vendor, a panel of Hindi listeners rated a premium neural voice on a scale of 1 to 5, where a higher score means the voice sounds more natural. It scored 4.4 out of 5 before phone compression and 3.6 after G.711. A drop of almost a full point on a five-point scale is easy to hear. The same voice that sounds smooth in a browser demo sounds clearly worse to a caller on the phone, so listen to your chosen voice over a real call before launch.

How to plan for phone audio

Pick a speech-to-text model that holds up on narrowband audio, and give it a custom dictionary of the names and terms your callers say. Then test the whole agent on a real phone line before launch, because a browser test call will flatter it every time. Testing Voice Agents Before You Go Live puts that phone test in order with the other checks.

Voice agents vs IVR: what changed

An interactive voice response (IVR) system routes callers through a fixed menu, while a voice agent understands what the caller says and acts on it.

An IVR plays a menu and listens for keypad tones, called DTMF (dual-tone multi-frequency) signals. Every path is built in advance, so a caller whose problem is not on the menu presses 0 for a person. A voice agent lets the caller explain in their own words, then acts through tools. Where exact wording matters, it can still speak fixed text, such as the greeting or a pre-recorded clip.

An IVR can still be the right tool for very simple routing. AI voice agent vs traditional IVR covers when to keep an IVR and how to move off one.

Where a voice agent can run

Once you know the parts, the main choice is who runs the orchestration layer, and where.

Hosted platforms run it on their own cloud. Most charge a platform fee of around 5 to 7 cents a minute, with the models billed separately, and the call audio stays with the vendor. Open-source frameworks give you code to build an orchestration layer yourself, along with everything around it.

An open-source platform sits between the two. Dograh comes with a visual workflow builder, telephony, call transfer and quality assurance (QA), and you can self-host it with no per-minute platform fee, though servers, models and phone lines still cost money. Pair it with self-hosted open-weight models and no call audio leaves your infrastructure.

Whichever route you take, judge it on a real phone call, with the words your own callers use.

Each part of voice AI above has its own deep dive:

• How the Voice AI Pipeline Works (STT to LLM to TTS)
• Cascade (STT LLM TTS) vs Speech-to-Speech: Tradeoffs Explained
• What Is an Orchestration Layer in Voice AI
• Barge-In, Endpointing, and Turn-Taking: Voice UX Fundamentals
• AI Voice Agent vs Traditional IVR: What Changed

Voice AI glossary

The 20 terms below are grouped by where they show up in a call. Where a deep dive covers a term in full, the entry links to it.

Models and the platform

• Voice agent: Software that holds a spoken conversation on a phone or web call and can take actions during it.
• Speech-to-text (STT) / automatic speech recognition (ASR): The model that turns the caller's speech into text in real time. It sends out words while the caller is still speaking, so the next model can start early. Any word it mishears is passed on to every step after it, so the reply can only be as good as the transcript. More in how the voice AI pipeline works.
• Large language model (LLM): The brain of the agent. It reads the transcript and the step's instructions, then decides what to say next and when to use a tool, such as looking up an order.
• Text-to-speech (TTS): The model that turns the reply text into audio.
• Cascade pipeline: Speech-to-text, a language model and text-to-speech chained as separate, swappable models. Compared in cascade vs speech-to-speech.
• Speech-to-speech (S2S) / realtime model: One model that hears the caller's audio and answers with audio directly, skipping the written text in between. It is better at picking up tone of voice, but weaker than a cascade on calls with complex tool use, and not automatically faster.
• Orchestration layer: The software that runs the call around the models. The models themselves are the inference layer. Dograh is an open-source orchestration layer you can self-host. More in what is an orchestration layer in voice AI.

Timing and turn-taking

• Latency: The delay between the caller finishing and the agent starting to speak. In our experience, under about 800 ms feels natural.
• Voice activity detection (VAD): A small model that decides whether someone is speaking right now.
• Endpointing: Deciding that the caller has finished speaking. More in barge-in, endpointing and turn-taking.
• Turn detection: The wider rules for who holds the floor, using the words as well as silence.
• Barge-in: The caller talking over the agent, and the agent stopping to listen.
• Backchannel: A short "mm-hm" that shows attention without taking the turn.

Phone lines and audio

• Sample rate: How many times per second audio is measured. Most phone calls arrive at 8 kHz.
• Narrowband and wideband: Narrowband carries about 300 to 3,400 Hz. Wideband, often sold as HD voice, carries about 50 to 7,000 Hz. The range is how much of the voice's sound gets through, from low pitches to high ones. Narrowband cuts off the high sounds, so "s" and "f" blur together. Wideband keeps more of them, so speech sounds clearer to the caller and to speech-to-text.
• Codec: The method that compresses audio for transport, such as G.711 on phone lines.
• Word error rate (WER): The share of words speech-to-text gets wrong. Lower is better.
• SIP trunk: A connection that links your system to the public phone network using SIP.
• WebRTC: The browser standard for live audio and video. It is how a web page sends your voice over the internet, so a demo call to a voice agent from a browser usually runs on it, at 16 kHz.
• IVR and DTMF: The "press 1 for billing" menu system, and the keypad tones it listens for. More in AI voice agent vs traditional IVR.