How the Voice AI Pipeline Works: STT to LLM to TTS, Step by Step

A voice AI pipeline passes each caller turn through three models. Speech-to-text turns the caller's audio into text, a large language model writes the reply, and text-to-speech turns it back into audio. The stages stream into each other, so the reply starts before it is finished. In our experience, replies feel natural when they start within about 800 ms.

Key Takeaways

• Streaming overlaps the stages, so the first words play before the reply is done.
• A word misheard by speech-to-text travels through every later stage.
• Swap each stage on its own, per agent in Dograh, even to a self-hosted model.

This post is part of our guide to voice AI. The guide names the three models inside a voice agent. Here we follow one caller's sentence through them, time each step, and show how each stage is set up in Dograh, an open-source, self-hostable voice agent platform.

Following one sentence through the pipeline

A caller's sentence passes through three models in order, and each one hands its output to the next.

Take a caller who says: "Hi, I need to move my Thursday appointment to Friday morning." Most phone calls still reach the agent as 8 kHz audio, while a browser demo runs at 16 kHz, and that difference hits the first stage hardest.

Speech-to-text turns sound into words

Speech-to-text (STT) writes down the caller's words as the audio arrives. While the caller is still saying "Thursday", it is already sending partial results, a provisional transcript it corrects as more audio comes in.

Voice activity detection (VAD) listens for whether anyone is speaking. Dograh uses a 0.2-second silence setting for it by default. A separate turn-detection step then judges whether the caller has actually finished, so a breath after "Thursday" does not set off a reply. Our post on barge-in, endpointing and turn-taking covers that decision in depth. Once the turn is over, STT hands a final transcript to the next stage.

The language model writes the reply

The large language model (LLM) reads the final transcript, the conversation so far and the instructions for the current step of the call. In Dograh, those instructions come from the prompt of the node the call is on. The number that matters here is time to first token, the wait until the first piece of the reply comes back. A good reply is short: "Sure, I can help with that. Can I have your date of birth so I can find your booking?" The agent confirms who it is talking to before it reads out any appointment details.

If the step needs outside data, such as the caller's bookings, the model calls a tool, which adds time of its own (more on that below).

Text-to-speech turns the reply back into sound

Text-to-speech (TTS) turns the reply into audio. Its key number is time to first byte, the wait before the first sound plays. Most pipelines feed TTS a phrase or a sentence at a time, so "Sure, I can help with that" can start playing while the model is still writing the question about the date of birth.

Some lines can skip TTS entirely. In Dograh, the model can pick a pre-recorded human clip when one fits the moment, which plays with no synthesis wait, and fall back to TTS for everything else.

Why streaming makes the pipeline fast enough

Streaming lets the three stages overlap, so the caller hears the start of a reply long before the whole reply exists.

Salesforce AI Research measured the difference in a 2026 tutorial on a cascade built from Deepgram for STT, Qwen2.5-7B-Instruct served on vLLM, and ElevenLabs for TTS. The paper estimated about 950 ms with the stages run one after another. Streamed into each other, the same pipeline measured 755 ms to first audio, with a best case of 729 ms.

By our math, that saving of roughly 200 ms came from overlap alone. The authors' own conclusion is that realtime speed comes from streaming and pipelining across components more than from any single fast model.

The same paper also tried a self-hosted speech-to-speech model. Under a local Transformers setup it took about 146 seconds to first audio, so the authors chose the cascade. Cascade vs speech-to-speech compares the two designs on speed and tool use.

Open Source Alternative to Vapi / Retell

Self-hosted voice agent platform — no per-minute fees

dograh-hq/dograh

Star on GitHub

Where the time goes in one turn

Each stage takes a slice of the turn, and together the slices have to fit inside a delay the caller will accept.

Twilio publishes median starting targets for a cascaded voice agent at launch, not the best you can reach. Time to first token is how long the model takes to start its reply, and time to first byte is how long the voice takes to start speaking.

Source: Twilio's guide to core latency in AI voice agents, vendor-published targets.

Speech-to-text and the language model take 725 of the 885 ms in the platform target, about four-fifths of the turn by our math. Text-to-speech needs only 100 ms, because it only has to start speaking, not finish the reply.

The three stages add up to 825 ms, which is 60 ms short of the platform turn gap. Twilio's guide does not break that gap down, but it names what fills it: short hops of about 10 ms each between services, plus the buffering and orchestration around them.

The 230 ms between the platform gap and the mouth-to-ear gap is the phone network itself, and no model choice removes it.

These numbers are Twilio's targets. On real calls, in our experience, speech-to-text is usually the slowest stage, more often than the language model. Our speech latency playbook explains why and how to cut it.

In our experience, replies feel natural when they start within about 800 ms, which is tighter than these starting targets. Getting there takes streaming at every stage and services placed close together: the same server for self-hosted open models, or the same cloud region for hosted providers.

Measure your own numbers before trusting any budget. In Dograh, View Trace shows how long each stage took on every turn of a call, so you can find the slow stage from there.

What goes wrong between the stages

Each stage trusts the one before it, so a problem early in the pipeline comes out the other end as a confident mistake.

The language model only ever sees text. If STT hears "Thursday" as "Tuesday", the model will gladly help move the wrong appointment, and TTS will say so in a calm, sure voice. Phone audio makes this more likely, because 8 kHz audio blurs the "s" and "f" sounds that tell "sixty" from "fifty". Dograh sends a custom dictionary of your names and terms to the STT provider, and the prompt can tell the model to read dates and numbers back for the caller to confirm.

Sometimes the language model returns an error or goes quiet, and on a phone call that silence sounds like a dropped line. When you use your own LLM keys, Dograh can move the call to a backup LLM in either case: when the main model returns an error, or when it produces no output within a set time (1.5 seconds by default, adjustable from 0.1 to 10). You can add up to two rules, one for each condition. Dograh's own managed LLM handles its fallbacks itself.

Tool calls add time the speech budget does not cover. A lookup in a database placed next to the agent can fit inside the turn. A call to a third-party API often cannot, and a longer one needs a short spoken filler so the caller does not hear dead air.

When the caller talks over the agent, the later stages have to stop. The voice goes quiet and the half-written reply is dropped. In Dograh, interruptions are set for each step and are on by default, and the agent stops once the caller has said two words.

Catching these failures is the work of the orchestration layer, covered in what an orchestration layer does in voice AI.

Choosing the model for each stage

Each stage of a cascade can be swapped on its own, and who controls that choice shapes your cost and where the audio goes.

Hosted voice platforms bundle the pipeline and run the call audio through their own cloud. You choose from the providers they support and pay a platform fee of around 5 to 7 cents a minute on top of the models. A stage you want to change is only as open as the vendor's list.

Dograh is the orchestration layer: it runs the call flow, tools and telephony, and the speech and language models are a separate layer you choose. You set them for each agent on the Models page, as a named configuration or your organisation's default. A workflow node holds the prompt and tools for its step and never picks its own model.

There are three ways to fill each stage. Dograh's own models cover all of them through one connection. With bring your own key (BYOK), you connect separate accounts, for example Deepgram for STT, OpenAI or Groq for the LLM, and ElevenLabs for TTS. Or you point a stage at a model you host yourself, such as an OpenAI-compatible vLLM or Ollama server for the LLM and a Speaches server for STT and TTS.

Only that last option keeps the call audio on your own servers. BYOK moves the contract to you, and the audio still reaches the provider.

Swapping a provider leaves the call flow alone, and a pre-call data fetch can even change the LLM, its fallback and the TTS voice for a single call.

For one model in place of three, Dograh also runs speech-to-speech models such as Gemini 3.1 Flash Live and OpenAI Realtime, with a separate LLM for data extraction and quality checks. Handing a call to another agent still needs a cascade agent.

Whichever stack you choose, time each stage on a real phone call before launch.