What Is an Orchestration Layer in Voice AI? Every Job It Runs

An orchestration layer is the software that runs a voice call around the AI models. It connects the phone line, decides when the caller has finished, calls tools, transfers calls, handles voicemail and records data. Dograh is an open-source one you can self-host. The models themselves, cascade or speech-to-speech, form a separate inference layer you choose.

Key Takeaways

• A custom dictionary sends your brand terms to speech-to-text as keyterms.
• Transfers start cold, and a spoken introduction on Twilio makes them warm.
• Speech-to-speech support differs by platform, so check it before you commit.

This post is part of our guide to Voice AI Explained: The Complete Glossary and Beginner's Guide. Its orchestration section names a few of the layer's jobs in passing. Here we go through each one, and how to judge a platform on it.

Orchestration layer vs inference layer: where the line sits

The models produce words and audio, and the orchestration layer decides what happens with them on a live call.

On each turn, the orchestration layer streams the caller's audio to speech-to-text, decides when the caller has finished, and hands the transcript to the language model with the instructions for the current step. It sends the reply to text-to-speech and stops that voice if the caller cuts in. The models never see the phone line or the call flow.

Dograh, an open-source, self-hostable voice agent platform, sits on the orchestration side of that line. Dograh is the orchestration layer: it runs the call flow, tools and telephony, and the speech and language models are a separate layer you choose. You pick them for each agent on the Models page, from Dograh's own models, any provider with your own API keys (BYOK), or open-weight models you host yourself.

The split lets you swap a model without rebuilding the agent. It also decides where audio goes: with BYOK the contract moves to you, while the audio still reaches the provider. Only self-hosted open-weight models keep every call on your servers, as our guide to running voice AI on local, self-hosted models shows. For the inside of the inference layer, see how the voice AI pipeline works.

How the agent hears the right words and knows the caller is done

Two listening jobs sit in the orchestration layer: telling speech-to-text which words to expect, and deciding when the caller's turn is over.

Speech-to-text models are trained on general speech, so they slip on brand names and industry terms such as KYC (know your customer) or AUM (assets under management). In Dograh's dictionary, you add the words your agent should actively listen for, and Dograh sends them to the speech-to-text provider as keyterms on each call. When the audio is unclear, the model leans toward your spellings, which lowers the word error rate on exactly those terms.

Keep the list short. AssemblyAI's keyterms guide warns that long lists, or common words, can cause overcorrections, where the model "hears" a keyterm nobody said. Start with the terms a mishearing would break, and add more only when transcripts show a real miss.

The second job is deciding that the caller is done, and Dograh offers two strategies. The default is transcription-based: the turn ends when the transcript shows a finished statement, which suits short answers like "yes". The other is the Smart Turn Analyzer, an open model that judges from the audio whether the turn sounds complete. If the caller seems mid-thought, it waits, up to an Incomplete Turn Timeout of 2 seconds by default.

Daily, which maintains the model, reports that Smart Turn v3, the version Dograh runs, reached 94.31% English accuracy on its own test set, from a model of only 8 MB. Barge-in, endpointing and turn-taking covers the trade-offs in depth.

Open Source Alternative to Vapi / Retell

Self-hosted voice agent platform — no per-minute fees

dograh-hq/dograh

Star on GitHub

Acting on the call: data, tools, transfers and voicemail

A voice agent earns its place by what it does during and after the call, and those actions run through the orchestration layer.

Variable extraction makes an agent useful to the rest of your business. In Dograh you switch it on for a step and name the typed fields you want, such as a callback time or whether the caller agreed to a renewal. Extraction is an AI step, so it records each answer as the caller stated it, with the transcript and recording as the exact record. A Webhook node sends the values to your CRM.

During the call, the agent can call your HTTP APIs, use tools on any Model Context Protocol (MCP) server, and answer questions from a knowledge base of your own documents.

Cold and warm transfers to a person

Dograh transfers cold by default: the caller is connected to the person with no summary. On Twilio you can turn on a transfer introduction, a short spoken line both parties hear before they are connected, which makes it a warm transfer. It is off by default and may be skipped. The destination can be fixed, or picked per call by your own service.

Transfer to a person works on Twilio. Test it on a real phone call, since browser test calls cannot transfer.

Voicemail and call screening

Many pickups are not people. In CALLR's 2026 marketplace benchmark, 14.0% of 17,202 outbound callbacks reached voicemail, and 19.9% of 407,573 inbound calls ended on voicemail yet showed as answered on dashboards. CALLR is a telephony vendor, and the data comes from property and home-improvement marketplaces.

Dograh tells voicemail, a call screener and a person apart. On voicemail it leaves the message you set and marks the call voicemail_detected, so a campaign can retry later. On a call screener, the phone feature that asks callers who they are, it speaks your screening message and waits for the person.

Building and running many agents

Past a single call, the orchestration layer is where you build agents and run them at volume.

Dograh can generate a starting agent from a plain-language description of your use case. You then shape it in the visual workflow builder, where each step holds its own prompt, so you change one step without touching the rest. Test calls and call traces sit alongside, so you can try a change and see where a call went wrong.

Dograh also runs as an MCP server, so Claude Code or Cursor can list your agents, read their definitions and pull call transcripts from your editor.

Transfer to Agent chains agents: it hands the call to another Dograh agent with a labelled transcript, so a reception agent can pass a billing question on without the caller repeating themselves. Campaigns run outbound calls from a CSV list, with limits on concurrent calls and calling hours, retries, and pause and resume.

Switching models, including speech-to-speech

Because the models sit in their own layer, a good orchestration layer lets you change them per agent and fall back when one fails.

In Dograh, you choose models on the Models page. Your organisation sets a default, and any agent can use its own models instead. When you start a call through the API, you can also swap the language model or the voice for that one call.

LLM fallback protects the call when the large language model (LLM) fails. If the main language model returns an error, or gives no output within a set time (1.5 seconds by default, adjustable from 0.1 to 10), the call switches to a backup model. You can set up to two rules, one per condition. It applies when you bring your own LLM keys. Dograh's managed models handle their own fallbacks.

Dograh runs speech-to-speech and full-duplex models out of the box, including Gemini 3.1 Flash Live, OpenAI Realtime, Azure Realtime, Grok Voice Agent and GPT-Live, which runs full duplex. A speech-to-speech agent keeps the call flow and telephony, with a separate language model for variable extraction and quality review (QA). A few features still need a cascade agent: handing the call to another agent, the spoken introduction before a transfer, and the text transition lines between steps.

This is a real difference between platforms. In our experience, Vapi and Retell do not support speech-to-speech or full-duplex models today. Cascades stay better at complex tool calling, so choose per agent. Cascade vs speech-to-speech covers which calls suit each.

How to judge a voice AI orchestration platform

Judge an orchestration layer by the jobs in this post, tested on a real phone call, because an easy builder can still leave the hard jobs to you.

Open-source frameworks give you the parts of the layer as code, and production use needs a lot of plumbing on top, from voicemail handling and transfers to campaigns and a builder. Hosted platforms run the layer for you for a platform fee of around 5 to 7 cents a minute, with models billed separately, and what they cover differs.

Put the voice AI orchestration tools you are considering through one phone test. Check the transcript for your brand names. Read out a long account number slowly and see whether the agent waits. Let one call go to voicemail, transfer another over your production line, and check that the extracted fields arrive.

Dograh puts these jobs into one open-source platform you can self-host, and self-hosted voice agent orchestration has no per-minute platform fee. Servers, models and phone lines still cost money. Pair it with open-weight models you host yourself, and no call audio leaves your infrastructure.