Record the lines your agent repeats on every call, then let the orchestrator play the matching clip and synthesise only what is left. On a scripted flow, roughly two thirds of spoken audio can come from that library, so the text-to-speech line falls to about a third. Clone the same speaker for the fallback voice.
Key Takeaways
- How repetitive your script is decides the size of the saving.
- Text to speech is one of four line items on a call.
- Self-hosted open-weight voices are the ones listeners flag as robotic.
This post is part of our guide to Running Voice AI on Local, Self-Hosted Models. This one goes after a single line on the bill, and the quality problem sitting next to it. We build Dograh, an open-source orchestrator for voice agents, and this clip-or-synthesise decision runs on every turn.
What the hybrid setup does on a call
Your agent plays a real human recording whenever one fits the moment, and synthesises only the lines nothing in the library covers.
Most of a scripted call is not improvised. The greeting never changes, the compliance disclosure is fixed word for word because legal signed off on that wording, and hold messages barely move between calls.
Record those with a real person. At runtime the orchestrator checks, turn by turn, whether a clip covers what the agent needs to say next. When one does it plays the file. When nothing does, text to speech generates the line in a clone of that same speaker.
The clone matters. Cutting between a warm human recording and a different synthetic voice is worse than synthesising everything, because the seam is audible.
That decision sits in the orchestration layer. Dograh holds the workflow, the turn-taking, the tool calls and the call lifecycle, and it is not a model. The inference underneath is a separate choice between Dograh's own models, a third-party provider through your own key, and open weights on hardware you own. Only the last one keeps the audio inside your network, because your own key moves the commercial contract and leaves the audio path where it was.
Playing a cached file also takes synthesis out of the turn, which matters when the whole voice-to-voice budget is under 800 milliseconds.
Coverage is what moves the bill
What actually moves your bill is coverage, rather than the price of any single clip. Generating a typical ten second line with text to speech costs about a fifth of a cent on Amazon Polly's standard Neural pricing. Playing the same ten seconds from a pre-recorded file costs a tiny fraction of that, well under a hundredth of a cent on standard cloud file-serving rates.
That is hundreds of times cheaper per clip. Treat that number carefully, though. It only proves the library itself is cheap to run. It does not tell you how much you will actually save, because that depends entirely on how much of the call comes from the library in the first place.
Whatever fraction of the call's audio comes from the library, your text-to-speech spend drops by roughly that same fraction. On a well-built scripted flow, we typically see about two thirds of the audio coming from the library, cutting the text-to-speech cost to about a third of where it started. That is a claim about how repetitive your script is. On a more open-ended conversation, that fraction shrinks a lot.
The orchestrator checks each line against the clip library and synthesises only what it cannot find there. The share it finds, shown along the bottom, is the number that decides your saving.
Text to speech is one line on the bill
Cutting two thirds off your text-to-speech cost does not cut two thirds off your total cost per call, because other meters keep running regardless: the phone connection itself, at $0.0140 a minute on Twilio's standard US rate, and the speech-to-text step, at $0.0048 a minute on Deepgram's promotional streaming rate. Add Amazon Polly Neural synthesis at $0.0132 a minute and a typical low-cost minute totals $0.0320, about 3 cents. Cut two thirds off just the synthesis line and it drops to $0.0044, bringing the total to $0.0232, about 2 cents. Real, but nowhere close to three times cheaper, because text to speech was never the whole bill.
The picture changes completely if you start from an expensive voice provider instead. Swap in our derived ElevenLabs rate of about $0.136 a minute for synthesis, and the same minute totals $0.1548, about 15 cents, with text to speech now the dominant cost on the call. Cut two thirds off that synthesis line and it drops to $0.0453, bringing the total to $0.0641, about 6 cents. That is a genuinely large saving, because on an expensive voice provider text to speech really was most of the bill to begin with.
The same two-thirds cut, applied twice. It barely dents a call that was already cheap, and it transforms one where synthesis was most of the bill.
Open Source Alternative to Vapi / Retell
Self-hosted voice agent platform — no per-minute fees
dograh-hq/dograh
Star on GitHub
Changing TTS tier is the bigger single lever
Every vendor price in this post was read from the vendor's own pricing page on 22 or 23 September 2026. Prices move, so check them again before you budget on them.
Published per-character prices span about forty times, so moving tier shifts more money in one step than a clip library will.
Amazon Polly Standard is $4 per million characters and Neural is $16. Google Cloud lists Studio voices at $160 per million characters. Our derived ElevenLabs subscription rate lands at $165.
Anyone with a spreadsheet will ask why they should record anything. The tiers are not interchangeable on quality. Cheap voices sound cheap, and callers decide whether to trust you in the first few seconds.
A clip library lets you refuse the trade. Human recordings cover the majority of the audio, the engine handles only the residue, and that residue can run on a cheap tier or on open weights you host yourself. Neither lever alone does that.
One caution on per-minute numbers. A voice platform's published all-in rate is a marketing choice rather than a neutral measurement, so treat it with some scepticism. Build your own estimate the way we just did here, from named component prices with the date you checked them. That same approach, weighing named component costs against the systems automation replaces, is the basis for the broader case for keeping voice AI on your own infrastructure.
The robotic voice problem belongs to self-hosted models
The best commercial voice-cloning services already sound convincingly human. It is the free, self-hostable models that still sound noticeably synthetic, which is exactly why this recording pattern matters here.
The Interspeech 2025 study on human fooling rates tested this directly. 135 native US-English listeners gave more than 30,300 ratings on short clips of conversational speech, guessing human or AI each time. Real human recordings were taken for human 70.68 percent of the time, and the top commercial clones matched that, with PlayHT at 71.49 and ElevenLabs at 69.85. Inside that test, listeners genuinely could not tell them from a real person.
The open-weight models tested fell well short of that baseline. The best of them, F5-TTS, reached 50.26, about half of listeners. The weakest, VoiceCraft, managed 30.52, barely three in ten. Listeners flagged open-source output for sounding digital 36.1 percent of the time against 6.9 percent for human recordings, and that was the single biggest giveaway they named.
So whether AI voices sound robotic now depends entirely on which model you mean. For the best paid services the gap has closed. For self-hosted open weights it is still wide open, and it is measured rather than impressionistic. That is why leaning on real recorded human audio matters most in a self-hosted stack. A bigger open checkpoint will not buy the naturalness back, real recordings will, and the quality fix matters more here than the money saved.
Join the Dograh Community
Dograh is an OSS alternative to Vapi. Join our Slack community for queries, releases, best practices & community interactions.
Voice cloning is where open-weight licences actually bite
Your fallback voice has to sound like the same person you recorded, and that one requirement rules out several models that would otherwise be obvious choices.
NVIDIA's Magpie TTS, the model recommended elsewhere in this guide, is a sensible default for local voice in general. It cannot do the job here, though. NVIDIA's published model card states that zero-shot voice cloning was removed for security reasons, so it ships a small set of fixed speakers instead, under NVIDIA's own licence terms rather than an OSI-approved open source one. Good model, wrong job.
The engines that can actually clone a voice each come with their own catch. F5-TTS sounded the most convincing of the open models tested earlier, but its weights are CC BY-NC 4.0, non-commercial only, so you legally cannot use it inside a paid product. Coqui's XTTS-v2 clones from about six seconds of reference audio, and it carries the custom Coqui Public Model License, which you have to read rather than recognise. Chatterbox from Resemble AI is the one that actually works here. It is MIT licensed, free for commercial use, and clones from a short audio prompt, which makes it the one open self-hostable engine you can safely build this pattern on.
Licence terms on text-to-speech models vary far more than on speech-to-text models, and a non-commercial clause is the most common reason a self-hosted stack ends up unshippable. Price matters less than you would think here. Artificial Analysis normalises TTS prices per million characters on its blind-preference leaderboard, and as measured on 22 September 2026 the cheapest entry there was Kokoro 82M at $0.7 per million, which ships fixed voices with no cloning at all, so it cannot be used for this either. Check the licence before you book the studio to record your voice actor, not after. Get the release in writing at the same session. A voiceprint is a regulated biometric in its own right, and why voice data needs to stay local covers what that obliges you to do.
Glossary
- Voice cloning
- Reproducing a specific speaker's voice from a short reference recording instead of picking a stock voice. Some open-weight engines ship it and others have deliberately removed it, so it is a model choice rather than a feature you can assume.
- TTS caching
- Storing the synthesised audio for a phrase the agent repeats and replaying that file instead of paying to generate it again. It removes the per-character charge and takes the synthesis step out of the turn budget.
- Time to first audio byte
- How long after the text is ready before the first audio sample leaves the server. Vendors publish it measured server-side, so it excludes the network leg out to the caller, and a cached clip has almost none of it.
- Per-character pricing
- Billing synthesis by the characters you submit rather than by audio minutes. Punctuation and markup usually count toward the total and silence does not, so the same script costs differently under per-character and per-minute plans.
Work out your coverage fraction first. Count the sentences your agent repeats on every call, divide by everything it says, and that fraction tells you whether this is worth the effort. Above half, record them with a real person and clone that voice for the rest. Below a quarter, spend the time on engine choice instead.
