Voice Cloning for AI Voice Agents: The Complete Guide

Voice cloning helps an AI voice agent when one consenting person's voice has to carry both recorded clips and generated lines across many calls. It backfires when callers know the real person, or when that person could not freely say no. Judge any clone on a real phone line, where 8 kHz audio changes how it sounds.

Key Takeaways

• Clone a voice when one consenting speaker must cover recorded and generated lines.
• Never clone a voice callers know as a real member of staff.
• Judge "sounds like the person" apart from "sounds natural", over a real phone.

This post is part of our guide to AI voice agents, where a cloned voice is one of three choices for the voice callers hear. We cover when a clone helps or backfires, consent, phone-audio quality and the open models that clone, then how each runs in Dograh.

When a cloned voice helps a phone agent

The strongest case for cloning is the hybrid call. Fixed lines such as the greeting, the opening hours or a required notice are recorded by a real person. Everything that changes per call, such as a name, a date or an answer, is generated. If the generated half uses a clone of the same speaker, the caller hears one voice from start to finish.

The language model picks a recorded clip when one fits the moment and falls back to text-to-speech when none does. Without a clone, that fallback sounds like a second person joining the call. With one, the switch is hard to hear.

A clone also helps when one voice has to cover many agents. A business with a booking line, a reminder campaign and an after-hours line can keep the same consented voice on all three, so a customer hears a familiar sound on every call.

When cloning a voice backfires

A clone backfires when callers know the real person behind it. Clone a receptionist that regulars know, and callers assume they are talking to her. Some will ask after her family or share something private, and the trust that made the voice useful turns into a problem the day they find out.

It also backfires when the voice outlives the job. A staff member who leaves still speaks for the business on every call until someone removes the clone. Set the end date before you record anything.

The law treats the clone like any other AI voice. The FCC's ruling on AI-generated voices says a voice cloned with AI is an artificial voice under the Telephone Consumer Protection Act (TCPA). So outbound calls in a cloned voice need the same consent as any automated call, however human the voice sounds.

A familiar voice makes the automated part easy to miss. If you want to tell callers they are speaking with an automated assistant, or your location requires it by law, write it in the greeting along with the business's name.

Open Source Alternative to Vapi / Retell

Self-hosted voice agent platform — no per-minute fees

dograh-hq/dograh

Star on GitHub

Consent from the person you clone

The person you clone has to be able to say no without cost. For an employee, that is harder than it sounds, because a request from a manager rarely feels optional. Offer a paid voice actor as the default, and treat a staff member's voice as their choice, with no effect on their job if they decline.

The agreement should say what the voice will say, on which lines and agents, for how long, and how the person can withdraw. It should also say what happens when they leave. In practice, that means removing the voice ID from every agent and deleting the clone in the provider's account, on a date both sides know.

Our post on voice cloning for pre-recorded and TTS playback has the written-consent checklist and the full setup, from recording the sample to testing the hand-off between clip and clone.

Judging a clone on phone audio

A clone can sound natural and still sound like the wrong person. Score those two questions separately. Play a recorded line from the real speaker and the same sentence from the clone, then ask a listener which is which, and separately, whether either sounds robotic.

Run that test twice: once in the browser and once over a real phone line. A browser test runs at 16 kHz, but most phone calls still arrive at the agent as 8 kHz audio. On narrowband audio, a voice can sound flatter and lose some of the variation that made it recognisable. The gap depends on the voice model and the phone path, so the phone test is the one that counts.

Test the hard lines on their own. Dates, prices and phone numbers trip up text-to-speech whether the voice is cloned or not. In our TTS verbalization benchmark of seven open models, kokoro-82m read tricky text most accurately at 0.61, and the best date score of any model was 0.23. If your clone reads dates aloud, write them out in words in the prompt or record those lines.

Which open models can clone

Only some open models can clone, and the licence decides whether you can use them in a product. Kokoro cannot clone. Its model card describes an 82-million-parameter model under the Apache licence with 54 voices across 8 languages, and offers no way to clone from a reference clip.

Chatterbox can. Its repository is MIT-licensed, and Chatterbox-Turbo, a 350-million-parameter English model, takes a short reference clip to clone a voice. A 500-million-parameter multilingual version covers 23 languages. Every audio file Chatterbox generates carries Resemble AI's Perth watermark.

Other open options carry licence traps. F5-TTS is licensed for non-commercial use only, and XTTS-v2 has its own Coqui Public Model License. Our comparison of Kokoro vs Chatterbox vs Coqui XTTS sets the three side by side on licence, speed and cloning.

Running an open model on your own servers is also the only way the voice sample stays in-house. A hosted cloning provider keeps the sample and the clone in its own account.

Running a cloned voice in Dograh

Dograh handles orchestration by running the call flow, the tools and the telephony, while you choose the speech and language models as a separate layer. The voice is one of those choices, so a clone plugs in like any other voice.

A clone from a hosted provider such as Cartesia or ElevenLabs runs by its voice ID. In the voice settings, pick the provider you cloned with, choose Enter Custom Value in the Voice field and paste the ID.

A local voice runs through Dograh's local model setting, which points at an OpenAI-compatible endpoint on your own server. That server has to return raw 24 kHz audio from the OpenAI-style speech endpoint. Chatterbox is an open model that can clone. To use it, run it behind that endpoint, and test the voice on a real phone call before it goes live.

The recorded half goes on the Recordings page. Type "@" in a prompt to insert a clip, as the pre-recorded audio setup shows, or set the greeting itself to play a recording.