A deepfake caller can fool an AI agent only if the agent treats a familiar voice or caller ID as proof. Check what the caller knows or receives instead, and never change details or release money on voice alone. Dograh, the open-source AI voice agent platform you can self-host or run on Dograh Cloud, verifies callers through your own systems.
Key Takeaways
- A cloned voice beats voice checks, and NIST rules out voice biometric comparison.
- Caller ID finds the record. It never proves who is calling.
- Payout, contact and password changes go to a person or a verified code.
This post is part of our guide to compliant AI voice agents, where each safeguard is tied to a point in the call. Below, we list what an agent never does on voice alone, the rules on voice and caller ID, a verification flow and trusted callbacks, all in Dograh.
Can a cloned voice fool an AI agent?
It can, if the agent is built to trust a voice. Impersonation is already the costliest fraud consumers report. The FTC's impostor scam data shows people reported losing $3.5 billion to impostor scams in 2025, with nearly $1 billion lost to scammers posing as businesses. In Hiya's State of the Call survey of more than 12,000 consumers in six countries, one in three said they get deepfake calls.
A cloned voice sounds like your customer, and a spoofed number looks like theirs. Neither matters if the agent checks what the caller knows or receives, never how they sound. In Dograh, verification runs through your own systems, so a voiceprint never decides anything, as the flow below shows.
Not every bot caller is a fraudster. Customers now send AI assistants to call businesses for them, which raises a different set of questions, covered in our post on AI agents that call a business for real customers.
What the agent never does on voice alone
Some requests hand over money or an account, so a convincing voice is never enough for them. The table lists them with what the agent does instead.
| Request on the call | What happens instead |
|---|---|
| Change payout or bank details | Transfer to a person, after a verified code |
| Change the phone number or email on file | Code sent to the old contact, then a person |
| Reset a password or PIN | Your own reset flow, started by your system |
| Read out balances or account numbers | Only after verification, and only masked values |
| Approve a refund or a payment | Transfer to a person |
| Read out a one-time code | Never, under any story the caller tells |
In Dograh, tools attach to individual nodes, so the tools behind these requests sit only on steps after verification, or nowhere at all. A caller with a perfect voice and an urgent story reaches a step that cannot make the change.
Why a voice or caller ID proves nothing
The US standard for digital identity already rules voice out. NIST's digital identity guidelines, the National Institute of Standards and Technology's SP 800-63B-4, say biometric comparison based on voice must not be used to authenticate someone. They also class codes sent over the phone network as a restricted method and ask verifiers to check for SIM swaps and number porting before sending one. These are the rules US federal agencies follow, and they make a sensible benchmark for anyone else.
Caller ID is no better as proof. The FCC's Truth in Caller ID rules ban faking caller ID to defraud or cause harm, and that ban exists because faking it is easy. The Financial Crimes Enforcement Network's alert on deepfake fraud names deepfake voices in impersonation schemes and points financial institutions to multifactor checks. Generally, then, a voice match or a known number should only ever find a record, never open it. How these rules bind your business depends on your sector, so confirm your verification design with counsel.
Detection vendors sell voiceprints as the fix. A voiceprint can be one signal, but NIST's position means it should never be the check that opens an account. Our overview of voice AI security for phone agents covers the wider set of attacks.
Open Source Alternative to Vapi / Retell
Self-hosted voice agent platform — no per-minute fees
dograh-hq/dograh
Star on GitHub
A verification flow built for cloned voices
Here is the order a Dograh agent follows on a call that may end in a sensitive request:
- The Greeting Text names the business and nothing about the account, because it plays before anyone is verified.
- Pre-Call Data Fetch looks up the record by the caller's number. That only finds the record.
- The agent asks a knowledge question, and an HTTP API tool sends the answer with the account ID from call data to your system, which replies yes or no.
- For a high-risk request, a second tool asks your system to send a one-time code to the number or email already on file, and a third checks the code the caller reads back.
- After two failed checks, the agent shares nothing more and offers a callback to the number on file or a person.
- The change itself goes to a person through Transfer Call. While the caller listens to hold music, Dograh dials your team and connects the caller when someone picks up.
The model never sees the stored answer or the code, because the HTTP API tool returns only the verdict. The codes come from the system you already trust, sent to contact details it already holds. If the transfer rings out, meets a busy line or times out, the agent gets transfer_failed and the call continues, and the prompt books the callback instead. Write the two-failure limit into the node prompt and test it, since the prompt is what applies it.
Callbacks you can trust
A callback beats any check made on an inbound call, because the business dials a number it already holds. The rule is simple: call the number on file, never a number the caller reads out.
During the call, an HTTP API tool writes the callback request to your system. Your system then starts the call through Dograh's API Trigger, with the number from its own records. The fraudster who asked for the callback is not on that line. For outbound calls about suspicious activity, our post on fraud alert callbacks shows a flow that never asks the customer for details.
Your own agent's voice is a separate question. Cloning a real person's voice for your agent needs that person's consent, covered in our post on cloning your own agent's voice.
Join the Dograh Community
Dograh is an OSS alternative to Vapi. Join our Slack community for queries, releases, best practices & community interactions.
Check the flow on real calls
A flow only protects you if every call follows it. Add a QA check such as "were any account details shared before verification" and run it on a sample of live calls, and add a call disposition for failed verifications so they are counted. Each flagged call keeps its recording and transcript, and its trace shows the exact turn that went wrong.
Self-hosting with local models keeps those recordings and the verification traffic on your own servers. Self-hosted Dograh has no per-minute platform fee, but servers, model keys and phone minutes still cost money. Dograh Cloud charges a 1¢-a-minute platform fee. Either way, a cloned voice meets the same flow, and the flow never asks how the caller sounds.
Glossary
- Voice biometric comparison
- Matching a caller's voice against a stored voiceprint to confirm identity. NIST SP 800-63B-4 says it must not be used to authenticate someone, because voices can be recorded and cloned.
- Out-of-band code
- A one-time code sent through a second channel, such as a text to the number on file, and read back on the call. NIST treats phone-network codes as restricted and asks for SIM swap and porting checks first.
- Caller ID spoofing
- Making a call show a number that is not the caller's. It is banned when done to defraud, which is why an inbound number can find a record but never prove who is calling.
- Step-up verification
- A stronger check asked only for a risky request, such as a code before a payout change. Routine questions stay quick, and high-risk actions cost the caller one more step.
