AI Voice Agents for Fraud-Alert Callbacks: Never Ask, Only Tell

An AI voice agent for fraud-alert callbacks calls the customer when a transaction is flagged, states what the bank already knows, and takes the confirm or deny decision straight into a card lock or dispute. The hard part is not speed. It is sounding legitimate when scammers open their calls the same way.

Key Takeaways

• Bank impersonators took nearly $1 billion in 2025, the costliest impersonation category.
• A prompt cannot guarantee an agent never asks for a card number.
• Every vendor in the call path holds data that makes impersonation convincing.

A bank has seconds to prove a fraud call is real. The most expensive scam in the country opens with the same sentence. We build Dograh, an open source voice agent platform, for calls where the account details being read out cannot end up on somebody else's infrastructure. A fraud alert is the sharpest version of that, because those same details are what makes an impersonation convincing.

The fraud alert and the scam start the same way

The biggest obstacle to a fraud-alert callback is not latency. It is that the customer has been trained to hang up on it.

The Federal Trade Commission reported that people reported losing $3.5 billion to imposter scams in 2025, with nearly $1 billion of that going to business impersonators and the highest losses of all going to people pretending to be from a bank. The same release is blunt about how those scams begin. "Some of the costliest impersonation scams start with a fake security alert, often from a bank."

So the legitimate call and the costliest fraud pattern in America share an opening move. The FDIC spells out the sequence in its warning about bank impersonation scams, where a customer gets a message about a purchase they did not make and then a call from someone claiming to be the bank's fraud department.

Then there is the simpler problem of being picked up at all. Hiya's survey of more than twelve thousand consumers found 86% of unknown calls go unanswered. A faster dialer does not fix a phone that nobody answers.

There is an uncomfortable symmetry to sit with before building anything. Synthetic voice is already being used against banks, and the standard advice from fraud teams is to treat unexpected voice contact with suspicion. A bank that deploys a synthetic voice outbound is adding to the same pool of calls its own security page tells customers to distrust. That tension does not go away. It gets managed by making the call verifiable rather than persuasive, which is a design decision and not a voice quality decision.

This changes what you are building. The job is to earn belief in about ten seconds, which is the same discipline that governs the first fifteen seconds of any outbound call.

What the callback actually has to do

The call flow itself is short and well understood.

The fraud system flags a transaction. The agent calls the customer, names the bank, and reads back what the bank already holds: the merchant, the amount, the approximate time. The customer confirms or denies. A confirmation releases the hold. A denial locks the card, opens the dispute, and schedules a replacement.

The design rule underneath it is that the agent tells the customer what the bank knows and never asks the customer to reveal anything. No full card number, no password, no one-time passcode, no date of birth. That single constraint separates the real call from the fake one, because asking for those details is what a scammer does. It is the same principle we applied to verify before you talk in collections calls.

The wording matters as much as the rule. The opening that names the bank, the confirm-or-deny question, and the line after a denial should be fixed utterances the agent reads, not sentences the model composes fresh each call. A generated opening drifts, and eventually it sounds urgent the way scam calls sound urgent. That is the script the FTC describes. The model routes between fixed nodes instead of writing them, the same reasoning that keeps a pharmacy agent from improvising about a controlled substance it cannot refill.

Then give the customer a way out that does not depend on trusting you. The agent should be able to stop and tell them to hang up and call the number on the back of their card. A scammer cannot survive that instruction. The same logic governs voicemail: name no amount and leave no callback number, since a number in a voicemail is a template for the next scam.

The constraint is easy to write down. Holding it in production is harder than it looks.

A prompt cannot hold the line on what the agent must never say

You cannot guarantee that a generative model never says something by telling it not to.

Prompts leak by paraphrase. The model avoids the banned phrase and produces the banned meaning instead. An instruction that says "never request the full card number" survives testing, and then under an unusual customer reply the model asks the customer to "confirm the rest of the digits." The rule held on the surface and broke underneath.

This matters more here than almost anywhere else. On a fraud call, one request for sensitive information does not simply fail a policy check. It teaches the customer that the bank's real calls behave like scams, and it hands a future attacker a script that now sounds authentic.

The engineering answer is to stop treating the rule as a request. Split the sensitive moments into fixed nodes with fixed utterances. At the point where identity is confirmed, the agent has a small set of exact things it can say, and generation is not in the path. The model routes between nodes. It does not compose the sentence.

You also cannot prove a negative about a generative system. What you can prove is traversal. If every call produces a record of which nodes it passed through, you can demonstrate that no call reached a state where sensitive information was requested. That proof requires you to own the transcripts and the traces, which is a hosting question before it is a modelling question.

Open Source Alternative to Vapi / Retell

Self-hosted voice agent platform — no per-minute fees

dograh-hq/dograh

Star on GitHub

Who else hears the call

Every vendor in the call path is another copy of the data that makes impersonation believable.

Trace one fraud-alert call through a typical hosted build. Speech recognition runs at one company. The language model runs at a second. Text to speech runs at a third. Telephony runs at a fourth. The orchestration layer runs at a fifth. Each of them sees some combination of the customer's name, the merchant, the amount, and the fact that this person's card was just flagged.

That last detail has real value to an attacker. Knowing a specific customer expects a fraud call from a specific bank at a specific moment is close to a complete script.

The pressure is rising. BioCatch surveyed 292 US financial institutions serving more than 280 million users and found attempted impersonation scams up 130% over the year ending May 2026.

Banks reading this are working under the GLBA Safeguards Rule, not HIPAA. The distinction is worth holding onto, because healthcare has a business associate agreement that formally delegates obligations to a vendor, and that reasoning does not transfer cleanly. The Safeguards Rule requires a bank to pick providers that can actually maintain safeguards and to bind them to those safeguards by contract. It then adds a duty that does not expire, which is periodically assessing each provider against the risk it presents. The FTC puts it plainly in its own guidance, saying the buck still stops with you. Adding a vendor does not subtract a responsibility. It adds one more thing to supervise, and one more place the data sits.

Self-hosting here is fraud-surface reduction

Self-hosting a fraud-alert agent is not an engineering preference. It removes copies of the data that makes fraud credible.

The usual case for running your own stack is control, or cost, or procurement posture. Those are real, and they are not why a fraud team should care. The call carries the exact information an impersonator needs, and every hop is a place it can escape.

Running the platform yourself cuts the hop count, and it is worth being precise about which hops actually go. The workflow engine, the transcripts and the call traces sit on infrastructure the bank already audits for everything else it runs, so the record proving your agent never asked for a card number stays where a regulator or an internal auditor can inspect it. That is the same argument the on-prem enterprise voice AI case makes more broadly.

Telephony is the layer to watch. It only stays yours if you run Asterisk ARI against your own PBX. The hosted carriers most deployments start with are third parties holding call audio like any other vendor, and self-hosting the platform does not change that.

The models need the same care. True colocation only works for models you can host, and closed commercial weights are not yours. The best available version there is choosing providers in the same cloud region, which shortens the network path without reducing how many companies hold the data.

What hosted platforms leave on the table

A hosted voice platform can run this workflow. What it cannot do is let you prove things about it.

The economics are the visible part. Most closed platforms charge a platform fee of roughly five to seven cents per minute before any AI usage, with speech, language and synthesis billed separately on top. Their own published all-in ranges stretch from about seven cents to over thirty cents a minute depending on which models you pick. For a bank running fraud verification at volume, that is a per-call tax on a call meant to prevent losses.

What matters more is what you can show afterwards. Owning the platform means owning the traces, which turns "our agent never asks for card numbers" from a claim into evidence. It also means bringing your own keys, or moving to open models and cutting the AI cost too.

Latency stays a hard requirement. End-to-end response under 800 milliseconds is the bar for a call that feels like a conversation rather than a recording, and speech recognition is what sets the floor, since batch transcription alone can cost 500 milliseconds before the model sees any text. We covered how that budget is actually spent in the sub-800ms bar.

If you are starting this, build the identity-handling moment first. Everything else in the call is ordinary automation, and that one node is where the call either earns belief or becomes indistinguishable from the scam it exists to stop.