Dograh

The Complete Guide to Voice AI Data Security, Setup by Setup

The Complete Guide to Voice AI Data Security, Setup by Setup
TechnicalNovember 14, 2026·8 min read

The Complete Guide to Voice AI Data Security, Setup by Setup

Pritesh Kumar
Pritesh Kumar·Co-founder, Dograh AI

Building Dograh, the open-source alternative to Vapi. OSS and Voice AI. Exit founder and YC alum.

Voice AI data security comes down to who holds each copy of a call. Hosted platforms and their model vendors hold the audio, and your own API key changes only the contract. Dograh, the open-source AI voice agent platform, keeps every copy on your servers when self-hosted with open-weight models, or runs on Dograh Cloud.

Key Takeaways

  • Encryption protects a copy in transit and at rest. It never changes who holds it.
  • Your own model key moves the contract. The audio still reaches that provider.
  • Self-hosting with open-weight models is the setup where no outside model vendor hears calls.

One phone call leaves several copies behind, each landing somewhere different. This guide maps them across four setups, then covers what model and phone vendors keep, how to harden keys and your own install, and the legal duties that follow, with Dograh as the worked example.

Every copy a voice call leaves behind

A voice agent call is not one piece of data. The live audio streams to a speech-to-text model, which produces a transcript. The transcript, with the caller's words and any customer details your tools fetched, goes into the language model's prompt. The reply goes to a voice model as text, and the finished call leaves a recording, a transcript, a trace of every model step and the keys that let the platform reach each provider.

Each copy has its own owner and its own retention. Encryption matters for every one of them, in transit and at rest, but it protects a copy while it moves or sits somewhere. It does not change which company holds the copy or for how long. That is the question this guide answers setup by setup.

The stakes are real. IBM's Cost of a Data Breach report puts the global average breach at USD 4.99 million, a 12% increase on the year before and a record high. A call recording with a name, a date of birth and a reason for calling is exactly the kind of record that drives that number.

Where call data goes in four setups

The setups differ on who runs the platform and who runs the models, and the table follows each copy. Read across a row to see every company that touches a call in that setup.

SetupLive audio goes toTranscripts and recordings sit withModel keysOutside companies that hear the call
Hosted platform with its own modelsThe platform and its model providersThe platform's cloudThe platform'sThe platform and its subcontractors
Hosted platform with your keysThe platform, then your chosen providersThe platform's cloudSaved in the platformThe platform and your providers
Self-hosted platform with hosted modelsYour server, then your model providersYour own storageYour own databaseYour model providers
Self-hosted platform with open-weight modelsYour servers onlyYour own storageNo outside model keys neededNone at the model layer

The phone carrier sits under all four. It carries every call and keeps a copy only if it records or transcribes, which you control in the carrier's own settings.

Self-hosting first, the setup that keeps calls in

Dograh supports all four setups, which makes it a useful way to see the difference between them. It is the orchestration layer that runs the call flow, tools and telephony, and the speech and language models are a separate layer you choose. On the Cloud service, a new account starts on the platform's own managed models, so the audio goes to Dograh's model service.

To bring your own key (BYOK), add your provider account under Providers and Models and make it the default. Saved secrets are not sent back to the browser once stored. Your own key gives you your own contract and your own data terms with that provider, and the audio still travels to it on every call.

Self-hosting moves the platform onto your servers. Recordings and transcripts then land in storage you run, and the platform vendor drops out of the list of companies holding your calls. The model providers stay on it as long as the models are hosted.

The last step closes the gap. Open-weight speech and language models served from your own GPU, through an OpenAI-compatible server such as Speaches, keep the audio, the prompt and the reply on your hardware. Our post on why voice data needs to stay local explains why the voice itself counts as sensitive data.

Owning the copies means owning their protection. The server, the database and the upgrades are your team's to run and patch, and so are the backups and access rules. On cost, the self-hosted platform has no per-minute fee, although servers, model keys and phone minutes still cost money. Dograh Cloud charges a 1¢-a-minute platform fee. The community Slack answers open-source questions, and Enterprise customers get custom SLAs.

Dograh

Open Source Alternative to Vapi / Retell

Self-hosted voice agent platform — no per-minute fees

dograh-hq/dograh

Star on GitHub

What model and phone vendors keep

Once audio reaches a provider, that provider's own data rules decide what happens next. OpenAI does not use data sent to its API to train or improve its models. Abuse monitoring logs are kept for up to 30 days by default, and zero data retention is available on eligible endpoints, including transcription and realtime audio, subject to OpenAI's prior approval.

Speech and voice providers set their own terms in the same way. Some let you opt out of model improvement on each request with a flag, and some offer a zero retention mode that deletes request data once it completes, often only on enterprise plans. Read each provider's data page before you send it a single call, and set the flag in your configuration rather than trusting a default.

The carrier keeps a copy only if it records. Twilio can encrypt recordings with your public key as soon as the call ends, using a fresh key per recording, so only the holder of your private key can decrypt them, and nobody at Twilio can. Our post on where voice agent call data goes and how long it stays follows each copy through its retention.

After a call ends, the carrier records it and locks the recording with the customer's public key, a fresh key per recording. The only key that opens it is the private key kept in the customer's own office.
If the carrier records, Twilio can encrypt each recording with your public key as soon as the call ends, using a fresh key per recording. Only the holder of your private key can decrypt it, and nobody at Twilio can.

Keys, logs and traces

Keys are the copies that open all the others, so treat them as the most sensitive data in the stack. Dograh's own API keys are created under Developers, start with dgr_ and are shown only once, and the platform stores a hash of each key rather than the key itself. Provider credentials saved under Providers are never returned to the browser.

Traces hold the full prompt and reply for every turn, which makes them as sensitive as transcripts. Each organization's Langfuse tracing is private by default, through a Platform Settings toggle that starts off, and a self-hosted Langfuse keeps trace data on your own servers.

Logs are the copy teams forget. A self-hosted install logs at DEBUG level by default, which suits a test server, so set LOG_LEVEL to INFO or higher in production and limit who can read the logs. Our post on voice AI security for phone agents covers caller-side threats, signup and the usage telemetry switch.

Join the Dograh Community

Dograh is an OSS alternative to Vapi. Join our Slack community for queries, releases, best practices & community interactions.

Hardening a self-hosted install

A few settings decide how well a self-hosted install protects its copies, and most are one line in the environment variables. For production, use the S3 backend so recordings sit in a private bucket behind signed links. It works with AWS or any S3-compatible server you run. Change the MinIO access key and secret from their defaults before the first real call.

Sign the media socket next. Set TELEPHONY_WS_TOKEN_SECRET on every process that places or receives calls, wait until the unverified connection warnings stop, then set TELEPHONY_WS_TOKEN_ENFORCE to true so unsigned connections are rejected. The signed URL appears in access logs, so treat log access as access to the socket.

Recording is a server-wide choice. ENABLE_CALL_RECORDING_UPLOAD switches recording off for the whole install, and transcripts are still saved. Retention is your own deletion job working on your own storage, run on the schedule your policy sets.

Then keep up with upgrades. Verizon's 2026 Data Breach Investigations Report found that 31% of breaches now start with software vulnerabilities, ahead of stolen passwords as the top way in. A self-hosted platform that sits a year behind is exactly that kind of target.

Every copy you hold carries duties under the laws that cover it. Under the Health Insurance Portability and Accountability Act (HIPAA), the Security Rule's technical safeguards treat encryption of patient data at rest and in transit as addressable. You assess it, then either adopt it or document why another measure is reasonable. Audit controls are required, so your systems must record and let you examine activity on patient data. Our guide to the best HIPAA-compliant voice AI platforms covers which vendors sign a business associate agreement.

Under the General Data Protection Regulation (GDPR), a voice becomes special-category biometric data when it is processed to uniquely identify a person. Purpose decides it, so a voice agent that only transcribes is not identifying anyone by voice. These are our readings of the rules, and you should confirm with counsel how they apply to your own calls.

Proving who opened which copy is its own job. Our post on building an auditable voice AI stack shows the records a security review asks for.

Glossary

Bring your own key (BYOK)
Using your own account and API key with a model provider instead of the platform's. It gives you your own contract and data terms, while the call audio still reaches the provider.
Zero data retention
A provider setting or agreement under which request data is not kept after processing. OpenAI offers it on eligible endpoints, including transcription and realtime audio, after prior approval.
Signed link
A time-limited URL signed with a secret key that grants access to one object in a private bucket. Dograh's S3 backend uses them so the recordings bucket can stay private.
Addressable
A HIPAA Security Rule measure you must assess, then either adopt or document why an alternative is reasonable. Encryption of patient data at rest and in transit is addressable.

Frequently Asked Questions

Get started with Dograh

Build, deploy, and scale AI agents with Dograh. Join the community of developers building the future.