Deepgram vs Whisper for Voice Agents: Latency, Accuracy and Cost

Pick Deepgram for fast replies without GPUs, because it streams words while the caller talks, at $0.0077 a minute for Nova-3. Pick self-hosted Whisper when audio must stay on your servers or a busy GPU beats that rate. Dograh, the open-source AI voice agent platform we build, runs both. You can self-host it or run it on Dograh Cloud.

Key Takeaways

• Deepgram streams words during speech. Self-hosted Whisper in Dograh answers after each pause.
• Over G.711 phone audio, Whisper Large V3 and Deepgram Nova 2 lost 0.6 points each.
• One always-on L4 equals about 55,900 Deepgram minutes, if it carries your load.

A hosted streaming API and an open model on your GPU fail a live call in different ways. We compare Deepgram and Whisper on reply speed, accuracy on phone audio, cost at volume and who holds the audio, then show how to test both on the same calls in Dograh.

Deepgram vs Whisper at a glance

Deepgram is a hosted API that turns speech into text while the caller is still talking, and Whisper is an open model from OpenAI that you run on your own GPU. In Dograh, Deepgram runs with your own key, and Whisper-family models run on your servers through Speaches, an open-source server that answers OpenAI-style transcription requests with faster-whisper.

On accuracy the two sit close together. What decides most projects is when the words arrive and where the audio goes, and the sections below take those rows one at a time.

Latency: streaming words vs finished sentences

Deepgram sends words as the caller speaks, so the transcript is nearly finished the moment they stop. On Pipecat's speech-to-text benchmark, across 1,000 samples, Nova-3 delivered its final words a median 247 ms after the speaker stopped, and 298 ms at the 95th percentile. Flux, Deepgram's model built for voice agents, also decides inside the model when the caller's turn has ended.

That speed matters because a natural reply has to land in under about 800 ms end to end. In the cascades we build and debug, speech-to-text is usually the slowest part of that budget, more often than the language model.

Whisper reads audio in 30-second windows, so it has no native stream. With the Local Models (Speaches) provider, Dograh waits for the caller to pause, then sends that finished speech segment to your Whisper server as one HTTP request. The text comes back in one piece once the segment is transcribed. So the wait you add is the time one sentence takes on your card, after the pause.

That makes model size a speed choice. Start with Dograh's default, Systran/faster-distil-whisper-small.en, a distilled English model that answers quickly, and move to Systran/faster-whisper-large-v3 only when recorded calls show errors that matter or callers speak another language. Our tutorial on running faster-whisper locally with Speaches covers the setup.

OpenAI also sells Whisper as an API, which is a separate choice from running it yourself. Our comparison of OpenAI Whisper for voice agents explains why its batch API suits recordings after the call.

Accuracy on phone audio

Phone audio is harder than a browser demo, and both models hold up on it about equally. Most phone calls still reach the agent as 8 kHz audio, while a browser test runs at 16 kHz. In the WildASR benchmark on English speech, Whisper Large V3 went from a 4.2% word error rate on clean audio to 4.8% through the G.711 phone codec and 4.9% through GSM. Deepgram Nova 2, an older Deepgram model than the Nova-3 Dograh offers, went from 6.0% to 6.6% on G.711 and 7.9% on GSM.

Where you run Whisper moves the score as much as the model does. The Artificial Analysis Word Error Rate Index, run on recorded files, puts Nova-3 at 5.2% and the same Whisper Large v3 weights at 4.1% on fal.ai, 4.5% on Together AI and 10.1% on Replicate. Deepgram's own comparison sets Nova-3 at 5.26% against 10.6% for Whisper large-v2, an older Whisper version. The number to trust is the one from your own recorded calls on your own setup.

Both can write words nobody said. The WildASR authors found that models "hallucinate plausible but unspoken content under partial or degraded inputs", and that applied across the models they tested, Whisper included. Play silence and hold music through whichever one you pick, and check the agent does not answer words nobody spoke.

Hard words are where the two routes differ in Dograh. The Dictionary words you add in an agent's Settings go to Deepgram as keyterms, which Deepgram lists as a $0.0013-a-minute add-on on streaming. Whisper through Speaches works from the audio itself, so test your product names and account codes on recorded calls before you settle on a model size.

Open Source Alternative to Vapi / Retell

Self-hosted voice agent platform — no per-minute fees

dograh-hq/dograh

Star on GitHub

Cost at volume: when one GPU pays off

Deepgram costs the same per minute at any volume, and a rented GPU costs the same per month at any volume, so your monthly minutes decide which is smaller. Deepgram charges $0.0077 a minute for Nova-3 and Flux English streaming on pay as you go, billed per second, and $0.0065 on the prepaid Growth plan, which starts at $4,000 a year.

Check one setting first. Dograh's Deepgram language starts on multi, which auto-detects the language and is priced as Nova-3 multilingual, $0.0092 a minute. If your callers speak English, set the language to en to get the one-language rate. Our Deepgram pricing breakdown covers the other add-ons and plans.

Here is a worked example, using one GPU on RunPod's secure cloud and our own arithmetic.

So below about 55,900 minutes a month, the Deepgram bill stays under the price of one always-on card. On Growth's $0.0065 the crossover moves to about 66,300 minutes. The sum holds only if one L4 keeps up with your busiest hour, so load-test it at the number of calls you expect before you count on it. Engineering time to run the GPU is not in the sum either.

The GPU pays off sooner when it already sits beside your language model. Our guide to running voice AI on your own GPUs sizes a full self-hosted stack. Self-hosted Dograh has no per-minute platform fee, but servers, model keys and phone minutes still cost money. Dograh Cloud charges a 1¢-a-minute platform fee.

Deepgram also caps how many streams run at once, at 150 on pay as you go and 225 on Growth. Past that, a busy campaign needs a higher limit from Deepgram.

Who controls the call audio

With Deepgram, the caller's audio goes to Deepgram's cloud, and you choose the region that processes it. Dograh's Deepgram setting takes the endpoint, so https://api.eu.deepgram.com keeps processing inside the EU and https://api.au.deepgram.com keeps it in Australia, with the same key. Running Deepgram's models on your own servers is part of Deepgram's Enterprise plan, on NVIDIA GPUs.

Whisper on your own servers keeps the audio and the transcripts inside your network. Bringing your own Deepgram key changes whose contract covers the audio, while the audio itself still travels to Deepgram. Hosted Whisper works the same way. Dograh's Hugging Face provider, with openai/whisper-large-v3-turbo as its default, sends the audio to Hugging Face's inference providers.

This split works because Dograh is the orchestration layer that runs the call flow, tools and telephony, while the speech and language models are a separate layer you choose. Full control of the audio comes from self-hosted Dograh with self-hosted models. On that install, also set ENABLE_TELEMETRY=false to switch off the anonymous usage telemetry.

Test both on the same calls in Dograh

Speech-to-text is a model setting in Dograh, so one agent can run on each and you compare the results. Under Models in the sidebar, save two configurations, one with Deepgram on the STT tab and one with your Local Models (Speaches) connection. Then pick one in the agent's Model tab and swap later. If neither of these two fits, the Transcriber settings list every speech-to-text provider an agent can use.

Call the agent from a real phone on each setup, since a browser test hears 16 kHz audio and the phone path is where names and numbers get misheard. Each call's trace then shows the speech-to-text time for every turn, so you see Deepgram's streaming and Whisper's per-segment wait on your own calls.

Post-call QA scores each step of the call and tags problems such as misunderstandings, which brings misheard turns to the surface across many calls. If you are also weighing AssemblyAI, our AssemblyAI vs Whisper comparison runs the same test against a different billing meter.

Which one to pick

Pick Deepgram when reply speed matters most and your minutes sit below about 55,900 a month. Pick self-hosted Whisper when the audio has to stay in your network, or when a GPU you already run for the language model has room for speech-to-text. You can start on Deepgram and move to Whisper once volume or compliance makes the case, and in Dograh that move is a model setting, with the rest of the agent left as it is.