AssemblyAI vs Whisper for Voice Agents: Hosted API or Own GPU

Pick AssemblyAI when you want words while the caller is still talking and no servers to run. Pick Whisper on your own GPU when the audio must stay in your network or the card stays busy. Dograh, the open-source AI voice agent platform we build, runs both. You can self-host it or run it on Dograh Cloud.

Key Takeaways

• AssemblyAI streams the call. Whisper through Speaches transcribes each finished sentence.
• One always-on L4 costs as much as 57,400 minutes of Universal-3.6 Pro Realtime.
• Only Whisper on your own servers keeps the caller's audio in your network.

A hosted streaming API and an open model on your own GPU fail in different places on a phone line. We compare AssemblyAI and Whisper on live-call delay, phone-audio accuracy, cost at volume and data control, and the last section runs both on the same Dograh agent.

AssemblyAI vs Whisper: the short answer

The choice comes down to who runs the speech model and where the caller's audio goes. AssemblyAI is a hosted API that streams text back while the caller talks and bills for every hour a connection stays open. Whisper is an open model under the MIT licence that you run on your own GPU, and the card costs the same whether calls come in or not.

Starting on AssemblyAI is quick because there is nothing to host. Teams move to Whisper when call volume keeps a GPU busy all month, or when the audio has to stay on their own servers. The table sets the routes next to each other, including the two ways to run them in Dograh.

Delay on a live call

AssemblyAI returns words during the call, and Whisper waits for the caller to stop. AssemblyAI's streaming API sends turn messages over one WebSocket and marks a finished turn with end_of_turn, so the agent can start its reply as soon as the caller finishes.

The Pipecat STT benchmark measures that moment on 1,000 samples. AssemblyAI's universal-3-6-pro returned its final transcript a median 307 ms after the speaker stopped, with a P95 of 401 ms. Universal-Streaming English came back in 256 ms. Against an 800 ms budget for the whole turn, that leaves most of the time for the language model and the voice.

Whisper was built to read audio in 30-second windows and has no streaming mode of its own, so a live agent needs something to cut the call into pieces. With Speaches, the open-source server Dograh uses for Whisper, the platform waits for a pause and sends that finished speech segment to your Whisper server as one HTTP request. No partial words come back while the caller talks.

So the delay your caller hears is the time one sentence takes on your GPU, counted from the pause. That depends on the model and the card. Dograh's default, Systran/faster-distil-whisper-small.en, is a small distilled English model that answers quickly. Systran/faster-whisper-large-v3 hears better and adds its extra time to every turn. Time a few real sentences on your own card before you set that against AssemblyAI's 307 ms.

Accuracy on phone audio

Every accuracy score below comes from test audio rather than your callers, so treat it as a starting list. Most phone calls still reach the agent as 8 kHz audio, while a browser test call runs at 16 kHz, the most Dograh's pipeline uses. A model can sound clean in the browser and still mishear an account number on a real line.

The Pipecat benchmark scores semantic word error rate, which counts only the mistakes that change the meaning. There, universal-3-6-pro scored 0.80%, Universal-3.5 Pro 0.95% and Universal-Streaming English 2.42%. Its clips come from a turn-taking dataset, not phone lines. The benchmark also notes that gaps under about 0.25 points sit inside its confidence interval, so close scores are a tie.

For recorded files, the Artificial Analysis speech-to-text index, AA-WER v2, puts AssemblyAI's Universal-3.5 Pro at 3.0% and Universal-3 Pro at 3.1%. Whisper Large v3 scores 4.1% on fal.ai, 4.5% on Together AI and 10.1% on Replicate. The weights are the same on each host, and the scores are not. How Whisper is served changes what it hears, so the setup you will run is the one to test.

Phone codecs are the next test. The WildASR benchmark from Boson AI played English recordings through phone codecs. Whisper Large V3 went from a 4.2% word error rate on clean audio to 4.8% through G.711 and 4.9% through GSM, so the codec barely moved it. AssemblyAI was not part of that study, so its phone-line score is yours to measure.

Hallucination matters more than the headline rate on a phone agent, because an agent that hears words in silence answers nobody. AssemblyAI says Universal-3 Pro hallucinates about 30% less than Whisper in its own testing. WildASR found that all seven models it tested sometimes write plausible words nobody said when the audio is cut short or degraded. Play silence and hold music through both before a real caller does. If Whisper makes your shortlist, our comparison of Whisper vs Parakeet vs Voxtral covers the other open models that serve on the same route, with their GPU memory and languages.

Open Source Alternative to Vapi / Retell

Self-hosted voice agent platform — no per-minute fees

dograh-hq/dograh

Star on GitHub

Cost at volume and the GPU break-even

AssemblyAI bills open session time by the hour, and a GPU bills by the hour of the month. AssemblyAI charges $0.45 an hour for Universal-3.6 Pro Realtime, $0.0075 a minute, and $0.15 an hour for Universal-Streaming, $0.0025 a minute. The meter runs from the moment the WebSocket opens until it closes, so idle time counts, as our AssemblyAI pricing guide shows call by call.

On the Whisper side, the model is free and the card is not. RunPod lists an NVIDIA L4 at $0.59 an hour on its secure cloud. Kept on for a 730-hour month, that is $430.70, and our guide to running voice AI on your own GPUs sizes the rest of a self-hosted stack around a card like it.

Here is a labelled example of where the two lines cross, as our own arithmetic. The clinic line takes 200 calls a day, four minutes each, for 30 days.

At 24,000 minutes, both AssemblyAI bills stay under the price of one L4 that sits on all month. The GPU starts to pay off above about 57,400 minutes a month against Universal-3.6 Pro Realtime, and above about 172,300 against Universal-Streaming.

Several things move those lines. The break-even only holds if one L4 carries your busiest hour of calls, and how many live calls one card carries depends on the model and your own load test. Engineering time is missing from the sum, since someone has to run and patch the server. On the AssemblyAI side, any session left open between calls adds billed minutes that no caller used.

The sums shift toward Whisper when a GPU already sits beside your language model. Then the speech model runs on hardware you pay for anyway, and the question becomes whether the card has room for it.

Data control: who hears the call

With AssemblyAI the caller's audio goes to AssemblyAI, and with Whisper on your servers it stays with you. AssemblyAI narrows that gap in two ways. It runs a separate EU data zone at the same price as the US one, with processing in six EU regions, Frankfurt and Paris among them.

Retention is the second. AssemblyAI offers zero retention of streaming audio and transcripts to customers who opt out of model training, while some metadata is kept for logging and billing. Audio is also kept out of training for customers under a business associate agreement (BAA) and for those on its EU servers.

Those terms narrow the privacy gap, and the audio still leaves your network. Bringing your own AssemblyAI key moves the contract to you, while the audio still reaches AssemblyAI. Only Whisper, or another open model, on servers you run keeps the call inside your own network. For a clinic or a law firm whose rules forbid sending call audio to an outside company, that settles the choice before price comes up.

Running AssemblyAI and Whisper in Dograh

Dograh runs the call flow, tools and telephony, and you pick the speech and language models as a separate layer. That makes AssemblyAI and Whisper two choices for the same agent, both made in the transcriber settings.

AssemblyAI runs on your own key, added in model configurations. The option is u3-rt-pro, AssemblyAI's Universal-3 Pro streaming model, in English, Spanish, German, French, Portuguese and Italian. The Universal-3.6 Pro prices and benchmark figures above belong to AssemblyAI's newer model, so check your AssemblyAI usage page for what u3-rt-pro sessions cost on your account. The agent's Dictionary reaches AssemblyAI as its keyterms prompt, so brand names and product codes go there.

Whisper runs through a Local Models (Speaches) provider that points at your Speaches server. The default model is Systran/faster-distil-whisper-small.en, English only, and Systran/faster-whisper-large-v3 sits beside it for other languages and harder audio. The Whisper route works from the audio alone, so names and numbers come down to the model you pick and how it does on your calls. On Dograh Cloud the Speaches server needs a public address, while a self-hosted install reaches it on your private network.

Because switching is a setting, the fair test takes an afternoon. Record 20 to 30 of your own calls with real names and numbers, then run them through one agent on AssemblyAI and the same agent on Whisper. Each call's trace shows the speech-to-text time for every turn, and the transcripts show which model got your callers' names right.

The built-in QA then scores each step of every call, so a misheard word that sent the conversation the wrong way shows up as a low-scoring step. Self-hosted Dograh has no per-minute platform fee, but servers, model keys and phone minutes still cost money. Dograh Cloud charges a 1¢-a-minute platform fee. Your AssemblyAI bill stays on your own account, and the Whisper GPU is yours to size.

Which one to pick

Pick AssemblyAI if you want the agent to hear the caller while they talk and you have no GPUs to run. Below about 57,400 minutes a month, its Universal-3.6 Pro Realtime bill stays below the price of one always-on L4. Close every session when the call ends, because idle time is billed.

Pick Whisper on your own GPU if the call audio must stay in your network, or if your volume keeps a card busy all month. Start with the small English model and move up to large-v3 only when your recorded calls show errors that matter. Load-test your busiest hour before you move callers over.

You can also run both. A pilot on AssemblyAI gets an agent live this week, and the same agent moves to Whisper on the day your volume or your compliance team asks for it. Our post on Deepgram vs Whisper runs the same test against Deepgram's per-second meter, and OpenAI Whisper for voice agents prices OpenAI's hosted Whisper against your own card.