Open-Source Speech-to-Text Models in 2026: A Voice Agent Guide

The open-source speech-to-text models to shortlist for a voice agent are OpenAI's Whisper (MIT), NVIDIA's Parakeet TDT 0.6B v2 and v3 and Canary-Qwen 2.5B (CC-BY-4.0, so credit NVIDIA), and Qwen3-ASR 1.7B and Mistral's Voxtral Mini (Apache 2.0). For English calls with the LLM on the same GPU, start with Parakeet v2.

Key Takeaways

• Speech-to-text is usually the slowest stage, so choose it before the LLM.
• Test every candidate on 20 to 30 of your own recorded phone calls.
• Parakeet and Canary-Qwen allow commercial use under CC-BY-4.0 if you credit NVIDIA.

Here we sort open-source speech-to-text models for a phone agent by speed, phone-audio accuracy, languages, licence and GPU fit, and show how each part works in Dograh, the open-source AI voice agent platform we build. You can self-host it or run it on Dograh Cloud.

Why speech-to-text comes first

Speech-to-text sets the floor for how fast your agent can answer, because the agent cannot start working out a reply until it has the caller's words. A reply feels natural when the whole round trip stays under about 800 milliseconds. In our experience, speech-to-text eats more of that budget than the language model does.

The choice also fixes things you cannot change later without a rebuild. It decides which languages you can serve and how much GPU memory is left for the language model. It also sets the licence terms that ship inside your product. So we pick the speech model first and fit the rest around it.

Dograh works as the orchestration layer, running the call flow, tools and telephony, and leaves the speech and language models to a separate layer you choose. You can fill that layer with Dograh's own models, any hosted provider with your own key, or open-weight models on your own servers. Only the last option keeps the caller's audio inside your network the whole time.

You can see where the time goes on every call. Each call's View Trace opens a trace that shows the speech-to-text, AI and voice time for each turn, so you measure the speech step on real calls instead of guessing from a leaderboard. For the voice side of the same decision, see our guide to open-source text-to-speech models.

Streaming or one sentence at a time

A speech model either streams words while the caller is still talking, or it waits for a finished stretch of speech and transcribes it in one go. Streaming lets the agent start thinking before the sentence ends. The finished-clip kind is simpler to serve and often more accurate.

Most open models are the finished-clip kind. Whisper's repository explains that it reads audio in a sliding 30-second window, and the large-v3-turbo model card says it does not do real-time transcription out of the box. Parakeet TDT, Canary-Qwen and Voxtral Mini 3B also take a finished clip.

Dograh's local route works with that shape. With the Local Models (Speaches) option, Dograh cuts the call at each pause and sends that speech segment as one request to an OpenAI-compatible /v1/audio/transcriptions server. So the number that matters is how long one sentence takes to come back on your own card. Our post on live vs recorded speech-to-text explains why a model built for recordings behaves differently on a live call.

A few open models stream. Voxtral Mini 4B Realtime (Apache 2.0, 13 languages) streams over a WebSocket served by vLLM, with a recommended delay of 480 ms, and wants a GPU with 16 GB or more. Parakeet realtime EOU 120M is English only and returns text without punctuation or capitals, with 80 to 160 ms of streaming latency and an end-of-utterance token when the caller finishes speaking. Nemotron 3.5 ASR Streaming 0.6B covers 40 language-locales with chunks from 80 to 1,120 ms.

What works in Dograh today is the per-segment route, through Local Models (Speaches) or an OpenAI-compatible base URL. For these streaming models, the next step is a streaming server in front of them. NVIDIA's Nemotron 3.5 Transcribe sits on the other side, an offline model that suits a second pass on recordings after the call.

Accuracy on phone audio

Leaderboard accuracy is measured on clean recordings, and a phone call is not clean. Most phone calls still reach the agent as 8 kHz audio, while a browser test call runs at 16 kHz, the most Dograh's pipeline uses. How big the gap is depends on the speech model and the phone path.

One model card measures it directly. The Parakeet TDT 0.6B v2 card reports a 6.05% average word error rate on its normal 16 kHz test sets and 6.32% after the audio was squeezed through 8 kHz mu-law and back. In background noise at an SNR of 5, it scored 8.23%.

Other models fall much further. The WildASR benchmark played the same recordings through G.711, the standard phone codec. Whisper Large V3 went from 4.2% to 4.8%, while Qwen2-Audio went from 5.8% to 48.3%. One model barely moved and the other stopped being usable.

The server matters as much as the weights. On the Artificial Analysis speech-to-text index (AA-WER v2, a blend of agent-style talk, parliament speeches and earnings calls, run on recorded files), Whisper Large v3 scored 4.1% on fal.ai and 10.1% on Replicate. Canary Qwen 2.5B on Replicate scored 4.3%, and Parakeet TDT 0.6B v3 on Together AI scored 4.5%.

So record 20 to 30 of your own calls with the names and numbers your callers really say, run each candidate on them through the server you plan to use, and read the transcripts. Brand names and product words are where models slip most. Dograh's Dictionary passes hard words to hosted providers that accept them, such as Deepgram and AssemblyAI. With a local model, list the usual mishearings in the prompt so the language model reads past them.

Open Source Alternative to Vapi / Retell

Self-hosted voice agent platform — no per-minute fees

dograh-hq/dograh

Star on GitHub

The open-source shortlist

This table puts each model's own card next to the way it reaches a Dograh agent. Sizes and languages come from the model cards and Whisper's repository.

Accuracy and speed pull in different directions. The Open ASR Leaderboard paper compared 86 systems across 12 datasets and found that Conformer encoders with transformer decoders give the best average accuracy, while CTC and TDT decoders run fastest. That is why Canary-Qwen edges Parakeet v2 on the cards' averages (5.63% against 6.05%), yet Parakeet v2 processes audio about eight times faster. That ratio is our arithmetic on the two cards' throughput figures, an RTFx of about 3,380 against 418.

For a closer look at three of these rows, with faster-whisper's memory use and how each one is served, read Whisper vs Parakeet vs Voxtral.

Languages settle half the list

Language needs usually narrow the field before speed does. Parakeet v2, Canary-Qwen and the realtime EOU model are English only. Parakeet v3 covers 25 European languages and detects which one the caller speaks. Voxtral Mini 3B covers 8, including Hindi, and Qwen3-ASR lists 30 languages plus 22 Chinese dialects. Whisper still has the widest list at 99.

In Dograh, the local route's Language field takes a two-letter code such as en or de, and you can type one that is not in the list. The default model, Systran/faster-distil-whisper-small.en, is English only. For other languages, switch to Systran/faster-whisper-large-v3 or type the name of another model your server holds.

Check each language on recorded calls in that language. A model's language list says it was trained on the language, and your callers' accents and phone lines decide how well it actually hears them.

Licences in plain words

Licences decide what you may ship, so read them before you build on a model. Whisper is MIT, and Voxtral and Qwen3-ASR are Apache 2.0. Both allow commercial use with light conditions, such as keeping the licence notice with the software.

Parakeet v2, Parakeet v3 and Canary-Qwen are CC-BY-4.0. Commercial use is allowed, and attribution is required: credit NVIDIA wherever you ship the model, for example on your product's notices page. Nemotron 3.5 ASR Streaming and Nemotron 3.5 Transcribe use NVIDIA's OpenMDW-1.1 licence.

Parakeet realtime EOU 120M is different again. It comes under the NVIDIA Open Model License, NVIDIA's own agreement, so call it open weights rather than open source and have someone read the terms before you ship. A NIM container from NVIDIA carries its own licence on top of the model's. Dograh itself is BSD 2-Clause, so the orchestration layer adds no conditions of its own.

What fits beside your LLM on one GPU

Memory decides what can share one machine, and sharing one machine keeps the speech step close to the language model. On a 24 GB card, a 4-bit 27B language model running a 64K context leaves about 4.2 GiB free. Parakeet TDT 0.6B v2 loads in roughly 2 GB, so it fits beside it.

Voxtral Mini 3B needs about 9.5 GB in bf16, so it wants a second card or a bigger one. Voxtral Mini 4B Realtime asks for 16 GB on its own. Whisper large-v3 in the original code needs about 10 GB, though faster-whisper at int8 brings Whisper down far enough to try. Our guide to local, self-hosted voice models covers the full stack that card runs.

Concurrency is the other half of sizing. One GPU serves several calls at once, but each caller's sentence waits its turn on the card, so load-test at the number of calls you expect. A self-hosted Dograh install runs 10 concurrent calls by default, which you can raise with DEFAULT_ORG_CONCURRENCY_LIMIT. Our deep dive on Nvidia Parakeet for voice agents covers v2 against v3, streaming and serving.

Whisper on your own GPU or as an API

Whisper is the easiest open model to start with in Dograh, because the Local Models (Speaches) route was built around it. Speaches runs Whisper through faster-whisper, and the default model is a distilled, English-only small Whisper that answers quickly. Our tutorial on running faster-whisper locally covers the setup end to end. To keep transcription in house, see self-hosted speech-to-text for voice agents, running Whisper or another open model on your GPU inside your network.

You can also run Whisper without your own GPU. Dograh's Hugging Face provider runs hosted Whisper through Inference Providers, with openai/whisper-large-v3-turbo as the default model. OpenAI's own Whisper API charges $0.006 a minute and does not stream, which suits recordings better than live calls.

One Whisper habit is worth testing for. The turbo model card warns that it can produce text that was never spoken, so play silence, coughs and hold music through it before real callers do. Our post on OpenAI Whisper for voice agents works through when the API beats your own GPU on cost.

When a hosted API is the better call

Self-hosting is not free, and for some teams a hosted streaming API is the right answer. If your volume is low or nobody on the team runs GPUs, a hosted provider gets you there with a key and no hardware. The same holds if you need streaming words today. You pay per minute, and the audio goes to that provider.

Dograh runs hosted speech-to-text from providers such as Deepgram, AssemblyAI, Speechmatics, Soniox and ElevenLabs with your own key, alongside Dograh's own models. Add them under Providers in the sidebar, then pick one on the STT tab of a model configuration in Models. Moving an agent from a hosted model to a local one later changes a model name and an address, and the agent stays the same.

Self-hosted Dograh has no per-minute platform fee, but servers, model keys and phone minutes still cost money. Dograh Cloud charges a 1¢-a-minute platform fee. Our comparison of the best speech-to-text APIs for voice agents covers stream prices, concurrency limits and the providers our platform runs.

Where to start

Start from your callers' languages, then your GPU, then the licence you can ship. For English calls with the language model on the same card, start with Parakeet TDT 0.6B v2. For European languages, test Parakeet v3. For the widest language list or the shortest setup in Dograh, start with Whisper through Speaches.

If you need streaming words today and have no GPU team, start on a hosted provider and move to a local model once the calls and the volume justify the hardware. Whichever you pick, record your own calls, compare transcripts and read the speech-to-text time per turn in the trace before you commit.

Each part of the speech-to-text choice above has its own deep dive:

• Best speech-to-text APIs for voice agents
• OpenAI Whisper for voice agents: API vs self-hosted
• Nvidia Parakeet for voice agents