Why Whisper Hallucinates on Silence and How to Fix It

Introduction

OpenAI's Whisper is one of the most widely used speech-to-text models in production today. It also has a strange habit: sometimes it invents sentences nobody ever said.

This happens most often during silence or long pauses. Instead of returning blank text, Whisper fills the gap with a guess — and that guess can be bizarre, fabricated, or outright harmful.

Researchers found this isn't a rare edge case. A 2024 FAccT study on speech-to-text hallucination harms found hallucinations in roughly 1.4% of transcriptions, and 38% of those contained fabricated violence, invented names, or false claims of authority.

For teams running Whisper in healthcare, legal, or voice AI settings, that's not a rounding error. This post breaks down exactly why silence triggers these hallucinations and walks through the fixes that actually work in production.

Key Takeaways

• Whisper's decoder always outputs text, so it "fills" silence by guessing or looping the last phrase heard
• Roughly 1% of transcriptions contain hallucinations — and a real share of those are harmful
• Longer pauses raise error rates for people with aphasia, elderly speakers, and non-native speakers
• No single fix solves it — combine VAD preprocessing, decoding tweaks, and post-processing filters

Common Causes of Whisper Hallucination on Silence

Whisper is OpenAI's open-source automatic speech recognition (ASR) model, trained on 680,000+ hours of multilingual audio. It's become the default engine behind transcription tools, subtitling software, and the speech-to-text layer inside plenty of voice AI agents.

Hallucination on silence isn't random. It comes from a specific mix of architecture decisions, training data, and how real speech actually sounds.

Silence Produces Near-Zero Audio Embeddings

Whisper's encoder converts audio into embeddings that the decoder then turns into text. When a segment is silent or close to silent, those embeddings carry almost no useful signal.

The decoder, though, is built to keep producing output. Rather than stopping, it predicts the most statistically probable continuation, which is often just the last thing it recognized, repeated.

A common real-world example: a podcast with a 20-second pause gets transcribed as "Hello, welcome to our show," looped dozens of times, instead of coming back empty.

A 2025 study on Whisper ASR hallucinations from non-speech audio found this looping behavior in 34.1% of tested silence-augmented clips.

Training Data Bias From Caption-Heavy Datasets

Part of Whisper's training data came from transcribed online video. That leaves a fingerprint.

When Whisper hits non-speech audio, it frequently defaults to phrases that sound like video outros rather than staying silent:

• "Thank you": 24.76% of non-speech test outputs
• "Thanks for watching": 10.32%
• "Subtitles by the Amara.org community": 0.46%

A silent stretch in a customer service call can get transcribed as YouTube sign-off language instead of nothing at all.

No Built-In Voice Activity Detection or Confidence Threshold

Whisper does include internal signals meant to catch silence: a no-speech probability score and an average log-probability check. OpenAI's own documentation notes that no-speech probability alone wasn't reliable enough, so it's paired with a log-probability threshold before a segment gets treated as silent.

The problem: these are soft heuristics, not a dedicated voice activity detector. They don't reliably catch every non-speech moment, especially background noise, hold music, or dead air on live calls. When they miss, Whisper still generates a full sentence instead of nothing.

Speech Patterns With Longer Pauses Disproportionately Trigger Hallucinations

This is where the harm gets personal. The Careless Whisper study compared transcription segments from people with aphasia against a matched control group:

Group	Hallucination rate
Aphasia speakers	1.8%
Control group	1.1%

Aphasia speakers' audio also contained far more non-vocal time: 41% of total duration versus 15% for controls, with pauses averaging 6.8 seconds versus 1.3 seconds.

In healthcare and accessibility settings, that gap turns into biased transcription errors that disproportionately hit protected groups.

What Happens If Silence Hallucination Is Ignored

These aren't cosmetic glitches. Left unchecked, they create real downstream damage.

In healthcare: Medical transcription tools built on Whisper are already deployed at scale. Reporting from the Associated Press on AI transcription in hospitals found roughly 30,000 clinicians across 40 health systems using a Whisper-based tool that had already produced fabricated text, including invented medication names and unsupported clinical details.

When source audio is deleted for privacy reasons, those errors become uncatchable after the fact.

In legal and hiring contexts: A hallucinated quote attributed to a defendant, witness, or job candidate can skew decisions and violate fairness standards. The risk is structural whether or not a court case has surfaced yet.

In voice AI and call centers: A silence-triggered hallucination mid-call can derail the entire conversation. An agent that reacts to a phantom phrase instead of a customer's pause looks broken, erodes trust, and tanks conversion rates.

Warning Signs You're About to Experience a Hallucination

Watch for these three patterns in your transcripts:

1. Looping — the same phrase repeating dozens of times in a row
2. Caption bleed — video-style phrases like "subscribe" or "thanks for watching" appearing mid-transcript, out of context
3. Language drift — text suddenly switching to an unexpected language despite fixed, single-language audio

How to Fix and Prevent Whisper Hallucination on Silence

There's no single switch that eliminates this problem. Effective mitigation layers several defenses together.

Add Voice Activity Detection (VAD) Preprocessing

Run audio through a dedicated VAD model (Silero VAD or WebRTC VAD are the two most common choices) before it ever reaches Whisper. Pass only the detected speech segments through.

This removes near-silent audio entirely, so Whisper never has an empty stretch to "fill" with invented text. Implement this at the ingestion stage of your pipeline, particularly for long-form recordings and live streams where dead air is common.

Tune Whisper's Decoding Parameters

A few configuration changes reduce hallucination risk directly:

• Set beam_size to 1 and temperature to 0 to cut exploratory, low-confidence guessing
• Adjust no_speech_threshold (default 0.6) and logprob_threshold (default -1.0) to be stricter about what counts as silence
• Set a hallucination_silence_threshold so long silences near a suspected hallucination get skipped

Test these carefully. Overly aggressive thresholds can cause Whisper to skip genuinely quiet speech, which trades one error for another.

Post-Process With a "Bag of Hallucinations" Filter

Build (or borrow) a list of commonly hallucinated phrases ("thanks for watching," "subtitles by," looping patterns) and strip them out after transcription.

This catches what slips through inference. The 2025 Whisper hallucination study found combining Silero VAD with a delooping filter and this kind of phrase list cut word error rate from 104.8% to 6.5% on non-overlapping audio.

Apply this as a final layer on every transcript before it reaches downstream systems.

Trim and Normalize Silence Before Inference

Apply decibel-threshold silence trimming to raw audio, especially at the start and end of files. Research shows silences at file boundaries are a direct trigger for hallucination. Trimming them removes the triggering condition before Whisper ever sees it.

This matters most for short clips: voicemails, call recordings with dead air at the start, or clips with trailing silence after the speaker hangs up.

Use a Voice AI Platform With Hallucination Handling Built In

Hand-building a VAD-plus-Whisper-plus-filtering pipeline from scratch is doable, but it's also ongoing maintenance work that most teams don't want to own.

Dograh AI, an open-source voice AI orchestration platform, supports locally hosted Whisper alongside built-in turn-detection and interruption handling. A caller's pause during a live call gets treated as conversation-flow logic, not transcribed indiscriminately.

Its VAD and turn-taking model evaluate pauses, intonation, and context together, with a documented turn-taking latency of 50–200 milliseconds, before deciding a caller's turn has actually ended.

That distinction matters. A generic pipeline transcribes every audio frame. A conversation-aware platform decides which frames are worth transcribing in the first place, which is exactly the layer that stops silence hallucinations before they start.

This approach fits teams building inbound or outbound voice agents at scale who'd rather not maintain hallucination-mitigation infrastructure in-house.

Tips for Long-Term Monitoring and Control

Fixing hallucinations once isn't enough — Whisper deployments need ongoing checks:

• Routine monitoring: Run automated QA on a sample of transcripts or calls to flag repeated phrases or hallucination-pattern text. Platforms like Dograh AI offer automated post-call analysis with sentiment and miscommunication detection to catch these patterns at scale.
• Testing standards: Benchmark any STT model against diverse speech samples — accents, disfluencies, elderly speakers — before rollout. Aggregate Word Error Rate alone hides these problems.
• Documentation: Version-lock Whisper model checkpoints and decoding parameters. Log every change so a hallucination regression can be traced back to a specific update.
• Model evaluation: Periodically evaluate alternatives like Voxtral or Canary for accents or domains where Whisper underperforms, rather than assuming one model fits every use case.

Conclusion

Silence-triggered hallucination in Whisper isn't bad luck. It comes from identifiable causes: a decoder that always outputs text, training data shaped by video captions, and soft internal heuristics that miss real silence.

Combining VAD preprocessing, decoding parameter tuning, and post-processing filters reduces the problem, though nothing eliminates it completely. Routine monitoring protects you from the costly downstream errors these hallucinations cause in healthcare, legal, and voice AI use cases.

Frequently Asked Questions

What is Whisper?

Whisper is OpenAI's open-source automatic speech recognition model, trained on hundreds of thousands of hours of multilingual audio. It's designed for near-human transcription accuracy across many languages.

What do people use Whisper for?

Common uses include transcription, subtitling, meeting notes, and medical documentation. It's also widely used as the speech-to-text engine inside voice AI agents and call automation tools.

Why does Whisper hallucinate more than other speech-to-text tools?

Researchers testing the same 187 known hallucination-triggering audio segments found zero comparable hallucinations from Google, Amazon, AssemblyAI, and Rev AI. Researchers attribute this to Whisper's GPT-like decoder and its caption-heavy training data.

Can voice activity detection (VAD) completely eliminate Whisper hallucinations?

VAD dramatically reduces hallucinations by removing silent segments before they reach Whisper. It works best paired with decoding parameter tuning and post-processing filters, not used alone.

Are Whisper hallucinations more common in certain languages or accents?

They disproportionately affect speech with longer pauses rather than any specific language. This includes people with speech disorders like aphasia, elderly speakers, and non-native speakers.

How common are Whisper hallucinations in real-world use?

Studies found hallucinations in roughly 1% of transcriptions. A notable share of those (nearly 40% in one study) were classified as actively harmful rather than harmless noise.