Experiments / TTS verbalization benchmark

EXP-001Benchmark✓ Run Sep 29, 2026 · 78 of 78 checks pass

Nobody's text-to-speech can say a date with confidence

We gave seven open text-to-speech models 1,080 sentences a voice agent might read back, then checked what they actually said.

7models

1,080sentences

27types of tricky text

2independent listeners

View data and code Methodology

Cite as text

Dograh (2026). TTS verbalization benchmark: do open text-to-speech models say dates, prices and phone numbers correctly? Dograh Experiments. https://www.dograh.com/hub/experiments/tts-verbalization-benchmark

BibTeX (for academic papers)

@misc{dograh2026tts,
  author = {{Dograh}},
  title  = {TTS verbalization benchmark: do open text-to-speech models say dates, prices and phone numbers correctly?},
  year   = {2026},
  howpublished = {Dograh Experiments},
  url    = {https://www.dograh.com/hub/experiments/tts-verbalization-benchmark},
  note   = {Data and code: https://github.com/dograh-hq/experiments/tree/main/benchmarks/tts-verbalization}
}

Summary

This experiment tested whether seven open text-to-speech models say tricky text correctly: dates, prices, phone numbers, hashtags and 23 other types. Each model read the same 1,080 sentences, and two independent speech recognizers wrote down what it actually said.

Key findings

The smallest model is the most accurate.kokoro-82m (82 million parameters) scores 0.61, ahead of orpheus-3b (3 billion) at 0.54, and uses roughly 18x less compute.

No model can say a hashtag.0 correct out of 279.

Dates are broken everywhere.The best score any model reached is 0.23: three out of four dates come out wrong.

The ranking is reliable.Two independent listeners rank the models identically, even though they disagree on 11% of individual clips.

What we found

Even the best model gets only six in ten right

Which model fits your industry?

Each industry needs different kinds of text read correctly. Pick yours to rank the models on just those types.

Overall score across all 1,080 sentences and 27 types.

0.61

0.54

0.49

0.49

0.44

6chatterbox-multilingual

0.25

7parler-tts

0.03

Switch to Table to see each model’s likely range and cost.

Where it breaks

The four hardest types of text. Each number is the best score any of the seven models reached, so every model did this badly or worse.

0.00

Hashtags

No model got a single one right

0.23

Dates

Even the best model: about 2 in 10 right

0.30

Version numbers

Even the best model: about 3 in 10 right

0.40

Fractions

Even the best model: about 4 in 10 right

Every model, every type

One model can be perfect at one type and score zero on another, so choose by the types your agent reads, not the average.

How to read this: each row is a type of text and each column is a model, so each square is one model on one type. The brighter the orange, the more often that model said it right. Dark squares mean it almost never did. Click any square for the details.

	Models →
Type of text ↓	kokoro-82m	orpheus-3b	qwen3-tts	chatterbox-turbo	chatterbox	chatterbox-multilingual	parler-tts
Initialism or Acronym
Musical Notation
Stock Ticker
Currency
Ordinal
Vehicle or Product Code
Decimal
Sports score
Roman Numeral
Abbreviation
Cardinal
Legal Reference
License Plate or Serial Numbers
Time
Unit
Mathematical Expression
Geographic Coordinates
Chemical Formula
Phone Number
Address
ISBN
URL or Email
Biological Classification
Fractions
Version Numbers
Date
Hashtag or Mention

Never rightAlways right

Date

kokoro-82m0.10

Right on about 4 of 40 date sentences

6 / 7rank on this type

-0.51vs its overall 0.61

0.03–0.2095% range

All 7 models on date

chatterbox

0.23

orpheus-3b

0.20

qwen3-tts

0.17

chatterbox-multilingual

0.17

chatterbox-turbo

0.15

kokoro-82m

0.10

parler-tts

0.00

A real miss

Her birthday falls on 2009-02-26 this year.

"her birthday falls on two thousand nine zero two twenty six this year"

The smallest model wins on both counts

kokoro-82m is the most accurate and the cheapest to run: about 18× less compute per clip than orpheus-3b, the runner-up.

← Compute per clip (shorter is cheaper)ModelCorrect (longer is better) →

0.31s

kokoro-82m

0.61

5.50s

orpheus-3b

0.54

4.99s

qwen3-tts

0.49

1.06s

chatterbox-turbo

0.49

3.48s

chatterbox

0.44

2.40s

chatterbox-multilingual

0.25

3.06s

parler-tts

0.03

Reading the results

An average across 27 types is a headline, not a decision

The spread inside one model is wider than the spread between models: kokoro-82m scores 1.00 on musical notation and 0.00 on hashtags in the same run. So look at the type your product actually reads.

If your product reads order numbers

look at ID & serial numbers

chatterbox

0.78

kokoro-82m

0.72

orpheus-3b

0.55

Top 3 of 7 on ID & serial numbers

If it books appointments

look at Dates

chatterbox

0.23

orpheus-3b

0.20

qwen3-tts

0.17

Top 3 of 7 on Dates

Want a different type? Use the industry picker or the heatmap above.

Hear it fail

Listen to what each model actually said

Pick a type of text to see a real sentence from the test, how it should be read, and what each of the seven models said.

0 of 7 models said it right

The text

Her birthday falls on 2009-02-26 this year.

Should be said as

Her birthday falls on February twenty sixth two thousand nine this year.

kokoro-82m"her birthday falls on two thousand nine zero two twenty six this year"Wrong

orpheus-3b"her birthday falls on two thousand and nine oh two twenty six this year"Wrong

qwen3-tts"her birthday falls on two thousand and nine dot r six this year"Wrong

chatterbox-turbo"her birthday falls on two thousand and nine two twenty six this year"Wrong

chatterbox"her birthday falls on two thousand nine two two six this year"Wrong

chatterbox-multilingual"her birthday falls on two thousand and nine nif-two twenty six this year"Wrong

parler-tts"her birthday fais and voslo vargy vogue"Wrong

Real sentences from the run. What each model said is shown as the listener heard it, with numbers written out as words. Both listeners agreed on every verdict shown.

How it worked

One sentence, four steps, repeated 7,560 times

Here's one real sentence from the test, followed all the way through.

01 · THE TEXT

We write a tricky sentence

1,080

sentences across 27 types, each created by code together with its correct spoken form

Written

Her birthday falls on 2009-02-26 this year.

Should be said as

Her birthday falls on February twenty sixth two thousand nine this year.

02 · THE VOICE

A model reads it aloud

7,560

audio clips: every sentence read by all 7 open text-to-speech models

kokoro-82m

reads the sentence aloud

03 · THE LISTENER

A recognizer writes down what it heard

15,120

transcription attempts: each clip sent to 2 speech recognizers from different companies

It heard

"her birthday falls on two thousand nine zero two twenty six this year"

04 · THE VERDICT

We compare it to the right answer

0.61

is the best score: the share of clips a model got right

Needed

"february twenty sixth two thousand nine"

Wrong

Why you can trust it

Two independent listeners. The same ranking.

A single speech recognizer deciding what a model "said" is a weak foundation, so each clip was sent to two from different companies.

Same ranking from both listeners

1kokoro-82m0.61

2orpheus-3b0.54

3qwen3-tts0.49

4chatterbox-turbo0.49

5chatterbox0.44

6chatterbox-multilingual0.25

7parler-tts0.03

Every record, accounted for

Records (2 listeners × 7,560 clips)15,120

Scored15,051

Excluded · listener rate limits47

Excluded · empty or invented transcripts22

Clips both listeners scored7,497

Clips the listeners disagreed on832

But the two listeners disagreed on 11% of individual clips (832 of 7,497). Only a human ear settles those, so we publish the disagreement set for you to check.

Behind the numbers

Four results that need a closer look before you rely on them.

Why do hashtags score zero?

The models split "#GreenBreak" into "Green Break" correctly, but none of them ever says the word "hashtag", and the test's correct reading includes it. That's a judgement call: plenty of people don't say "hashtag" out loud. If your product doesn't, read the other 26 types.

Why does parler-tts score 0.03?

Its speech is garbled overall, not just on the hard part. Read its score as the floor that proves the test can tell good from bad.

Example: "Her birthday falls on 2009-02-26 this year." came out as "Herbal tea flowers enswore so, very gee. Thok!"

Why is only one listener the official scorer?

Deepgram nova-3 "repairs" some mistakes on its own, for example turning "mph" into "miles per hour", which would turn a model's failure into a pass. So it's used as a cross-check, and OpenAI whisper-1 is the scorer.

What does this not tell you?

It is not general accuracy: every sentence is chosen to be hard. Models were tested as hosted on Replicate, in English only. Compute figures come from a 10-clip timing run per model, and 69 of 15,120 records (0.5%) never produced a score. Assume the speech recognizers (ASR) have an error rate of about 5%, so a small share of verdicts may reflect the listener mishearing rather than the model.

Data and code

The data and code behind this experiment

The test sentences, the code and the full results are published on GitHub, along with a free script that recalculates the numbers.

View data and code Listen to all 7,560 clips Methodology

Cite as text

Dograh (2026). TTS verbalization benchmark: do open text-to-speech models say dates, prices and phone numbers correctly? Dograh Experiments. https://www.dograh.com/hub/experiments/tts-verbalization-benchmark

BibTeX (for academic papers)

@misc{dograh2026tts,
  author = {{Dograh}},
  title  = {TTS verbalization benchmark: do open text-to-speech models say dates, prices and phone numbers correctly?},
  year   = {2026},
  howpublished = {Dograh Experiments},
  url    = {https://www.dograh.com/hub/experiments/tts-verbalization-benchmark},
  note   = {Data and code: https://github.com/dograh-hq/experiments/tree/main/benchmarks/tts-verbalization}
}

Each modelkokoro-82morpheus-3bqwen3-ttschatterbox-turbochatterboxchatterbox-multilingualparler-tts

Type of text	kokoro-82m	orpheus-3b	qwen3-tts	chatterbox-turbo	chatterbox	chatterbox-multilingual	parler-tts	Best
Initialism or Acronym	1.00	0.89	0.80	0.97	0.97	0.82	0.15	1.00
Musical Notation	1.00	0.40	0.23	0.38	0.40	0.33	0.00	1.00
Stock Ticker	1.00	0.82	0.92	0.85	0.82	0.85	0.03	1.00
Currency	0.97	0.88	0.62	0.68	0.47	0.03	0.00	0.97
Ordinal	0.90	0.97	0.82	0.93	0.88	0.70	0.00	0.97
Vehicle or Product Code	0.95	0.72	0.51	0.78	0.88	0.28	0.00	0.95
Decimal	0.90	0.50	0.82	0.62	0.33	0.00	0.00	0.90
Sports score	0.53	0.78	0.90	0.82	0.45	0.38	0.00	0.90
Roman Numeral	0.88	0.55	0.31	0.38	0.57	0.15	0.00	0.88
Abbreviation	0.85	0.69	0.59	0.80	0.60	0.62	0.57	0.85
Cardinal	0.80	0.82	0.50	0.62	0.42	0.12	0.00	0.82
Legal Reference	0.78	0.64	0.50	0.72	0.60	0.25	0.00	0.78
License Plate or Serial Numbers	0.72	0.55	0.55	0.45	0.78	0.17	0.00	0.78
Time	0.72	0.64	0.38	0.45	0.33	0.62	0.00	0.72
Unit	0.62	0.65	0.72	0.53	0.42	0.38	0.00	0.72
Mathematical Expression	0.68	0.40	0.70	0.40	0.00	0.00	0.00	0.70
Geographic Coordinates	0.00	0.53	0.67	0.23	0.00	0.00	0.00	0.67
Chemical Formula	0.62	0.47	0.61	0.45	0.62	0.50	0.00	0.62
Phone Number	0.42	0.60	0.41	0.38	0.60	0.03	0.00	0.60
Address	0.55	0.24	0.17	0.25	0.17	0.05	0.00	0.55
ISBN	0.15	0.53	0.26	0.28	0.40	0.00	0.00	0.53
URL or Email	0.42	0.40	0.16	0.50	0.15	0.00	0.00	0.50
Biological Classification	0.45	0.46	0.33	0.40	0.45	0.20	0.00	0.46
Fractions	0.10	0.10	0.40	0.03	0.15	0.03	0.00	0.40
Version Numbers	0.30	0.28	0.23	0.28	0.20	0.20	0.00	0.30
Date	0.10	0.20	0.17	0.15	0.23	0.17	0.00	0.23
Hashtag or Mention	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00