Dograh
Experiments  /  TTS verbalization benchmark  /  orpheus-3b
Model · EXP-001

orpheus-3b

Ranked 2 of 7 on the TTS verbalization benchmark. Strongest on ordinal, weakest on hashtag or mention.

At a glance
Said right54%share of all 1,080 sentences
Rank2 of 7among the seven models
Cost per clip5.50scomputing time per audio clip
Likely range51%–57%where the true score most likely sits
Strengths and weak spots

Where orpheus-3b does best and worst

Its five strongest and five weakest types of text, by the share of sentences it said right.

Best at
Ordinal
0.97
Initialism or Acronym
0.89
Currency
0.88
Cardinal
0.82
Stock Ticker
0.82
Breaks on
Hashtag or Mention
0.00
Fractions
0.10
Date
0.20
Address
0.24
Version Numbers
0.28
A real miss

Hashtag or Mention: what orpheus-3b actually said

The text
Follow #GreenBreak for updates during the conference.
Should be said as
Follow hashtag green break for updates during the conference.
orpheus-3b said
"follow green break for updates during the conference"
All 27 types

Every type of text, best to worst

The share of sentences orpheus-3b said right on each of the 27 types. Each type has 40 test sentences.

Ordinal
0.97
Initialism or Acronym
0.89
Currency
0.88
Cardinal
0.82
Stock Ticker
0.82
Sports score
0.78
Vehicle or Product Code
0.72
Abbreviation
0.69
Unit
0.65
Legal Reference
0.64
Time
0.64
Phone Number
0.60
License Plate or Serial Numbers
0.55
Roman Numeral
0.55
Geographic Coordinates
0.53
ISBN
0.53
Decimal
0.50
Chemical Formula
0.47
Biological Classification
0.46
Mathematical Expression
0.40
Musical Notation
0.40
URL or Email
0.40
Version Numbers
0.28
Address
0.24
Date
0.20
Fractions
0.10
Hashtag or Mention
0.00