Dograh

Small Language Models for Voice Agents: When Small Is Enough

Small Language Models for Voice Agents: When Small Is Enough
TechnicalNovember 19, 2026· Updated November 20, 2026·9 min read

Small Language Models for Voice Agents: When Small Is Enough

Abhishek Kumar
Abhishek Kumar·Co-founder, Dograh AI

Co-founder of Dograh, building the future of open-source voice AI agents. Own your voice AI stack.

A small language model is enough for a voice agent when the call is short and fixed, such as a reminder or a simple lookup. It falls short on long instructions, chains of tool calls and callers with two requests. Dograh, the open-source AI voice agent platform, runs any OpenAI-compatible model you host, self-hosted or on Dograh Cloud.

Key Takeaways

  • Small models handle short fixed flows, lookups and confirmations well enough.
  • On aiewf-eval, gpt-4.1-mini passed 81.8% of turns and gemini-3.7-flash 99.9%.
  • Dograh runs a small model on your own GPU through any OpenAI-compatible endpoint.

A small model answers sooner and fits on a smaller GPU, but it breaks in predictable places on a phone call. We cover where small models hold up, where they fail and what the published numbers show. Then we talk about how to test one and run it in Dograh.

What counts as small, and what it buys you

There is no agreed size at which a language model becomes small. In this post, small means the mini, lite and nano tiers of the hosted model families, and open models light enough to run on one modest GPU next to speech-to-text and a voice. No study gives a parameter count below which phone calls go wrong, so judge a model by the calls it handles, not its size.

A small model buys you a few things that matter on a phone line. It usually starts speaking sooner, which shortens the pause after the caller stops. It needs less hardware, so it can run on your own servers next to the rest of the voice stack. And on your own servers, the conversation never goes to an outside model provider. For hosted models ranked by speed, see our list of the best LLMs for voice agents by latency.

What it costs you is headroom. A small model follows simple instructions well and loses track sooner when there are many, which is exactly what a long phone call piles up.

Where a small model holds up

Small models do well when the call follows a short, fixed path. An appointment reminder that confirms a time and takes a yes or no is a good fit. So is an order-status call that looks up one record and reads back one field, or a front desk agent that answers hours, address and parking from a short knowledge base.

A small model handles an appointment reminder call. It has a short prompt on a sticky note and only two tools on a pegboard, a calendar and one record, and the call runs along a short row of stepping stones with two signposts, confirm a time and a yes or no, to a caller who answers yes.
An appointment reminder that confirms a time and takes a yes or no is a good fit. So is an order-status call that looks up one record and reads back one field, or a front desk agent that answers hours, address and parking from a short knowledge base.

These calls have a lot in common. The prompt is short, there are one or two tools with few parameters, and the caller rarely asks for something the agent was not built for. A small model also works well as the greeter in a larger setup, finding out what the caller wants and handing the call to a stronger agent for the hard part.

Where small models break

Small models break first on tool calls. A booking that needs three tools in a row, each with several required fields, gives the model three chances to pick the wrong tool, drop a field or invent a value. Long prompts are the next weak spot. An agent with twenty business rules may follow the first ten and quietly forget the rest by the fifth minute.

Mixed intent is the third. A caller who says "I want to move my appointment and also ask about my bill" needs the agent to hold two jobs and finish both, and small models often finish one and drop the other. Interruptions make it worse, since the model has to work out what was said, what was cut off and where the conversation now stands.

Some small models also write a tool call out as text instead of making it, so the caller hears something like a function name read aloud. Your platform should catch that, and the Dograh setup below shows how.

Dograh

Open Source Alternative to Vapi / Retell

Self-hosted voice agent platform — no per-minute fees

dograh-hq/dograh

Star on GitHub

What the published numbers show

The most useful public numbers for this question come from aiewf-eval, an open benchmark that runs 30-turn voice agent conversations with tool calls and a knowledge base of about 12,000 tokens. Each turn is judged on tool use, instruction following and grounding. These are rows from its October 2026 results.

Model and settingPass rateMedian time to first answerConversations
gemini-3.7-flash, low thinking99.9%821 ms30
gemini-3.5-flash-lite, minimal thinking94.3%617 ms30
gpt-4o-mini83.0%603 ms11
gpt-4.1-mini81.8%638 ms13
gpt-5-mini80.9%690 ms11
nemotron-3-nano-30b, NVFP471.9%934 ms7
nemotron-3.5-lightning, thinking off, NVFP469.6%63 ms16

Rows with fewer than 15 conversations are a small number of runs, so read them as a direction, not a final score. The pattern is still clear. The mini tiers passed about four turns in five, and gpt-4.1-mini had a tool error rate of 13.1%. They were not much faster than gemini-3.5-flash-lite either, which passed 94.3%. The fastest row, nemotron-3.5-lightning at 63 ms, passed the fewest turns.

Size is not the only lever. On the same board, qwen3.8-27b in NVFP4 passed 95.1% at a 101 ms median, which shows how much the serving hardware and quantization matter. Our post on open-weight LLMs for voice agents covers the open rows and their licenses.

Pairing a small model with a large one

The strongest case against going small is that you do not have to choose. The Thinking While Speaking paper by Srinivas, Englhardt, Iyer and Patel pairs a small talker model, from 135M to 1.7B parameters, with a large reasoner. The talker answers at once and folds in the reasoner's answer as it streams. The pair got within 6.3% of the large reasoner's accuracy, and in a live study 18 people rated it more responsive, with the talker running on an Apple M2 chip. That is a small study away from a phone line, but the idea holds.

The production version is simpler. Run a small model where the call is simple and a stronger one where it is not, as two agents with a handoff between them. Our guide on using different models in one voice agent shows how that split works.

Testing a small model on your own calls

Benchmarks shortlist a model, and your own calls decide. Take twenty real transcripts from the call type you want to automate, including the messy ones with interruptions and two requests at once. Turn each into a scripted test, and run every script several times, because a language model can give a different answer to the same input.

Score each run on whether it picked the right tool with every field filled, and whether it finished every request. Count tool errors separately from wrong wording, since a wrong field costs more than an awkward phrase. Look at the slowest calls as well as the median, since callers remember the long pause.

Then repeat the best two candidates over a real phone line. Phone audio reaches the agent as 8 kHz sound, transcripts get noisier, and a small model has less room to recover from a misheard word.

Join the Dograh Community

Dograh is an OSS alternative to Vapi. Join our Slack community for queries, releases, best practices & community interactions.

Running a small model in Dograh

Dograh can run a small model you host yourself or one you reach with your own key. For a model on your own GPU, start Ollama or vLLM, then open Providers in the sidebar, click Add Provider and choose Local Models (Speaches). Its base URL defaults to Ollama's http://localhost:11434/v1, and the model field accepts any name your server exposes. The LLM settings cover OpenAI-compatible endpoints such as these. If you serve with vLLM, start it with --enable-auto-tool-choice and the tool-call parser for your model family, or tool calls come back as plain text.

Next, open Models, click Add Configuration and choose BYOK, then pick the local connection on the LLM tab. Each agent uses that configuration from its Model tab. If the model writes a tool call out as text, Dograh catches it and runs it as a tool call, so the caller never hears a function name.

Add a backup for the moments a small model stalls. Click Configure fallbacks and set Slow response to start a stronger hosted model if the local one gives nothing within your limit, 1,500 ms by default. That backup needs its own key and sends the conversation to its provider, so choose one you are allowed to use. Our guide to running voice AI on local, self-hosted models covers the speech-to-text and voice side of the same stack.

Where it runs and what it costs

Dograh is the orchestration layer that runs the call flow, tools and telephony, and the speech and language models are a separate layer you choose. A small model on your own GPU keeps the language layer inside your network, and only full self-hosting of every layer keeps all call data there.

Running Dograh yourself means no per-minute platform fee. You still pay for servers, model keys and phone minutes, and your team looks after the GPU, the database and upgrades. Support comes from the open-source community, and Enterprise adds custom SLAs. Telemetry can be switched off when self-hosting. If you would rather not host it, Dograh Cloud runs it for a 1 cent a minute platform fee.

Pick your simplest call type, run a small model on it with a stronger fallback, and move up a size only where your scripts show it failing.

Glossary

Small language model
A language model from the mini, lite or nano tier of a family, or an open model light enough for one modest GPU. There is no agreed size cutoff.
Tool error rate
The share of turns in which a model calls the wrong tool, misses a required field or sends a bad value. aiewf-eval reports it per model.
Talker model
A small model that answers the caller at once while a larger reasoner works on the hard part and passes its answer forward.
OpenAI-compatible endpoint
A model server, such as Ollama or vLLM, that accepts the same chat requests as OpenAI's API, so a platform can call it without custom code.

Frequently Asked Questions

Get started with Dograh

Build, deploy, and scale AI agents with Dograh. Join the community of developers building the future.