On-Prem Voice AI for Logistics: When Your Own Servers Earn Their Keep

On-prem voice AI for logistics means running Dograh and the speech and language models on servers you own, so shipment, address and call data never leave your infrastructure. Self-hosting Dograh in your own cloud already gives most logistics companies data ownership. On-prem earns its keep when the cargo is high-value or a customer or contract requires it.

Key Takeaways

• Self-hosting in your own cloud gives most logistics companies data ownership.
• On-prem is for high-value cargo and contracts that require it.
• One 24GB card runs a 27B language model and a speech model together.

This post is part of our guide to AI voice agents for logistics and delivery. The guide ends on where the platform runs; this post sizes the servers and GPU and says plainly when on-prem is worth the upkeep. It assumes Dograh, an open-source, self-hostable voice agent platform, as the orchestration layer, with the speech and language models as a separate layer you place where you choose.

Self-hosted first, on-prem when it is required

No logistics regulation forces call data to stay on-site, so on-prem is a choice, never a default. Self-hosting Dograh in your own cloud account already gives data ownership for most logistics companies. On-prem is worth the servers and upkeep when the cargo is high-value, or when customers or contracts require it, for example pharma, defence or government shipments, or enterprise shippers whose security reviews demand it. The regulatory case for on-prem in healthcare and finance is made in Why On-Prem Will Win Enterprise Voice AI; this post is about freight and parcels, where the argument is commercial.

Dograh is the orchestration layer: it runs the call flow, tools and telephony, and the speech and language models are a separate layer you choose. That split is why the phrase "never leaves your infrastructure" needs a condition. It is true only when you self-host Dograh and run self-hosted open-weight models with it. That is the difference from bringing your own key to a hosted model provider, which moves the contract, not the data; the audio still reaches the provider. The code is on GitHub under the BSD 2-Clause licence, so a security reviewer can read what they are approving.

What a delivery call carries

The data in a logistics call is worth more to a thief than most people assume. A delivery coordination call carries a home address, a gate code and a delivery window. A carrier check call carries the current location of a loaded trailer and the time it will sit at a dock. A load booking call carries a rate. A driver callback carries a personal phone number. None of this is health data, and all of it is useful to someone planning a theft.

That threat is no longer theoretical. Cyble's 2025 report on transport and logistics, as reported by Indian Transport and Logistics News, counted 283 verified ransomware attacks on transport and logistics firms in 2025, more than the 120 in 2023 and 122 in 2024 combined, with trucking and freight services nearly 71% of victims. The same report describes cyber-enabled cargo theft, where attackers use network access to track, reroute and seize physical shipments. A voice platform that stores every call's audio and transcript in a vendor's cloud adds one more place that data can be taken from. The general case for keeping voice data local is in Why Voice Data Needs to Stay Local.

Open Source Alternative to Vapi / Retell

Self-hosted voice agent platform — no per-minute fees

dograh-hq/dograh

Star on GitHub

The three conditions test, applied to logistics

One consultancy's model for the self-hosting decision names three conditions that favour it: a data boundary that no vendor contract can satisfy, sustained call volume high enough to keep owned GPUs busy, and voice as core product infrastructure. Its view is that hosted is cheaper and faster for most companies otherwise, and its cost model assumes a net-new dedicated team to run the stack.

Apply that to a 3PL or a regional carrier honestly. Most fail the volume test, and that is fine; a few hundred delivery calls a day will not keep a GPU busy. Voice is rarely core product infrastructure in logistics; it is an operations tool. The condition logistics does meet, when it meets one, is the first. A pharma shipper's quality agreement, a defence contract's data-handling clause or an enterprise customer's security review can draw a boundary that says call audio and shipment data are processed on systems the carrier controls. No per-minute contract satisfies that.

The staffing assumption is the lever. If the stack needs a new team, the hosted platform wins on cost at any volume a 3PL will reach. If an existing operations or IT team absorbs it, and the platform supplies the production plumbing rather than leaving it to be built, the cost of on-prem is a server, a GPU and part of someone's week.

Servers and GPUs: what fits on one card

A single 24GB GPU runs a complete English voice stack. The language model is the big tenant: Qwen3.8 27B, quantised to 4-bit and run at a 64K context, leaves about 4.2 GiB free on a 24GB card. Parakeet TDT 0.6B v2 for speech-to-text takes roughly 2GB and fits beside it. If your drivers and consignees speak more than English, the Parakeet TDT 0.6B v3 model card lists 25 European languages at a 6.34% average word error rate on the Open ASR Leaderboard; it is CC-BY-4.0, so attribution is required. Voxtral Mini 3B is the other common choice, and its model card says it needs about 9.5GB of GPU memory in bf16, which means a larger card or a smaller language model.

Text-to-speech has a licence decision inside it. Magpie TTS is open weights under NVIDIA's own licence, with five fixed English speaker voices and no voice cloning. Chatterbox is MIT-licensed and clones voices, so it is the one that serves the pattern where a pre-recorded human clip falls back to a synthetic clone of the same voice. Call Magpie open weights, never open source.

Capacity is the next number. A self-hosted Dograh instance runs 10 concurrent calls by default, adjustable with a setting; the GPU and your phone provider set the real ceiling, and a dispatch desk at peak needs headroom above its average. Plan for the audio you will actually get: most phone calls still reach the agent as 8 kHz audio, while a browser demo runs at 16 kHz, and speech recognition is harder on the narrower audio. Test the speech model on your own call recordings, with your depot names and road noise, before you size anything. The model choices in more depth are in Running Voice AI on Local, Self-Hosted Models.

Who owns uptime and upgrades

On-prem moves the pager to your desk. A dispatch operation runs around the clock, so the server that answers the night line needs someone on call, a patching schedule that does not take the line down at 3 a.m., spare GPU capacity for peak season and a plan for updating models without changing how the agent behaves. A hosted platform does all of that for you, and the platform fee is what you pay for it. On one hosted platform that fee is $0.05 a minute, which at 1,000 minutes a day is about $18,000 a year before any model, phone or add-on cost.

Self-hosting removes that line from the bill and adds the work to your week. The sensible order is to start in your own cloud account, where the provider owns the hardware and you own the data and the platform, and move on-prem only when a contract or a customer requires it. The agent does not change between the two; what changes is who owns the uptime. What self-hosting changes for a team coming from a hosted platform is set out in Open-Source Alternative to Vapi: What Self-Hosting Really Changes.

When on-prem beats a hosted platform

On-prem wins when a boundary you did not draw says it must. A pharma shipper whose quality agreement names where data is processed, a defence or government contract with a data-handling clause, an enterprise shipper whose security review will not sign off a vendor cloud, or a yard that is air-gapped by policy: in those cases a self-hosted platform you can move onto your own hardware is the only option that passes, and the open code is what the reviewer reads.

Everywhere else, self-host in your own cloud and keep the models there too. You get the same data ownership, the same agent and the same answer to "where does the call audio go", without a server room. When the first contract that needs on-prem arrives, you move the deployment, not the agent.