Provisioned throughput vs on-demand: when it saves money

Provisioned throughput looks like a discount: reserve dedicated model capacity, pay a flat hourly rate, stop paying per token. Teams see the per-token math on a busy day and assume a reservation will be cheaper. Then the reservation sits half-idle overnight and on weekends, and the flat fee costs more than on-demand would have.

The decision is not about which mode is cheaper in the abstract. It is about your sustained utilization. Below a break-even point, on-demand wins by a wide margin. Above it, a reservation saves money and adds predictable latency. Here is how to work out where your workload actually sits.

Two different billing models

On-demand bills per token: a rate for input tokens and a higher rate for output tokens, quoted per million. You pay only for what you send and receive, and you inherit shared-pool rate limits that scale with your spend tier. It is the right default for anything spiky or low-volume.

Provisioned throughput bills per unit of dedicated capacity per hour, whether or not you use it. On Amazon Bedrock that unit is a Model Unit (MU). A single MU for Claude 3.5 Sonnet lists around $39.60 per hour, which is roughly $28,500 per month for a one-month no-commit reservation. Under a typical mixed load, one MU delivers on the order of 16 million input plus 4 million output tokens per hour. Longer commitment terms lower the hourly rate in exchange for a fixed contract.

The trade you are making: on-demand converts idle time into zero cost but gives you no capacity guarantee, while provisioned throughput guarantees capacity and latency but charges for idle time. Everything downstream follows from that.

Where the break-even sits

Sources converge on a break-even around 60 to 85 percent sustained utilization, depending on the model and commitment term. A practical reading: below roughly 60 percent sustained capacity use, on-demand or batch is almost always cheaper. Above roughly 80 percent, provisioned throughput typically saves 20 to 40 percent against on-demand and removes throttling risk.

Translate that into tokens. If one MU can serve about 16 to 20 million tokens per hour, break-even lands near 8 to 10 million tokens per hour of steady demand. If your peak hour touches that number but your average hour is a fraction of it, you are paying for capacity you do not use for most of the day.

Right candidates for a reservation

Steady, high-volume workloads with predictable load: a document-processing pipeline that runs continuously, a classification service behind a busy product surface, a batch job that saturates capacity for a known window. These keep utilization high enough to earn the discount.

Wrong candidates

Spiky, interactive, or low-volume workloads: a chat feature with daytime peaks and dead nights, an internal tool used a few hundred times a day, anything with unpredictable traffic. The unused hours waste the reservation. For most companies, on-demand stays cheaper until you are well into the millions of requests per month.

Cut the bill before you reserve anything

Reserving capacity is the last optimization, not the first. Most teams have cheaper wins available before they commit to an hourly contract, and those wins also lower the utilization math in on-demand's favor.

Batch is the obvious one. Both Bedrock and OpenAI offer roughly 50 percent off on-demand rates for asynchronous batch requests, which fits any non-urgent workload such as backfills, evaluations, or overnight data enrichment. We walk through it in using the batch API to cut LLM costs in half. Routing is the next lever: send easy requests to a smaller, cheaper model and reserve the frontier model for hard ones, as in model routing to cut AI costs. Both reduce the sustained volume you would need to justify a reservation.

If your real question is whether to leave managed APIs entirely and run your own inference, that is a separate break-even with its own fixed costs and operational burden. We covered it in self-hosting an LLM versus using an API. And if you are choosing between managed providers in the first place, Bedrock versus the OpenAI API for production cost compares the per-token economics.

A short decision process

Work through it in order. First, measure sustained utilization over a representative week, not a peak hour. Second, move every non-urgent workload to batch and confirm the remaining traffic is genuinely latency-sensitive. Third, apply routing so only the requests that need the expensive model reach it. Fourth, look at what steady demand remains: if it holds above roughly 70 to 80 percent of a Model Unit's capacity around the clock, price a reservation and compare the monthly flat fee against your projected on-demand spend. If it does not, stay on-demand and revisit when volume grows.

The mistake to avoid is reserving on the strength of a busy afternoon. Provisioned throughput rewards flat, predictable, high load. If your traffic has quiet hours, on-demand's zero-cost idle time is the feature, not a bug. For the broader picture of where inference money goes, see our overview of AI infrastructure costs.

Frequently asked questions

What utilization do I need before provisioned throughput pays off?

Roughly 60 to 85 percent sustained capacity use, depending on model and commitment term. Below about 60 percent, on-demand or batch is almost always cheaper. Above about 80 percent, a reservation usually saves 20 to 40 percent and removes throttling.

How much traffic is that in tokens?

A single Bedrock Model Unit serves roughly 16 to 20 million tokens per hour under mixed load, so break-even lands near 8 to 10 million tokens per hour of steady demand. Measure your average hour, not your peak, before committing.

Does provisioned throughput improve latency?

Yes. Dedicated capacity gives you predictable latency and no shared-pool throttling, which is often the real reason to reserve even when the cost is close. If latency matters more than cost, that can tip the decision on its own.

What should I try before reserving capacity?

Move non-urgent work to the batch API for about 50 percent off, apply model routing so cheap requests hit cheap models, and confirm the remaining traffic is truly latency-sensitive. Those steps often remove the need for a reservation entirely.