Right-size LLM reasoning effort to cut cost and latency

Reasoning effort is the dial that decides how many hidden thinking tokens a model spends before it answers. Set it too high and simple requests turn slow and expensive; set it too low and hard requests come back wrong. The fix is to right-size reasoning effort per request, not per model: send easy tasks on a low budget, reserve deep reasoning for the queries that actually need it, and measure the trade in latency, cost, and accuracy. Done well, this trims p95 latency and spend on a production AI feature without moving your quality bar.

Most teams discover reasoning effort the hard way. They swap a fast model for a reasoning model to fix a few hard cases, ship it, and watch the average response time double and the monthly bill climb. The reasoning model is not broken. It is thinking as hard about "reset my password" as it does about a multi-step billing dispute, because nobody told it to do otherwise.

What reasoning effort actually controls

Reasoning models generate a private chain of thought before the answer the user sees. Those thinking tokens are billed like any other output tokens, and they take wall-clock time to produce. A reasoning effort setting (commonly exposed as low, medium, or high, or as an explicit thinking-token budget) caps or steers how much of that private thinking the model does before it commits to a final answer.

The important part for a production SaaS feature is that this is a per-request lever, not a one-time model choice. The same model can answer a trivial classification on a shoestring budget and grind through a gnarly root-cause analysis on a large one. Treating reasoning effort as a fixed global setting is the mistake. You end up paying deep-reasoning prices on the 80 percent of traffic that never needed it.

This is a different lever from the ones most cost posts talk about. It is not about picking a cheaper model, which is model routing, and it is not about trimming the visible answer, which is output token control. Reasoning effort governs the tokens the user never sees but always pays for.

Why one global setting always loses

Production traffic is not uniform. In a support-triage feature we built for a B2B SaaS team, the request mix looked roughly like this: about 70 percent were simple intents (password resets, plan questions, where-is-my-invoice), 25 percent needed a couple of steps, and 5 percent were genuine multi-part investigations. When every request ran at high reasoning effort, the p95 latency sat near 14 seconds and the thinking tokens outnumbered the answer tokens more than three to one.

The failure modes cut both ways. On the simple 70 percent, the model overthought: it spent seconds second-guessing a one-line answer, which added latency and cost for zero quality gain. On the hard 5 percent, a low global budget would have made it underthink and cut corners on exactly the cases where reasoning earns its keep. A single setting cannot serve both ends of that distribution. You either overpay everywhere or underperform where it matters.

There is a reliability angle too. Longer reasoning is not free of risk. The more steps a model takes unsupervised, the more chances it has to wander, which is a cousin of the problem we covered in how reasoning models collapse long agent chains. More thinking is not automatically better thinking.

Right-size the budget per request

The pattern that works is to classify the request first, then pick a reasoning budget to match. You do not need a fancy difficulty model to start. A cheap, fast pre-classifier that sorts requests into three or four buckets gets most of the win.

Sort by task, not by user

Bucket by what the request needs, not who sent it. A concrete starting map for a support feature: known simple intents run at low effort, anything touching billing math or policy exceptions runs at medium, and flagged multi-part cases or explicit escalations run at high. Route the buckets, not the individuals.

Set a ceiling, then let the model stop early

A reasoning budget is a maximum, not a target. Good models stop thinking once they are confident, so a generous ceiling on the hard bucket does not mean every hard request burns the whole budget. The ceiling protects you from the pathological case where the model loops on itself, while still letting easy requests inside a bucket finish fast.

Escalate instead of guessing high

When you are unsure how hard a request is, default low and escalate on signal rather than defaulting high. If a low-effort pass returns low confidence, hits a validation failure, or asks for a tool it could not resolve, re-run that single request at a higher budget. Most traffic never triggers the escalation, so you pay for deep reasoning only on the requests that prove they need it.

Measure the trade, or you are guessing

Reasoning effort is a three-way trade between latency, cost, and accuracy, and you cannot tune what you do not measure. Instrument three things per request from day one: thinking tokens (separate from answer tokens), end-to-end latency, and the reasoning bucket the request landed in. Without the thinking-token count broken out, your cost dashboards will blame the wrong thing.

The accuracy side is where teams get lazy. It is tempting to lower reasoning effort until the bill looks good and call it done, but you have no idea whether you just quietly degraded quality on the hard cases. This is exactly what an offline eval set is for. Before you drop the budget on a bucket, replay it against a labeled set built from real production failures and confirm accuracy holds. Ship the cheaper setting only when the eval says the quality bar did not move.

Tie the numbers back to unit economics. Thinking tokens land in your cost of goods for the feature, so they flow straight into the gross margin on your AI feature. A reasoning model at a flat high setting can quietly turn a healthy-looking feature into one that loses money on power users. Attributing thinking-token spend per request is what makes the overthinking visible.

A realistic before-and-after

Back to that support-triage feature. We kept the same reasoning model and changed only the effort policy: low on the simple 70 percent, medium on the 25 percent, high on the 5 percent, with a low-to-high escalation path on low-confidence answers. The simple bucket stopped overthinking, the hard bucket kept its depth, and the escalation caught the handful of misclassified hard cases that started low.

The measured result over a week of live traffic: p95 latency fell from roughly 14 seconds to under 5, total thinking-token spend dropped by more than half, and the eval accuracy on the hard bucket was flat within noise. The win did not come from a cheaper model or a shorter answer. It came from refusing to pay for reasoning that no request had asked for.

Perceived latency helps here too. On the buckets that still take a few seconds, streaming the answer as it generates hides much of the remaining wait, so the reasoning budget you do spend feels shorter than the clock says.

Where this fits in a production AI stack

Reasoning effort is not a silver bullet, and it is not the first thing to reach for. If your feature is slow because you are calling a frontier model for a task a small model could do, fix the model choice first. If it is expensive because your prompts are bloated, fix the prompt. Reasoning effort is the lever you pull once the model and prompt are right and you still see a long tail of overthought simple requests dragging your p95 and your bill.

The mental model that keeps this simple: match the compute to the difficulty of the request, and make the model prove it needs more before you pay for more. That is the whole discipline. It is boring, it is measurable, and it is one of the highest-leverage tuning moves available on a reasoning-model feature in production.

Frequently asked questions

Is reasoning effort the same as picking a smaller model?

No. Model choice decides the base capability and price per token. Reasoning effort decides how many hidden thinking tokens that model spends before answering. You tune both, but they are separate dials. A small model with the wrong effort setting can still be slow and wrong, and a large model at low effort can still be fast and cheap on easy work.

Will lowering reasoning effort hurt quality?

Only if you lower it on requests that needed the depth. That is why you bucket by task difficulty and validate each bucket against an eval set before shipping the cheaper setting. Lowering effort on genuinely simple traffic usually costs no measurable accuracy, because the model was overthinking answers it already knew.

How do I start without building a difficulty classifier?

Begin with rules you already trust. Known intents, request length, and whether the request touches money or policy are enough to define three buckets on day one. Add a learned classifier later if the rules leave obvious cost on the table. The escalation path covers the cases your rules misjudge.

Does this replace model routing?

No, it composes with it. Routing picks which model handles a request; reasoning effort tunes how hard that model thinks. A mature setup does both: route the request to the right model, then set the reasoning budget for the task inside that model.