Serverless GPU cold starts: the LLM cost nobody budgets

Answer first: serverless GPU inference bills you per second the GPU is allocated, and a cold start can allocate the GPU for 15 to 30 seconds before it does any useful work. On a request that only needs 5 seconds of inference, a 30-second cold start means about 85 percent of that call's cost went to waiting, not answering. If you self-host an open model to escape API prices, cold starts are the tax nobody puts in the spreadsheet.

• Cold start is the time to pull a model into GPU memory and become ready. For a large model loaded from network storage, expect 15 to 30 seconds without weight caching.
• Keep-warm removes cold starts but you pay for idle GPU time, so it only pays off above a traffic threshold.
• Below roughly 40 to 50 percent sustained GPU utilization, serverless per-second billing usually wins. Above it, a dedicated GPU is cheaper per token.

Where the cold-start tax comes from

Serverless GPU platforms scale to zero when idle, which is why they look cheap. The catch is that a scaled-to-zero worker has no model in VRAM. When a request arrives, the platform has to allocate a GPU, pull the model weights, and load them before it can generate a single token.

Small models on platforms with fast boot paths can hit sub-200ms cold starts. Large language models are a different story. A 70B-parameter model loaded from network storage into GPU memory commonly averages 15 to 30 seconds to become ready, and you are billed for that window. That is the arithmetic behind the 85 percent figure: 30 seconds of paid startup wrapped around 5 seconds of paid inference means most of the invoice is waiting. It also blows up your time to first token, so it is a user-experience problem before it is a cost problem.

The keep-warm tradeoff

The obvious fix is to stop scaling to zero. But keep-warm is not free, and the right setting depends entirely on your traffic shape.

Minimum workers

Setting a floor of one always-on worker eliminates cold starts for your baseline traffic, because there is always a loaded model ready to answer. The cost is that you now pay for that GPU 24 hours a day whether or not anyone is using it. For bursty, low-volume features this can cost more than the cold starts it prevents. For steady traffic it is usually the right call, and it starts to resemble a provisioned-throughput decision on a managed API.

Snapshots and weight caching

The middle ground is to make the cold start cheaper rather than eliminate it. Caching model weights on fast persistent storage next to the GPU can cut a cold start from around 90 seconds to about 15. GPU memory snapshots, where the platform freezes a loaded process and restores it, can restart up to 10x faster than loading from scratch. These techniques do not remove the tax, but they shrink the paid waiting window enough that scale-to-zero becomes viable for spiky workloads.

The 40 percent utilization rule

The cleanest way to decide is to estimate sustained GPU utilization. Serverless bills you only for the seconds you use, so it wins when your GPU would otherwise sit idle most of the time. A dedicated GPU bills you for every hour whether busy or not, so it wins when you keep it busy.

The practical crossover sits around 40 to 50 percent sustained utilization. Below that, serverless per-second billing is cheaper because you are not paying for idle capacity. Above it, a dedicated or reserved GPU is consistently cheaper per token, and you stop paying the cold-start tax entirely because the model stays resident. This is the same break-even logic that drives the self-host versus API decision, just at the level of how you run the GPU rather than whether you run one.

A worked example

Say a feature handles 8,000 inference calls a day, each taking about 4 seconds, on a GPU that costs roughly 2 dollars an hour. Actual compute is 8,000 x 4 = 32,000 seconds, about 8.9 hours a day, or about 37 percent of the day. On paper serverless looks right for that 37 percent utilization.

Now add cold starts. If traffic is spiky enough that 1 in 10 calls hits a cold worker and pays a 20-second startup, that is 800 cold starts x 20 seconds = 16,000 extra billed seconds a day, nearly a 50 percent add to your paid GPU time. That single assumption can flip the decision toward a keep-warm floor or a dedicated GPU. The lesson is that utilization alone does not decide it, your traffic burstiness does. Handle the bursts deliberately, the same way you would handle rate limits in production, rather than letting the platform absorb them at cold-start prices.

If you are weighing serverless against a managed API for a feature that also needs low latency, streaming responses can hide some of the wait from users, and a delivery plan on what we build can pin the numbers to a real workload.

Frequently asked questions

What causes a serverless GPU cold start?

The worker scaled to zero has no model in GPU memory. When a request arrives the platform allocates a GPU and loads the model weights before it can respond, and you are billed for that load time.

How long is a cold start for a large model?

For a large model loaded from network storage, commonly 15 to 30 seconds, and up to around 90 seconds without weight caching. Small models with a fast boot path can be under a second.

When should I keep a GPU warm instead of scaling to zero?

When your traffic is steady enough that idle GPU cost is less than the accumulated cold-start cost, or when cold-start latency would hurt the user experience. Bursty, low-volume features often do better scaling to zero with weight caching.

When is serverless cheaper than a dedicated GPU?

Below roughly 40 to 50 percent sustained utilization, because you avoid paying for idle time. Above that threshold a dedicated or reserved GPU is cheaper per token and removes cold starts because the model stays loaded.