Handling LLM rate limits (429s) in production

The first time an AI feature gets real traffic, it does not fail because the model is wrong. It fails because the provider returns a 429. A batch job, a demo spike, or one enterprise customer running a bulk import pushes you past the tokens-per-minute cap, and suddenly a chunk of requests are erroring while the rest slow to a crawl. This is a throughput problem, not a model problem, and it has a well-understood set of fixes.

This is different from a provider outage. When OpenAI or Anthropic is actually down, you fail over to a second model, which we covered in building resilient LLM API features. A 429 means the provider is up and healthy and is telling you that you are sending too much, too fast, for your account tier. The job is to shape your own traffic so you stay under the ceiling.

• Rate limits are enforced two ways at once: requests per minute (RPM) and tokens per minute (TPM). You can be well under one and blow past the other.
• The fix is four layers: client-side rate limiting, backoff that respects Retry-After, a queue with a concurrency cap, and key or provider pooling.
• Under sustained overload, the only real choice is which requests wait and which fail fast. Decide that on purpose.

What actually triggers a 429

Every major provider meters two independent quantities. OpenAI publishes both RPM and TPM per model, and either one can trip. On a Tier 1 account, GPT-4o-class access is roughly 500 RPM and 200,000 TPM. Tier 2 (after about $50 spent and a week of history) moves you to roughly 5,000 RPM and 450,000 TPM. Tier 5 goes to 10,000 RPM and tens of millions of TPM. Anthropic meters requests, input tokens per minute, and output tokens per minute separately: Tier 1 is around 20,000 input TPM and 4,000 output TPM for a Sonnet-class model, doubling at Tier 2 and doubling again at Tier 3.

Two things surprise teams here. First, the token limit is per minute and measured against a rolling window, so a burst of ten large requests in the same second can exceed a limit that your one-minute average sits comfortably under. Second, output tokens count, and for Anthropic they have their own tighter ceiling. A feature that streams long completions can hit the output TPM cap while the input cap has plenty of headroom. Read the specific limit named in the 429 body before you assume which one you hit.

The provider tells you what it hit. An Anthropic 429 carries type rate_limit_error with a message that names the limit ("number of request tokens has exceeded your per-minute rate limit"). Most 429 responses also include a Retry-After header with the number of seconds to wait. That header is the single most useful piece of data in the whole exchange, and the most commonly ignored.

The four levers that keep you under the limit

Client-side rate limiting

The cheapest fix is to never send the request that would 429 in the first place. Put a token-bucket limiter in front of your provider client, sized just under your account's real RPM and TPM. Before a call, you deduct its estimated token cost from the bucket; if the bucket is empty, the call waits. This turns a hard provider error into a soft local delay you control. Size the bucket to about 80 to 90 percent of your published limit so you leave headroom for estimation error and other callers sharing the key.

Backoff that respects Retry-After

When a 429 does slip through, retry with exponential backoff and jitter, not a fixed sleep. If the response includes Retry-After, honor it exactly rather than guessing. Without it, back off on a growing schedule (for example 1s, 2s, 4s, 8s) with random jitter added to each step so a thundering herd of retries does not re-synchronize and hit the wall together. Cap the number of retries so a genuinely overloaded path fails fast instead of piling up latency.

A queue with a concurrency cap

Backoff handles the occasional spike. Sustained load needs a queue. Put LLM calls on a bounded work queue with a fixed number of concurrent workers, so the number of in-flight requests can never exceed what your tier allows. This also gives you a natural place to shed load: when the queue depth crosses a threshold, you can start rejecting low-priority work immediately instead of accepting it and timing out later. The concurrency cap is what keeps latency bounded; without it, every new request just makes the existing ones slower.

Key and provider pooling

When one account's ceiling is genuinely too low, spread the load. An AI gateway pools rate limits across multiple API keys and providers: when one key hits its limit, traffic routes to another key or another provider automatically. This is the same machinery as cost-based routing, and it pairs well with routing traffic across models to cut cost. Pooling buys real headroom, but it adds a moving part, so add it when a single account is actually the bottleneck, not preemptively.

Decide which requests wait and which fail

No amount of engineering makes an over-quota account infinite. Once you are genuinely at capacity, the only decision left is prioritization. An interactive request where a user is watching a spinner should jump the queue; a nightly bulk re-embedding job should yield. Tag every LLM call with a priority and let the queue drain high priority first. For the low-priority background work, a Retry-After delay of a few seconds is invisible; for the interactive path, it is the whole experience.

Two techniques reduce how often you reach that decision at all. Cutting token throughput directly helps: prompt caching a stable prefix lowers billed input tokens and, with it, pressure on your TPM ceiling. And moving bulk, non-urgent work off the synchronous path entirely (for example onto a batch endpoint) takes it out of the rate-limit budget your live traffic competes for.

What to monitor

You cannot tune what you cannot see. Track your 429 rate as a first-class metric, not a log line. Watch queue depth and worker saturation so you know when you are approaching the cap before customers do. Record the split between input and output tokens per minute against your tier's separate ceilings, because the one you hit first tells you which limit to raise or which traffic to shape. These signals belong on the same dashboard as the rest of your production LLM monitoring, next to tail latency and cost per request.

The order to add these in matters. Client-side rate limiting and Retry-After-aware backoff are a day-one investment for any feature that will see bursty traffic. The queue and concurrency cap come when a single spike can knock out the feature. Key or provider pooling is last, when your account tier is the real ceiling and raising it is not fast enough.

Frequently asked questions

What is the difference between RPM and TPM limits?

RPM caps how many requests you send per minute; TPM caps how many tokens (input, and often output separately) you process per minute. They are enforced independently. A feature sending many tiny requests hits RPM first; a feature sending a few very large prompts hits TPM first. A 429 can come from either, so read the error body to see which one tripped.

Should I always retry a 429?

Retry only idempotent calls, and only with exponential backoff plus jitter, honoring the Retry-After header when present. Cap the retries so an overloaded path fails fast instead of amplifying the pileup. For non-idempotent operations, or when the queue is already deep, it is better to fail fast and let the caller decide.

Does prompt caching help with rate limits?

Indirectly, yes. Caching a stable prompt prefix lowers the billed input tokens on each call, which reduces pressure on your tokens-per-minute ceiling as well as your bill. It does not change your requests-per-minute limit, so it helps most when TPM is your binding constraint.

When do I need an AI gateway?

When a single account's rate limit is genuinely your bottleneck and raising the tier is too slow. A gateway pools limits across multiple keys and providers and routes around a key that has hit its cap. Below that point, client-side limiting, backoff, and a bounded queue are simpler and usually enough.