Timeouts, retries, and fallbacks that keep an AI feature up

• An LLM call is a network call to a shared, rate-limited service, so treat it like any other unreliable dependency: bound it, retry it carefully, and have a plan for when it fails.
• Set aggressive timeouts and escalate on retry, retry only transient errors with exponential backoff and jitter, and trip a circuit breaker when failures cluster.
• Design a degraded mode users can live with, and watch for the 200 OK that is quietly broken.

A retrieval-augmented feature that answers in 800 milliseconds on your laptop can time out, rate-limit, or return garbage the week you ship it to real traffic. Nothing about the model changed. What changed is that you are now calling a shared, rate-limited service over the public internet, at a concurrency your local testing never touched. Reliability for AI features is mostly the same discipline you would apply to any flaky dependency, plus one failure mode that is unique to language models.

Why a working demo still falls over in production

An LLM call is a network call. It can be slow, it can be throttled, and the provider can have a regional incident that has nothing to do with your code. Under load, the naive version of the feature, one synchronous call with no timeout and an automatic retry loop, is the worst possible design. When the provider slows down, every request piles up waiting, your worker pool exhausts, and the retries add load to a service that is already struggling. That is how a minor upstream blip becomes a full outage on your side.

The goal is not to make the model never fail. It is to contain each failure so it stays local: one slow request should not take down the feature, and one provider incident should not take down the product.

Set aggressive timeouts, then escalate

Never make an LLM call without a timeout. A common starting point is 30 seconds for a standard prompt and up to 60 seconds for a heavy reasoning task, with much tighter budgets, around 10 seconds, for internal calls between agents where a human is not waiting. Pick the number from your latency budget, not from the provider's default.

A useful refinement is to start tight and escalate. If the first attempt times out at 30 seconds, the retry can allow 60. Most calls finish well under the limit, so an aggressive initial timeout fails fast on the ones that have already gone wrong, while the longer retry still gives a genuinely slow request room to complete. For anything user-facing, pair this with token streaming so the person sees output arriving rather than a spinner, a pattern we cover in streaming LLM responses as a SaaS feature.

Retry the right errors, with jitter

Retries recover from transient failures: rate limits, brief network errors, and 503s. They must not be used on errors that will fail again the same way, such as a malformed request or an authentication failure. Retrying those just burns budget and delays the inevitable error.

When you do retry, use exponential backoff with jitter. Backoff spaces attempts out so you are not hammering a recovering service; jitter, a small random offset added to each delay, stops every client from retrying in the same instant after a shared outage. Without jitter, a provider that recovers gets hit by a synchronized wall of retries, the thundering herd, and falls over again. Rate-limit responses deserve special handling because the provider often tells you how long to wait; our guide to handling LLM 429 rate limits in production covers reading the retry-after header and shaping traffic.

Circuit breakers stop the bleeding

Retries help with brief blips. They make a sustained outage worse. If the provider is down for five minutes and you keep retrying, you generate load and tie up workers for nothing. A circuit breaker fixes this by tracking failures and opening the circuit once they cross a threshold, after which requests fail immediately instead of hanging.

The arithmetic is stark. Consider a feature making 100 requests per minute during a five-minute outage. Without a breaker, 500 to 1000 requests each hang for the full 30-second timeout, exhausting your capacity. With a breaker, the first 10 to 15 failures trip the threshold and every request after that fails fast in under 10 milliseconds, freeing workers to serve the rest of the product. After a cooldown the breaker lets a trial request through to test whether the provider has recovered. A per-tenant kill switch is the manual version of the same idea, which we describe in staged rollout and the AI feature kill switch.

Graceful degradation: a plan B users can live with

When retries are exhausted and the breaker is open, you still owe the user a response. Graceful degradation means the feature keeps working at reduced capability instead of returning a 500. The right fallback depends on the feature. A support assistant can fall back to keyword search over your help center. An AI summary can show the raw record it was going to summarize. A drafting tool can offer a template. A multi-provider setup can fail over to a secondary model, which is often the cleanest option when the outage is provider-specific.

Whatever the fallback, make it honest. Tell the user they are seeing a simpler result, and log the degradation so you can measure how often plan B fires. Routing all of this through an AI gateway in your production architecture keeps timeout, retry, breaker, and failover logic in one place instead of scattered across every call site. For a deeper treatment of fallbacks specifically, see what to do when the model fails.

Watch for the 200 OK that is actually broken

The failure mode unique to language models is the successful-looking bad response. The provider returns 200 OK, the latency is normal, and the content is hallucinated, empty, or malformed JSON. Standard HTTP error tracking sees nothing wrong. A reliability layer built only on status codes will report perfect uptime while users get nonsense.

Guard the output, not just the transport. Validate structure when you expect structured output, run cheap sanity checks such as a schema parse or a required-field check, and consider a lightweight quality signal that can trip the circuit breaker on a run of low-quality responses, not only on HTTP errors. Feed those signals into whatever you already use to monitor an LLM feature in production so a quality regression pages you the same way a latency spike would.

Frequently asked questions

What timeout should I set for an LLM call?

Start from your latency budget, not the provider default. Around 30 seconds is reasonable for a standard user-facing prompt and 60 for heavy reasoning, with much tighter limits for background or inter-agent calls. Escalate the timeout on the retry rather than setting one large value everywhere.

How many times should I retry a failed LLM request?

Two or three attempts with exponential backoff and jitter is usually enough for transient errors. Beyond that, a circuit breaker is the better tool: it stops retrying entirely once failures cluster, instead of adding load to a service that is already down.

Do I need a circuit breaker if I already have retries?

Yes. Retries handle brief, isolated failures; a circuit breaker handles sustained outages. Without one, an extended provider incident turns your retry logic into a load generator that keeps your own workers pinned and slows the rest of the product.

How do I detect a bad response that still returns 200 OK?

Validate the output. Parse structured responses against a schema, check for required fields and empty content, and track a quality signal alongside your HTTP metrics. Let that signal trip the breaker and alerting so silent quality failures are treated like any other outage.