When your AI feature has a bad day: an incident runbook

Your AI feature was fine yesterday. This morning support tickets say the summaries are wrong, and your dashboard latency looks normal. Nothing in your code changed. This is the moment a standard SRE runbook leaves you blind, because the thing that broke is not your server; it is the model, the prompt, the retrieved data, or a provider you do not control.

This is a runbook for AI feature incidents: how to know one is happening, how to find the failing layer fast, which mitigation lever to pull, and how to write a postmortem that stops the same failure from shipping twice.

Why AI features need their own runbook

A normal outage is loud: the service 500s, the pager fires, you roll back the deploy. AI incidents are often quiet. The API returns 200, latency is fine, and the output is simply worse than it was, because a provider updated a model without notice, your retrieval index drifted, or a prompt change regressed a case you were not testing. Detection and triage both need AI-specific signals, which is why the generic runbook does not fit.

Detection: know what healthy looks like

You cannot detect degradation without a baseline. Before an incident, you should already be tracking output quality on a small rolling eval set, error rates by type, latency at p95, cost per request, and the user thumbs-down rate. When any of these moves off its baseline, that is your first signal. If you have not wired this up yet, start with monitoring an LLM feature in production.

The most valuable early warning is the one users hand you for free: a spike in thumbs-down, or in a particular complaint. Route those into the same place as your metrics so a quality regression pages someone, not just a 500. The loop for capturing them is in turning user feedback into an eval dataset.

Triage: which layer is failing?

Once you know something is wrong, the job is to localize it. Work down this list in order, because the cheapest fixes are at the top.

Is it the provider?

Check the provider status page and your own error logs for a rise in 429s, 500s, or timeouts. Provider incidents are real and recurring: OpenAI had a multi-hour degradation in late 2024, and rate-limit caps have tightened without notice. If the provider is down or throttling, your lever is failover, covered below and in handling 429s in production.

Is it a silent model swap?

If you pin a moving alias like a "latest" model, the provider can shift it under you, and output drifts even though your code is unchanged. Compare current outputs against your frozen eval set. A drop with no deploy on your side points here. This is the strongest argument for pinning explicit model versions and testing every upgrade, which we cover in catching regressions when models update.

Is it your change?

Look at what you shipped in the window before the regression: a prompt edit, a new retrieval index, a changed tool schema. This is why every AI change should carry a version. If a prompt change is the culprit, you want to roll back the prompt, not the whole service, which requires versioned prompts as described in prompt versioning for production AI features.

Is it the data?

If retrieval feeds the model, a broken ingestion run, a stale index, or a document that changed shape can wreck answers while the model itself is fine. Spot-check what the retriever returned for a failing query. Bad context in means bad answer out, and no amount of prompt tuning fixes it.

Mitigation: pull the cheapest lever that works

You do not need to fix the root cause to stop the bleeding. Order your levers from least to most disruptive.

Start by rolling back the last AI change. If a prompt or index version caused it, revert to the previous version; that is a matter of seconds when your changes are versioned rather than baked into a deploy.

Next, fail over to another model. If a provider is down or a specific model regressed, route to a backup, which is the pattern in what to do when the model fails and model routing.

If nothing safe is left, flip the feature off. A kill switch that disables the feature and shows a graceful message beats serving wrong answers to every user; build it before you need it, per staged rollout and kill switches. Whatever lever you pull, tell affected users in plain language. A short in-product note that the feature is degraded and being worked on costs nothing and prevents a support pile-up.

The postmortem that stops the repeat

The incident is not over when the metric recovers. An AI postmortem needs more than a timeline; it needs the full version snapshot at the time of failure: model version, prompt version, retrieval index version, tool schema version, and the eval set version you were checking against. Without those, you cannot reproduce what happened.

Then close the loop that matters most: take the exact inputs that failed and add them to your frozen eval set. The single rule that compounds over time is that a failure which reached production once becomes a test case that blocks the next release. That is how an incident makes the feature permanently harder to break, and it is the same discipline as gating model updates in CI.

Frequently asked questions

How is an AI incident different from a normal outage?

A normal outage is loud and code-shaped: the service errors and you roll back a deploy. An AI incident is often quiet: the API returns 200 and latency is normal, but output quality drops because a model, prompt, index, or provider changed. Detection needs quality signals, not just uptime.

What should I check first when AI output gets worse?

Work top down: provider status and error rates, then whether a pinned model moved under you, then your own recent prompt or index changes, then the retrieved data. The order runs from the cheapest fix to the most involved.

Can I roll back a prompt without redeploying the app?

Only if your prompts are versioned and served independently of your code. That is the main reason to version prompts: it turns a rollback from a full deploy into a one-line config change during an incident.

What belongs in an AI feature postmortem?

A timeline, the root cause by layer, and a full version snapshot (model, prompt, index, tool schema, eval set). Most important, the failing inputs go into your eval set so the same regression cannot ship again.