How to roll out an AI feature safely with flags and a kill switch

Most SaaS teams ship an AI feature the way they ship a settings page: merge, deploy, tell everyone. That works until the model does something you did not test for in front of a paying customer. An LLM feature fails differently than a CRUD form. The output is non-deterministic, the failure modes are open-ended, and the cost is a live meter. A staged rollout with a real kill switch is how you get the feature in front of users without betting the account on the first prompt that goes sideways.

This is the release mechanism we use when we ship an LLM feature into an existing product. It is not about the model quality. It is about controlling blast radius while you learn how the feature behaves on real traffic.

Why an AI feature needs a different rollout

A normal feature has a bounded set of states you can enumerate and test. An AI feature has an input space you cannot enumerate: every customer document, every phrasing, every edge case in their data. You will not find the bad outputs in staging because staging does not have your customers' data or their weird inputs.

Three things make LLM features risky to ship in one shot. The output can be wrong in ways that look confident and correct. The cost per request is variable and can spike from a prompt-injection loop or a runaway retrieval. And latency under real concurrency is often 3-5x what you saw in a demo with one user. A staged rollout lets you watch all three on a small slice of traffic before the whole customer base is exposed.

The four gates of a staged rollout

Treat the rollout as four gates, not a single switch. Each gate is a wider audience, and you only widen when the previous stage is clean on the metrics that matter.

Gate 1: internal only

Enable the feature for your own team and a synthetic tenant seeded with realistic data. This is where you catch the obvious breakage: prompts that error, retrieval that returns nothing, a schema that does not validate. If you have a set of acceptance evals, this is where they run against production infrastructure rather than a laptop. We wrote about turning fuzzy AI requirements into checkable tests in acceptance criteria and evals for AI features.

Gate 2: design partners

Turn it on for three to five friendly accounts who know they are early. This is the first contact with real customer data and real phrasing. You are looking for the outputs your synthetic tenant could never produce: the industry jargon, the malformed inputs, the questions your prompt never anticipated. Keep this gate for at least a few days. Volume is low, so signal comes from reading transcripts, not from dashboards.

Gate 3: percentage rollout

Now widen by percentage of tenants: 5 percent, then 25, then 50. At this scale the dashboards become the primary signal. You are watching p95 latency, error rate, refusal rate, and cost per request against a budget. Hold each step long enough to see a full business cycle of traffic, usually a day, before you widen again.

Gate 4: full plus default-on

Ship to 100 percent, then decide separately whether the feature is on by default or opt-in. Default-on is a bigger commitment than full rollout, because it changes the product for people who never asked for it. Keep the flag in the codebase even after full rollout. It is your fastest path back to zero if something regresses later.

Building the kill switch

A kill switch is not a deploy. If your only way to disable the feature is to revert and redeploy, you do not have a kill switch, you have a ten-minute outage with a code review in the middle. The switch has to flip at runtime.

Concretely, the feature reads a flag on every request. When the flag is off, the feature path is skipped entirely and the UI degrades to its pre-AI state. The important detail is what the switch turns off. A good kill switch disables the feature and shows the fallback experience, not a spinner and not an error. If your feature summarizes a ticket, the kill switch should return the raw ticket, not a broken summary card. We covered designing that degraded path in what to show when the model fails.

Test the kill switch before you need it. Flip it in production during Gate 2 and confirm the UI falls back cleanly and no error hits the user. A kill switch you have never pulled is a guess.

Per-tenant enablement and runtime config

Percentage rollout is coarse. What you actually want is to enable and disable per tenant, so you can turn the feature on for a specific account that asked for it and off for one that reported a problem, without touching anyone else. Store enablement as tenant state, not as a hardcoded list, so support can flip it without a deploy.

The same runtime config that holds the flag is where you keep the model name, the prompt version, and the token budget. Being able to change the model or roll back a prompt without a deploy is what makes a staged rollout fast. When a new model version regresses, you point one tenant back at the old one and compare, rather than rolling back the whole release. If you run one model across many customers, pair this with per-tenant limits so one account cannot burn the shared budget; we go into that in per-tenant LLM cost caps.

What to check before you widen

Do not widen on a schedule. Widen on evidence. Before each gate, confirm the current stage is clean on a short, fixed list.

• Error rate and timeout rate are flat or lower than the previous stage.
• The p95 latency is inside the budget you promised the product team, not just the average.
• Cost per request is within range, with no spikes that point at loops or oversized retrieval.
• Refusal rate and thumbs-down feedback are not climbing as the audience grows.
• You have read a sample of real transcripts from this stage, not only the metrics.

The last point matters most in the early gates. Dashboards tell you the feature is up. Reading transcripts tells you whether it is any good. Both have to pass. Ship the guardrails before the feature is public, not after; we listed the ones that matter in the guardrails to add before launch.

Frequently asked questions

How long should a staged rollout take?

For a new AI feature, plan on one to two weeks from internal to full. The design-partner gate and the first percentage step are where the real learning happens, so do not rush them. Later features on the same infrastructure move faster because the flag, kill switch, and dashboards already exist.

Do I need a feature-flag platform to do this?

No. A flag can be a column on your tenant table read on each request. A managed platform helps once you are running many flags and want percentage targeting and audit logs out of the box, but the pattern works with a database row and a config value. Start simple and add tooling when the manual version hurts.

What is the difference between a kill switch and a fallback?

The kill switch is the control that disables the feature at runtime. The fallback is what the user sees when it is off. You need both: a switch that flips instantly and a degraded experience that is usable rather than broken. A switch that leaves users staring at an error is only half a safety mechanism.

Should the AI feature be on by default after full rollout?

Treat default-on as its own decision, made after the feature has run at 100 percent opt-in and the metrics are stable. Default-on changes the product for users who never opted in, so it deserves its own review of accuracy, cost, and support load, not an automatic flip the moment rollout hits 100 percent.