How to A/B test an AI feature in B2B SaaS without fooling yourself

Short answer: to A/B test an AI feature in a B2B SaaS product, randomize by account rather than by user, pick one outcome metric that reflects work getting done (not clicks on the AI button), and run cost, latency, error rate and support tickets as guardrails. Plan for fewer units than you think you have, run long enough to get past the novelty spike, and decide in advance what result would make you ship, iterate or pull the feature.

Offline evals tell you whether the model output is good. They do not tell you whether shipping the feature made customers better off. That second question needs an online experiment, and AI features break several assumptions that standard A/B testing advice quietly relies on. This post covers the ones we see trip up SaaS teams most often.

Why AI features are harder to A/B test than a button change

Three things make it harder.

First, the treatment itself is not fixed. The same prompt can return different answers on different calls, the provider can update a model under the same alias, and retrieval results change as customers add documents. Your treatment group is not getting one experience; it is getting a distribution of experiences. Pin model versions for the duration of the test and log the prompt version on every call, as covered in prompt versioning for production AI features, so you know what each user actually saw.

Second, the costs are per use. A button costs nothing to click. An AI feature costs money every time it runs, so a variant that improves your outcome metric by a small amount while doubling token spend may be a net loss. Cost has to be part of the decision, not an afterthought.

Third, the obvious metric is misleading. Usage of the AI feature almost always goes up when you make it more visible. That tells you people clicked, not that their work got easier.

Randomize by account, not by user

In B2B products, users inside the same account talk to each other, share documents and see each other's work. If half of a team gets an AI summary feature and the other half does not, the control users read the summaries their colleagues pasted into Slack. Your control group is contaminated, and the measured effect shrinks toward zero.

Account-level randomization fixes contamination but cuts your sample size. A product with 40,000 active users might have only 1,200 active accounts, and that is your real number of independent units. Two consequences follow:

• You need either a larger effect or a longer test to reach a clear result. Run a power calculation on accounts, not users, before you start.
• A few very large accounts can dominate the result. Stratify assignment by account size so both arms get a similar mix, and report results with and without your top accounts.

Use the same assignment mechanism you use for staged rollouts and kill switches, so the experiment flag and the emergency off switch are the same system.

Pick metrics that reflect real work

One primary outcome metric

Choose a single metric that captures the job the feature is meant to help with, measured at the account level. Some examples:

• For an AI ticket reply drafter: median time to first response, or tickets resolved per agent per week.
• For AI search over a knowledge base: share of searches followed by a document open with no repeat search within two minutes.
• For an AI report builder: reports created and then shared or exported, per active account.

Write it down before the test starts. Choosing the metric after seeing results is the fastest way to ship a feature that did nothing.

Output-level quality signals

Track acceptance signals on the AI output itself: share of drafts sent with light edits versus heavy edits or discarded, thumbs up and down, regenerate clicks. These are diagnostic, not the decision metric, because they only exist in the treatment arm. They help you explain why the primary metric moved. Our post on turning user feedback into an eval dataset covers how to capture them cleanly.

Guardrails

Guardrails are metrics that must not get worse beyond a set threshold, whatever happens to the primary metric. For AI features we always include:

• Cost per active account per week, using per-feature cost attribution as described in LLM cost attribution per feature.
• p95 latency of the AI call and of the page it sits on.
• Error and timeout rate, including fallbacks triggered.
• Support tickets that mention the feature, and any increase in churn-risk signals for treatment accounts.

Running the test without fooling yourself

Expect a novelty spike

New AI features get tried by curious users in the first week, then usage settles. If you stop the test at day five you will measure curiosity. Run for at least two full business cycles, which for most B2B products means two to four weeks, and look at the effect week by week. A real effect holds steady or grows; a novelty effect fades.

Use pre-period data to reduce noise

Account behavior before the test is a strong predictor of behavior during it. Variance reduction methods such as CUPED adjust for each account's pre-period value and can meaningfully shrink the sample you need. Most experimentation platforms support this; if you analyze by hand, a regression with the pre-period metric as a covariate does the same job.

Do not peek and stop early

Checking the dashboard daily and stopping the moment a result looks significant inflates false positives. Either fix the duration in advance or use a sequential testing method that is designed for repeated looks.

Decide what each result means before you see it

Agree on three outcomes up front. Ship: the primary metric improves by at least the minimum effect you care about and no guardrail is breached. Iterate: the primary metric is flat but output acceptance is strong, which usually means a placement or workflow problem rather than a model problem. Pull: a guardrail is breached, or the metric is flat and acceptance is weak.

A worked example

Here is an illustrative setup for a support platform testing an AI reply drafter. The numbers are for planning, not results from a specific customer.

• Unit: account, 900 eligible accounts, stratified into small, mid and large by ticket volume, split 50/50.
• Primary metric: median first-response time per account, with the 4 weeks before launch as the CUPED covariate.
• Minimum effect worth shipping: 10 percent faster first response.
• Guardrails: AI cost under 2 dollars per account per week, p95 draft latency under 6 seconds, no rise in reopened tickets.
• Duration: 4 weeks, with a pre-agreed kill switch if error rate passes 2 percent.

If the drafter cut first-response time but reopened tickets rose, that is a quality problem hiding behind speed, and the right call is iterate, not ship. That is exactly the kind of result a plain usage chart would never show.

Offline checks still come first. An experiment is expensive, so only test variants that already pass your acceptance criteria and evals. The logging this depends on is the same you need anyway to monitor an LLM feature in production.

FAQ

How many accounts do I need to A/B test an AI feature?

It depends on the variance of your metric and the smallest effect you care about. Run a power calculation using accounts as the unit. As a rough guide, products with a few hundred active accounts can usually detect only large effects, so they should test bold changes or rely more on staged rollouts and qualitative feedback.

Can I compare two models or prompts with an A/B test?

Yes, and the same rules apply. Pin both model versions, log which one served each call, and include cost and latency as guardrails, since a stronger model that is slower and pricier can lose overall even when its answers are better.

Should the control group see nothing at all?

For a new feature, yes: the control gets the current product. For a change to an existing AI feature, the control gets the current AI behavior and the treatment gets the new one.

Is usage of the AI feature a good success metric?

No. Usage measures attention, and it rises with visibility regardless of value. Use it as a diagnostic, and make the decision on an outcome metric tied to the user's actual work.