A support-ticket classifier that is 4 points more accurate can still lose money if each call costs three times what the test reported. AI features can fail in several ways at once. The output changes with the input. Response time drifts under load. Cost per call moves with prompt length. And now and then the model says something it should not. So an A/B test on an AI feature is less about tuning and more about a gate. The hard part is deciding, before anyone sees a result, which numbers decide ship or hold and where each limit sits. Once a promising number appears, every line drawn afterwards bends toward it.
An AI feature has to clear four gates at the same time
Before any limit matters, ask a blunter question: has the feature earned live traffic at all? Four things must hold together, and they pull in different directions:
- the model behaves: it is precise, rarely makes things up and responds fast enough;
- users get something out of it;
- the money still works when everyone uses it;
- nothing gets worse on safety or compliance.
A lift in task completion that triples the cost to serve is a loss. So is one that raises the share of toxic answers. The PM's job is to run one sound experiment that reads all four, without letting a strong number on one hide a failure on another.
That starts with the hypothesis. A usual hypothesis assumes one fixed outcome that the new version either lifts or does not. An AI system gives different answers each time, so it needs a range of acceptable behavior instead: an expected range for response time, a target drop in made-up answers, a lift in ranking quality and the kinds of errors you will accept. Without those ranges you cannot tell a real effect from model noise when the results arrive.
Tie the model capability to a user outcome and state the mechanism. For example: if the model classifies support tickets more accurately, then resolution speed rises, because better routing removes an internal handoff. The "because" makes the result interpretable. If resolution speed does not move, you know where to look: either routing accuracy never improved, or handoffs were never the bottleneck. Two different fixes, and the mechanism tells you which one to try.
Then write down what would make you kill it: error types you will not accept, a maximum rate of made-up answers, a cost limit per request and the safety signals that force a rollback. The point is to decide this while you are still neutral. After a promising lift, every limit starts to look negotiable. Which leads to the next question: which of the many numbers a test produces gets a vote at all?
Two or three numbers decide; the rest only explain
The ranges in the hypothesis fall into three groups of metrics. Keep them apart, because they fail separately. A model can get more accurate while users ignore it. Users can love a feature the margin cannot carry.
Outcome metrics show value for users and the business: tasks completed, retention or engagement, time saved, conversion, ratings of answer quality, time to resolve a support ticket. This is what the feature is for.
Model metrics sit underneath and often move on their own: precision and recall, the rate of made-up answers (hallucinations), ranking relevance, the spread of response times (not just the average), cost per call, whether the model's confidence matches its real accuracy and signs that it is drifting. A feature can pass a product bar and fail a model bar. That is why both gates exist.
Guardrail metrics catch a headline number that improves at the cost of something you refuse to trade: harmful answers, signs of bias, unsafe responses, user frustration, technical errors, spikes in computing cost. They define when you roll back, and each one can veto a positive result on its own.
Listed out, that is twenty-odd numbers. With that many readings, something will have moved in the direction somebody wanted, and the test turns into an argument. The arithmetic is unforgiving:
P(at least one false win) = 1 minus 0.95^20 = 1 minus 0.36 = 0.64
Twenty independent metrics at a 5% significance level give you about a 64% chance that one of them shows a "significant" move on pure noise. So pick one outcome metric as the decision metric, one model metric that has to clear its threshold, and treat the guardrails as vetoes. Everything else is diagnostic. It explains why the decision metric moved, or failed to. It does not decide the call.
The number that surprises everyone deserves the most suspicion. On an AI test it usually turns out to be a logging change, a prompt-version mismatch or a cache artifact, and checking those three takes an hour. Do it before the result goes into a slide.
Variance, not traffic, sets the sample size
AI tests usually need more users than ordinary ones. The effect itself changes with the prompt, the kind of question and how confident the model is. That variation adds to normal random noise. Fix the minimum sample size, the power calculation and how you will judge the effect before launch, not after peeking at day-three numbers.
Here is how fast it grows. Say task completion is 40%, and you want to detect a 2-point lift at 5% significance and 80% power. The usual rule of thumb (about 16 x p x (1 minus p) divided by the squared effect) gives 16 x 0.24 / 0.0004, roughly 9,600 users per group. Now suppose the same user gets noticeably different answers on different days, so results cluster by user and type of question. A design effect of 1.5 (a plausible guess, not a measured figure) raises that to about 14,400 per group. With 3,000 eligible users a day split evenly, the test needs almost ten days instead of six and a half. Teams that plan for the first number stop early. Early stops on noisy metrics are where false wins come from.
Test offline first, on data you already have:
- check precision and recall against a labeled set;
- count made-up answers on a fixed set of reference questions;
- compare relevance with the current version;
- confirm the safety rules hold;
- check that the cost per request is affordable.
Only a version that passes all five gets live traffic. The live test then measures what offline cannot: how real users behave and how reliable the feature is in the real world.
Once traffic flows, most of your reliability comes from keeping conditions still. Process inputs the same way. Lock prompt versions. Use one caching setup. Keep the same confidence thresholds and a stable traffic split. Anything that drifts mid-test adds noise you will later mistake for a result. A prompt edited on day four, for example, splits the test group into two different products, and no analysis can put them back together.
A model judge is cheap, and biased in known directions
Every AI test eventually hits the same bottleneck: somebody has to read the answers. Human review is the most trustworthy signal and the hardest to scale. So teams hand the first pass to a second model that grades answers for a fraction of the cost. The mistake is to think cheap means neutral. An LLM judge has measurable, repeatable biases. Lianmin Zheng and colleagues described three of them in "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" (NeurIPS 2023). Position bias: it prefers whichever answer it sees first. Verbosity bias: it rewards longer answers. Self-enhancement bias: it scores its own model family higher. None of this shows in an average. All of it tilts the comparison in a direction you did not choose.
The same paper reports that a strong judge such as GPT-4 agreed with human preferences more than 80% of the time, about as often as humans agree with each other. That is the honest case for judges, and it sets the bar. A judge is good enough when it agrees with your human reviewers about as often as two reviewers agree with each other, on your own data.
So use the judge to extend human judgment, not replace it. First compare it with a human-labeled set and measure how often they agree. Only then let it sort the volume. Shuffle the order of answers in every comparison to cancel position bias. Control for length so that a long answer does not pass for a good one. A workable rule: every week, re-score a fixed sample of, say, 200 human-labeled pairs. If agreement falls more than 5 points below the level you recorded at the start, send the judge back for recalibration, however fast it is. Used this way, it lets you check far more answers. Used as an oracle, it quietly decides what "good" means and hides that choice inside a number.
Ship, retrain or stop: four rules and one signed record
Many AI tests stall at sign-off. The feature touches so many areas that several teams must approve it: product, data science, ML engineering, legal and compliance, data governance and AI design. The PM's job is to run these reviews in parallel instead of queuing them for months.
They sign off on one written record. It holds the hypotheses and expected ranges, offline results, the test metrics, guardrail limits, the reason for the sample size and the rollback conditions. Next to it go the AI-specific compliance questions: how personal data is handled, whether decisions must be explainable, which content risks apply, where the training data came from and how exposed you are to made-up answers. Checking this before launch costs a meeting. Finding it after rollout costs a rollback.
With the record in place, four rules settle most calls:
- Ship only when everything holds at once. Outcome metrics improve, model metrics clear their limits and the cost to serve stays affordable. AI costs grow with use, so model the money before launch.
- Any guardrail breach means rollback. More toxic answers, more made-up facts, more bias or a safety problem forces a rollback even if the main metric is up. This veto is not negotiable, which is why you set it early.
- Check scale. Simulate more traffic, cost spikes, long inputs and setups where one action calls several models. Do this before you trust today's margin.
- Check repeatability. A version is ready when offline and live results roughly agree, behaviour is predictable and the model copes with changing inputs. When results disagree, the answer is usually retraining or a design change. A wider rollout only makes the gap more expensive.
The rules bite only when they are pinned to moments in the test:
- Before traffic opens: define the user and model hypotheses, name the outcome, model and guardrail metrics, run the offline evaluation, confirm economic feasibility, secure governance approvals, and set duration and sample size.
- While it runs: watch guardrails daily, track the cost trend, verify data quality, confirm prompt versions and inference consistency, and treat any interim comparison as exploratory.
- When it closes: validate significance, analyze where the variance came from, check behavior against your error taxonomy, simulate the economics at scale, and document the decision and its next steps.
Cost and latency that pass at 5% can fail at 100%
Of the four rules, the scale rule is the one teams underweight. The test produces a reassuring number and nothing in the dashboard argues with it.
Yet the money side of an AI feature changes with traffic, and the reasons are mechanical. Cost per request depends on tokens, the chunks of text the model reads and writes. Anything that makes prompts longer raises the bill faster than the number of calls: extra context pulled from documents, longer conversations, retries after an unsure answer. Caching hides this at low traffic. A small set of common questions gets answered from the cache. As traffic grows and questions get more varied, fewer hit the cache, and each new request costs closer to the full price. Designs where one user action triggers several model calls multiply the effect, because each call has its own tokens and its own wait.
Response time works the same way. The average says almost nothing about the experience. The users who give up are in the slowest few percent of requests (the p95 and p99): the ones that stall under load, on long inputs or when the provider slows down. A feature that looks fast on average can be losing its heaviest users, who are often the most valuable. So the ship decision looks at p95 and p99, not the average, and asks how they would look with the longer inputs full traffic brings. Slow tails and growing token counts tend to arrive together. A slice that looked fast and cheap can turn slow and expensive in the same week, when the feature is already in front of everyone.
A worked example: the cost that clears at 5% and breaks at 100%
The figures here are invented to show the mechanism, not taken from a real system. Say a request costs about $0.010 at full inference and about $0.001 served from a warm cache. At 5% of traffic the query mix is narrow and the cache stays hot, so roughly 70% of requests hit it: blended cost is 0.70 x $0.001 + 0.30 x $0.010, about $0.0037 per request. Comfortable, and it is the number the test reports.
At 100% the mix widens, the cache-hit rate slides toward 30%, and retrieved context plus low-confidence retries lengthen prompts, so full-inference cost drifts up to roughly $0.014. The same formula now gives a different answer:
- Blended cost = hit rate x cached cost + (1 minus hit rate) x full cost
- At 5% traffic: 0.70 x $0.001 + 0.30 x $0.010 = $0.0037
- At 100% traffic: 0.30 x $0.001 + 0.70 x $0.014 = $0.0101, near 2.7x the tested figure
- Three model calls per user action: 3 x $0.0101 = about $0.030
- At 2 million requests a month: about $7,400 versus $60,000
Engagement never moved, because cost was not the metric anyone was watching. The only place the problem could have shown up before launch was a cost simulation run on the widened query mix, which is why that simulation belongs on the sign-off record and not in a post-mortem.
Small products are better served by shadow mode and staged rollouts
Teams with small user bases push back on all of this, and they have a point. A B2B product with 1,500 weekly active accounts cannot reach 14,400 users per arm in any reasonable time, and a two-month test on an AI feature measures a model that has already been replaced. For them, the argument goes, offline evaluation plus careful qualitative review beats a statistically underpowered experiment that only produces false confidence.
That objection is right more often than experimentation teams admit. If the required sample per arm exceeds what you can collect in about four weeks, a 50/50 split is the wrong tool. Two alternatives hold up better. Shadow mode runs the new model on live inputs without showing its output, so you get real latency, cost and error distributions with zero user risk. A staged rollout (1%, then 5%, then 25%) with hard guardrail stops turns the test into a safety check instead of an effect estimate, which is all a small product can honestly measure.
The decision rule is short. Enough traffic for the sample size within a month and a reversible change: run the A/B test. Not enough traffic, or a change you cannot undo for users who saw it: shadow first, then stage with vetoes, and say plainly in the sign-off record that the effect size was never measured.
The scorecard on one page
Everything above fits on a single page that the reviewers sign before traffic opens. Each line holds a number or a named person, never a sentence of intent:
- Decision metric: one outcome metric, its baseline and the smallest lift that pays for the feature. For the ticket classifier, that might be median resolution time with a bar of at least 8% faster.
- Model gate: one model metric and its floor, such as routing precision on the labeled set no lower than the rules engine it replaces.
- Vetoes: each guardrail with a hard number (hallucination rate, harmful-output rate, p99 latency). Crossing any one ends the test whatever the lift.
- Cost ceiling: cost per user action at full traffic, taken from the simulation in the previous section, not the figure the 5% slice reported.
- Sample and end date: the per-arm size with the design effect included (14,400 in the earlier example) and the day the test closes regardless of the dashboard.
- Judge check: the human-agreement level recorded at calibration and the drop that sends the judge back for recalibration.
- Rollback owner: who can pull the feature and how quickly traffic returns to control.
Reading a result against the card
At close, the card turns a long meeting into a short one, because the arguments already happened. Decision metric over its bar, model gate cleared, no veto tripped, cost under the ceiling: ship. Decision metric over its bar while the model gate is missed, or offline and online results disagree: retrain, since the lift rests on behavior nobody can predict yet. Any veto tripped, or simulated full-traffic cost above the ceiling: stop. A lift below the bar with every other line green is also a stop, and it is the line teams most want to renegotiate.
Set the threshold before you see the numbers
The card earns its keep on the test that disappoints. A 1.5-point lift against a signed bar of 2 points reads as a near miss on the day, and the temptation is to call it good enough. The card already says what happens next: the feature stops, and the team decides whether a retrain is worth another test. A bar moved once after the numbers arrive can be moved again on every test that follows.