An AI feature does not fail the way a button does. Its output shifts with the input, its latency drifts under load, its cost per call moves with prompt length, and every so often it says something it should not. That is why an A/B test on an AI feature is less an optimization loop and more a gate: the numbers you read at the end decide whether the model is safe enough, useful enough, and cheap enough to serve at full traffic. This piece is about that decision layer, which measurements actually settle a ship-or-hold call, and how to set the thresholds before the results tempt you.
Deciding whether an AI feature has earned its place
Four things have to hold at once, and they pull in different directions. The model has to behave (precision, hallucination rate, latency). Users have to get something out of it. The unit economics have to survive scale. And nothing can regress on safety or compliance. A win on user value that quietly triples cost-to-serve, or lifts task completion while raising the toxic-output rate, is not a win. The PM's job is to run one statistically sound experiment that reads all four without letting a strong number on one bury a failure on another.
Writing a hypothesis you can actually falsify
Conversion-and-retention framing is too coarse for a probabilistic system. Because the output varies, the hypothesis has to name a band of acceptable behavior rather than a single point: an expected latency range, a target hallucination reduction, a ranking-quality lift, and the confidence thresholds or error modes you will tolerate. Without those bands, you cannot tell a real effect from model noise when the results land.
Tie the model capability to a user outcome and state the mechanism. For example: if the model classifies support tickets more accurately, then resolution speed rises, because better routing removes an internal handoff. The \"because\" is what makes the result interpretable. If resolution speed does not move, you know to check whether routing accuracy actually improved or whether handoffs were never the bottleneck.
Then write down what would make you kill it. Unacceptable error types, an upper bound on hallucination rate, a cost ceiling per request, and the safety triggers that force a rollback. Deciding these while you are still neutral is the whole point; after you have seen a promising lift, every threshold suddenly looks negotiable.
Picking the two or three numbers that will decide it
Three families of metric run in parallel, and they answer different questions.
Outcome metrics capture user and business value: task completion rate, retention or engagement uplift, time saved per workflow, conversion change, output-quality ratings, support resolution time. These are what the feature is for.
Model-performance metrics sit underneath the outcome and often move independently of it: precision and recall, hallucination rate, ranking relevance, the latency distribution (not just the mean), cost per inference, confidence calibration, drift indicators. A feature can clear a product threshold while quietly failing a model one, which is why both gates exist.
Guardrail metrics catch the case where a headline number improves at the expense of something you refuse to trade: harmful-output rate, bias indicators, unsafe-response flags, user-frustration signals, infrastructure errors, compute-cost spikes. These are the metrics that define your rollback conditions, and they are allowed to veto a positive result on their own.
Sample size, power, and dirty data
AI experiments usually need larger samples than their deterministic cousins, because the effect itself varies by prompt structure, query distribution, data diversity, and the model's own confidence intervals: variance stacks on top of the usual sampling noise. Fix the minimum sample size, the power calculation, and how you will read effect size before launch, not after a peek at day-three numbers.
Run offline first. Evaluate precision and recall against a labeled set, probe hallucination behavior on golden datasets, check relevance against the current baseline, confirm the safety categories hold, and verify cost per request is viable. Only a candidate that survives all five earns live traffic, where the online test measures the thing offline cannot: real user behavior and real-world reliability.
Once traffic is flowing, most of your reliability comes from holding conditions still: consistent preprocessing, version-locked prompts, a standardized caching strategy, harmonized confidence thresholds, and stable traffic allocation. Every one of those that drifts mid-test adds variance you will later mistake for signal.
Scaling evaluation with a model as judge
Human review is the most trustworthy signal and the one that scales worst, so teams increasingly hand the first pass to a second model that grades outputs at a fraction of the cost. The mistake is to read cheap as neutral. An LLM judge carries measurable, repeatable biases: it favors whichever answer it sees first when you swap the order, rewards longer responses even when the extra length adds nothing, and scores outputs from its own model family higher than a blind human would. None of that shows up in an average; all of it tilts the comparison in a direction you did not choose.
The defensible use is as an amplifier of human judgment, not a replacement for it. Calibrate the judge against a human-labeled set, measure agreement, and only then let it triage volume, always shuffling answer order to cancel position bias and controlling for length so verbosity does not masquerade as quality. When agreement with the human panel drops below the line you set in advance, the judge goes back to calibration rather than staying in the chair because it is fast. Used that way it widens how much you can evaluate; used as an oracle, it quietly decides what \"good\" means and hides the decision inside a number.
Who signs off, and on what evidence
An AI experiment touches enough surfaces that the sign-off is cross-functional by necessity: product, data science, ML engineering, legal and compliance, data governance, and AI-focused design. The PM's role is to orchestrate them rather than serialize them into a months-long queue.
What they sign off on is a documented record: hypotheses and expected behavior ranges, offline evaluation results, the experiment metrics, guardrail thresholds, the sample-size justification, and the rollback conditions. Alongside it sits the compliance surface specific to AI: PII handling, any explainability requirement, content-risk categories, dataset provenance, and hallucination-risk exposure. Folding these into the pre-launch check is cheaper than discovering them after rollout.
The moment you decide to ship, retrain or stop
Four rules resolve most calls. Ship only when outcome metrics improve, model metrics clear their thresholds, and cost-to-serve stays viable together: AI's variable cost structure means you model the economics before, not after. Any guardrail regression (toxic output, hallucination, bias, safety) forces a rollback even when the primary metric is up; that veto is non-negotiable, which is why you set it early.
The third rule is about scale: simulate traffic growth, cost spikes, long-context queries, and multi-agent workflows before you trust today's margin, because the number that works at 5% of traffic can invert at 100%. The fourth is repeatability: a variant is shippable only if offline and online results agree, behavior stays predictable, and drift sensitivity is acceptable. When they diverge, the answer is usually retraining or an architecture change, not a wider rollout.
Why the cost and latency that pass at 5% can fail at 100%
The economics of an AI feature are not linear in traffic, and the reasons are mechanical. Cost per request is driven by tokens, so anything that lengthens prompts (retrieved context, longer conversations, retries after a low-confidence answer) raises the bill faster than call volume alone does. Caching hides this at low traffic, where a small set of common queries hits warm; as the query mix widens with scale, the cache-hit rate falls and the marginal request costs closer to its full inference price. A multi-agent design multiplies the effect, because one user action fans out into several model calls that each carry their own tokens and their own latency.
Latency has the same shape. The mean tells you almost nothing about the experience, because the users who abandon are sitting in the tail: the p95 and p99 requests that stall under load, on long contexts, or during a provider slowdown. A feature that looks responsive on average can be quietly losing its heaviest users, who are often the most valuable ones. This is why the ship decision reads the distribution and the scaled economics before trusting a margin measured on a slice of traffic: the number that clears at 5% can invert once the tail and the token growth arrive together, and by then the feature is already in front of everyone.
The workflow condensed into something you can follow
The rules only bite once they are pinned to a moment in the test: what is fixed before traffic opens, what is watched while it runs, and what is recorded when it closes.
Before traffic opens, define the user and model hypotheses, name the outcome, model, and guardrail metrics, run the offline evaluation, confirm economic feasibility, secure governance approvals, and set duration and sample size.
While it runs, watch guardrails daily, track the cost trend, verify data quality, confirm prompt versions and inference consistency, and treat any interim metric comparison as exploratory only.
When it closes, validate significance, analyze where the variance came from, check the model's behavior against your error taxonomy, simulate the economics at scale, and document the decision and its next steps.
What product managers ask before their first AI test
Why is A/B testing AI harder than testing traditional features?
Outputs vary by context, query distribution, and model state, so the same variant produces different results across users. That extra noise, plus the safety and cost dimensions, is what forces deeper evaluation and governance than a UI test needs.
Should PMs use offline or online experiments?
Both, in order. Offline tests validate model quality and safety cheaply on fixed data; online tests then measure the behavior, economics, and reliability that only real traffic reveals.
What if an AI model improves value but increases cost?
Run the cost–value trade-off across scaling scenarios. If margin collapses once traffic grows, the model is not ready, whatever the current lift says.
How do guardrail failures affect decisions?
They override. A safety or compliance regression forces a rollback even when the primary metric improves.
What skills does this take?
Enough model literacy to read performance metrics, statistical judgment, metric design, economic modeling, and the cross-functional orchestration to move all of it through sign-off.
Set the threshold before you see the numbers
The single habit that separates a reliable AI experiment from a hopeful one is deciding the bar in advance (the value lift that counts, the guardrail line that vetoes, the cost ceiling that holds at scale) and then reading the results against it rather than around it. Do that consistently and the experiment stops being a search for a story in the data and becomes the thing your organization trusts to make the call.