Send the same prompt to a language model twice and you can get two different answers. That one fact breaks the habits a PM brings from deterministic features, where a change is isolated and read against a small, stable set of metrics.
An AI product's output is probabilistic, it shifts as data changes, and it varies with user context, so quality, safety and cost all move in production at once. Read only one of them and you can ship a model that wins on engagement and loses money on every fifth request. What follows is one loop from hypothesis to rollout call, for PMs who want a single process instead of a bag of tactics for each AI surface.
A model change moves four numbers at once
Four domains collapse into one decision, and different people usually own them. Statistical rigor sits with data science, qualitative evaluation with design and research, the cost model with finance or platform, governance with legal and policy. Run them as four sequential reviews and each can pass a variant another would block. Say a new model gains 3 points of accuracy and adds 40% to cost per request (illustrative numbers). The accuracy review signs off, the cost review sees it a month later, and nobody connects the two until the four readouts sit on one page.
So the hypothesis has to span three layers and name the link between them. At the model layer, state the change you expect: fewer hallucinations, better relevance, lower p95 latency, safer outputs. At the experience layer, say what the user sees differently: a more relevant recommendation, a shorter flow, clearer guidance. At the outcome layer, name the metric and the size, for example task completion up 2 points. Writing all three pays off when the outcome stays flat. You can then tell whether the model failed to improve or the improvement never reached the user.
Then list the failure modes you expect. AI has its own: hallucinated answers, unsafe content, off-topic responses, latency regressions, wrong predictions, and cost spikes from long prompts or runaway reasoning chains. Each becomes a guardrail later. A failure mode you forget here is a guardrail you will not have when it fires.
The test design depends on how much can go wrong
Risk decides the structure:
- Classic A/B suits a contained change, such as a new system prompt.
- A/B/C compares two candidate models against production.
- A/B with gating adds a limit on confidence, safety or capacity. Requests that fall below it go back to the old path.
- Multi-armed bandits suit high-variance personalization. They shift traffic toward the winner during the run, and you give up a clean effect estimate in return.
- For a new model family or architecture, shadow testing comes first. The candidate answers live traffic in parallel, users only ever see the production answer, and you log real behavior before anyone is exposed.
Model randomness is a traffic bill
AI tests usually need more traffic than their deterministic equivalents. Sampling temperature and retrieval variation add noise on top of ordinary user noise, and noise is paid for in sample size. A worked sketch: say baseline task completion is 40%, and you want to detect a 2-point absolute lift at a 5% significance level with 80% power. The standard approximation, n ≈ 16 × p(1-p) / δ² per arm, gives 16 × 0.24 / 0.0004 = 9,600 users per arm. Now suppose replaying the same prompts shows model randomness inflating outcome variance by 25%. The requirement rises to about 12,000 per arm, 2,400 more users in each arm before you can read anything.
Variance reduction buys part of it back. CUPED, described by Alex Deng, Ya Xu, Ron Kohavi and Toby Walker of Microsoft in a 2013 WSDM paper, adjusts each user's outcome with a pre-experiment covariate and cuts variance by a factor of 1-ρ², where ρ is the correlation between covariate and outcome. With ρ = 0.5, variance falls by a quarter and the 12,000 drops back to about 9,000 per arm. The catch: a brand-new AI surface has no pre-period behavior to use as a covariate. For a first launch, budget the full traffic.
Two metric families exist only because a model is involved
Five metric families run in parallel. The first two would not exist for a button change.
Model quality is the entry ticket for any model change: precision and recall (or F1 where both matter), relevance scores, hallucination rate weighted by severity, the pattern of false positives against false negatives, calibration of stated confidence, and the latency distribution. Read p95 and p99. A mean of 800 ms can hide a tail where one request in twenty takes six seconds, and that tail is where users give up.
Safety and guardrail metrics decide whether the test may continue at all: toxic or harmful outputs, bias indicators, privacy leaks, unsafe recommendations, brittleness on edge cases, and how often the fallback fires. One violation forces a rollback whatever the other numbers say.
Behavioral and product metrics (funnel conversion, task completion, search success, retention cohorts, satisfaction) still answer whether users are better off, and they catch downstream effects a model score never sees. Economic metrics close the loop. Cost per output and margin belong in the readout from day one, since spend swings with token count, context length, model size, retrieval load and the number of reasoning steps.
Drift can void the comparison mid-run
The fifth family, drift and stability metrics, protects the experiment itself. If the candidate's accuracy slides during a 14-day run, users in week two were served a different system from users in week one, and the pooled lift describes neither. Track distribution shift between arms, embedding drift, accuracy by day, hallucination rate on new query types and the spread of confidence scores. One practical trigger: if day-over-day accuracy on a fixed probe set moves by more than the effect you are trying to detect, stop reading the A/B and investigate with the DS/ML team before trusting any result.
Offline benchmarks fail in a predictable way
Offline evaluation checks intrinsic quality cheaply: labeled and golden-set tests, hallucination detection, relevance benchmarks, adversarial prompts, safety-classifier pre-checks and cost profiling. It de-risks the launch. It cannot see real users.
A golden set is a frozen sample of the past. One built from last quarter's logs says nothing about query types that appeared this month, and nothing about how a person reacts to an answer that is correct but phrased with too much confidence.
Online testing is where the live query distribution, edge-case behavior, trust signals, funnel movement, cost spikes and latency under real load finally appear, and where significance, confidence intervals and effect size get assessed.
When offline looked strong and online falls flat, the cause is usually on a short list: the model misread real intent, met prompt variants it never saw, hit a new data distribution, ran into UX friction or a weak explanation, or tripped a routing failure. Each sits at a different leaf, so diagnose before you roll back or retrain.
Offline strong, online flat: a decision path
Walk the divergence down one branch at a time, because the fix lives at a different leaf each time.
- Did the online model-quality metric (accuracy, relevance, hallucination rate) actually beat baseline? If no, the model never generalized to real intent or the live query distribution. Retrain on representative data before shipping and leave the surface alone.
- Quality improved but the outcome metric stayed flat, and the better output does reach users? If they do not notice or act on it, the block is UX friction or a weak explanation. Fix the surface, not the model.
- Output is good and visible but reaches the wrong requests? The gating or routing rule is misfiring. Correct the routing logic and keep the model as it is.
- One segment gained while the blended number stayed flat? Check whether the split was planned before launch and whether that cohort is large enough to carry its own estimate. If both hold, read the result per segment and ship for the cohort that gains instead of killing the variant.
Tier the metrics before the readout, not after
A mixed result needs a rule written before launch. Primary metrics carry the decision: user value, conversion, engagement, retention. Secondary metrics qualify it: precision and recall, hallucination rate, latency. Guardrails must stay green whatever happens: safety, bias, cost thresholds, compliance, drift stability. A variant ships only when all three tiers agree.
Most real decisions are trade-offs: accuracy against latency, relevance against cost, coverage against risk, personalization depth against fairness. The weighting is a business choice that a model score cannot make. In automation-heavy workflows, where an AI answer triggers an action nobody reviews, hallucinations carry the most weight. In recommendations, relevance dominates. Enterprise tools put safety and compliance first, and in low-margin products inference cost decides. Write the weights down, and the experiment stops rewarding whichever number happened to move most.
Teams that ship many small model updates a week push back on all this. Three tiers and a veto list slow every release, they argue, and most prompt tweaks carry little risk.
They are right when the change can be reverted in minutes, touches no regulated decision and leaves cost per request unchanged; there one primary metric plus automated guardrail alerts is enough. Everywhere else the tiers earn their overhead. Experimentation teams call this an overall evaluation criterion: a definition of success agreed before the experiment runs, so the readout is judged against a standard set while nobody knew the answer. The tiered rule is that criterion with the AI-specific vetoes attached.
Inference cost belongs in the primary readout
Cost is the number teams defer most often, and AI margins are fragile enough that it has to be modeled before value is declared. The drivers are token count, context-window size, model family, retrieval calls, prompt complexity, cascading model calls and throughput under concurrency. Read them as cost per output and cost per successful task. Raw spend grows with traffic and tells you nothing about the variant.
A worked example: the segment the test undersampled
The numbers here are illustrative, and they show how an average hides the failure. The production path costs $0.004 per request. The variant adds a second reasoning call worth $0.0015, so a normal request now costs $0.0055. Long-context requests (pasted contracts, full chat histories) cost six times that, $0.033 each, and made up 3% of test traffic. The blended test cost is 0.97 × $0.0055 + 0.03 × $0.033 ≈ $0.0063 per request. Half of requests end in a completed task, so cost per successful task is about $0.0127, comfortably inside budget.
Then the variant rolls out to an enterprise cohort where long-context requests are 20% of traffic. The blend becomes 0.8 × $0.0055 + 0.2 × $0.033 = $0.011 per request, and cost per successful task climbs to $0.022, up about 74%. At one million requests a month that is roughly $4,700 of extra spend the test never showed. The engagement lift was identical in both worlds. Engagement simply never measured cost.
Set cost guardrails the way you set safety ones: a ceiling on cost per request, on cost per successful task, on cost as a share of revenue, and a peak-load budget. Cross any of them and the experiment pauses even if engagement is up. Stratify the readout by context length and stress it against weekend surges, enterprise batch jobs, long-context abuse and a launch-day burst. If any stratum costs more than twice the median request (a tunable threshold), project its cost at the rollout traffic mix before you declare a winner.
Some variants should never get real traffic
However good the offline scores look, some candidates should never reach a user. Before traffic opens, the PM works with DS, legal, compliance and policy to verify content safety, confirm bias thresholds across segments, validate data provenance and check explainability where a regulation or contract requires it. Each check needs a named owner and a written pass condition, for example "no segment's refusal rate more than 2 points above the overall rate", because a review without a pass condition turns into a discussion. The record that comes out (hypotheses, evaluation criteria, risk scenarios, offline results, cost thresholds, guardrails, rollback plan) is what makes the decision auditable six months later, when someone asks why the model shipped.
The veto is real. Biased outcomes, unsafe edge cases, privacy-sensitive behavior or severe hallucinations force an immediate no-go even when primary KPIs are green, and an engagement gain that arrives with rising hallucinations cannot ship. That is why guardrails get written while everyone is still neutral about the candidate.
Ship, hold or kill: three calls and their triggers
Each call has its own trigger set, and all of it fits on one card:
- Ship when the primary KPIs improve, model metrics beat baseline, cost per successful task stays under its ceiling, no safety or bias issue appears, drift holds steady, and offline and online agree.
- Hold and retrain when drift emerges, the hallucination distribution worsens, cost turns unpredictable, relevance splits by segment, or offline and online diverge. That pattern signals instability, and a retrain on representative data often fixes it.
- Kill when guardrails fail, trust signals degrade, margin collapses, frustration rises or the model behaves inconsistently under load.
Whichever call you make, document it. The next experiment starts from this one's record.
Where a signed decision rule still breaks
A one-page decision rule signed by the PM, the DS lead and whoever owns the cost model covers most of what can go wrong: the primary metric and the lift that justifies shipping, the guardrails that veto (safety, bias, drift, cost per request), and which of the three calls each combination of results triggers. Its blind spot is the offline-strong, online-flat path described above. Nobody likes planning for a candidate they already believe in, so that branch is often left without an owner. When it shows up, the page is still signed and every guardrail is still listed, yet the argument gets settled by whichever number moved most. Before launch, write one more line on the page: who diagnoses that branch, and within how many days of the readout.