A machine learning model behaves differently in production than on a benchmark. Data distributions shift, user intent varies, and the model's outputs move product behaviour, cost structure, and user trust all at once. So A/B testing a model in production is not a contest of accuracy scores. It has to read model quality, user value, operational reliability, safety, and cost-to-serve in the same experiment, because a change that looks like an upgrade on one of those can be a regression on another. This is a working framework for replacing or upgrading a model that already serves live traffic: how to compare candidates with shadow tests and gating, when offline evidence runs out, and how to make the ship-or-hold call on business impact rather than a leaderboard.
Swapping a live model without breaking what works
Model testing in production is a layered evaluation, not a single measurement. The offline score, the online user result, the guardrails, and the cost model each answer a different question, and the PM's job is to connect them into one decision rather than let the most convenient number win. An accuracy gain that raises cost per successful task, or a relevance lift that erodes trust through inconsistency, is not an upgrade, and you only see that when the four views are read together.
Why a model behaves differently the moment it goes live
A model that looks strong offline on precision, recall, F1, latency, and ROC-AUC can still behave unexpectedly once it meets real-time input, noisy distribution shifts, and variable traffic. Production testing is where real-world accuracy, hallucination rate, relevance on unseen inputs, trust and behaviour changes, and stability under load actually appear. The evaluation therefore spans several dimensions that a single metric cannot: model quality (accuracy, precision, recall, ranking relevance, error severity, latency and reliability); user behaviour (task completion, engagement depth, retention, conversion, and frustration signals); economics (inference cost, compute and memory footprint, retrieval overhead, and cost per successful task); and safety, where guardrails ensure that "better accuracy" does not arrive with harmful content, biased predictions, unsafe recommendations, or privacy violations. Cost is not a finance review after the fact: the PM models unit economics and cost scenarios as part of the experiment.
Where the offline score stops being evidence
Offline evaluation validates the model's intrinsic quality cheaply and safely. Run it against historical datasets, compute precision and recall, measure hallucinations, test ranking accuracy, profile cost, simulate edge cases, and score bias and fairness. It is fast and it de-risks the launch, but it cannot see behaviour: it tells you the model is competent, not that users are better off.
Online A/B testing is where the rest of the truth lives (real distribution shifts, genuine decision pathways, engagement and retention impact, business outcomes, operational cost-to-serve, and latency under real load) and where significance, confidence intervals, and effect sizes finally mean something. When an offline improvement fails to translate online, the gap is diagnostic rather than random: it usually points to personalisation drift, a query distribution that differs from the training data, UX friction that blunts a technically better output, or a downstream funnel effect. Naming which one broke is what tells you whether to retrain, re-prompt, or fix the surface.
Running the candidate in the dark first
Shadow testing lets a new model run on production inputs without affecting anyone. Both the baseline and the candidate receive the same traffic, only the baseline's output reaches users, and the candidate's output is logged and compared offline. That reveals stability, latency, quality distribution, hallucination patterns, and unexpected failure modes before a single user is exposed, which is why it belongs first for large architectural changes, new model families, uncertain safety behaviour, unknown cost implications, or compliance-sensitive domains. Its limit is equally clear: shadow mode cannot measure behavioural impact, UX changes, long-term retention, or funnel movement, because no user ever sees the candidate. It is a precursor to an A/B test, never a substitute for one.
Letting traffic through a slice at a time
Gating turns a rollout into a controlled release. Static gating exposes the candidate only when fixed conditions hold (metadata criteria, a matching user segment, appropriate task complexity) so that, for instance, complex queries route to a stronger model. Dynamic gating uses real-time signals instead: confidence thresholds, safety-classifier checks, model-uncertainty scores, cost thresholds, and latency tolerance windows, which contain the damage when the candidate behaves unexpectedly. Traffic gating then ramps exposure in stages (a small percentage, then more, then more again) advancing only while guardrails stay green and effect sizes trend positive. Each layer buys evidence at a bounded cost.
Setting up the comparison itself
A model test is worth exactly what the decisions made before it opens are worth, and each one constrains the next. First fix the variant structure: most tests are a baseline A against a new version B, but multiple candidates call for A/B/C, high-variance personalisation suits bandit allocation, and routing questions need contextual model-routing tests. Then state the hypothesis with its mechanism ("if the new ranking model better captures semantic relevance, then search engagement rises, because users reach relevant results earlier") and attach to it the expected quality uplift, the behaviour change, the cost range, and the safety constraints.
Only then choose the metrics, sorted into tiers so a mixed result has a rule waiting for it: primary metrics that carry the decision (conversion, engagement, task completion, quality ratings), model metrics that qualify it (precision and recall, hallucination rate, relevance, latency), guardrails that must stay green regardless (safety flags, frustration signals, error patterns, bias or fairness regressions), and economic metrics (inference cost, cost per task, compute variability). Size the test last: minimum sample, expected power, minimum detectable effect, and runtime, and expect to need more traffic than a deterministic feature would, because model variance and personalisation stack on top of ordinary sampling noise.
What a model change does downstream in the funnel
A model rarely changes one step in isolation. It can collapse irrelevant steps, accelerate task completion, filter users into new flows, or reroute discovery entirely, so the honest readout analyses the whole funnel rather than a top-level KPI that may simply have relocated a gain. A more accurate model can still confuse users, produce results too complex to act on, or add latency that harms flow. Accuracy and UX fit are separate questions, and behavioural metrics answer the second. Retention and trust are the longer test: a short-term uplift means little if errors accumulate, explanations stay unclear, or users quietly revert, so watch cohort retention after the change rather than the launch-week spike.
The relocation problem is concrete. A ranking change can lift final conversion simply by pulling forward users who would have converted anyway, while adding friction that pushes a quieter segment out three steps earlier. Read only the endpoint and it looks like a clean win; read the funnel and you see one cohort accelerated and another lost: a different decision entirely, and one the headline metric would have hidden.
What the better model costs you per prediction
A model change moves operational cost in either direction, and margins are fragile enough that cost has to be modelled before value is declared. The drivers are model size, token throughput, retrieval operations, latency scaling, concurrency load, and batch execution; read them as cost per successful task and margin, not raw spend, and simulate margin scenarios, load spikes, cost elasticity, and worst-case performance. The point is not to prove the model is expensive but to find where it becomes so: a variant that pencils out at test volume can invert under a weekend surge, a high-volume enterprise workload, a multi-step agent chain, or a long-context task that is rare in the sample and common at scale. Where the model genuinely improves relevance, automation, or personalisation, that cost becomes a revenue lever rather than a drain, but only the scenario modelling tells you which case you are in.
A worked sketch shows how the trap springs. Say the candidate adds one extra retrieval-and-rerank pass per request, a fraction of a cent on the test sample, and the online test (running on well-behaved traffic) shows a solid engagement lift at a comfortable blended cost. If long-context requests were a small slice of that sample but turn out to be a large share of a major enterprise cohort, each costing several times the average, cost per successful task can quietly double at full rollout while engagement never wavers, because engagement was never the metric under strain. Where the extra pass genuinely lifts relevance enough to raise conversion or cut downstream support load, the same cost reads as an investment rather than a leak: the number does not change, only what it buys. A cost ceiling per successful task, set before launch alongside the safety limits, is what converts that into a guardrail instead of a post-mortem.
Ship it, retrain it, or roll it back
Ship when the primary KPIs improve, guardrails hold, model metrics beat the baseline, cost-to-serve stays sustainable, no fairness or safety regression appears, and offline and online agree. Retrain when value improves but the signal is instability rather than failure: drift emerging, hallucinations spiking, cost turning unpredictable, relevance splitting by segment, or gating catching frequent fallbacks. Kill the variant when a primary or guardrail metric regresses, safety risk escalates, frustration rises, margin collapses, or the offline–online mismatch persists. Whichever way it goes, document the outcome, because the next model test starts from this one's record.
What ML teams ask about gating and rollback
Why not rely on offline evaluation alone?
Because user behaviour, input diversity, and distribution shifts cannot be simulated reliably offline. Offline evidence establishes safety and feasibility; production reveals whether the model actually helps.
What is the safest way to test a new model?
Shadow testing first, then gated exposure, then a staged A/B rollout, each step widening exposure only once the previous one holds.
How do you judge a model's economic impact?
Through cost-to-serve analysis read as cost per successful task, plus scenario modelling and margin simulation for the loads that only appear at scale.
Shadow first, gate second, compare third
Evolving a production model safely is a sequence, not a leap: run the candidate in shadow until its behaviour is known, gate it onto a controlled slice of traffic, then compare it on the metrics that actually decide (quality, behaviour, safety, and margin together). Teams that treat that as one experiment, with the decision rule written down before the results arrive, can keep improving their models without discovering the cost, the drift, or the trust problem after everyone is already on the new version.