Swapping out a model that already serves real users is the hardest model test to call. You have a working model to protect. Lab tests only tell you so much. And one swap changes accuracy, speed, cost and user trust all at once. A model that wins on a benchmark can still lose with real users. So the decision to ship or hold must rest on business results, not on a leaderboard.
The decision breaks before the model does
In a model swap, the first thing to fail is usually the decision, not the model. Lab scores, user results, safety limits and costs each answer a different question. The PM's job is to join them into one call. In practice that means one owner for the decision. It also means a written rule, agreed before launch, for which view wins when they disagree. Without that rule, the team with the most flattering number wins. Then a model that is more accurate but more expensive per completed task ships as an "upgrade".
The views disagree because they measure different things:
- Model quality (accuracy, ranking relevance, how bad the errors are, speed) shows whether the model does its job better.
- User behaviour (tasks completed, engagement, retention, conversion, signs of frustration) shows whether that improvement reaches anyone.
- Economics (the cost of each model call, compute, cost per successful task) shows whether the business can afford it.
Safety cuts across all three. Harmful content, biased predictions, unsafe recommendations and privacy leaks can all arrive together with "better accuracy". That is why the evaluation spans several dimensions that a single metric cannot.
These views pull against each other. A bigger model can win on quality simply because it spends more on compute. So cost cannot wait for a finance review after launch. The PM works out the unit economics inside the experiment, next to the quality numbers.
Lab wins often do not survive real users
A model can score well in the lab and still misbehave with live users and changing traffic. Lab scores here mean the usual accuracy measures such as precision, recall, F1 and ROC-AUC. The benchmark was built from yesterday's data at a steady load. Real use brings inputs nobody labelled, in volumes nobody tested. Real accuracy, made-up answers, relevance on new inputs and stability under load only show up after launch.
Booking.com put numbers on this in its 2019 paper "150 Successful Machine Learning Models: 6 Lessons Learned at Booking.com". The company deliberately slowed its own product down. A response about 30% slower cost roughly 0.5% of conversions. Booking.com also found that a better lab score does not always bring business gain, and named reasons any PM will recognise:
- The value has a ceiling. A better model on a problem that is already solved adds nothing.
- The model chases a stand-in goal, such as clicks, instead of what the business needs.
- A model that guesses users too well can feel creepy (the "uncanny valley" effect).
Each reason explains a specific gap between lab and live numbers.
The benchmark still has a job, as long as nobody treats it as the verdict. Lab tests check the model's quality cheaply. You run it on past data, measure accuracy and made-up answers, test ranking, estimate cost, try rare cases and check for bias. This tells you the model is competent. It cannot tell you users are better off.
A live A/B test covers the rest. It shows how real users behave, how engagement and retention change, and what the model costs and how fast it is under real load. Only here do significance, confidence intervals and effect sizes mean something. When a lab gain disappears in the live test, the gap tells you where to look. The usual causes are:
- personalisation that drifted away from what users want now;
- a query distribution that differs from the training data, meaning real users ask different things than the model learned from;
- a clumsy interface that hides a better answer;
- a side effect further down the funnel.
The second cause has a name, training-serving skew. Its cure is dull but reliable: log the data exactly as the live model saw it, and train on those logs. Knowing which cause broke tells you what to do next: retrain the model, change the prompt or fix the screen.
Run it in the shadows first, then open the doors in small steps
In a shadow test, the new model gets real requests, but users never see its answers. The old and new models receive the same traffic. Only the old model's answer reaches the user. The new model's answer is logged and compared. You still need safety controls on data access and system load, because the new model handles real requests even though its answers go nowhere. The shadow test shows stability, speed, answer quality, made-up answers and odd failures before any user is exposed. Start here for a new type of model, a big redesign, unclear safety, unknown cost or a sensitive area such as finance or health.
Its limit is just as clear. A shadow test cannot show how users react, because no user ever sees the new model. It cannot measure retention or funnel changes. It comes before the A/B test and never replaces it.
Next come gates: rules that decide who sees the new model. There are three kinds:
- Fixed gates show the new model only when set conditions hold, for example a certain user segment or a hard question. That way, complex questions can go to a stronger model.
- Live gates check each request as it comes: is the model confident enough, did the safety filter pass it, is it within the cost and speed limits.
- Traffic gates raise the share of users step by step. You move to the next step only while the safety limits hold and the effect looks positive.
Each gate buys you evidence at a cost you can control.
The rollout as a checklist
The steps run in order. The percentages are a common starting point, and teams adjust them. Each step opens only when the previous check passes.
- Shadow, 0% of users. Copy live traffic to the new model, show users only the old one, and log the new one. To move on: the slowest 5% of responses (p95) stay within the speed budget, answer quality and the rate of made-up answers are at least as good as the old model's, and the cost per request stays under its limit. If it fails here, the model goes back to training, not to users.
- Small slice, about 1-5%. Show the new model behind live gates: a confidence threshold, a safety filter, and cost and speed limits. To move on: the safety limits hold for the agreed period and the effect looks positive.
- Ramp, 5 to 25 to 50 to 100%. Widen one step at a time. Move on only while the safety limits hold and the cost per successful task stays under its cap.
- Rollback triggers, set before launch. Any safety limit broken, cost per successful task over the cap, the slowest 1% of responses (p99) too slow, or a lasting gap between lab and live results sends traffic back to the last safe step.
Each check needs a number, a minimum waiting time and a person who can pull the switch. Here is how it might look for a swap of a search-ranking model. The team sets these numbers for its own product; no standard prescribes them.
- Leaving shadow: at least seven days, so the model sees a full week, including the busiest day. The slowest 5% of responses are no more than 10% slower than the old model's. People rate a sample of 500 logged answers, and they are no worse than the old model's.
- Leaving the small slice: 72 hours with no safety limit broken. The safety filter steps in on fewer than 2% of requests. If it steps in more often, the filter is doing the work the model should do.
- Each ramp step: at least 48 hours, started on a weekday, so the people who can roll back are at work.
- Rollback: a switch that sends all traffic back to the old model within minutes. The old model stays running until the new one has handled 100% of traffic for two weeks.
Teams skip that last line most often. Turning off the old model the day the ramp ends saves a little compute. But then every later rollback means a full redeploy.
Set up the comparison before you switch anything on
A model test is only as good as the choices made before it starts, and each choice shapes the next. First decide what you compare. Most tests compare the current model A with a new version B. If you have several candidates, you run A/B/C. If you want traffic to shift towards the winner during the test, you can use a "bandit" setup that does this automatically. If the question is which model should handle which request, you test the routing rules.
Then write the hypothesis and say why it should work. For example: "If the new ranking model understands what people mean better, search engagement will rise, because users find relevant results sooner." Add the expected quality gain, the change in behaviour, the cost range and the safety limits. A hypothesis without a "because" gives you a result you cannot explain when it disappoints.
Only then choose the metrics. Sort them into groups, so a mixed result already has a rule waiting for it:
- Main metrics make the decision: conversion, tasks completed, quality ratings.
- Model metrics explain it: accuracy, made-up answers, relevance, speed.
- Safety limits must hold no matter what: safety flags, signs of frustration, new bias.
- Cost metrics put a price on it: the cost of model calls, cost per task.
Size the test last: the minimum sample, expected power, minimum detectable effect (the smallest change the test can reliably spot), and how long it must run. Expect to need more traffic than for an ordinary feature. A model's answers vary, and personalisation adds even more noise on top of normal chance. Booking.com's speed finding shows the scale. The usual rule of thumb for sample size at 80% power is:
n per arm = 16 x p x (1 minus p) / d², where p is the baseline rate and d the absolute effect
Take a 3% baseline conversion and a 0.5% relative change (d = 0.00015): 16 x 0.03 x 0.97 / 0.0000000225 gives about 20 million sessions per group. Few products have that much traffic. That is why speed belongs among the safety limits with a hard ceiling. You will not measure its cost precisely, so you cap it.
A model can move a gain around without creating one
A model rarely changes just one step. It can remove useless steps, speed up tasks, send users down new paths or change what they discover. So an honest readout looks at the whole funnel. The top number may only show a gain that moved from one group of users to another.
Here is how that can look. Say a site has 100,000 sessions. 40,000 come from new users, who convert at 4% (1,600 orders). 60,000 come from returning users, who convert at about 7.3% (4,400 orders). That is 6,000 orders in total. The new ranking model serves returning users better: they rise to 8.5%, or 5,100 orders. New users now see results tuned to habits they do not have yet, and they slip to 3%, or 1,200 orders. The total is 6,300 orders, a clean 5% lift on the dashboard.
Split by group, the picture changes. The product gained 700 orders from people who were already loyal. It lost 400 from the group that becomes next year's loyal customers. If new users who had a weak first visit come back less often, this "win" shrinks the funnel that produces future winners. The right call is to show the new model only to returning users and keep the old one for new users. The headline number would have hidden that.
Retention and trust take longer to test. A short-term lift means little if errors pile up, explanations stay unclear or users drift back to old habits. Watch retention by user group for several weeks after the change, not just the spike in launch week.
Cost per successful task decides more launches than accuracy
Every gain above has a price per answer. That price is the number most likely to change between the test and the full launch. It depends on model size, the amount of text processed, extra lookups, speed and how many requests run at once. Look at it as cost per successful task and as margin. Then simulate traffic spikes and the worst case. The goal is to find where the model gets expensive. A model that pays off at test volume can lose money during a weekend rush, a big business client, a chain of several model calls or a long-document request that is rare in the test and common at scale.
A worked example shows how the trap works. Say the new model adds one extra search-and-sort step. A short request now costs $0.005, and a long-document request costs $0.030. In the test, long requests are 5% of traffic, so the average cost is 0.95 x $0.005 + 0.05 x $0.030, about $0.0063. The number that matters divides that by the success rate:
Cost per successful task = blended cost per request / share of requests that end in a successful task
If 55% of requests end in a successful task, that is $0.0063 / 0.55, about $0.011 per success. Cheap enough. For now.
After launch, a large business client arrives, and long requests make up 30% of its traffic. The average cost becomes 0.70 x $0.005 + 0.30 x $0.030, or $0.0125. Cost per successful task rises to about $0.023. It has doubled, and engagement has not moved, because engagement does not care what a request costs.
The same cost can still be a good deal. If the extra step makes results relevant enough to lift conversion or cut support tickets, the extra cent per success buys revenue. The number stays the same; what it buys changes. Set a cost ceiling per successful task before launch, next to the safety limits. Then it works as an alarm that goes off before launch, not after.
When you can skip the full A/B test
Teams that retrain often have a fair complaint about all this. Say a fraud or ranking model is refreshed every week on new data, with the same design and the same inputs. A full A/B test for each refresh would take longer than a week. By the time the result comes in, the tested model is two versions old. A shadow test plus a gated ramp, they argue, gives most of the protection with much less delay.
In that case they are right. A routine refresh can go straight from shadow to a step-by-step ramp with hard safety limits, if it keeps the same type of model, inputs and goal, and its lab scores stay within an agreed range of the current model. It gives up a measured effect, but a refresh was never meant to produce one.
The exception ends when the change is real: a new type of model, a new goal, new inputs or a new screen for users. Those get the full comparison, because that is exactly where lab gains and business gains part ways.
Ship it, retrain it, or roll it back
Ship when the main metrics improve, the safety limits hold, the model beats the old one, the cost stays affordable, there is no new bias or safety problem, and lab and live results agree.
Retrain when value goes up but the model is unstable. Signs of that: results drift, made-up answers spike, cost becomes hard to predict, relevance differs a lot between user groups, or the gates often have to fall back to the old model.
Roll back when a main metric or a safety limit gets worse, safety risk grows, frustration rises, margin collapses or the gap between lab and live results does not go away.
Whatever you decide, write down the result together with the gap between the lab and live numbers. After a dozen launches, that log becomes your own version of Booking.com's lesson. It shows how much lab improvement your product needs before users notice anything.
Shadow first, gates second, comparison third
For the next model swap on your roadmap, ask which step it can skip. A routine refresh can skip the full comparison. A new type of model, or one with unknown costs, skips nothing. It stays in shadow until you know its speed and how often it makes things up. Then it moves to a small gated slice with rollback triggers already written. Only after that does it face the full comparison on quality, behaviour, safety and margin. A team that jumps straight to a 50/50 split saves two weeks. It usually spends them later, explaining a cost or trust problem that appeared after everyone was on the new model.