The same question put to a model ten times can come back with ten different answers, several of them fine. "Different" is not automatically "worse", and that is what makes generative features hard to test. A conversion experiment assumes a fixed treatment: everyone in variant B sees the same screen, so any lift belongs to it. Swap the screen for a model that improvises and the treatment becomes a distribution of outputs, partly subjective in quality, drifting over time, and costing a different amount on every call. Get the design wrong and you ship a model your raters admire and your users quietly edit into something else.
A conversion test assumes a treatment that holds still
A generative feature breaks a plain A/B test in five ways at once:
- Outputs vary. Users in the same variant get different answers to the same prompt.
- Quality is subjective. Correctness, tone, creativity and helpfulness depend on what the user was trying to do, so no single pass mark fits.
- Cost and speed swing with the model and the prompt length. Read them next to quality, not after it.
- Behaviour changes over days as trust grows or fades, so a first-session number can mislead.
- Better-sounding can mean riskier. A model that reads better can quietly make up more facts (hallucinate) or say more unsafe things.
A conversion-focused framework sees none of this on its own. It needs ranking, rubric-based scoring, a multi-metric readout, and an evaluation pipeline that runs before any user is exposed. The rest of this piece builds that pipeline and puts numbers on the two places teams most often guess: how many human judgements a comparison needs, and what an improvement really costs.
Four kinds of generative experiment, four levels of risk
Most generative tests take one of four shapes, and mixing them up wastes time:
- Prompt experiments change tone, length, system instructions, how much context the model sees or how it searches documents. They are cheap, can be undone in minutes and suit early tuning.
- Model-version experiments swap in a larger model, a different architecture, a fine-tuned version or a new safety layer. They change behaviour more deeply and need guardrails and close monitoring.
- Output-quality experiments aim at a broad improvement: better reasoning, fewer made-up facts, more accuracy, cleaner summaries.
- AI-driven UX experiments put generated content into the product itself, such as generated onboarding, changing screens or chat-style flows. Here activation and retention matter more than any quality score.
The shape sets how much evidence you need. A prompt tweak in an internal drafting tool can go live behind a feature flag after a short offline check. A model swap in a customer-facing assistant should pass all three gates below, with a safety floor written down first.
Run three gates before the result means anything
None of the four should meet live users first. Use three gates, in order:
- Offline evaluation. The cheapest gate: automatic metrics on synthetic test sets and benchmarks. It cannot tell whether users are better off, but it removes obvious failures in hours.
- Human evaluation. Output quality depends on context and has many sides. Use one consistent scoring guide (a rubric), pairwise "A versus B" ranking, safety checks for toxic or off-policy answers and labels for whether each task was done correctly.
- Online A/B test. Only now does real traffic see the candidate. It measures behaviour, retention, perceived quality and the cost and speed changes that only production reveals.
What the offline metrics actually measure, and what they miss
- BLEU: n-gram overlap with a reference answer, introduced by Papineni and colleagues at IBM in 2002 for machine translation. It rewards wording that matches the reference, so it punishes a correct answer phrased differently and cannot see factual accuracy.
- ROUGE: recall-oriented overlap, published by Chin-Yew Lin in 2004 and common for summarisation. It shows how much reference content is echoed, not whether the summary is faithful or useful.
- Perplexity: how well a model predicts held-out text. A fluency and confidence proxy, not a measure of correctness or safety.
- Embedding similarity: semantic closeness between output and reference vectors. It catches paraphrase that overlap metrics miss, but still cannot judge truthfulness, tone-fit or user trust.
Each is a cheap filter for pruning a clearly broken candidate before it reaches a human reviewer. None of them stands in for the behavioural and safety readouts that only human and live evaluation provide, because all four measure distance to a fixed text, and users do not grade against a fixed text.
When offline and live disagree, the gap is the diagnosis
A model that ranked well on a curated set but disappoints in production has usually hit one of three things. It met prompt phrasings it never saw. The real query distribution differs from the benchmark. Or UX friction blunts an output that is technically correct. Each points to a different response: retrain, re-prompt, or fix the surface. A single flat conversion number leaves you guessing between them, so log enough context per request (prompt length band, intent class, whether the output was edited) to tell them apart afterwards.
Pairwise preference needs fewer judgements than teams fear
Pairwise ranking tends to beat numeric rating, because people compare more consistently than they score in the abstract. Their agreement rate is also a check on the rubric: if two trained raters disagree on a third of pairs, the rubric is unclear, not the raters. The practical question is how many comparisons to buy.
A worked example. Say you want to detect whether the new model wins 55% of head-to-head comparisons against the old one, where 50% means no difference, at a 5% significance level with 80% power. The usual approximation is n ≈ (1.96 + 0.84)² × 0.25 / 0.05², which comes to 7.84 × 0.25 / 0.0025 = 784 decisive comparisons. If raters call 30% of pairs a tie and you drop ties, you need about 784 / 0.7 = 1,120 judgements. Want to detect a 52% win rate instead? The denominator shrinks to 0.02², and the requirement grows more than sixfold, to about 4,900. Small quality differences are expensive to confirm by hand, which is the honest argument for an LLM judge.
That judge brings its own failure. Zheng and colleagues, in their 2023 MT-Bench and Chatbot Arena study, found strong model judges agreeing with human preferences about as often as humans agree with each other, while showing stable position and verbosity biases. Randomise the order of each pair, run both orders, and calibrate the judge on a few hundred human-labelled pairs before you let it replace raters.
Four metric families, and one north star worth tracking
No single number settles a generative test. Use four families of metrics:
- Quality: correctness, relevance, coherence, tone, factual accuracy, rate of made-up answers. Human or mixed human-and-model scoring captures these best, and pairwise ranking is the steadier tool.
- Behaviour: did quality change what users did? Activation, task completion, repeat use, session depth and trust signals, such as how often people edit, reject or abandon the output.
- Efficiency: cost per generation, response time, compute use and throughput. These tie straight to margin.
- Safety: toxic answers, following harmful instructions, drift on sensitive topics, policy violations. This is the only family that can veto a positive result on its own.
How many metrics should the decision use? Enough to cover quality, behaviour, cost and safety together, and few enough that each has a named threshold. A win on one that hides a loss on another is the failure this structure exists to catch.
The share of outputs a user keeps without editing is a useful north star, and steadier than a periodic satisfaction survey. A kept output usually signals usefulness. Some edits are part of the task, though: a user adding their own figures to a drafted email has not rejected the draft. So read the edit rate per use case, and count an edit as a failure only where the task expects a finished output, such as a reply suggestion or a product description.
Price the output users keep, not the generation
Cost per generation is the number on the invoice. Cost per kept output is the number that decides whether an upgrade pays. Say the current model costs $0.0020 per generation and users keep 50% of outputs without editing, so each kept output costs $0.0040. The candidate costs 60% more, $0.0032 per generation, and lifts the kept rate to 62%. Its cost per kept output is $0.0032 / 0.62 ≈ $0.0052, an increase of about 29%, half the headline figure.
At 400,000 generations a month, the candidate costs $1,280 against $800, an extra $480. It also produces 248,000 kept outputs instead of 200,000, so the extra money buys 48,000 additional usable outputs at one cent each. Whether one cent is cheap depends on what a kept output saves the user or earns the business, and that comparison belongs in the readout, not in a finance review three months later.
Averages still hide the tail. Test traffic rarely carries the same prompt-length mix as production, and long prompts cost several times a short one. Stratify cost by prompt-length band, then reweight the test figures to the production mix before you read them. If the heaviest band is a larger share of production than of the test, your blended cost per kept output is optimistic, and a ceiling on it, agreed before launch next to the safety floor, is what turns that surprise into a guardrail.
Running one experiment end to end
Start with a hypothesis precise enough to be wrong. "Model B reduces hallucinations and lifts task success without raising cost per kept output" states a direction and a constraint. Add the size of improvement you would accept, say a 5-point gain in kept rate, so that "better" has a number attached.
Before any user sees the candidate, put the guardrails in place: the safety layer, fallback prompts, rate limits and real-time monitoring. They are what let you run the test at all. Filter candidates offline, then have humans rank or score the survivors against the rubric, mixing pairwise preference with task-success and safety-factuality labels so a fluent but wrong answer is caught.
Run the online test with a stable traffic split. Keep prompt caching the same for both groups. Use fixed seeds where you need repeatable outputs. Look at new and returning users separately, because their trust and behaviour differ a lot. Then read the result as one picture: quality, behaviour, cost and safety together. A model that users like more but that doubles the cost per call can wipe out the margin, unless pricing or the product design absorbs it. Keep monitoring after rollout. Drift, shifts in what users ask, unexpected behaviour and scaling effects show up only later. Compare day, week and month windows before you declare victory.
How a generative test misleads without anyone noticing
The usual traps are less about statistics than about what you choose to look at:
- Trusting quality scores alone. They flatter a model people admire but do not use. Pair every rubric with behaviour data.
- Ignoring a safety drop because the model "reads better". That ships harm behind a good headline. Put a safety score into the decision rules.
- Testing only offline. The most common shortcut and the most expensive. Automatic metrics never capture trust or what users do next.
- Leaving cost out. You find the margin problem after rollout.
- Treating a subjective task as if it had one right answer. That creates false precision, and the next model quietly breaks it.
Small teams push back on all of this. A lightweight human-evaluation loop and a feature flag, they argue, beat a three-gate pipeline they cannot staff. For prompt and UX tests on low-risk surfaces, they are right: a few hundred pairwise judgements plus a live kept-rate readout is enough.
Do prompt changes still need A/B testing? For user-facing output, yes, because a small wording change can shift correctness or trust more than it looks, and only a controlled test shows the direction. What a small team should not do is swap the underlying model on a customer-facing surface without the safety floor and cost ceiling in writing.
A growth-stage company can add structured rubrics, a repeatable safety workflow and enough cost modelling to justify a bigger model on economics. An enterprise folds safety review, compliance and observability into every experiment and standardises evaluation datasets across teams so results are comparable. The pipeline keeps the same shape at every size; what changes is how much of it is formalised.
Fix the rubric before the first comparison
Generative experiments that hold up share one discipline: the standard is fixed in advance (the quality rubric that counts as better, the safety floor that vetoes, the cost ceiling per kept output that has to hold at scale), and the outputs are read against it instead of being searched for a flattering story.
The reason has less to do with statistics than with attachment. By launch day most teams already have a favourite model or prompt, and generative output varies enough that a motivated reader can find support for almost any conclusion in it. When the first real conversation about thresholds happens in front of the dashboard, the ship, hold or rollback call drifts toward that favourite, and nobody has to bend a single number for it to happen. Write the three thresholds into the test plan before the first comparison is collected.