The same prompt can return a different answer on every call, and "different" is not automatically "worse", which is exactly what makes generative features hard to test. A conversion experiment assumes a fixed treatment: everyone in variant B sees the same screen, so any lift is attributable to it. Replace the screen with a model that improvises, and the treatment becomes a distribution of outputs whose quality is partly subjective, drifts over time, and nudges user behaviour in ways a single click-through never captures. Testing it well means pairing behavioural data with structured human judgement and watching cost per generation as closely as quality. This piece is for PMs and ML teams shipping LLM features who need an experiment that survives non-determinism.
Why a conversion test alone tells you almost nothing here
A generative feature fails a plain A/B test in five specific ways at once. Its outputs vary, so the same prompt yields different responses to different users inside one variant. Its quality is subjective (correctness, tone, creativity, and helpfulness depend on what the user was trying to do) so no single success threshold fits. Its cost and latency swing with model choice and prompt length, and have to be read next to quality rather than after it. Its effect on behaviour builds over days as trust forms or erodes, so a first-session number can mislead. And a model that looks better can quietly raise the hallucination or unsafe-output rate. A conversion-focused framework sees none of this on its own; it has to be augmented with ranking, rubric-based scoring, a multi-metric readout, and a controlled evaluation pipeline that runs before users are ever exposed.
Four kinds of generative experiment, and when each fits
Most generative tests fall into one of four shapes, and confusing them is where teams waste cycles. Prompt experiments vary tone, length, system instructions, context windows, or retrieval prompts; they are cheap and suit early product tuning. Model-version experiments (a larger model, a different architecture, a fine-tuned versus base checkpoint, a modified safety layer) change behaviour more deeply and cannot run without guardrails and close monitoring. Output-quality experiments target a systemic improvement such as stronger reasoning, fewer hallucinations, higher factual accuracy, or cleaner summarisation. AI-driven UX experiments put generated content into the product surface itself (auto-generated onboarding, dynamic UI states, personalised or conversational flows) and here behavioural metrics like activation and retention matter more than any intrinsic quality score.
The three-stage pipeline: offline, human, then live
Reliable evaluation runs in stages so that a weak candidate is filtered before it costs anything. Offline evaluation is first and cheapest: automatic metrics such as BLEU, Rouge, perplexity, and embedding similarity, run over synthetic test sets and model benchmarks. It cannot judge whether users are better off, but it removes obvious failures fast. Human evaluation comes next, because output quality is contextual and multi-dimensional: use a consistent rubric, pairwise "A versus B" ranking, safety scoring for toxicity and alignment, and task-based correctness labels. Pairwise ranking tends to be more reliable than a numeric rating, because people compare more consistently than they score in the abstract, and their agreement rate is itself a check on whether the rubric is well defined. Only then does online A/B testing expose real traffic, measuring behaviour, retention, perceived quality, and the cost and latency shifts that only production reveals.
When offline scores and live results disagree, the gap itself is diagnostic. A model that ranked well on a curated set but disappoints in production has usually met prompt phrasings it never saw, a real query distribution that differs from the benchmark, or UX friction that blunts an output that is technically correct. Reading which of those broke tells you whether to retrain, re-prompt, or fix the surface: three different responses that a single flat conversion number would leave you guessing between.
The metrics that decide it: quality, behaviour, efficiency, safety
No single number settles a generative test; four families answer questions the others cannot. Quality metrics (correctness, specificity and relevance, coherence, tone alignment, factual accuracy, and hallucination rate) are best captured through human or hybrid scoring, with pairwise ranking as the more stable instrument. Behavioural metrics show whether that quality changed what users did: activation, task completion, repeat usage, session depth, and trust signals such as how often people edit, reject, or fall back from the output. The share of outputs a user keeps without editing is a useful north-star here, and a steadier one than a periodic satisfaction survey: a response people keep is working; one they rewrite is not, however good it looks on a rubric. Efficiency metrics (cost per generation, latency per request, compute utilisation, throughput) tie directly to the product's margin and belong in the readout from the start. Safety metrics (toxicity, compliance with harmful instructions, drift on sensitive topics, policy-violation frequency) are the ones allowed to veto a positive result on their own.
Running one experiment end to end
Start with a hypothesis precise enough to be wrong: something like "Model B reduces hallucinations and lifts task success without raising cost per generation" states a direction, a magnitude you would accept, and a constraint, rather than a vague hope that the model is "better." Before any user sees it, put the guardrails in place (the safety layer, fallback prompts, rate limits, and real-time monitoring) because these are what let you run the test at all. Filter candidates offline, then have humans rank or score the survivors against the rubric, mixing pairwise preference with task-success and safety-factuality labels so a fluent-but-wrong answer is caught.
Launch the online test on stable traffic splits, hold prompt caching constant, use deterministic seeds where reproducibility matters, and segment new users from returning ones, since trust and behaviour differ sharply between them. Read the result holistically: weigh quality lift, behavioural change, cost, and safety together, because a model that improves perceived quality while doubling inference cost can erase contribution margin unless pricing or UX absorbs it.
That cost failure is easy to miss because it hides in an average. Suppose a variant adds one extra reasoning pass per request (a fraction of a cent on a well-behaved test sample) and the online numbers show a healthy quality lift at a comfortable blended cost. If a long-context segment that was a small slice of test traffic turns out to dominate a heavier real-world cohort, each of those requests costing several times the average, the cost per successful task can double at full rollout while the quality metric never flinches, because it was never the number under strain. Setting a ceiling on cost per successful task before launch, right next to the safety floor, is what turns that surprise into a guardrail. Then roll out with monitoring that keeps running: drift, distribution changes, emergent behaviour, and scaling effects surface only after launch, so measure across day, week, and month windows rather than declaring victory on day one.
Where a generative test quietly misleads you
The recurring traps are less about statistics than about what you choose to look at. Over-indexing on subjective quality scores flatters a model that people admire but do not actually use, so pair every rubric with real behavioural data. Ignoring a safety regression because the model "reads better" ships harm behind a good headline; put a safety score directly into the experiment's decision criteria. Evaluating only offline is the most common shortcut and the most expensive: automatic metrics never capture trust or downstream behaviour, so test live even at small volume. And leaving cost unmodelled means discovering the margin problem after rollout, when a long-context or high-throughput segment turns a comfortable blended cost into a loss. Treating a subjective task as if it had one objective answer belongs on the same list: it manufactures false precision that the next model quietly breaks.
Matching the method to the team you have
The right rigour depends on scale. A small team is better served by a lightweight human-evaluation loop and tests aimed at prompts and UX rather than large model swaps it cannot safely monitor. A growth-stage company can afford structured quality rubrics, a repeatable safety-evaluation workflow, and enough cost modelling to justify or reject a bigger model on economics rather than vibes. An enterprise needs governance in the loop: safety review, compliance, and observability folded into every experiment, with evaluation datasets standardised across teams so results are comparable. The pipeline is the same shape at every size; what changes is how much of it you formalise.
Questions teams raise once results land
Why can't a generative feature rely on offline metrics alone?
Offline metrics measure intrinsic quality on fixed data. They cannot see user trust, subjective satisfaction, or behavioural impact: the things that actually decide whether a generative experience works.
How many metrics should the decision use?
Enough to cover quality, behaviour, and cost and safety together. A win on one that hides a loss on another is the failure this structure exists to catch.
Do prompt changes really need A/B testing?
For user-facing output, yes: a small wording change can shift correctness or trust more than it looks, and only a controlled test tells you which way.
Decide the rubric before you run the test
The habit that separates a reliable generative experiment from a hopeful one is fixing the standard in advance (the quality rubric that counts as better, the safety floor that vetoes, and the cost ceiling that has to hold at scale) and then reading the outputs against it rather than searching them for a flattering story. Do that, and experimentation stops being a way to justify a model you already like and becomes the thing that tells you, honestly, whether it is ready to serve.