Sample Ratio Mismatch in A/B Tests: A Diagnostic Tree and Chi-Square Check

A test planned for 50/50 should land close to 50/50. When it drifts, the split itself is suspect, and no lift is worth reading until the drift is explained. Sample Ratio Mismatch is the alarm that assignment went wrong somewhere between bucketing and analysis. Ignore it and every downstream number inherits the same doubt.

An allocation check comes before any lift estimate

Sample Ratio Mismatch (SRM) means observed variant counts differ from planned allocation after eligibility rules. It is an integrity alarm; it says nothing about whether a product change worked. In a 50/50 test, 48.9% control and 51.1% treatment may be ordinary small-sample variation, but at scale the same split can indicate bucketing defects, lost assignments, targeting changes, or analysis joins that unevenly remove users.

Freeze the population, verify allocation, test counts, find the first divergent layer, then estimate conversion, revenue, retention, or other outcomes. A clean dashboard cannot repair a broken randomized denominator.

Expected counts are the experiment contract

A split can only be wrong relative to the number it was supposed to hit, and that number comes from the configuration-analysis contract, which specifies:

• Unit of randomization: user, account, device, session, or another stable entity.
• Eligibility rule: conditions for entering the experiment.
• Planned allocation: for example, 50/50, 90/10, or 34/33/33.
• Population timestamp: period and event state used to count assignments.

Expected count = Total eligible randomized units × Planned allocation share

With 100,000 eligible assigned users, 50/50 expects 50,000 per arm; a 90/10 rollout expects 90,000 and 10,000. Do not assume 50/50: a valid ramp can otherwise look like SRM.

Declare one population entry point. Assignment is usually the cleanest integrity population because it is closest to randomization. Exposure can be the estimand for users who saw a rendered variant, but users who never reached the page cannot disappear from an assignment-level check.

If allocation changed mid-flight, say 10% treatment Monday and 50% from Tuesday, then one period-wide ratio is wrong. Split by configuration interval, calculate expected counts in each, then combine them. Include configuration history in the diagnostic dataset.

The chi-square calculation in a worked example

The chi-square goodness-of-fit test compares observed assignment counts with the allocation contract:

X² = Σ (Observed count - Expected count)² / Expected count

For k fixed-allocation variants, degrees of freedom are k - 1; for two arms, 1. The test asks whether departure exceeds what random assignment reasonably produces. It identifies neither the defect nor the treatment effect. It belongs to the family of trust checks that Kohavi, Tang and Xu cover in "Trustworthy Online Controlled Experiments" (Cambridge University Press, 2020): checks that run before anyone reads a result.

In this synthetic assignment-level 50/50 test, 100,000 unique eligible users were assigned:

Control contributes:

(48,900 - 50,000)² / 50,000 = 24.2

Treatment contributes 24.2; total X² = 48.4 with 1 degree of freedom and p-value below 0.001, below a conventional 0.05 alert threshold. Stop outcome analysis and trace the data path; hunting for a flattering outcome cut at this point only hides the defect.

At lower volume the same percentage gap means far less. With only 50 expected users per arm, a chi-square check is more sensitive to small expected cell counts, so confirm the expected counts are adequate or use an exact or simulation-based check. Count unique randomized units, not raw event rows: duplicates create fake certainty.

Three denominators expose different failures

A significant chi-square proves the split is off but not where, so the next move is to compare three denominators in sequence:

Start with assignments. Matching assignments suggest randomization worked; later exposure divergence points to the product or instrumentation path, not allocation. If assignment and exposure match but the final table has SRM, inspect query logic and eligibility, for example a left join quietly turned into an inner join by a WHERE condition.

Within each layer, count one row per randomization unit. Joining account-level randomization to user-ID events can multiply linked users; deduplicate before variant counts and inspect duplicate identities separately.

Follow the mismatch from noise to defect

Walk the mismatch from noise to defect one question at a time:

1. Was the expected ratio calculated from active configuration for every date interval?
   • No: rebuild expected counts from configuration history and rerun.
   • Yes: continue.
2. Are counts unique at the randomization unit?
   • No: deduplicate assignments and inspect identity mappings.
   • Yes: continue.
3. Does assignment-level SRM exceed the predeclared alert threshold?
   • No: record a difference compatible with random fluctuation and proceed to exposure validation.
   • Yes: inspect randomizer, assignment persistence, target rules, traffic exclusions, and assignment logging.
4. Do assignments match while exposures mismatch?
   • Yes: inspect client/server exposure events, page errors, redirects, consent logic, flag evaluation, and version-specific rendering.
   • No: continue.
5. Do assignment and exposure match while analysis eligibility mismatches?
   • Yes: inspect joins, event-time windows, null filters, bot exclusions, geography rules, and analysis-model deduplication.
   • No: if all layers mismatch, start at the earliest divergent timestamp.
6. Did the mismatch begin at a known deployment or configuration change?
   • Yes: compare before and after using the same eligibility rule, then roll back or repair the path.
   • No: inspect identity changes, traffic routing, and missing event partitions before blaming the product.

A passing assignment check only narrows the failure surface; a failed check pauses effect estimation until explained and the valid population is rebuilt.

SQL checks for the assignment, exposure, and join layers

Each branch of that decision tree has a concrete query behind it. These PostgreSQL-style queries use generic names; replace events and timestamps with warehouse fields. Each counts distinct randomization units.

Check 1: assignment counts against configured allocation

This finds duplicate assignments and observed counts for the configured window.

WITH assigned AS (
  SELECT experiment_id, user_id, variant,
    MIN(assigned_at) AS first_assigned_at,
    COUNT(*) AS assignment_rows
  FROM experiment_assignments
  WHERE experiment_id = 'checkout-copy-v3'
    AND assigned_at >= TIMESTAMP '2026-08-01 00:00:00'
    AND assigned_at <  TIMESTAMP '2026-08-08 00:00:00'
  GROUP BY 1, 2, 3
)
SELECT variant,
  COUNT(*) AS unique_assigned_users,
  SUM(CASE WHEN assignment_rows > 1 THEN 1 ELSE 0 END) AS users_with_duplicate_rows,
  ROUND(100.0 * COUNT(*) / SUM(COUNT(*)) OVER (), 2) AS observed_share_pct
FROM assigned
GROUP BY 1
ORDER BY 1;

A user in multiple variants is a separate integrity failure. Do not let MIN(assigned_at) hide it; run:

SELECT user_id,
  COUNT(DISTINCT variant) AS assigned_variants,
  ARRAY_AGG(DISTINCT variant ORDER BY variant) AS variants_seen
FROM experiment_assignments
WHERE experiment_id = 'checkout-copy-v3'
  AND assigned_at >= TIMESTAMP '2026-08-01 00:00:00'
  AND assigned_at <  TIMESTAMP '2026-08-08 00:00:00'
GROUP BY 1
HAVING COUNT(DISTINCT variant) > 1;

Cross-variant assignment can follow a changed hashing key, deleted cookies, merged identities, or a flag service that does not persist allocation. A clean split is less reassuring when users contaminate both arms.

Check 2: exposure loss by assigned variant

Exposure should mean a genuine chance to experience the assigned version; a page load before variant evaluation is not proof.

WITH assignment AS (
  SELECT DISTINCT user_id, variant
  FROM experiment_assignments
  WHERE experiment_id = 'checkout-copy-v3'
    AND assigned_at >= TIMESTAMP '2026-08-01 00:00:00'
    AND assigned_at <  TIMESTAMP '2026-08-08 00:00:00'
), exposure AS (
  SELECT DISTINCT user_id
  FROM product_events
  WHERE event_name = 'checkout_copy_v3_exposed'
    AND event_at >= TIMESTAMP '2026-08-01 00:00:00'
    AND event_at <  TIMESTAMP '2026-08-08 00:00:00'
)
SELECT a.variant,
  COUNT(*) AS assigned_users,
  COUNT(e.user_id) AS exposed_users,
  COUNT(*) - COUNT(e.user_id) AS no_valid_exposure,
  ROUND(100.0 * COUNT(e.user_id) / NULLIF(COUNT(*), 0), 2) AS exposure_rate_pct
FROM assignment a
LEFT JOIN exposure e USING (user_id)
GROUP BY 1
ORDER BY 1;

Lower treatment exposure can indicate failed rendering, redirects, or a different event name. Higher treatment exposure can mean control lacks an equivalent event, exposure precedes assignment, or re-entry creates inconsistent logs. The difference calls for investigation: it may reflect instrumentation loss, delivery failure, or a treatment-induced change in exposure.

Check 3: eligibility joins that remove one arm

This makes join loss visible. It requires a qualifying checkout start after assignment; adapt the event but retain the left join while debugging.

WITH assignment AS (
  SELECT DISTINCT user_id, variant, assigned_at
  FROM experiment_assignments
  WHERE experiment_id = 'checkout-copy-v3'
    AND assigned_at >= TIMESTAMP '2026-08-01 00:00:00'
    AND assigned_at <  TIMESTAMP '2026-08-08 00:00:00'
), eligible AS (
  SELECT DISTINCT user_id
  FROM product_events
  WHERE event_name = 'checkout_started'
    AND event_at >= TIMESTAMP '2026-08-01 00:00:00'
    AND event_at <  TIMESTAMP '2026-08-08 00:00:00'
)
SELECT a.variant,
  COUNT(*) AS assigned_users,
  COUNT(e.user_id) AS retained_after_eligibility_join,
  COUNT(*) - COUNT(e.user_id) AS excluded_by_join,
  ROUND(100.0 * COUNT(e.user_id) / NULLIF(COUNT(*), 0), 2) AS eligibility_rate_pct
FROM assignment a
LEFT JOIN eligible e USING (user_id)
GROUP BY 1
ORDER BY 1;

A final outcome query may correctly use an inner join for a defined per-protocol analysis, but that changes the population: document it and test SRM on that denominator. If eligibility is treatment-affected, excluding non-eligible users can introduce post-treatment selection bias. Assignment-level intent-to-treat remains the safer diagnostic anchor.

Allocation failures that hide in ordinary data work

Knowing the usual shapes of an imbalance shortens the hunt. Five of them recur. Target rules drift after launch. An operating-system exclusion for an unsupported client is nested only under treatment. The imbalance begins at the rule-edit timestamp in raw assignments.

Exposure instrumentation differs by variant. Control emits checkout_viewed; treatment emits checkout_copy_v3_exposed. Querying only the new event removes control from the exposed denominator, making treatment larger and potentially more likely to convert by selecting a working event path.

Eligibility is evaluated after treatment. A variant routes users to a new payment page while analysis retains downstream page views. Treatment users are lost or counted through another event; the inclusion condition may be on the causal path to outcome.

Identity stitching is asymmetric. Logged-in users receive user-ID assignments and anonymous visitors device-ID assignments. If treatment triggers login earlier, a downstream map resolves treatment sessions more often; joined user-ID counts can show SRM despite balanced device-level assignment.

Event delays imitate missing users. Mobile events can arrive after prompt server-side assignments, so a same-day alert may disappear after the agreed lateness window. Set a completeness cutoff and rerun after late partitions settle.

The earliest divergent layer and timestamp are more useful than a long cause list. Build a timeline of configuration edits, deploys, schema changes, and pipeline incidents.

Why outcome analysis must pause

Any one of those defects poisons the result as well as the split. In a synthetic 50/50 checkout test of 100,000 users, assignments are balanced but a defective eligibility join retains 48,900 control and 51,100 treatment users. The outcome table reports 10.0% control versus 11.0% treatment purchase.

The lift is not interpretable. The extra 2,200 treatment records could be returning customers whose prior-purchase status was resolved by an identity join, while equivalent control records were dropped for a null event field. Their higher pre-test purchase propensity can produce 11.0% with no checkout-copy effect.

No outcome breakdown repairs an unresolved population defect. Device, channel, or country segments may locate the join issue but cannot validate lift. Reweighting is not automatic: it assumes the composition difference and every causal variable are known. Repair the assignment table and eligibility logic, then rerun or rebuild a population with an inclusion rule fixed before exposure. A false positive can create later trust, support, revenue, and roadmap costs.

Name the owner who can pause the readout

Those costs land well after the readout, when nobody remembers who owned the data path. Give the experiment owner authority to pause readout, the data owner responsibility for the analysis dataset, and the flag or application owner responsibility for assignment and exposure. Maintain an incident record with experiment ID, window, configuration ratio, observed counts, query version, defect hypothesis, and resolution.

The release record must state which decision population was valid and why. Store timestamped allocations and target rules, version population SQL or models, and preserve raw pre-deduplication counts. A final significance-chart screenshot is not allocation-integrity evidence.

For predictive products, place allocation checks beside model and feature monitoring. This framework for A/B testing ML models in production situates validity in the wider production decision process.

A release gate before effect estimation

The release must clear this gate before anyone estimates an effect:

• The randomization unit is named and stable across assignment, exposure, and analysis tables.
• Planned shares use timestamped configuration, including ramps.
• Assignment counts use unique entities and identify multi-variant users.
• Expected/observed counts, chi-square statistic, degrees of freedom, and alert rule are recorded for every arm.
• Assignment SRM is absent or explained by a documented valid configuration change.
• Exposure events represent actual delivery and have comparable arm definitions.
• Exposure loss is calculated by variant, not hidden in conversion denominators.
• Eligibility joins are tested as left joins before accepting an inner-join dataset.
• Late events, identity stitching, bot filters, and timezone boundaries have declared rules.
• The earliest mismatch time and layer have an owner, defect record, and remediation decision.
• Outcome analysis stays paused until the estimation population passes the integrity gate.

This is stricter than a one-line SRM p-value: the signal triggers investigation; the layer-by-layer count trail establishes whether a product decision is supportable.

Treat unresolved SRM as invalid experiment data

When the mismatch cannot be explained, the hardest part is often the conversation, because someone has already seen the tempting uplift. Report it in the terms the gate uses: planned and observed counts, the chi-square result, the earliest layer and timestamp where the ratio changed, and the defect hypothesis with its owner. Then state the decision plainly. The effect estimate is withheld, the population will be rebuilt or the test rerun, and the 11.0% versus 10.0% from the outcome table is evidence neither for nor against the change.

Calling it invalid data, rather than a failed test, keeps the product idea alive for a clean rerun. Discarding a tempting uplift costs less than shipping on an allocation defect, and an experiment that cannot vouch for its own denominator has nothing trustworthy to say about its numerator.