Sample Ratio Mismatch in A/B Tests: A Diagnostic Tree and Chi-Square Check

An allocation check comes before any lift estimate

Sample Ratio Mismatch (SRM) means observed variant counts differ from planned allocation after eligibility rules. It is an integrity alarm, not an outcome metric or proof a product change failed. In a 50/50 test, 48.9% control and 51.1% treatment may be ordinary small-sample variation, but at scale can indicate bucketing defects, lost assignments, targeting changes, or analysis joins that unevenly remove users.

Freeze the population, verify allocation, test counts, find the first divergent layer, then estimate conversion, revenue, retention, or other outcomes. A clean dashboard cannot repair a broken randomized denominator.

Expected counts are the experiment contract

The configuration-analysis contract specifies:

• Unit of randomization: user, account, device, session, or another stable entity.
• Eligibility rule: conditions for entering the experiment.
• Planned allocation: for example, 50/50, 90/10, or 34/33/33.
• Population timestamp: period and event state used to count assignments.

Expected count = Total eligible randomized units × Planned allocation share

With 100,000 eligible assigned users, 50/50 expects 50,000 per arm; a 90/10 rollout expects 90,000 and 10,000. Do not assume 50/50: a valid ramp can otherwise look like SRM.

Declare one population entry point. Assignment is usually the cleanest integrity population because it is closest to randomization. Exposure can be the estimand for users who saw a rendered variant, but users who never reached the page cannot disappear from an assignment-level check.

If allocation changed mid-flight—10% treatment Monday and 50% from Tuesday—one period-wide ratio is wrong. Split by configuration interval, calculate expected counts in each, then combine them. Include configuration history in the diagnostic dataset.

The chi-square calculation in a worked example

The chi-square goodness-of-fit test compares observed assignment counts with the allocation contract:

X² = Σ (Observed count - Expected count)² / Expected count

For k fixed-allocation variants, degrees of freedom are k - 1; for two arms, 1. The test asks whether departure exceeds what random assignment reasonably produces. It identifies neither the defect nor treatment effect.

In this synthetic assignment-level 50/50 test, 100,000 unique eligible users were assigned:

Control contributes:

(48,900 - 50,000)² / 50,000 = 24.2

Treatment contributes 24.2; total X² = 48.4 with 1 degree of freedom and p-value below 0.001, below a conventional 0.05 alert threshold. Stop outcome analysis and trace the data path rather than seek a flattering outcome cut.

At lower volume, the same absolute gap differs. With only 50 expected users per arm, small expected cells can make the chi-square approximation unstable; use an exact or simulation-based check. Count unique randomized units, not raw event rows: duplicates create fake certainty.

Three denominators expose different failures

Compare layers in sequence:

Start with assignments. Matching assignments suggest randomization worked; later exposure divergence points to the product or instrumentation path, not allocation. If assignment and exposure match but the final table has SRM, inspect query logic and eligibility—for example, a left join turned into an inner join by a WHERE condition.

Within each layer, count one row per randomization unit. Joining account-level randomization to user-ID events can multiply linked users; deduplicate before variant counts and inspect duplicate identities separately.

Follow the mismatch from noise to defect

1. Was the expected ratio calculated from active configuration for every date interval?
   • No: rebuild expected counts from configuration history and rerun.
   • Yes: continue.
2. Are counts unique at the randomization unit?
   • No: deduplicate assignments and inspect identity mappings.
   • Yes: continue.
3. Does assignment-level SRM exceed the predeclared alert threshold?
   • No: record a difference compatible with random fluctuation and proceed to exposure validation.
   • Yes: inspect randomizer, assignment persistence, target rules, traffic exclusions, and assignment logging.
4. Do assignments match while exposures mismatch?
   • Yes: inspect client/server exposure events, page errors, redirects, consent logic, flag evaluation, and version-specific rendering.
   • No: continue.
5. Do assignment and exposure match while analysis eligibility mismatches?
   • Yes: inspect joins, event-time windows, null filters, bot exclusions, geography rules, and analysis-model deduplication.
   • No: if all layers mismatch, start at the earliest divergent timestamp.
6. Did the mismatch begin at a known deployment or configuration change?
   • Yes: compare before and after using the same eligibility rule, then roll back or repair the path.
   • No: inspect identity changes, traffic routing, and missing event partitions before blaming the product.

A passing assignment check only narrows the failure surface; a failed check pauses effect estimation until explained and the valid population is rebuilt.

SQL checks for the assignment, exposure, and join layers

These PostgreSQL-style queries use generic names; replace events and timestamps with warehouse fields. Each counts distinct randomization units.

Check 1: assignment counts against configured allocation

This finds duplicate assignments and observed counts for the configured window.

WITH assigned AS (
  SELECT experiment_id, user_id, variant,
    MIN(assigned_at) AS first_assigned_at,
    COUNT(*) AS assignment_rows
  FROM experiment_assignments
  WHERE experiment_id = 'checkout-copy-v3'
    AND assigned_at >= TIMESTAMP '2026-08-01 00:00:00'
    AND assigned_at <  TIMESTAMP '2026-08-08 00:00:00'
  GROUP BY 1, 2, 3
)
SELECT variant,
  COUNT(*) AS unique_assigned_users,
  SUM(CASE WHEN assignment_rows > 1 THEN 1 ELSE 0 END) AS users_with_duplicate_rows,
  ROUND(100.0 * COUNT(*) / SUM(COUNT(*)) OVER (), 2) AS observed_share_pct
FROM assigned
GROUP BY 1
ORDER BY 1;

A user in multiple variants is a separate integrity failure. Do not let MIN(assigned_at) hide it; run:

SELECT user_id,
  COUNT(DISTINCT variant) AS assigned_variants,
  ARRAY_AGG(DISTINCT variant ORDER BY variant) AS variants_seen
FROM experiment_assignments
WHERE experiment_id = 'checkout-copy-v3'
  AND assigned_at >= TIMESTAMP '2026-08-01 00:00:00'
  AND assigned_at <  TIMESTAMP '2026-08-08 00:00:00'
GROUP BY 1
HAVING COUNT(DISTINCT variant) > 1;

Cross-variant assignment can follow a changed hashing key, deleted cookies, merged identities, or a flag service that does not persist allocation. A clean split is less reassuring when users contaminate both arms.

Check 2: exposure loss by assigned variant

Exposure should mean a genuine chance to experience the assigned version; a page load before variant evaluation is not proof.

WITH assignment AS (
  SELECT DISTINCT user_id, variant
  FROM experiment_assignments
  WHERE experiment_id = 'checkout-copy-v3'
    AND assigned_at >= TIMESTAMP '2026-08-01 00:00:00'
    AND assigned_at <  TIMESTAMP '2026-08-08 00:00:00'
), exposure AS (
  SELECT DISTINCT user_id
  FROM product_events
  WHERE event_name = 'checkout_copy_v3_exposed'
    AND event_at >= TIMESTAMP '2026-08-01 00:00:00'
    AND event_at <  TIMESTAMP '2026-08-08 00:00:00'
)
SELECT a.variant,
  COUNT(*) AS assigned_users,
  COUNT(e.user_id) AS exposed_users,
  COUNT(*) - COUNT(e.user_id) AS no_valid_exposure,
  ROUND(100.0 * COUNT(e.user_id) / NULLIF(COUNT(*), 0), 2) AS exposure_rate_pct
FROM assignment a
LEFT JOIN exposure e USING (user_id)
GROUP BY 1
ORDER BY 1;

Lower treatment exposure can indicate failed rendering, redirects, or a different event name. Higher treatment exposure can mean control lacks an equivalent event, exposure precedes assignment, or re-entry creates inconsistent logs. It indicates a broken path, not more engaged users.

Check 3: eligibility joins that remove one arm

This makes join loss visible. It requires a qualifying checkout start after assignment; adapt the event but retain the left join while debugging.

WITH assignment AS (
  SELECT DISTINCT user_id, variant, assigned_at
  FROM experiment_assignments
  WHERE experiment_id = 'checkout-copy-v3'
    AND assigned_at >= TIMESTAMP '2026-08-01 00:00:00'
    AND assigned_at <  TIMESTAMP '2026-08-08 00:00:00'
), eligible AS (
  SELECT DISTINCT user_id
  FROM product_events
  WHERE event_name = 'checkout_started'
    AND event_at >= TIMESTAMP '2026-08-01 00:00:00'
    AND event_at <  TIMESTAMP '2026-08-08 00:00:00'
)
SELECT a.variant,
  COUNT(*) AS assigned_users,
  COUNT(e.user_id) AS retained_after_eligibility_join,
  COUNT(*) - COUNT(e.user_id) AS excluded_by_join,
  ROUND(100.0 * COUNT(e.user_id) / NULLIF(COUNT(*), 0), 2) AS eligibility_rate_pct
FROM assignment a
LEFT JOIN eligible e USING (user_id)
GROUP BY 1
ORDER BY 1;

A final outcome query may correctly use an inner join for a defined per-protocol analysis, but that changes the population: document it and test SRM on that denominator. If eligibility is treatment-affected, excluding non-eligible users can introduce post-treatment selection bias. Assignment-level intent-to-treat remains the safer diagnostic anchor.

Allocation failures that hide in ordinary data work

Target rules drift after launch. An operating-system exclusion for an unsupported client is nested only under treatment. The imbalance begins at the rule-edit timestamp in raw assignments.

Exposure instrumentation differs by variant. Control emits checkout_viewed; treatment emits checkout_copy_v3_exposed. Querying only the new event removes control from the exposed denominator, making treatment larger and potentially more likely to convert by selecting a working event path.

Eligibility is evaluated after treatment. A variant routes users to a new payment page while analysis retains downstream page views. Treatment users are lost or counted through another event; the inclusion condition may be on the causal path to outcome.

Identity stitching is asymmetric. Logged-in users receive user-ID assignments and anonymous visitors device-ID assignments. If treatment triggers login earlier, a downstream map resolves treatment sessions more often; joined user-ID counts can show SRM despite balanced device-level assignment.

Event delays imitate missing users. Mobile events can arrive after prompt server-side assignments, so a same-day alert may disappear after the agreed lateness window. Set a completeness cutoff and rerun after late partitions settle.

The earliest divergent layer and timestamp are more useful than a long cause list. Build a timeline of configuration edits, deploys, schema changes, and pipeline incidents.

Why outcome analysis must pause

In a synthetic 50/50 checkout test of 100,000 users, assignments are balanced but a defective eligibility join retains 48,900 control and 51,100 treatment users. The outcome table reports 10.0% control versus 11.0% treatment purchase.

The lift is not interpretable. The extra 2,200 treatment records could be returning customers whose prior-purchase status was resolved by an identity join, while equivalent control records were dropped for a null event field. Their higher pre-test purchase propensity can produce 11.0% with no checkout-copy effect.

No outcome breakdown repairs an unresolved population defect. Device, channel, or country segments may locate the join issue but cannot validate lift. Reweighting is not automatic: it assumes the composition difference and every causal variable are known. Repair the assignment table and eligibility logic, then rerun or rebuild a population with an inclusion rule fixed before exposure. A false positive can create later trust, support, revenue, and roadmap costs.

Assign an owner and preserve the evidence

Give the experiment owner authority to pause readout, the data owner responsibility for the analysis dataset, and the flag or application owner responsibility for assignment and exposure. Maintain an incident record with experiment ID, window, configuration ratio, observed counts, query version, defect hypothesis, and resolution.

The release record must state which decision population was valid and why. Store timestamped allocations and target rules, version population SQL or models, and preserve raw pre-deduplication counts. A final significance-chart screenshot is not allocation-integrity evidence.

For predictive products, place allocation checks beside model and feature monitoring. This framework for A/B testing ML models in production situates validity in the wider production decision process.

A release gate before effect estimation

• The randomization unit is named and stable across assignment, exposure, and analysis tables.
• Planned shares use timestamped configuration, including ramps.
• Assignment counts use unique entities and identify multi-variant users.
• Expected/observed counts, chi-square statistic, degrees of freedom, and alert rule are recorded for every arm.
• Assignment SRM is absent or explained by a documented valid configuration change.
• Exposure events represent actual delivery and have comparable arm definitions.
• Exposure loss is calculated by variant, not hidden in conversion denominators.
• Eligibility joins are tested as left joins before accepting an inner-join dataset.
• Late events, identity stitching, bot filters, and timezone boundaries have declared rules.
• The earliest mismatch time and layer have an owner, defect record, and remediation decision.
• Outcome analysis stays paused until the estimation population passes the integrity gate.

This is stricter than a one-line SRM p-value: the signal triggers investigation; the layer-by-layer count trail establishes whether a product decision is supportable.

Treat unresolved SRM as invalid experiment data

A/B testing earns trust through comparability, not chart volume. Calculate expected counts from the real allocation schedule, test assignments, and trace the first denominator where the ratio changes.

If mismatch remains, pause effect readout, repair the data path or setup, and rebuild a valid population. Discarding tempting uplift is cheaper than treating an allocation defect as customer evidence.