Switchback Experiment Design Calculator for Block Length and Power

The calculator starts with the market’s memory

A marketplace cannot always randomize buyers, sellers, drivers, couriers, or inventory providers as isolated users. Treatment on one side can change availability, prices, matching speed, or behavior for everyone active then. This interference means a conventional user-level A/B test answers the wrong question.

A switchback assigns treatment and control by time block: the whole market, or a defined geography and operating window, receives one condition and then switches. The task becomes managing time dependence: choose block length, discard washout time where needed, represent carryover, and estimate power from time-series behavior rather than event-table rows.

Treating 500,000 sessions during 20 blocks as 500,000 independent observations is costly and wrong. Shared demand shocks, day-of-week patterns, supply reactions, and residual effects relate nearby sessions; effective evidence can be closer to independent block-level contrasts than logged sessions.

Time randomization changes the estimand

The cleanest estimand is the difference in a market outcome during eligible treatment versus eligible control time under a pre-specified random schedule. Eligibility matters: if ranking changes at 10:00 and supplier behavior settles only at 10:40, that interval is neither stable treatment nor stable control.

The outcome should reflect the interference mechanism. In ride-hailing dispatch, completed trips per eligible rider request and driver acceptance can both matter. In a two-sided retail marketplace, orders per search session can hide treatment-induced changes in seller exposure; seller conversion, cancellation rate, and fulfillment delay belong in diagnostics. Demand uplift with worse fulfillment is not a clean product win.

Randomization needs a calendar rule. Mechanical weekday alternation can confound condition with weekly seasonality. Instead, form matched block pairs and randomly assign treatment then control or control then treatment. Pairing balances time while breaking a predictable sequence. Include fixed effects for pair, day-of-week, or operating window when defined before exposure.

Formula sheet for a defensible design

This planning sheet exposes calculator inputs; no closed-form equation captures all marketplace dynamics. Fix units before entering numbers.

For early sizing, P ≈ ((z_(1-α/2) + z_power) × SD(D_p) / Δ)² gives required randomized pairs. It is valid only when SD(D_p) comes from historical paired blocks or a time-series model reflecting serial dependence. A session-level standard deviation and session count improperly pretend shared market conditions do not exist.

Show calendar cost: a 12-hour block with two-hour washout has 10 usable hours after each switch. A six-hour block with the same washout loses one-third of assigned time before power, sample quality, or operational risk are considered.

Carryover turns short blocks into biased contrasts

Short blocks create more transitions, which can balance time trends but let the market remember the prior condition. Ranking can alter seller repricing; a delivery incentive can shift driver positioning; a fraud rule can create manual-review queues that persist after removal. These effects often contaminate control immediately after treatment, pulling a contrast toward zero or, less commonly, reversing it.

Washout is a causal design claim: after W, residual influence is small enough for the estimand. One aggregate chart cannot prove it. Estimate results at multiple defensible washouts and assess whether sign, magnitude, and uncertainty remain credible.

Use a business-informed sensitivity range, not one optimistic estimate. If a ship decision becomes stop when washout moves from one hour to two, the result identifies a design dependency, not a reliable product effect. Extend only after checking whether longer blocks can isolate the measured state.

A marketplace example with three schedules

Consider a synthetic test of a new matching policy. The primary outcome is completed orders per eligible request. The team seeks a 2.5% relative lift from baseline; demand has a pronounced daily cycle, historical hourly outcomes have positive adjacent-hour correlation, and supply positioning may need one to three hours to settle after a switch.

This illustrative comparison is not a universal duration rule; it shows why a calculator needs calendar and carryover views.

A supplies frequent reversals but spends seven of 28 days in washout and assumes the first analyzed hour afterward is settled. If supply response lasts two or three hours, apparent precision is bias. C retains more measurement time but far fewer contrasts; one weather event, promotion, or service incident can dominate a daily result unless pair controls absorb it.

B is often sensible here because six usable hours follow the two-hour exclusion, but it is not a verdict. Simulate B alongside A and C using historical volatility, realistic autocorrelation, and every prior carryover path. Highest nominal power is not necessarily preferable: biased high power is still wrong.

If the policy changes subsidies, fees, or fulfillment costs, pair the readout with a unit economics metric architecture. Gross order growth can coexist with weaker contribution margin, which should be a guardrail rather than an afterthought.

Simulated power replaces session-count fiction

Simulation asks how often this exact design detects an effect under stated effect size, noise, and carryover, revealing fragile assumptions. A test may be well powered if carryover vanishes after one hour but weak or biased if it persists for three.

This compact notebook simulates paired randomization, an AR(1) market shock, treatment effect, and treatment-to-control carryover; it repeatedly estimates a block-pair contrast. Replace synthetic parameters with estimates from clean historical windows and add day-of-week terms or richer errors where data require them.

import numpy as np

rng = np.random.default_rng(41)

def simulated_power(block=8, washout=2, pairs=42, effect=0.025,
                    carry=0.015, tau=1.5, rho=0.55, sigma=0.08,
                    runs=4000):
    hits = 0
    for _ in range(runs):
        order = np.concatenate([rng.permutation([0, 1]) for _ in range(pairs)])
        z = np.repeat(order, block)
        eps = np.zeros(len(z))
        for t in range(1, len(z)):
            eps[t] = rho * eps[t - 1] + rng.normal(0, sigma)
        y = 1.0 + effect * z + eps
        keep = np.ones(len(z), dtype=bool)
        for t in range(1, len(z)):
            if z[t] != z[t - 1]:
                keep[t:t + washout] = False
                if z[t] == 0 and z[t - 1] == 1:
                    k = np.arange(min(block, len(z) - t))
                    y[t:t + len(k)] += carry * np.exp(-k / tau)
        means = [y[i*block:(i+1)*block][keep[i*block:(i+1)*block]].mean()
                 for i in range(2 * pairs)]
        d = np.array(means)[order == 1] - np.array(means)[order == 0]
        t_stat = d.mean() / (d.std(ddof=1) / np.sqrt(pairs))
        hits += abs(t_stat) > 1.96
    return hits / runs

The normal critical value is for readability. Production should use the planned analysis, such as randomization inference over paired assignments or regression with pair fixed effects and time-series-aware uncertainty, and simulate the null. A false-positive rate under zero effect materially above chosen alpha means the method is miscalibrated before power matters.

Run a grid: 4, 8, 12, and 24-hour blocks; one- to four-hour washouts; plausible rho values; and carryover amplitudes matching the mechanism. Record power, bias of Δ̂, usable calendar time, and null rejection rate. Power without bias is insufficient, while unbiased designs with no power merely postpone a decision.

Different operating states need different schedules

A mature marketplace with stable hourly volume can support shorter blocks than a thin market where a few large orders determine outcomes. The issue is not traffic alone, but whether block outcomes have stable observations and the market returns to comparable states between switches.

For real-time matching, memory may be minutes. For seller incentives, inventory positioning, or worker scheduling, it can exceed a session. Do not force switchbacks: if the induced state outlasts a practical block, a geo experiment, staggered rollout, or long-horizon policy evaluation may identify effects more honestly.

Scope matters too. A nationwide switchback may be operationally clean yet vulnerable to one incident affecting every market. Independently randomizing sufficiently separated geographies can add replicates if spillovers are limited. Separate dashboard rows do not establish independence; test shared shocks, central promotions, and common supply pools.

More calendar time can expose new failures

Longer tests improve precision only while intervention, instrumentation, and business conditions are stable. An app release, payment outage, unusual promotion, or supply-policy change can alter outcomes and carryover. Predefine exclusions for known incidents, but never remove blocks after seeing favored conditions.

Predictable schedules can cause adaptation: staff may learn the rhythm and sellers may observe recurring changes. Fixed treatment-control-treatment-control sequences invite anticipation and weak calendar balance. Randomized paired order, concealed where ethical and operationally appropriate, reduces this risk.

Guardrails matter. Completed orders can rise by accepting orders with greater cancellation risk. Median fulfillment time, cancellation rate, customer contact rate, seller acceptance, and fairness-sensitive distribution metrics can reveal the mechanism. Inspect them by block and relevant segment, not just pooled averages. An aggregate gain driven by one geography while another absorbs harm warrants a narrower release.

A credible design names its failure conditions

Treat the calculator as an assumption record, not a black box for a preferred block length. State expected settling time, outcome aggregation, assignment unit, washout rule, historical noise-estimation period, and sensitivity ranges for autocorrelation and carryover. Preserve the simulation notebook with the decision.

A mature decision can be conditional: run eight-hour blocks with two-hour washout only if simulations show acceptable null calibration, adequate power at the minimum useful effect, and little estimator bias across the stated carryover range. If not, lengthen blocks, change design, or decline to test through switchbacks.

A switchback earns trust when precision reflects what the market can forget, not how many events logging can count.