Switchback Experiment Design Calculator for Block Length and Power

In a marketplace, treating one rider changes what every other rider sees. A user-level A/B test there quietly answers the wrong question. Switchbacks flip the whole market between conditions in time blocks, and their evidence is counted in blocks: a 28-day test with eight-hour blocks yields about 83 switches, not hundreds of thousands of sessions. The design below works through block length, washout and power.

The calculator starts with the market’s memory

A marketplace cannot always randomize buyers, sellers, drivers, couriers, or inventory providers as isolated users. Treatment on one side can change availability, prices, matching speed, or behavior for everyone active then. Randomize users here and the control group is itself contaminated by the treatment.

A switchback assigns treatment and control by time block: the whole market, or a defined geography and operating window, receives one condition and then switches. The task becomes managing time dependence: choose block length, discard washout time where needed, represent carryover, and estimate power from time-series behavior rather than event-table rows.

Treating 500,000 sessions during 20 blocks as 500,000 independent observations is costly and wrong. Shared demand shocks, day-of-week patterns, supply reactions, and residual effects relate nearby sessions; effective evidence can be closer to independent block-level contrasts than logged sessions.

Time randomization changes the estimand

Because the unit of evidence is a block rather than a session, the quantity you are actually estimating shifts with it. The cleanest estimand is the difference in a market outcome during eligible treatment versus eligible control time under a pre-specified random schedule. Eligibility matters: if ranking changes at 10:00 and supplier behavior settles only at 10:40, that interval is neither stable treatment nor stable control.

The outcome should reflect the interference mechanism. In ride-hailing dispatch, completed trips per eligible rider request and driver acceptance can both matter. In a two-sided retail marketplace, orders per search session can hide treatment-induced changes in seller exposure; seller conversion, cancellation rate, and fulfillment delay belong in diagnostics. Demand uplift with worse fulfillment is not a clean product win.

Randomization needs a calendar rule. Mechanical weekday alternation can confound condition with weekly seasonality. Instead, form matched block pairs and randomly assign treatment then control or control then treatment. Pairing balances time while breaking a predictable sequence. Include fixed effects for pair, day-of-week, or operating window when defined before exposure.

Formula sheet for a defensible design

No closed-form equation captures all marketplace dynamics, so the planning sheet below lists the calculator inputs one by one. Fix units before entering numbers.

For early sizing, P ≈ ((z_(1-α/2) + z_power) × SD(D_p) / Δ)² gives required randomized pairs. It is valid only when SD(D_p) comes from historical paired blocks or a time-series model reflecting serial dependence. A session-level standard deviation and session count improperly pretend shared market conditions do not exist.

Show calendar cost: a 12-hour block with two-hour washout has 10 usable hours after each switch. A six-hour block with the same washout loses one-third of assigned time before power, sample quality, or operational risk are considered.

Carryover turns short blocks into biased contrasts

That usable-time arithmetic argues for shorter blocks, but the market’s memory argues back: short blocks create more transitions, which can balance time trends but let the market remember the prior condition. Ranking can alter seller repricing; a delivery incentive can shift driver positioning; a fraud rule can create manual-review queues that persist after removal. These effects often contaminate control immediately after treatment, pulling a contrast toward zero or, less commonly, reversing it.

Washout is a causal design claim: after W, residual influence is small enough for the estimand. One aggregate chart cannot prove it. Estimate results at multiple defensible washouts and assess whether sign, magnitude, and uncertainty remain credible.

Choose the washout from how the market’s memory decays, and test a range rather than trusting one guess. When residual influence fades within about 30 minutes, a one-hour washout leaves limited contamination, so test one and two hours and keep the usable time. When the effect falls gradually across two hours, a 30-minute washout almost certainly mixes states; compare two- and three-hour exclusions and inspect the pair-level effects before trusting the sign. When the intervention persists through most of a day, as with inventory repositioning or a manual-review queue that drains slowly, intraday switching measures transitions rather than a stable state, so move to daily blocks or reconsider the intervention. Queues can be sized before the test. By Little's law (1961), average time in a system equals the number of items in it divided by throughput: a backlog of 600 cases cleared at 100 an hour holds a case for about six hours, and no washout shorter than that is honest. The awkward case is asymmetry: when treatment leaves a longer tail than control, a symmetric washout can still leave directional bias despite equal exclusions, and the honest response is to model direction-specific carryover and report it.

Use a business-informed sensitivity range, not one optimistic estimate. If a ship decision becomes stop when washout moves from one hour to two, the result identifies a design dependency, not a reliable product effect. Extend only after checking whether longer blocks can isolate the measured state.

A marketplace example with three schedules

Take a synthetic test of a new matching policy. The primary outcome is completed orders per eligible request. The team seeks a 2.5% relative lift from baseline; demand has a pronounced daily cycle, historical hourly outcomes have positive adjacent-hour correlation, and supply positioning may need one to three hours to settle after a switch.

This illustrative comparison is not a universal duration rule; it shows why a calculator needs calendar and carryover views.

Three schedules over a 28-day window make the calendar cost concrete. Schedule A uses four-hour blocks with a one-hour washout: three usable hours per switched block and roughly 167 block boundaries. The reversals are frequent, but residual supply response can survive into the analyzed time, so A is too aggressive unless decay is demonstrably fast. Schedule B uses eight-hour blocks with a two-hour washout: six usable hours per block across about 83 switches, which lowers transition contamination while still balancing intraday demand, the strongest candidate to carry into simulation. Schedule C uses daily blocks with a three-hour washout: 21 usable hours but only about 27 switches in the month, trading many randomized contrasts for greater day-level trend exposure, and it earns its place only when the market settles slowly.

A supplies frequent reversals but spends seven of 28 days in washout and assumes the first analyzed hour afterward is settled. If supply response lasts two or three hours, apparent precision is bias. C retains more measurement time but far fewer contrasts; one weather event, promotion, or service incident can dominate a daily result unless pair controls absorb it.

So which one should run? None yet. B is often sensible here because six usable hours follow the two-hour exclusion, but that makes it the first schedule to simulate, nothing more. Simulate B alongside A and C using historical volatility, realistic autocorrelation, and every prior carryover path. Judge the schedules on bias as well as power: high nominal power with a biased estimator simply reaches the wrong answer more confidently.

If the policy changes subsidies, fees, or fulfillment costs, pair the readout with a unit economics metric architecture. Gross order growth can coexist with weaker contribution margin, which should be a guardrail rather than an afterthought.

Simulated power replaces session-count fiction

Simulation asks how often this exact design detects an effect under a stated effect size, noise level, and carryover path, and in doing so it shows which assumptions the result leans on. A test may be well powered if carryover vanishes after one hour but weak or biased if it persists for three.

This compact notebook simulates paired randomization, an AR(1) market shock, treatment effect, and treatment-to-control carryover; it repeatedly estimates a block-pair contrast. Replace synthetic parameters with estimates from clean historical windows and add day-of-week terms or richer errors where data require them.

import numpy as np

rng = np.random.default_rng(41)

def simulated_power(block=8, washout=2, pairs=42, effect=0.025,
                    carry=0.015, tau=1.5, rho=0.55, sigma=0.08,
                    runs=4000):
    hits = 0
    for _ in range(runs):
        order = np.concatenate([rng.permutation([0, 1]) for _ in range(pairs)])
        z = np.repeat(order, block)
        eps = np.zeros(len(z))
        for t in range(1, len(z)):
            eps[t] = rho * eps[t - 1] + rng.normal(0, sigma)
        y = 1.0 + effect * z + eps
        keep = np.ones(len(z), dtype=bool)
        for t in range(1, len(z)):
            if z[t] != z[t - 1]:
                keep[t:t + washout] = False
                if z[t] == 0 and z[t - 1] == 1:
                    k = np.arange(min(block, len(z) - t))
                    y[t:t + len(k)] += carry * np.exp(-k / tau)
        means = [y[i*block:(i+1)*block][keep[i*block:(i+1)*block]].mean()
                 for i in range(2 * pairs)]
        d = np.array(means)[order == 1] - np.array(means)[order == 0]
        t_stat = d.mean() / (d.std(ddof=1) / np.sqrt(pairs))
        hits += abs(t_stat) > 1.96
    return hits / runs

The normal critical value is for readability. Production should use the planned analysis, such as randomization inference over paired assignments or regression with pair fixed effects and time-series-aware uncertainty, and simulate the null. A false-positive rate under zero effect materially above chosen alpha means the method is miscalibrated before power matters.

Run a grid: 4, 8, 12, and 24-hour blocks; one- to four-hour washouts; plausible rho values; and carryover amplitudes matching the mechanism. Record power, bias of Δ̂, usable calendar time, and null rejection rate. High power is not enough if the estimator is biased; an unbiased design with low power may still leave the decision unresolved.

Different operating states need different schedules

A mature marketplace with stable hourly volume can support shorter blocks than a thin market where a few large orders determine outcomes. Raw traffic matters less than two other questions: does each block collect enough observations for a stable outcome, and does the market return to comparable states between switches?

For real-time matching, memory may be minutes. For seller incentives, inventory positioning, or worker scheduling, it can exceed a session. Do not force switchbacks: if the induced state outlasts a practical block, a geo experiment, staggered rollout, or long-horizon policy evaluation may identify effects more honestly. Hohnhold and colleagues at Google (KDD 2015) made the broader point for ordinary experiments: short-run readouts can misstate long-term effects.

Scope matters just as much. A nationwide switchback may be operationally clean yet remain vulnerable to a single incident that hits every market at once. Independently randomizing sufficiently separated geographies can add replicates if spillovers are limited. Separate dashboard rows do not establish independence; test shared shocks, central promotions, and common supply pools.

More calendar time can expose new failures

Extending the window is the natural reflex when you want more precision, yet longer tests improve precision only while intervention, instrumentation, and business conditions are stable. An app release, payment outage, unusual promotion, or supply-policy change can alter outcomes and carryover. Predefine exclusions for known incidents, but never remove blocks after seeing favored conditions. Fix the readout date too. Checking the contrast every morning and stopping on a good one is the peeking problem Johari et al. analyzed in "Peeking at A/B Tests" (KDD 2017), and pair-level data does not exempt a switchback from it.

Predictable schedules can cause adaptation: staff may learn the rhythm and sellers may observe recurring changes. Fixed treatment-control-treatment-control sequences invite anticipation and weak calendar balance. Randomized paired order, concealed where ethical and operationally appropriate, reduces this risk.

Guardrails matter as much as the headline number, because completed orders can rise simply by accepting orders that carry greater cancellation risk. Median fulfillment time, cancellation rate, customer contact rate, seller acceptance, and fairness-sensitive distribution metrics can reveal the mechanism. Inspect them by block and relevant segment, not just pooled averages. An aggregate gain driven by one geography while another absorbs harm warrants a narrower release.

A credible design names its failure conditions

Treat the calculator as an assumption record. Once it is used to justify a block length someone had already picked, it has stopped doing its job. State expected settling time, outcome aggregation, assignment unit, washout rule, historical noise-estimation period, and sensitivity ranges for autocorrelation and carryover. Preserve the simulation notebook with the decision.

A mature decision can be conditional: run eight-hour blocks with two-hour washout only if simulations show acceptable null calibration, adequate power at the minimum useful effect, and little estimator bias across the stated carryover range. If not, lengthen blocks, change design, or decline to test through switchbacks.

A switchback earns trust when its precision reflects what the market can actually forget, not how many events the logging pipeline happens to count.