Hire for judgment, not statistical vocabulary
An analyst can define a p-value yet make poor product decisions. Interviews should test reasoning under uncertainty: inspecting measurement before inference, separating noise from effects, and knowing when results do not justify rollout.
Experiment analysis links product behavior to business action. A weak readout can ship harmful onboarding, waste acquisition budget, or reject a useful feature after a noisy test. Test the chain from question and metric definition through design, analysis, trade-offs, and decision.
Use job-relevant scenarios, not trivia. Give an incomplete but plausible experiment brief and note what candidates ask before calculating. Questions about eligibility, exposure, event definitions, and conversion windows show more discipline than immediately reaching for significance.
Start with the measurement contract
Before variance or power, establish whether candidates treat a metric as a defined measurement rather than a dashboard label. Present this case: a checkout screen raised purchase conversion from 8.0% to 8.6%. Ask what they need before calling it a win.
Strong answer pattern: They ask who is in the denominator, whether users were randomized before exposure, whether purchases are deduplicated, which window applies, and whether allocation or tracking changed. They distinguish the 0.6-percentage-point absolute lift from the 7.5% relative lift: (8.6% - 8.0%) / 8.0%.
Weak answer pattern: They call it a 7.5% gain and recommend rollout without checking whether bot traffic, duplicate users, failed events, or changed eligibility altered the denominator.
Ask which metric is primary. A capable candidate names one metric tied to the hypothesis, plus diagnostic metrics and guardrails. For checkout, purchase conversion may be primary; refund rate, payment failures, support contacts, and revenue per eligible user guard against misleading improvement.
Ask candidates to explain variance
Use a small lift with uneven daily results. Ask: Why can two groups with identical product experience produce different conversion rates? Test whether candidates connect random variation to a business decision rather than recite a definition.
Strong answer pattern: Observed conversion estimates a sample, not fixed behavior for every eligible user. Chance differences in user mix and outcomes create variation; smaller samples are less precise, conversion variance depends on the underlying rate, and high-spend outliers can widen revenue-metric uncertainty.
They may describe a confidence interval as effects compatible with the data and chosen method, not a guarantee that the true value lies in one specific interval. For binary metrics, they may mention a standard error based on conversion rate and sample size while recognizing that formulas cannot fix bad data.
Weak answer pattern: They say variance means messy data or that a larger sample always proves a result. Another warning sign is treating significance as proof of commercially useful causation without checking randomization and instrumentation.
For sharp daily swings, thoughtful candidates check sample counts, allocation, release timing, logging failures, channel mix, weekday effects, and whether a small user cluster drove revenue. They do not invent a causal story from a chart.
Test statistical power before launch
Ask: How would you decide whether an experiment has enough traffic? This shows whether sample size is treated as a pre-test design choice rather than a post-test excuse.
Strong answer pattern: They define Statistical Power as the probability of detecting a specified true effect under the design. They ask for baseline rate, minimum detectable effect (MDE), significance threshold, desired power, allocation ratio, and expected traffic. Smaller MDEs need more observations because small signals are harder to separate from random variation.
They also ask whether the MDE is commercially meaningful. If a 0.2-point conversion lift generates less gross margin than engineering and operational shipping costs, designing to detect it may not be worthwhile. Power should serve a decision threshold, not ritual.
Weak answer pattern: They give a universal sample size, say two weeks is enough, or say a non-significant result means no effect exists. An underpowered test can miss a meaningful effect; it cannot prove equivalence.
If traffic cannot support the MDE, options include extending the test if conditions remain stable, narrowing eligibility only with defensible rationale, choosing a more sensitive valid metric, or declining to test until expected impact warrants the cost. Do not change success criteria after results arrive.
Catch peeking before it becomes p-hacking
Present a four-week test that is significant on day six and a stakeholder wants to stop early. Ask what the candidate would do.
Strong answer pattern: Repeatedly checking a fixed-horizon test and stopping at the first favorable threshold raises false-positive risk; the result may be a random high point. They ask whether a sequential design, approved interim analysis, or stopping rule was planned before launch. If not, they recommend completing the planned sample or having the appropriate statistical owner revise the analysis plan before a new decision.
They distinguish operational monitoring from inferential peeking. Teams should watch for broken payments, exposure failures, and material guardrail harm to protect users and data, but these do not authorize opportunistic success claims.
Weak answer pattern: Stop whenever p is below 0.05, or never inspect data until the final day. The first invites p-hacking; the second ignores product risk and tracking failures.
Ask how they would document the choice. Look for the original hypothesis, duration, primary metric, stopping rule, exposure definition, exclusions, and any departure from plan in writing. This makes later debate auditable.
Expose segment fishing with one chart
Show an aggregate result near zero, then a strong lift for Android users in one acquisition channel. Ask whether to target that segment.
Strong answer pattern: They ask whether the segment was specified before the test and how many segments were inspected. Slicing until a favorable result appears raises false-positive risk. They check segment sample size, confidence interval, baseline differences, and whether the pattern repeats in a fresh sample or holdout.
They know segmentation can be legitimate: a platform-specific rendering change has prior rationale for platform analysis; a post-hoc slice by browser, city, campaign, and weekday does not. The issue is planned questions versus searching for a story, not segmentation itself.
Weak answer pattern: They choose the largest observed lift, or reject all segment analysis because the aggregate was flat. The first overfits noise; the second can hide meaningful heterogeneous effects.
A reliable next step is a confirmatory segment-focused test with a predeclared metric and adequate power. Until then, the finding is a hypothesis, not a rollout instruction.
Separate lift from the shipping decision
A candidate who reports lift without consequences has not finished analysis. Give this scenario: treatment raises trial starts by 4% relative, reduces activation among trial users, and leaves revenue uncertain. Ask whether it should ship.
Strong answer pattern: They define metrics and populations: trial-start conversion is trial starters divided by eligible visitors; Activation Rate is activated users divided by eligible trial users. They ask whether activation is a meaningful first-value event and whether conversion windows align.
More trial starts can coexist with fewer activated users if lower-intent visitors enter the funnel. Candidates may estimate activated users per eligible visitor rather than read rates separately. They examine confidence intervals, magnitude, retention signals, support burden, and revenue or margin where the model permits.
Ship, iterate, stop, or run a targeted follow-up can all be valid. A conditional answer might ship only if activation decline stays within a pre-agreed guardrail and expected long-term value exceeds cost. Uncertainty tied to a defined next measurement is not indecision.
Weak answer pattern: Ship because the headline metric is significant, or reject because one secondary metric declined without checking precision, magnitude, or metric hierarchy. Lift is evidence about a metric; a decision requires value, risk, confidence, and reversibility.
Use assessments that resemble the job
Product and analytics assessments have moved beyond formula recall toward work samples, critique exercises, and live readouts. A guide to modern assessment formats and the signals they capture explains why formats reveal different parts of judgment.
For analytics hiring, provide a compact case with a hypothesis, event dictionary, experiment table, and ambiguous chart. Ask for a five-minute readout, then challenge one assumption. This tests whether candidates notice a denominator mismatch, state uncertainty, and change a recommendation when evidence changes.
Do not grade polish as analytical skill. Candidates with fewer presentation habits may still identify the central measurement flaw. Score reasoning, not only delivery confidence.
Build a repeatable hiring scorecard
Consistency matters because unstructured impressions reward familiarity and speaking style. Use the same prompt, follow-ups, and scoring anchors for each role family, recording answer evidence rather than vague impressions of intelligence.
Rate five dimensions on a defined scale:
- Measurement discipline: Checks event definitions, eligibility, denominators, and tracking integrity.
- Experimental design: Defines hypothesis, primary metric, guardrails, power inputs, and stopping rules.
- Statistical reasoning: Explains variance, uncertainty, multiple comparisons, and limits of non-significant results.
- Decision quality: Connects effect size to customer value, economics, risk, and a concrete next action.
- Communication: States assumptions and caveats in language product, marketing, and engineering teams can use.
Anchor scores in observable behavior. High decision quality is not always saying ship, but explaining when shipping, iteration, or stopping is warranted. Low quality is not lacking jargon, but confidently recommending action unsupported by design or data.
Have panelists submit scores before discussion so the most senior voice cannot rewrite recollections. Resolve disagreements through the transcript or notes: what did the candidate ask, calculate, assume, and decide?
Make the interview predict better analysis
The goal is not a human statistics glossary, but people who protect the business from false certainty and turn imperfect evidence into proportionate action.
Use variance, Statistical Power, peeking, segment fishing, and trade-off questions as a connected system. Someone who handles each separately may still struggle in experimentation work; someone who links measurement quality, uncertainty, user impact, and economics is more likely to produce trusted analysis.
Before the next hiring loop, write one job-relevant experiment case, define evidence for strong and weak answers, and align interviewers on the scorecard. That turns an analytics conversation into a test of experimentation skill.