Hire for judgment, not statistical vocabulary
Plenty of analysts can define a p-value cleanly and still make poor product decisions. An interview should probe the reasoning underneath: whether a candidate inspects the measurement before trusting an inference, separates noise from real effects, and recognizes when a result simply does not justify a rollout.
Experiment analysis is where product behavior gets connected to business action, and a weak readout is expensive. It can ship harmful onboarding, burn acquisition budget, or bury a useful feature after one noisy test. None of those failures show up as a statistics error on paper; they arrive as a confident recommendation the data never supported, which is exactly the habit an interview has to surface. So test the whole chain: from how a question and its metrics are defined, through design and analysis, to the trade-offs and the final call.
The analytics interview questions that actually separate candidates favor job-relevant scenarios over trivia. Hand the candidate an incomplete but plausible experiment brief and watch what they ask before they reach for a calculator. Questions about eligibility, exposure, event definitions, and conversion windows reveal far more discipline than an instinct to compute significance first.
Start with the measurement contract
The first scenario tests exactly that instinct to inspect before computing. Before variance or power comes up, find out whether the candidate treats a metric as a defined measurement or just a dashboard label. Put this in front of them: a checkout screen raised purchase conversion from 8.0% to 8.6%. What do they need before calling it a win?
A strong response digs into the denominator: who is in it, whether users were randomized before exposure, whether purchases are deduplicated, which window applies, and whether allocation or tracking shifted mid-test. It also separates the 0.6-percentage-point absolute lift from the 7.5% relative lift, (8.6% - 8.0%) / 8.0%, rather than blurring the two. A weaker one announces a 7.5% gain and recommends rollout without ever asking whether bot traffic, duplicate users, failed events, or changed eligibility moved the denominator underneath the number.
Then ask which metric is primary. A capable candidate names one metric tied to the hypothesis and surrounds it with diagnostics and guardrails. For checkout, purchase conversion might be primary, with refund rate, payment failures, support contacts, and revenue per eligible user standing guard against an improvement that only looks good.
Ask candidates to explain variance
Once a candidate interrogates the denominator, the next scenario checks whether they can explain why honest numbers still wobble. Hand over a small lift with uneven daily results and ask the plain question: why can two groups with an identical product experience post different conversion rates? The point is to see whether the candidate ties random variation to a business decision or just recites a textbook line.
The answer you want treats observed conversion as an estimate of a sample, not fixed behavior for every eligible user. Chance differences in who showed up and what they did create the spread; smaller samples are less precise, conversion variance depends on the underlying rate, and a few high-spend outliers can blow up the uncertainty on any revenue metric. A candidate on solid ground might describe a confidence interval as the range of effects compatible with the data and the chosen method (not a promise that the true value sits inside one particular interval) and might mention a standard error built from the conversion rate and sample size, while noting that no formula rescues bad data.
The weaker version says variance just means messy data, or that a bigger sample always proves a result. Another red flag is treating significance as proof of commercially useful causation without pausing to check randomization and instrumentation. Faced with sharp daily swings, the careful candidate checks sample counts, allocation, release timing, logging failures, channel mix, weekday effects, and whether one small cluster of users drove the revenue, instead of inventing a causal story from the shape of a chart.
Test statistical power before launch
Understanding variance leads straight to the design choice that tames it. Ask how they would decide whether an experiment has enough traffic. The tell is whether sample size shows up as a design choice made before the test or an excuse offered after it.
A strong candidate defines Statistical Power as the probability of detecting a specified true effect under the design, then asks for the inputs it needs: baseline rate, minimum detectable effect (MDE), significance threshold, desired power, allocation ratio, and expected traffic. Smaller MDEs demand more observations, because a faint signal is harder to pull out of random variation. Better still, they ask whether the MDE is even worth detecting: if a 0.2-point conversion lift throws off less gross margin than the engineering and operational cost of shipping it, designing a test to catch it is a poor use of the traffic. Power should serve a decision threshold, not a ritual.
The weak answer quotes a universal sample size, declares two weeks enough, or reads a non-significant result as proof that no effect exists. An underpowered test can miss a real effect; it cannot demonstrate equivalence. When traffic genuinely cannot support the MDE, the honest options are to extend the test if conditions hold steady, narrow eligibility only with a defensible reason, switch to a more sensitive but still valid metric, or decline to run it until the expected impact justifies the cost, never to move the success criteria after the numbers land.
Catch peeking before it becomes p-hacking
Power is fixed before launch; the next scenario tests discipline once the test is running and the numbers start to tempt. Describe a four-week test that crosses significance on day six, with a stakeholder itching to stop early. Ask what they would do.
A sound candidate explains that repeatedly checking a fixed-horizon test and stopping at the first favorable threshold inflates false-positive risk, because that early crossing may just be a random high point. They ask whether a sequential design, an approved interim analysis, or a stopping rule existed before launch; if none did, they push to complete the planned sample or to have the right statistical owner choose a method that accounts for the interim look before anyone claims significance. And they keep operational monitoring separate from inferential peeking: watching for broken payments, exposure failures, and material guardrail harm protects users and data, but it never licenses an opportunistic success claim.
The weak candidate either stops the moment p drops below 0.05 or refuses to look at the data until the final day. The first invites p-hacking; the second ignores product risk and tracking failures. Ask, too, how they would document the choice, and listen for the original hypothesis, duration, primary metric, stopping rule, exposure definition, exclusions, and any deviation from plan written down: the record that makes a later argument auditable.
Expose segment fishing with one chart
Peeking inflates false positives over time; slicing inflates them across the data at a single glance. Show an aggregate result sitting near zero, then a strong lift for Android users in a single acquisition channel, and ask whether to target that segment.
The disciplined answer asks whether the segment was named before the test and how many segments were examined, since slicing until something favorable appears drives up false-positive risk. It checks the segment's sample size, its confidence interval, any baseline differences, and whether the pattern survives in a fresh sample or holdout. It also knows segmentation can be perfectly legitimate: a platform-specific rendering change has a prior reason to be analyzed by platform, whereas a post-hoc slice by browser, city, campaign, and weekday does not. The line is between planned questions and hunting for a story, not segmentation as such.
The weak answer grabs the largest observed lift or throws out all segment analysis because the aggregate came back flat. One overfits noise; the other can bury a genuine heterogeneous effect. The reliable next move is a confirmatory, segment-focused test with a predeclared metric and adequate power. Until that runs, the finding is a hypothesis, not an instruction to roll out.
Separate lift from the shipping decision
Measurement, variance, power, and multiple comparisons all feed the same final act of turning an effect into a call. A candidate who reports lift and stops there has not finished the analysis. Give them a scenario with teeth: treatment raises trial starts by 4% relative, reduces activation among trial users, and leaves revenue uncertain. Should it ship?
A strong candidate first pins down metrics and populations (trial-start conversion is trial starters divided by eligible visitors, and Activation Rate is activated users divided by eligible trial users) then asks whether activation is a meaningful first-value event and whether the conversion windows line up. More trial starts can sit happily alongside fewer activated users if lower-intent visitors are entering the funnel, so they may estimate activated users per eligible visitor rather than reading the two rates in isolation. From there they weigh confidence intervals, magnitude, retention signals, support burden, and revenue or margin wherever the model allows. Ship, iterate, stop, or run a targeted follow-up can all be defensible; a conditional call might ship only if the activation decline stays inside a pre-agreed guardrail and expected long-term value clears the cost. Uncertainty pinned to a defined next measurement is not the same as indecision.
The weak candidate ships because the headline metric is significant, or rejects the whole thing because one secondary metric slipped, without checking precision, magnitude, or the metric hierarchy. Lift is evidence about a metric. A decision needs value, risk, confidence, and reversibility.
Use assessments that resemble the job
Testing all of this well means the exercise itself has to resemble the job. Product and analytics assessments have moved well past formula recall toward work samples, critique exercises, and live readouts. A guide to modern assessment formats and the signals they capture lays out why different formats expose different parts of judgment.
For analytics roles, hand over a compact case: a hypothesis, an event dictionary, an experiment table, and one deliberately ambiguous chart. Ask for a five-minute readout, then challenge a single assumption and see what happens. The exercise shows whether the candidate spots a denominator mismatch, states their uncertainty out loud, and revises a recommendation once the evidence shifts.
Resist grading polish as analytical skill. A candidate with fewer presentation habits may still be the one who names the central measurement flaw. Score the reasoning, not the delivery.
Build a repeatable hiring scorecard
A fair readout of any single answer still needs a consistent yardstick across candidates. Consistency matters here because unstructured impressions quietly reward familiarity and a confident speaking style. Use the same prompt, the same follow-ups, and the same scoring anchors across a role family, and record the evidence in a candidate's answers rather than a vague sense of how smart they seemed.
Rate five dimensions on a defined scale. Measurement discipline is whether they check event definitions, eligibility, denominators, and tracking integrity. Experimental design is whether they can state a hypothesis, a primary metric, guardrails, power inputs, and stopping rules. Statistical reasoning covers variance, uncertainty, multiple comparisons, and the limits of a non-significant result. Decision quality is the link from effect size to customer value, economics, risk, and a concrete next action. Communication is whether they can state assumptions and caveats in language that product, marketing, and engineering can all use.
Anchor every score in observable behavior. High decision quality is not a reflex to say ship; it is explaining when shipping, iterating, or stopping is the right move. Low quality is not an absence of jargon; it is confidently recommending an action the design and data do not support. Have panelists submit scores before the group discusses, so the most senior voice cannot quietly rewrite everyone's recollection, and settle disagreements against the transcript: what did the candidate actually ask, calculate, assume, and decide?
Calibrate the interview against the work analysts actually do
The goal was never a walking statistics glossary. It is people who shield the business from false certainty and turn imperfect evidence into action that fits the size of the evidence.
Treat variance, Statistical Power, peeking, segment fishing, and the trade-off question as one connected system rather than five separate quizzes. A candidate who handles each in isolation may still come apart in real experimentation work; one who links measurement quality, uncertainty, user impact, and economics is far likelier to produce analysis a team can trust. Before the next hiring loop, write one job-relevant experiment case, define what strong and weak answers look like, and get the interviewers aligned on the scorecard. That is what turns an analytics conversation into a real test of experiment judgment.