100 Experiment Design Interview Questions and Answers (2026)

An A/B test is a randomized controlled experiment that splits users into two (or more) groups, exposes each to a different variant, and compares a metric to measure the causal effect of a change.

• The mechanics:

  • Control (A) sees the current experience; treatment (B) sees the proposed change.

  • Users are randomly assigned, then you compare an outcome metric between groups.

• Why we run it:

  • To establish causality: randomization isolates the change as the only systematic difference between groups.

  • To make decisions with evidence instead of intuition or HiPPO opinions.

  • To quantify impact and risk before a full rollout.

• The alternative it beats: Before/after comparisons confound the change with seasonality, trends, and other launches; A/B testing avoids this by measuring both groups over the same period.

Randomization assigns each unit to control or treatment by chance, so on average both groups have the same distribution of every characteristic (known and unknown) except the treatment itself.

• Balances confounders in expectation: Age, device, tenure, and even unmeasured traits are distributed similarly across groups, so they can't systematically bias the comparison.

• Removes selection bias: Assignment doesn't depend on user attributes or self-selection, so the groups aren't different for reasons other than the treatment.

• Works better at scale: Balance is a large-sample property; small samples can still be imbalanced by chance.

• Verify, don't assume: Run balance checks (A/A tests, covariate comparisons) to confirm randomization worked and the pipeline isn't biased.

A well-formed A/B test needs a hypothesis, a randomization unit, defined metrics, a sizing plan, and a decision rule agreed before launch.

• Hypothesis: A specific, testable prediction about how the change moves a metric.

• Randomization unit and assignment: The unit (user, session, geo) and the split (e.g. 50/50), applied consistently.

• Metrics: A primary metric (the decision driver), plus guardrail metrics to catch harm and secondary metrics for context.

• Sample size and duration: Derived from a power analysis using the minimum detectable effect (MDE), baseline rate, and significance level.

• Statistical plan: Significance threshold (alpha), power (1 - beta), and the analysis method fixed in advance to avoid p-hacking.

• Decision rule: What ship/no-ship outcome each result triggers, defined before you look at data.

A good A/B test is one whose result you can trust and act on: it's adequately powered, cleanly randomized, tied to a meaningful pre-registered metric, and free of common validity threats.

• Sufficient power: Sized to detect a realistic effect, so a null result actually means "no meaningful effect," not "too small to see."

• Valid randomization: Balanced groups, verified with A/A tests or sample-ratio-mismatch checks.

• Pre-registered metric and hypothesis: One primary metric and decision rule chosen upfront to prevent cherry-picking and multiple-comparison inflation.

• Runs long enough: Covers full weekly cycles and accounts for novelty and primacy effects.

• Controls validity threats: Avoids peeking (fixed horizon or sequential methods), interference, and instrumentation bugs; includes guardrails to catch unintended harm.

• Actionable and interpretable: The result maps cleanly to a business decision, ideally with confidence intervals to convey magnitude and uncertainty.

Correlation means two variables move together; causation means changing one actually changes the other. Randomization is the gold standard because it removes confounders and reverse causation, so an observed difference can only be attributed to the treatment.

• Why correlation isn't causation:

  • Confounding: a third variable drives both (ice cream sales and drownings both rise with heat).

  • Reverse causation: the effect may actually cause the supposed cause.

  • Selection bias: the groups being compared differed to begin with.

• What randomization buys you:

  • By assigning treatment by chance, groups are equivalent in expectation on all other factors, so those factors can't explain the difference.

  • It breaks the link between treatment and any confounder, and rules out reverse causation because assignment precedes the outcome.

• The result: A statistically significant metric difference in a randomized test is a causal estimate of the change's effect, not just an association.

• Caveat: Randomization guarantees internal validity only; generalizing to other populations or contexts (external validity) still requires judgment.

A feature flag is the delivery mechanism (a runtime switch that controls who sees a feature), while an experiment is the measurement framework (a randomized comparison that tells you whether the feature is worth shipping). They are related because experiments are usually implemented on top of feature flags.

• Feature flag: a control switch:

  • Turns a feature on/off or exposes it to a subset of users without a redeploy.

  • Primary uses: gradual rollout, kill switch, decoupling deploy from release.

• Experiment: a measurement:

  • Randomly assigns users to variants and compares metrics to establish causal impact.

  • Requires a hypothesis, randomization, and statistical analysis, not just toggling.

• How they relate:

  • An A/B test typically uses a flag to route users into control vs. treatment.

  • A flag can exist without an experiment (pure rollout), but a clean experiment almost always rides on a flag.

• Key distinction: a flag answers "who sees it?"; an experiment answers "did it work?"

Early design questions clarify what you're testing, on whom, how you'll measure it, and whether the test is even feasible. Asking them up front prevents a badly scoped experiment that can't produce a decision.

• Objective: What decision will this experiment inform, and what's the hypothesis?

• Metrics: What is the primary metric, secondary metrics, and guardrails we must not harm?

• Population and unit: Who is eligible, and what is the randomization unit (user, session, cluster)?

• Sizing: What's the MDE we care about, and do we have enough traffic to detect it in a reasonable time?

• Risks and edge cases: Novelty effects, seasonality, network/interference effects, and how we'll ramp safely.

• Validity checks: How will we detect sample ratio mismatch or instrumentation bugs?

Treat each prompt as a variant, randomly assign users (or sessions/requests) to prompt A or B, then compare them on a pre-declared primary metric with proper statistical testing before shipping the winner.

• Define the hypothesis and metrics first: Pick a primary metric (e.g. task success, thumbs-up rate, conversion) plus guardrails (latency, cost per call, refusal/safety rate).

• Randomize at the right unit: Usually randomize by user so a person sees a consistent prompt; use a stable hash of user ID to bucket deterministically.

• Control confounders: Hold the model, temperature, and retrieval context constant so only the prompt differs.

• Handle LLM-specific noise: Outputs are stochastic and quality is hard to measure; combine automated eval/LLM-as-judge with human ratings and log full inputs/outputs.

• Size, run, and decide: Compute sample size for your minimum detectable effect, ramp traffic gradually, then test significance before rollout; watch cost and latency guardrails throughout.

My approach is to define one clear hypothesis and metric, pick the right randomisation unit, then split traffic randomly and deterministically into a control (current experience) and treatment (change), sized for adequate power and validated for balance before trusting any results.

• Define upfront:

  • State the hypothesis, primary metric, guardrail metrics, MDE, significance level, and power (e.g. 5% and 80%).

  • Compute required sample size and expected runtime from baseline rate and variance.

• Choose the randomisation unit: Usually user-level for consistency; coarser (account, geo) when there are shared features or spillover.

• Assign groups:

  • Deterministic hashing with a per-experiment salt so assignment is random, stable, and independent of other tests.

  • Control keeps the current experience; treatment gets the change and only the change (isolate one variable).

• Validate before analysing: Run an SRM check that the observed split matches intended, and check pre-experiment covariates (an A/A test or balance check) for no systematic differences.

• Run and monitor: Let it run the pre-committed duration (avoid peeking/early stopping without correction), watch guardrails, then analyse at the randomisation unit.

A/B testing is ideal when you can randomly assign many independent units, reliably measure a metric tied to the change, and the effect shows up within a reasonable window.

• You can randomize at the right unit: Users, sessions, or accounts can be assigned independently without contaminating each other.

• Sufficient sample size: Enough units and events to detect the expected effect with adequate statistical power.

• A clear, measurable metric: A well-defined outcome (conversion, retention) that responds to the change and can be logged accurately.

• Short feedback loop: The effect materializes in days or weeks, not years.

• Low interference and reversible change: One user's treatment doesn't affect another's outcome, and you can safely roll back if it hurts.

Avoid an A/B test when you can't randomize cleanly, can't get enough power, the effect can't be measured in time, or the test itself is unethical or too costly.

• Randomization isn't feasible: Network effects or spillover (marketplaces, social features) mean treatment leaks to control, violating independence.

• Too little traffic: Underpowered tests can't detect realistic effects, so results are inconclusive.

• Effect is slow or diffuse: Brand changes or long-term retention effects outlast a practical test window.

• Ethical or legal concerns: Withholding a beneficial change or exposing users to harm is not acceptable.

• High cost or one-way doors: Rare, irreversible, or expensive-to-build changes (major rebrands, legal compliance) may not justify a controlled test.

Start from a clear hypothesis and a primary metric, size the test with a power calculation, run it with proper randomization and instrumentation, then analyze and decide. The goal is a decision, not just a p-value.

1. Define the goal and hypothesis: State the change, expected direction, and the mechanism (why it should work).

2. Choose metrics: One primary (decision) metric, a few secondary metrics, and guardrails (latency, revenue, retention).

3. Pick the unit and randomization: Usually randomize by user for a consistent experience; ensure independence of units.

4. Power and duration: Set the minimum detectable effect (MDE), significance, and power; compute sample size and run at least one to two full business cycles.

5. Instrument and validate: Log assignment and events; run an A/A test or check for sample ratio mismatch before trusting results.

6. Analyze and decide: Compare primary metric, check guardrails, then ship, iterate, or abandon.

The lifecycle runs from idea to hypothesis to design to execution to analysis to decision, with each stage gating the next. The point is to convert a belief into a causal, data-backed launch decision.

1. Idea and prioritization: An observation or opportunity motivates a change; prioritize by expected impact vs. cost.

2. Hypothesis: Turn the idea into a falsifiable claim with a predicted metric movement.

3. Design: Choose metrics, randomization unit, MDE, power, sample size, and duration.

4. Implementation and QA: Build behind a flag, instrument logging, and validate with an A/A test.

5. Execution: Ramp traffic, monitor guardrails, and watch for sample ratio mismatch or bugs.

6. Analysis: Assess statistical and practical significance, segment carefully, and check novelty/primacy effects.

7. Decision and rollout: Ship, hold, or iterate; document the learning even when the result is null.

A good hypothesis is a specific, testable prediction that a defined change will move a defined metric in a defined direction, for a defined population, ideally with a rationale. Falsifiable means the data could clearly prove it wrong.

• Core components:

  • The change: exactly what is different in treatment.

  • The population: which users it applies to.

  • The metric and direction: which primary metric moves, and up or down.

  • The mechanism: why you expect that effect (the causal story).

• Falsifiable: Framed so a specific outcome would refute it; avoid vague claims like "improves the experience."

• Testable: The predicted effect must be measurable and large enough to detect with feasible sample size.

• Example form: "Adding one-click checkout for mobile users will increase purchase conversion by at least 2% because it removes form friction."

Testing one change at a time keeps attribution clean: if you bundle changes and the metric moves, you cannot tell which change caused it. Bundling trades interpretability for speed and can hide offsetting effects.

• Clean attribution: One change means a significant result maps to a single cause you can learn from and reuse.

• Costs of bundling:

  • Offsetting effects: one change helps, another hurts, and the net looks flat, so you wrongly discard both.

  • No learning: even a win doesn't tell you which piece drove it, hurting future decisions.

  • Debugging is harder when a guardrail regresses.

• When bundling is acceptable:

  • Changes are inseparable, or you only care about the combined ship decision, not attribution.

  • To isolate interactions deliberately, use a factorial design instead of a blind bundle.

Define success metrics that reflect the feature's intended value plus guardrails against harm, propose experiments that isolate that value, and interpret outcomes by weighing statistical significance, practical significance, and guardrail impact together.

• Define success metrics:

  • Pick a primary metric tied to the feature's goal (e.g. conversion, engagement).

  • Add guardrails (latency, retention, revenue) so a local win doesn't cause a global loss.

  • Prefer metrics that are sensitive, attributable, and hard to game.

• Propose experiments:

  • Design the smallest test that isolates the change; ramp from small to full traffic.

  • Consider holdouts for long-term effects and staged rollouts for risk.

• Interpret outcomes:

  • Statistical significance: is the effect real, not noise?

  • Practical significance: is it big enough to justify shipping and maintenance?

  • Check guardrails and segments; beware novelty effects and multiple-comparison false positives.

  • A null result is still a decision and a learning, not a failure.

A full experiment outline states the goal, a falsifiable hypothesis, a primary success metric with guardrails, the design parameters, and an explicit plan for confounds like seasonality and novelty. The edge cases are what separate a naive test from a trustworthy one.

• Goal: The business/user problem and the decision the test informs.

• Hypothesis: Specific predicted metric movement with a mechanism, e.g. "feature X raises 7-day retention by 1%."

• Success metric: One primary metric plus guardrails; define the MDE, power, and required duration.

• Design: Randomization unit, eligibility, sample size, and validity checks (A/A, sample ratio mismatch).

• Edge cases:

  • Seasonality: run across full business cycles (weekends, paydays, holidays) to avoid time-of-period bias.

  • Novelty/primacy: users react to newness; watch the effect stabilize over time before deciding.

  • Interference: network or marketplace spillover can violate independence; consider cluster randomization.

  • Ramp and kill switch: start small and be ready to roll back on guardrail regressions.

Balance them by treating user experience as a constraint and learning speed as the objective: run experiments small and fast enough to learn quickly, but with guardrails that stop you from harming real users.

• Learning speed favors small, frequent, cheap tests: Iterate rapidly, ship changes behind flags, and read results early to compound knowledge.

• User experience is protected by guardrails and exposure limits:

  • Ramp traffic gradually (e.g. 1% then 5% then 50%) and monitor guardrail metrics like latency, errors, and churn.

  • Set automatic stop conditions so a bad variant is killed before it reaches many users.

• Resolve the tension with the ramp-up strategy: Early small exposure catches disasters cheaply; wider exposure buys the power needed to detect real effects.

• Match rigor to risk: Low-risk UI tweaks can learn fast with light process; changes touching trust, safety, or revenue deserve slower, more careful rollout.

A/B tests are only as trustworthy as the data feeding them: without correct instrumentation and reliable assignment infrastructure, you get biased or missing measurements that silently invalidate results.

• Instrumentation defines what you can measure: Metrics require correct, complete event logging; a missing or mislabeled event means the metric is wrong no matter how good the analysis.

• Engineering owns valid assignment and exposure: Randomization, bucketing, and consistent exposure logging must be correct or you get sample ratio mismatch (SRM) and dilution.

• Early collaboration prevents wasted experiments: Adding instrumentation after launch means you can't recover data; designing metrics and logging together up front avoids reruns.

• Shared trust in the results: When engineers understand the metric definitions and data pipeline, debugging anomalies (logging bugs vs. real effects) is far faster.

Metric sensitivity is how reliably a metric detects a real treatment effect with a feasible sample size; metric alignment is how faithfully improving that metric reflects real progress toward the business goal. A good decision metric needs both.

• Sensitivity (statistical property):

  • Depends on effect size and variance: low-variance, frequently-occurring metrics move detectably with fewer users.

  • Improve it via variance reduction (CUPED), better metric definitions, or choosing metrics closer to the change.

  • An insensitive metric gives false 'no effect' results even when the treatment works.

• Alignment (validity property):

  • The metric must move in the same direction as true long-term value; a well-aligned metric that goes up means the business genuinely benefited.

  • Poorly aligned metrics are gameable or short-sighted (clickbait raises clicks but lowers satisfaction).

• The tension: The most aligned metric (e.g. lifetime revenue) is often the least sensitive; the trick is finding sensitive proxies that stay aligned.

Pick metrics that faithfully capture the goal and are sensitive enough to move, then set thresholds from a mix of statistical power, business value, and cost, not arbitrary round numbers.

• Metric selection:

  • Alignment: does it reflect the true objective, not just a convenient proxy?

  • Sensitivity: can the intervention realistically move it given its variance?

  • Measurability and timeliness: available reliably within the experiment window.

• Setting the success threshold:

  • Practical significance: the minimum effect (MDE) worth shipping, driven by business impact vs. cost.

  • Statistical significance: chosen alpha and power so the threshold is detectable at your sample size.

  • Costs and risks: engineering, maintenance, and downside if the effect is a false positive.

• Structure and safeguards:

  • Name one primary (decision) metric to avoid multiple-comparison cherry-picking.

  • Add guardrails and set thresholds before running, not after seeing data.

Leading metrics move fast and give early signal but are noisier proxies; lagging metrics are the outcomes you truly care about but take too long to observe in a single experiment. The trade-off is speed and sensitivity versus fidelity to the real goal.

• Leading (early) metrics:

  • Pros: measurable within the experiment window, higher event rate so more statistical power.

  • Cons: only a proxy; a leading win may not translate to the real outcome (Goodhart risk).

  • Examples: activation, first-week engagement, add-to-cart.

• Lagging (outcome) metrics:

  • Pros: directly capture business value (retention, LTV, churn).

  • Cons: slow to accumulate, often underpowered in a short test, harder to attribute.

• How to resolve the trade-off:

  • Decide on validated leading metrics that have been shown to predict the lagging outcome.

  • Monitor lagging metrics as guardrails or via long-term holdouts to confirm the proxy held up.

Guardrail metrics are metrics you don't intend to improve but must not harm: they protect core business and user health while you optimize the primary metric, catching cases where a "win" causes hidden damage.

• What they are:

  • Constraints, not targets: acceptable as long as they stay within bounds (no significant regression).

  • Examples: page latency, crash rate, revenue, unsubscribe/complaint rate, overall retention.

• Why they matter:

  • Prevent local optimization from causing global harm (boosting clicks while tanking satisfaction).

  • Counter Goodhart's Law by watching the true objective while the primary metric is being gamed.

  • Catch unintended side effects the experiment wasn't designed to test (performance, trust).

• How to use them:

  • Define acceptable-harm thresholds up front; a primary win that breaches a guardrail should not ship.

  • Ensure they have enough power to detect a meaningful regression, not just the primary metric.

Count metrics tally events per unit, rate metrics are a ratio of two counts, and binary metrics are 0/1 per unit. The type dictates the distribution, how you compute variance, and which statistical test applies.

• Binary (proportion) metrics:

  • Did the user convert? Each unit is 0 or 1; follows a Bernoulli/binomial distribution.

  • Variance is p(1-p), maximized near p=0.5, so extreme rates are cheaper to estimate.

• Count metrics:

  • Number of events per user (sessions, purchases); often Poisson-like or heavy-tailed/overdispersed.

  • High variance and skew; a few heavy users dominate, so consider capping/winsorizing or a log transform.

• Rate (ratio) metrics:

  • A ratio like clicks/impressions where the denominator isn't the randomization unit.

  • Naive variance is wrong because numerator and denominator are correlated; use the delta method or cluster/bootstrap methods.

• Why type affects analysis:

  • Variance formula and test differ (proportion test vs. t-test on capped counts vs. delta-method for ratios).

  • Higher-variance metrics need larger samples for the same MDE, so metric type drives power planning.

Some metrics barely budge because they're high-variance, dominated by factors outside the experiment, or already near a ceiling: the signal you can create is tiny relative to the noise, so detecting it needs impractically large samples.

• High variance drowns the signal:

  • Required sample size scales with variance divided by effect squared, so a noisy metric needs enormous n for a small true effect.

  • Skewed/heavy-tailed metrics (revenue per user) have huge variance from a few outliers.

• The effect is genuinely small or diluted:

  • A change affecting one small step barely moves an aggregate metric driven by many other factors.

  • Dilution: only a fraction of users are exposed to the changed feature.

• Ceiling/floor effects: Metrics already near 99% (or 0%) have little room and low variance to exploit.

• Mitigations: Variance reduction (CUPED), capping outliers, triggered analysis on exposed users, or choosing a more sensitive proxy metric.

Sample size comes from a power calculation: you fix your significance level, desired power, baseline rate, and the smallest effect you care about, then solve for the n per group that lets you reliably detect that effect.

• Key inputs:

  • Significance level (alpha): tolerated false-positive rate, usually 0.05.

  • Power (1 - beta): probability of detecting a true effect, usually 0.80 or 0.90.

  • Baseline rate/mean and its variance: the current conversion rate or metric level in control.

  • Minimum Detectable Effect (MDE): the smallest change you want to be able to catch.

• How each moves sample size:

  • Smaller MDE: n grows fast (roughly with 1/MDE^2), the dominant driver.

  • Higher power or lower alpha: larger n.

  • Higher variance in the metric: larger n; more balanced baseline rates (near 50%) also need more.

• Practical steps:

  • For proportions, use the two-proportion formula; for means, use variance-based formulas or a calculator/simulation.

  • Compute n per variant, then multiply by number of variants to get total traffic needed.

Duration is driven by two things: how long it takes to accumulate the required sample size at your traffic rate, and covering the natural rhythms of user behavior. You run at least a full business cycle (typically a week, often two) so the sample isn't biased toward one slice of behavior.

• Traffic-based floor:

  • Duration = required sample size / daily eligible users per variant.

  • If that yields only 1-2 days, still don't stop early.

• Full business cycle:

  • Behavior differs by day of week (weekday vs weekend) and time of month (paydays, billing cycles).

  • A partial cycle over-weights whoever happens to be active, biasing results and hurting generalizability.

  • Run in whole weeks so each variant sees the same mix of days.

• Other timing factors:

  • Novelty and primacy effects: early behavior may not reflect steady state, so longer runs let these settle.

  • Avoid peeking and stopping the moment significance appears: fix the horizon in advance (or use a sequential/group-sequential method).

The MDE is the smallest true effect your test is designed to reliably detect at your chosen power and alpha. It trades directly against sample size and runtime: halving the MDE roughly quadruples the sample needed, so you pick the smallest effect that would actually change a business decision, not the smallest that's statistically imaginable.

• What MDE means:

  • A design-time input, not the observed effect: "if the true lift is at least X, we'll catch it with 80% power."

  • Effects smaller than the MDE will often go undetected.

• The trade-off:

  • n scales roughly with 1/MDE^2: smaller MDE means much larger sample and longer runtime.

  • Setting an unrealistically tiny MDE can make the test infeasible; setting it too large risks missing real, worthwhile wins.

• Choosing a practically significant MDE:

  • Anchor to business impact: what lift justifies the cost of shipping and maintaining the change?

  • Sanity-check feasibility against your traffic and acceptable runtime.

  • Don't set the MDE from a hoped-for result; base it on the decision threshold.

An underpowered experiment has too little data to reliably detect the effect you care about, so it produces noisy, untrustworthy conclusions. A flat or non-significant result from it tells you almost nothing: it is not evidence of "no effect," just an inability to distinguish a real effect from noise.

• Why underpowered tests are problematic:

  • High false-negative rate (Type II): real wins get missed and abandoned.

  • Any result that does reach significance tends to be exaggerated (Type M / winner's curse).

  • Wide confidence intervals: the estimate is too imprecise to act on.

  • Wasted resources and false confidence in a null.

• What a flat result actually tells you:

  • "Absence of evidence, not evidence of absence." The effect could be real but below your detection threshold.

  • Check the CI: if it spans both meaningful gains and losses, the test was inconclusive, not negative.

  • Correct response: don't declare "no difference"; either gather more data or accept you couldn't resolve it.

Calculating sample size up front is what makes the test interpretable: it guarantees you have enough data to detect a meaningful effect and commits you to a stopping rule before you see results, protecting against both false negatives and biased peeking.

• Why it's crucial:

  • Ensures adequate power so a real effect isn't missed.

  • Sets an objective stopping point, preventing peek-and-stop that inflates false positives.

  • Lets you check feasibility (do you even have the traffic?) before investing in the build.

• What happens if you skip it:

  • Underpowered test: inconclusive results you can't act on.

  • Temptation to stop when the p-value dips below 0.05, dramatically raising false-positive risk.

  • Overpowered test: burning traffic/time or exposing users to a bad variant longer than needed.

  • Post-hoc rationalizing of noise, leading to shipping changes that don't hold up.

Sample size isn't a separate step at the end; it's baked into design by working backward from the decision you want to make. You define the metric and the effect worth acting on, compute the required n, check it against your traffic and timeline, and only then finalize the experiment.

• Design workflow:

  1. Pick the primary metric and its baseline value and variance from historical data.

  2. Set the MDE from the business decision threshold, plus alpha and power.

  3. Compute required n per variant.

  4. Divide by daily eligible traffic to get runtime, rounded to whole business cycles.

  5. Feasibility gate: if runtime is too long, revisit MDE, metric, or scope.

• Make it robust:

  • Account for multiple variants and any multiple-comparison correction.

  • Consider variance-reduction (e.g. CUPED) to lower the required n.

  • Document assumptions and the fixed stopping rule before launch.

The baseline rate sets the scale of the metric and (for proportions) determines its variance, while the variance directly controls how much noise you must overcome to detect an effect: higher variance and smaller baseline effects both demand more samples.

• Baseline rate anchors the effect size:

  • An absolute lift of 1% means something very different off a 2% baseline (a 50% relative lift) than off a 40% baseline (a 2.5% relative lift).

  • You usually design around a minimum detectable effect (MDE) expressed relative to this baseline.

• Variance governs the noise floor:

  • For a binary metric variance is p(1-p), so it is fully determined by the baseline rate and peaks near p=0.5.

  • For continuous metrics (revenue, time) variance is separate and often large or skewed, inflating the required sample.

• They feed the formula together:

  • Sample size scales roughly with variance divided by the squared effect size, times the (z_alpha + z_beta) factor for your significance and power.

  • Smaller MDE or larger variance both push n up sharply (the squared term makes halving the effect quadruple the sample).

• Why they matter:

  • Bad baseline/variance estimates give the wrong sample size: too small and you are underpowered, too large and you waste traffic and time.

  • Estimate both from historical data, and consider variance-reduction (e.g. CUPED) to cut required n.

An unequal split costs power because the smaller arm becomes the bottleneck: the variance of the measured difference is dominated by whichever group has fewer users, so a 90/10 split needs far more total traffic than 50/50 to reach the same power.

• Power is limited by the smaller arm:

  • The standard error depends on 1/n_A + 1/n_B; that sum is minimized at 50/50 and blows up as one arm shrinks.

  • A 90/10 split roughly needs ~2.8x the total sample of a 50/50 test for equivalent power.

• When you would still use one:

  • Risk control: a risky or unproven treatment gets only 10% exposure to limit blast radius.

  • Gradual ramp-ups: start at 1/5/10% to catch bugs and guardrail regressions before full launch.

  • Abundant traffic: if you have plenty of users, the extra time to power a 10% arm is acceptable.

  • Costly treatment: the variant is expensive to serve (compute, inventory), so you cap its share.

• Rule of thumb: Prefer 50/50 when you want maximum power fastest; use skewed splits when safety or cost outweighs speed.

The standard approach is deterministic hashing: hash a stable unit ID together with an experiment-specific salt, map the result into buckets, and assign arms by bucket range. This is stateless, reproducible, and gives a consistent allocation without storing a lookup table.

• Deterministic hashing:

  • Compute hash(user_id + experiment_salt), then take it modulo N (e.g. 1000) to get a bucket.

  • Same input always yields the same bucket, so a user stays in their arm across visits and servers, no database needed.

• Per-experiment salt: A unique salt per test makes assignments independent across concurrent experiments (avoids correlated splits).

• Consistent traffic allocation:

  • Assign bucket ranges to arms (e.g. 0-499 control, 500-999 treatment); use a uniform hash so buckets are evenly filled.

  • To ramp up, expand the treatment bucket range; keeping existing buckets fixed avoids reshuffling already-assigned users.

• Validation: Check a sample-ratio mismatch (SRM) test: observed split vs intended split should not differ significantly.

Dilution happens when your analysis includes users who never actually experienced the change (untriggered users): their unchanged behavior is averaged in with the affected users, shrinking the measured effect toward zero and hurting statistical power.

• The mechanism: If only 10% of assigned users hit the feature, a real 10% lift among them looks like ~1% across everyone.

• Why it weakens detection:

  • The true effect is diluted while variance stays roughly the same, so the signal-to-noise ratio drops and you need far more traffic to reach significance.

  • You risk a false negative: concluding "no effect" when there is one, just concentrated in a subset.

• The fix: trigger-based analysis:

  • Analyze only users who reached the trigger point (were eligible to see the change) in both arms, keeping the comparison unbiased.

  • Log the trigger for control users too (counterfactual logging), so you compare like with like.

• Caveat: The triggered (local) effect answers "does it work when seen"; the diluted (global) effect still matters for the overall business impact of shipping.

Ramping (staged rollout) means gradually increasing the fraction of users exposed to a treatment (1% then 5% then 50% then 100%) so you can catch problems early on a small population before committing full traffic.

• Why ramp:

  • Risk containment: a bug or bad UX hits few users at 1%, limiting blast radius.

  • Operational safety: check for latency, errors, and infra load before scaling.

• What each stage is for:

  1. 1%: sanity / guardrail check, catch crashes and severe regressions.

  2. 5%: confirm stability and rough metric direction.

  3. 50%: reach statistical power for a confident, precise effect estimate.

• Key nuances:

  • Early tiny stages are underpowered: don't read them as final proof, they are safety gates, not verdicts.

  • Watch for novelty effects that fade as ramp widens and time passes.

  • Keep control and treatment allocation clean at each step so comparisons stay valid; the 50% stage is usually where you conclude.

  • Have an automatic rollback / kill switch tied to guardrail metrics.

My primary concern is the multiple-comparisons problem: comparing five colors against control (or each other) means many simultaneous tests, which inflates the chance of a false positive, so some color looks like a "winner" purely by luck. I'd address it with correction and adequate power.

• Why it's a risk:

  • With 5 arms at α = 0.05 each, the family-wise false-positive rate climbs well above 5%, so a spurious "winner" is likely.

  • Traffic also splits across all arms, reducing power per comparison.

• How I'd address it:

  1. Apply a correction (Bonferroni, or Benjamini-Hochberg to control false discovery rate) to keep the overall error rate honest.

  2. Recompute sample size / power for the higher arm count so each cell can actually reach significance.

  3. Pre-register one primary metric and hypothesis to avoid cherry-picking after the fact.

• Push back on the premise:

  • Ask whether button color plausibly moves the metric enough to justify the traffic; is this the highest-value test right now?

  • If yes, treat it as an A/B/n test with proper correction rather than five casual A/B tests.

An A/A test splits traffic into two groups that receive the identical experience, so there should be no real difference; it validates that your experimentation system itself is unbiased and correctly calibrated.

• What it validates:

  • Randomization/assignment is fair: no sample-ratio mismatch between groups.

  • False-positive rate is calibrated: p-values should be roughly uniform, so significant results appear about 5% of the time at alpha = 0.05.

  • Metrics pipeline and variance estimates are correct (no pre-existing bias between buckets).

• How it's used:

  • Run before trusting a new platform, or continuously as a background sanity check.

  • Can also empirically size the null distribution for a metric.

• Red flag: If an A/A test shows "significant" differences more often than chance, something is broken (assignment, logging, or analysis) and A/B results can't be trusted.

Dogfooding (testing on employees) is used before or instead of external experiments when you need fast qualitative feedback, want to catch bugs, or can't safely/ethically expose real users to an unproven change.

• When it makes sense:

  • Early, risky, or unpolished features where external exposure could damage trust or revenue.

  • Catching functional bugs and usability issues before a controlled rollout.

  • Very small user populations where you can't get enough external traffic.

  • Sensitive changes needing qualitative depth (interviews, observation) not just metrics.

• Why it helps: Fast, cheap, high-context feedback and accountability ("eat your own dog food").

• Big caveat:

  • Employees are not representative: they know the product, are more tolerant, and are biased toward it.

  • Treat it as qualitative validation, not a substitute for a properly powered external A/B test.

Experiment review and institutional memory turn one-off tests into durable organizational learning: review guards against flawed or misinterpreted results, and documentation prevents repeating experiments, relearning the same lessons, or losing the reasoning behind decisions.

• Why review matters:

  • Catches p-hacking, peeking, SRM, and post-hoc metric shopping before decisions are made.

  • Ensures conclusions match the evidence and guardrails were respected.

• Why institutional memory matters:

  • Teams turn over; without a record, hard-won insights and "we already tried that" knowledge vanish.

  • Enables meta-analysis: patterns across many experiments inform priors and strategy.

• What to document per experiment:

  1. Hypothesis, the change tested, and pre-registered primary/guardrail metrics.

  2. Design: population, unit of randomization, sample size/power, duration.

  3. Results with effect sizes and confidence intervals, plus data-quality checks.

  4. The decision made (ship/hold/iterate) and the reasoning behind it.

  5. Surprises, caveats, and follow-up questions for future tests.

Simpson's Paradox is when a trend that appears within every subgroup reverses (or disappears) when the subgroups are combined, because a confounding variable is distributed unevenly across groups. In an A/B test it means the aggregate result can point the opposite way from the truth revealed within each segment.

• The mechanism: A lurking variable (segment mix, device, time) is unbalanced between the arms and correlates with the outcome, so pooling averages weight things differently.

• How it appears in A/B tests:

  • Ramp-up bias: treatment is rolled out gradually, so it accumulates more late-joining or different-mix traffic than control, skewing pooled rates.

  • Segment imbalance: treatment converts better on both desktop and mobile individually, but because it got more low-converting mobile users, its overall conversion looks worse.

• How to guard against it:

  • Ensure proper randomization and equal exposure timing (check for SRM within segments).

  • Analyze key segments as well as the aggregate; investigate when they disagree.

  • Use methods that adjust for the confounder (stratification / weighting) rather than a naive pooled rate.

A/B tests can fail to measure what you think for statistical, engineering, or behavioral reasons. Recognizing these validity threats up front is what separates a trustworthy result from a misleading one.

• Statistical threats:

  • Peeking / early stopping: repeatedly checking and stopping at significance inflates false positives.

  • Underpowered tests: too little sample to detect the real effect (false negatives).

  • Multiple comparisons: testing many metrics/variants without correction manufactures spurious wins.

• Data / instrumentation threats:

  • Sample Ratio Mismatch from broken assignment or logging.

  • Metric bugs, missing events, or bot traffic contaminating counts.

• Design / assumption threats:

  • Interference / network spillover breaking group independence (SUTVA).

  • Confounds and Simpson's Paradox from uneven segment mixes.

• Behavioral / temporal threats:

  • Novelty and primacy effects that fade or grow over time.

  • Short-term vs long-term divergence: a metric win that harms retention or revenue later.

• Interpretation threats:

  • Chasing a surrogate metric that doesn't move the true business goal.

  • Twyman's law: an unusually striking result is more often an error than a discovery, so verify it.

Novelty and primacy effects are temporary distortions caused by users reacting to change itself rather than to the change's lasting value. A novelty effect is an initial spike as curious users try something new; a primacy effect is initial resistance as habituated users are slowed by the unfamiliar. Both mean early data misrepresents the long-run steady-state effect.

• Novelty effect:

  • A shiny new button/feature gets extra clicks simply because it's new; engagement decays as the novelty wears off.

  • Risk: shipping a "winner" whose lift evaporates weeks later.

• Primacy effect:

  • Existing users are used to the old design, so a change initially underperforms while they relearn, then improves.

  • Risk: killing a genuinely good change based on a rough first week.

• How to identify them:

  • Plot the treatment effect over time: a decaying or rising trend (rather than a stable line) hints at these effects.

  • Segment new vs returning users: novelty/primacy mainly hit existing users; new users have no prior expectation.

  • Run the test long enough to reach a stable plateau before deciding.

• Implication for interpretation:

  • Base the decision on the steady-state effect, not the transient early window.

  • For big changes, consider a longer holdout to measure the durable long-term impact.

Both biases arise when the units you analyze are not a random, representative sample: selection bias comes from how units enter the study, and survivorship bias from how units drop out, so the observed effect reflects the sampling process rather than the true treatment effect.

• Selection bias:

  • Treatment and control groups differ systematically before treatment, so differences at the end confound the effect.

  • Common cause: non-random assignment or self-selection (users who opt in to a feature differ from those who don't).

  • Threatens internal validity: you can't attribute the outcome to the treatment.

• Survivorship bias:

  • You only measure units that 'survived' (stayed active), ignoring those who churned or crashed.

  • Example: measuring engagement only among retained users hides that treatment drove weak users away.

• Mitigations:

  • Randomize assignment and analyze intent-to-treat (keep everyone assigned, including dropouts).

  • Check pre-experiment covariate balance and run an A/A test.

  • Track attrition rates per arm: differential dropout is a red flag.

Instrumentation and logging bugs corrupt the raw data an experiment depends on, so even a perfectly randomized design produces wrong conclusions: garbage in, garbage out.

• Differential logging:

  • The worst case: a bug logs events differently in treatment vs control (e.g. the new UI fires an extra event), creating a fake effect.

  • Because it correlates with the arm, randomization can't cancel it out.

• Missing or dropped events: Client-side logging loss, ad blockers, or crashes silently remove data and bias metrics.

• Sample ratio mismatch (SRM): If the observed split deviates from the intended (e.g. not 50/50), a logging or assignment bug is likely; the results should not be trusted.

• Defenses:

  • Run A/A tests to validate the pipeline end to end.

  • Monitor SRM and event counts automatically as guardrails.

  • Instrument treatment and control with identical logging code paths.

The goal is to isolate the treatment as the only systematic difference between groups: you identify variables that affect the outcome, then control them through randomization, design, or statistical adjustment.

• Classify the variables: Independent (the treatment you manipulate), dependent (the outcome metric), and confounders (affect the outcome and correlate with the treatment).

• Randomization is the primary control: Randomly assigning units balances both known and unknown confounders in expectation, which is its key strength over observational methods.

• Design-based controls:

  • Blocking/stratification: randomize within strata (e.g. by country, device) to guarantee balance on strong covariates.

  • Hold constant everything else (same time window, same population, one change at a time).

• Statistical controls: Adjust for pre-experiment covariates with regression or CUPED to reduce variance and residual imbalance.

• Verify balance: Check covariate distributions across arms and run an A/A test before trusting results.

Revenue metrics are heavily right-skewed: a few whales can dominate the mean and blow up variance, so you handle outliers to get stable, trustworthy estimates without distorting the true effect.

• Understand why it matters: High variance reduces statistical power and makes results noisy; one big spender landing in one arm can fake a win.

• Capping/winsorizing: Cap values at a high percentile (e.g. 99th) to tame the tail; choose the threshold before seeing results to avoid bias.

• Transform or change the metric: Analyze conversion rate (did they pay?) separately from spend, or use bounded proxies less sensitive to extremes.

• Robust and nonparametric methods: Bootstrap confidence intervals or quantile comparisons don't assume normality.

• Report both, and investigate: Show capped and uncapped results; a genuine outlier may be signal, but check it isn't fraud or a logging bug.

Twyman's Law states that any figure that looks interesting or surprising is usually wrong: the more extreme or exciting a result, the more likely it stems from an error rather than a real effect.

• Why extremes are suspect: Real treatment effects are usually small; a huge jump more plausibly reflects a bug, a broken metric, or a data-processing error.

• Common culprits: Logging errors, SRM, bot/outlier contamination, a misdefined metric, or a leak between arms.

• How to respond:

  • Treat a surprising win as a hypothesis to debug, not a result to ship; check the pipeline and instrumentation first.

  • Replicate it in an independent test before believing it.

• Cultural value: It instills healthy skepticism and prevents celebrating false positives that are really measurement artifacts.

Seasonality, holidays, and shocks introduce time-correlated variation that can bias or add noise to an experiment if treatment and control aren't exposed to the same calendar. The core defense is concurrent randomization plus running long enough to average over cycles.

• How they threaten validity:

  • Confounding: comparing pre vs post periods conflates the treatment with the calendar (never do a before/after without a concurrent control).

  • Non-representative windows: a test run only over Black Friday won't generalise to a normal week.

  • External shocks (outages, PR events, competitor moves) can hit one arm's users harder or distort behavior for everyone.

• How to account for them:

  • Concurrent control: treatment and control experience the same holidays and shocks simultaneously, so most calendar effects cancel in the difference.

  • Run over full cycles: at least one to two weeks to cover weekly seasonality; longer for monthly patterns.

  • Avoid ending on anomalous days and watch for day-of-week composition differences.

  • Variance reduction with time covariates (e.g. CUPED on pre-period metrics) removes seasonal baseline differences between users.

  • Monitor and annotate: flag known shocks, and consider sensitivity analysis excluding shock periods.

Every hypothesis test carries a false-positive risk (usually 5%); when you test many metrics or variants, those risks compound, so the chance of at least one spurious "win" grows fast. Correction controls that inflated error rate so you don't ship noise.

• The problem: error inflation:

  • With m independent tests at α=0.05, the family-wise false-positive rate is 1 - (1 - 0.05)^m: 20 tests gives ~64% chance of at least one false positive.

  • Applies both to many metrics on one experiment and many variants in one test.

• Common corrections:

  • Bonferroni: divide α by the number of tests; simple but conservative, hurts power.

  • Benjamini-Hochberg (FDR): controls the expected fraction of false discoveries; less strict, better when scanning many metrics.

• Practical framing that reduces the burden:

  • Pre-declare a single primary metric so the ship decision hinges on one test, not many.

  • Treat secondary and guardrail metrics as exploratory or apply correction to that family.

• Caveat: don't over-correct until real effects vanish; balance false-positive control against power (Type II risk).

This is a trade-off decision, not a pure statistics call: you weigh the value of the primary gain against the cost of the guardrail regression, and you don't automatically ship just because the primary won. The guardrail exists precisely to catch this, so treat its decline seriously.

• First, verify both signals:

  • Is the guardrail drop statistically real or noise? Check its confidence interval and power, not just direction.

  • Confirm no instrumentation or randomization bug (e.g. sample-ratio mismatch).

• Quantify the trade-off in a common unit:

  • Convert both to dollars or a shared utility: does the primary gain outweigh the guardrail loss?

  • Respect hard guardrails (a pre-agreed non-negotiable line like latency or trust) that veto shipping regardless.

• Investigate the mechanism:

  • Segment to see if the regression concentrates in one group; a targeted fix may preserve the win.

  • Check whether it's a novelty artifact that fades.

• Decide with stakeholders:

  • Options: iterate to remove the regression, ship with monitoring if the trade is clearly net-positive and reversible, or hold.

  • Escalate genuine value-vs-risk trade-offs to product/leadership rather than deciding unilaterally.

A flat or inconclusive result means you failed to detect a significant effect: the confidence interval spans zero. Crucially, this is not proof of "no effect": it may mean no real effect, an effect too small to matter, or simply too little power to detect a real one. Your next step is to distinguish which.

• What flat does and doesn't mean:

  • Absence of evidence isn't evidence of absence: a wide CI is uninformative.

  • A tight CI centered near zero is genuinely informative: the effect is small if it exists.

• Diagnose the cause:

  • Underpowered? Recompute the minimum detectable effect for your sample; the true effect may be below it.

  • Weak treatment/dilution? Low exposure or a subtle change; check that users actually experienced it.

  • Heterogeneity? A win in one segment cancels a loss in another.

• What to do next:

  • Run longer or with more traffic if the CI is too wide to conclude.

  • Strengthen the treatment or target the responsive segment, then re-test.

  • If tightly flat, accept no meaningful effect and don't ship the added complexity; move on.

A Type I error is a false positive (concluding a feature works when it doesn't); a Type II error is a false negative (missing a real improvement). In A/B testing, α controls Type I risk and power (1 - β) controls Type II risk, and each carries a distinct business cost.

• Type I error (false positive):

  • You ship a feature that has no real effect (or a harmful one).

  • Cost: wasted engineering, added complexity/maintenance, and possible degradation of the metric you thought you improved.

  • Controlled by the significance level α (typically 0.05).

• Type II error (false negative):

  • You fail to ship a feature that genuinely helps.

  • Cost: an opportunity cost, the forgone gain, often invisible because you never see what you missed.

  • Controlled by power (1 - β), driven by sample size, effect size, and variance.

• The trade-off:

  • Lowering α (stricter) reduces false positives but raises false negatives unless you add sample size.

  • Set the balance by cost asymmetry: irreversible or risky changes warrant strict α; cheap reversible ones can tolerate more Type I risk.

You replicate a surprising win when the result is large, unexpected, or high-stakes enough that a false positive would be costly: a repeat experiment guards against the chance that the first result was a statistical fluke or a one-off artifact.

• Why replicate:

  • Any single significant result at p<0.05 still has a real false-positive probability, inflated further if you ran many metrics or peeked.

  • The winner's curse: an effect that crosses the bar tends to be overestimated, so the true lift is usually smaller than observed.

  • The first run may have coincided with a confound (holiday, outage, seasonality, a bug).

• When it's worth it:

  • The effect is suspiciously large or contradicts prior intuition/theory.

  • The decision is expensive or hard to reverse (infra cost, permanent UX change).

  • The original test was underpowered, ran briefly, or was one of many comparisons.

• When to skip replication: Low-risk, easily reversible changes with a plausible mechanism: just ship and monitor.

• How to replicate well: Pre-register the hypothesis and primary metric, run on fresh traffic/time period, and power it for the (smaller) expected true effect.

You stop early for harm as soon as there is credible evidence a key metric or guardrail is being damaged, because the cost of continuing is real user harm; stopping early for a win is far riskier statistically and demands a much higher, pre-planned bar.

• Stopping for harm (asymmetric, act fast):

  • Trigger on guardrail breaches: crashes, latency spikes, revenue or retention drops, or clear ethical/safety issues.

  • The bar is lower because the downside is ongoing damage; a suggestive negative signal justifies pulling the treatment.

  • Often automated via alerts or a formal futility/harm boundary in sequential monitoring.

• Stopping for a win (symmetric caution, high bar):

  • Naive peeking and stopping at the first significant p-value massively inflates false positives.

  • Only valid with a pre-committed sequential method (e.g. group sequential boundaries, always-valid p-values, Bayesian stopping rules) that controls error over repeated looks.

  • Even then, early winners overestimate the effect, so let it run for stability unless the win is urgent.

• The core asymmetry: For harm, a false alarm just costs you a paused experiment; for a win, a false positive ships a bad change: so err toward stopping harm and toward patience on wins.

Read the readout as a whole, not one number: the point estimate (lift) tells you the likely size and direction, the confidence interval tells you the range of plausible truth, the p-value tells you whether the effect is distinguishable from noise, and guardrails tell you whether the win came at an unacceptable cost.

• Lift (point estimate): The best single guess of impact; ask whether its magnitude clears your practical/business threshold, not just zero.

• Confidence interval: The most informative piece: check both ends. A tight CI fully above your threshold is a strong ship; a CI spanning zero or a trivial range means the result is inconclusive even if 'significant.'

• p-value: Answers 'is this distinguishable from noise?' but says nothing about size or importance; interpret against your pre-set alpha and account for multiple metrics.

• Guardrails: A primary-metric win with a guardrail regression (latency, revenue, complaints) can still be a no-ship; these often act as veto conditions.

• Combining into a decision:

  1. Was the test valid? Check sample ratio mismatch, power, and duration first.

  2. Is the effect real and big enough? CI clearly above the practical threshold, p under alpha.

  3. Any guardrail broken? If so, weigh the tradeoff explicitly or hold.

  4. Ship, iterate, or replicate based on the balance of magnitude, certainty, and cost.

When individual-level randomization breaks down (interference, no valid control, coarse units), turn to alternative causal methods that approximate randomization at a different level or use natural variation.

• Geo-tests (randomize by region): Use when there's cross-user interference or marketing spend can only be set at a market level; regions become the randomized units.

• Switchback tests (randomize over time): Use in marketplaces where treatment affects shared supply (surge pricing, dispatch); alternate the whole system between variants across time windows.

• Quasi-experiments (no randomization available): Difference-in-differences, regression discontinuity, or synthetic control exploit natural variation when a rollout was staggered or a threshold determined exposure.

• The trade-off: These require stronger assumptions and usually have fewer units, so they give weaker causal guarantees and less power than a clean A/B test.

Under time and resource pressure you trade statistical power, breadth, and precision for speed, but you protect the core threats to validity: randomization, a clean control, and an unbiased primary metric.

• Trade sample size against detectable effect: Smaller samples mean you can only detect larger effects; accept a bigger minimum detectable effect rather than run underpowered tests you can't interpret.

• Trade breadth for focus: Test one clear hypothesis with one primary metric instead of many variants that dilute power and invite false positives.

• Never trade internal validity: Keep proper randomization, a concurrent control, and correct assignment: without these the result is uninterpretable at any speed.

• Substitute cheaper evidence where valid: Use variance reduction (e.g. CUPED), reuse existing telemetry, or run quasi-experiments when a full A/B is too costly.

• Be explicit about external validity limits: Short tests may miss novelty effects or seasonality; state these caveats rather than overclaiming.

In mature programs at companies like Microsoft, Google, and Bing, only about 10-30% of well-reasoned ideas produce a statistically significant positive result: most ideas are flat or negative, which should make you plan for volume, humility, and cheap failure.

• Most ideas fail even when experts believe in them: Bing found roughly a third of ideas win, a third do nothing, and a third hurt; intuition is a poor predictor.

• Implication: run many experiments: Wins come from a portfolio, so lower the cost per experiment to increase throughput.

• Implication: fail fast and cheap: Kill losers quickly and free resources; a killed bad idea is a successful experiment.

• Implication: expect small effects: Real wins are often tiny (fractions of a percent), so invest in sensitivity and variance reduction to detect them.

• Implication: stay skeptical of big wins: A surprisingly large effect is more often a bug or logging error than a breakthrough; validate before celebrating.

Choose one primary metric that directly captures the hypothesis's intended effect, secondary metrics to explain and diagnose it, and guardrail metrics to ensure you didn't harm anything you care about; a good metric is sensitive, aligned with long-term value, and hard to game.

• Primary metric: The single measure that decides success; it must directly reflect the change's goal and drive the ship/no-ship call.

• Secondary metrics: Supporting and diagnostic measures that explain the mechanism (e.g. why the primary moved) but don't decide alone.

• Guardrail metrics: Protect health you must not break (latency, crashes, revenue, unsubscribe rate); a win that trips a guardrail is not shippable.

• What makes a good metric: Sensitive (moves when the treatment truly works), aligned with long-term business value, measurable in the test window, and resistant to gaming.

• Risks of surrogate metrics: A short-term proxy (e.g. clicks as a stand-in for satisfaction) may move opposite to the true goal; optimizing the proxy can degrade the real outcome (Goodhart's law).

The Overall Evaluation Criterion (OEC) is the single quantitative metric (or weighted combination) that defines success for an experiment and captures long-term value: having one avoids cherry-picking among many metrics and forces you to decide up front what 'better' really means.

• What it is: A pre-agreed primary decision metric that the organization trusts to represent user and business value over the long term.

• Why one metric matters:

  • With many metrics you can always find one that 'won', inflating false positives and enabling motivated reasoning.

  • A single OEC aligns teams and makes ship/no-ship decisions objective and consistent.

• How to construct one:

  • Start from the long-term goal (retention, lifetime value) and find sensitive short-term proxies that are causally predictive of it.

  • Combine components into a weighted or normalized score when no single measure suffices (e.g. sessions per user blending engagement and satisfaction).

  • Validate that moving the OEC actually predicts the long-term outcome, using historical experiments.

• Pair it with guardrails: The OEC decides success; guardrails ensure that success wasn't bought by harming latency, revenue, or trust.

You operationalize the abstract construct into concrete, observable proxies, then triangulate across several imperfect measures rather than trusting a single number.

• Define the construct first: State what 'creativity' means for this context (novelty, usefulness, diversity of ideas) before measuring anything.

• Operationalize into observable proxies:

  • Behavioral: number of distinct ideas generated, variety/uniqueness of outputs, edit patterns.

  • Rated: human judges score outputs on rubrics (novelty + quality), reporting inter-rater agreement.

• Triangulate, don't rely on one metric: Combine behavioral, rated, and self-report signals; agreement across them builds validity.

• Check validity and reliability: Construct validity (does it capture creativity?), and reliability (consistent across raters/time?).

• Beware gaming: A proxy like 'idea count' can be inflated by low-quality volume, so pair quantity with quality.

Goodhart's Law says "when a measure becomes a target, it ceases to be a good measure": once you optimize a metric, people (or the system) exploit its gaps so it drifts from the true goal it was meant to represent.

• The core mechanism: A metric is a proxy for a real objective; optimizing hard against the proxy widens the gap between proxy and objective.

• Relation to surrogate metrics:

  • Surrogates (clicks, sign-ups) stand in for slow goals (revenue, retention); they're especially vulnerable because they were never the true aim.

  • Example: optimizing click-through can drive clickbait that hurts satisfaction and long-term engagement.

• Implications for experimenters:

  • Validate surrogates: confirm the proxy genuinely correlates with (and ideally causes) the true outcome.

  • Use guardrails to catch the gaming (quality, complaints, downstream retention).

  • Prefer a small basket of metrics over one, so no single number can be exploited unchecked.

  • Periodically re-check that the metric still tracks the goal after teams optimize toward it.

You validate sensitivity with an A/A/B-style check: run a known intervention (or deliberately degrade the experience) and confirm the metric moves in the expected direction with the expected magnitude. If a metric can't detect a change you know exists, it can't be trusted to detect real ones.

• Intentional degradation test:

  • Inject a known-negative change (add latency, hide a feature) and check the metric drops as expected.

  • Optionally test dose-response: bigger degradation should move the metric more.

• A/A test for false positives: Run identical treatments; the metric should show no significant difference, confirming it isn't spuriously noisy or biased.

• Reuse past experiments: Check whether the metric moved in prior tests with known outcomes; a metric that never moves is likely insensitive.

• Quantify sensitivity:

  • Compute the MDE at your sample size: if the smallest detectable effect exceeds any realistic change, the metric is too insensitive.

  • Pair with variance-reduction to improve sensitivity where possible.

With low traffic you can't brute-force sample size, so you shift strategy: reduce the variance and noise in your metrics, choose more sensitive metrics, target bigger changes, and accept longer runtimes or alternative designs that squeeze more signal from fewer users.

• Lower the sample size you need:

  • Variance reduction: CUPED, covariate adjustment, or stratification.

  • Pick metrics closer to the change and less noisy (a proxy or continuous metric instead of a rare conversion).

  • Test bolder changes with larger expected effects (bigger MDE).

• Change the design:

  • Run longer to accumulate users (watch for seasonality and drift).

  • Within-subject / crossover or switchback designs where each unit acts as its own control.

  • Loosen power/alpha deliberately if the decision cost is low, and be explicit about the risk.

• When A/B isn't viable:

  • Bayesian methods to reason about probability and expected value rather than binary significance.

  • Quasi-experiments (difference-in-differences, interrupted time series) or qualitative evidence.

  • Roll up related changes into fewer, higher-impact experiments instead of many small ones.

The winner's curse (Type M, or magnitude, error) is the tendency of statistically significant results from underpowered tests to overstate the true effect. Because only large observed effects clear the significance bar when power is low, the ones you "win" on are systematically the lucky, inflated estimates.

• Why it happens:

  • With low power, the true effect sits near or below the detection threshold.

  • Only estimates that land far above the mean (helped by noise) reach significance.

  • Conditioning on "significant" therefore selects for exaggerated observed effects.

• Consequences:

  • Reported lift is bigger than reality, so post-launch impact disappoints.

  • In severe cases the estimate can even have the wrong sign (Type S error).

  • Distorts prioritization: overhyped wins get resources they don't deserve.

• How to guard against it:

  • Adequately power the test so significant effects are close to true ones.

  • Report confidence intervals, not just point estimates, and discount early/lucky wins.

  • Replicate or hold out to confirm before betting on the magnitude.

The randomisation unit is the entity you assign to control or treatment; it must match the level at which the intervention acts and at which you measure, and it must be independent across units. For product changes, the user (or account) is usually the right unit because a person experiences multiple sessions and pages, and consistency across them is what you care about.

• Match unit to the intervention and metric:

  • Randomise at the level where the change is experienced and where analysis units are independent.

  • Common units: user, session/request, device, cookie, page view, or cluster (store, city).

• Why the choice is critical:

  • Too fine a unit (e.g. per request) leaks treatment across the same user, contaminating both arms and biasing results.

  • The analysis unit should equal (or nest cleanly within) the randomisation unit, or your variance estimates and p-values are wrong.

• Why user-level is preferred for product changes:

  • Consistency: the user sees the same experience every visit, avoiding a jarring flip-flop.

  • No cross-contamination: a user can't be in both arms, so exposure is clean.

  • Captures long-run and cross-session behaviour (retention, repeat purchase) that session-level units miss.

• When a coarser unit is needed: Spillover/network effects (marketplace pricing, social features): randomise by cluster or geo to keep units independent.

Yes, running many experiments concurrently is normal at scale; the key is to make their assignments statistically independent so one test's split doesn't skew another, and to watch for genuine interaction between features that touch the same surface.

• Independent randomisation:

  • Hash each user with a per-experiment seed/salt so assignment in test A is uncorrelated with test B.

  • With independent hashing, other experiments' effects average out as noise across your arms rather than biasing the mean.

• Layered / orthogonal designs:

  • Use overlapping experiment layers (as in Google's system): experiments in different layers are crossed orthogonally so they don't collide.

  • Put mutually conflicting experiments in the same exclusive layer so a user is in at most one of them.

• Guarding against interference:

  • Flag experiments that modify the same UI element or metric; consider a full factorial or exclusivity if a true interaction is plausible.

  • Monitor for sample-ratio mismatch (SRM) per experiment as an early warning of collision or leakage.

• Practical note: Independence handles most cases; reserve exclusivity for the rare directly-interacting features to preserve traffic efficiency.

User-level randomisation is preferred for anything the user can see because it keeps their experience consistent across every session and device, and prevents the same person landing in both arms. Session- or cookie-based units are weaker proxies for a person and tend to split one individual across variants.

• Why user-level wins:

  • Consistent UX: the feature doesn't appear and disappear between visits, which both improves experience and avoids confounding.

  • Clean exposure: one person = one arm, so downstream metrics (retention, LTV) are attributable.

• Problems with session-level:

  • A returning user can be treatment today and control tomorrow, diluting the effect and confusing them.

  • Sessions from the same user aren't independent, so naive variance estimates understate uncertainty.

  • Can't measure cross-session outcomes like repeat purchase or churn.

• Problems with cookie-based:

  • Cookies are cleared, blocked, or expire, causing reassignment and split identities.

  • One user across phone, laptop, and browsers gets multiple cookies (and possibly multiple arms).

  • Inflated unique counts and leakage between arms bias results.

• When session/cookie is acceptable: Logged-out traffic where no stable user ID exists, or short single-session flows where cross-session consistency doesn't matter.

Randomising by account instead of individual user changes both the number of independent units and the correlation within them: accounts are fewer and contain multiple correlated users, which reduces effective sample size and typically lengthens runtime, but it is necessary when the feature is shared at the account level.

• Fewer independent units:

  • An account with 10 users counts as one randomisation unit, so your n drops sharply versus per-user assignment.

  • Fewer units means less power for the same calendar time.

• Intra-account correlation:

  • Users in one account behave similarly (clustering), so you must analyse at the account level or use clustered standard errors.

  • Ignoring clustering understates variance and produces false positives.

• Impact on sample size and runtime:

  • The design effect ~ 1 + (m-1)*ICC inflates the required number of user-equivalents (m = users per account, ICC = intra-cluster correlation).

  • Uneven account sizes add variance too; a few whales can dominate metrics.

  • Net effect: you often need to run longer to accumulate enough accounts.

• When it's the right call: Shared/collaborative features (a team workspace setting) where mixing arms within an account would contaminate exposure or confuse users.

Prioritize by expected value per unit of traffic: rank ideas by likely impact and confidence, weighted against the sample size (and time) each needs to reach power.

• Estimate expected impact: Score each experiment by potential lift on the primary metric times your prior confidence it will work (evidence, past wins, strength of hypothesis).

• Weigh the traffic cost:

  • Small expected effects on low-traffic surfaces need huge samples; a modest effect on a high-traffic page may be cheaper to detect.

  • Use a power calculation to convert each test's MDE into a required sample size, then compute impact-per-user.

• Frameworks help rank: ICE (Impact, Confidence, Ease) or PIE give a shared, defensible ordering across teams.

• Respect overlap and interaction: Non-overlapping surfaces can run concurrently; tests touching the same metric or UI may need mutual exclusion, which itself consumes traffic.

• Keep a strategic reserve: Reserve capacity for high-urgency or reversible-risk launches, and deprioritize experiments that can't reach power in a reasonable window.

Reusing the same buckets carries over correlations from one test to the next: users who were in "treatment" before may retain learned behavior or state, biasing the new experiment. Re-randomize whenever that carryover could confound the next test.

• Carryover / learning effects: Users habituated to a previous treatment behave differently, so a stable bucket assignment imports that history into the new test.

• Correlated assignment across tests: If bucket 1 is always "treatment," the same population is repeatedly exposed, and unmeasured traits of that group get confounded with the treatment.

• Novelty and fatigue leak forward: A group tired of constant change may under-respond to a genuinely good new feature.

• When to re-randomize:

  1. Between experiments that share a metric or surface, so prior exposure doesn't bias the split.

  2. When a prior treatment could have durable effects (habit, saved state, model personalization).

  3. Use a fresh hash seed (salt) per experiment so bucketing is independent across tests.

• When reuse is fine: Truly independent, unrelated surfaces with no lingering state, where stable IDs simplify analysis.

Carryover contamination is when effects of a previous experiment persist into a later one, so the new test's baseline or behavior is shifted by history rather than by the current treatment. Guard against it with washout periods, re-randomization, and careful monitoring.

• How it contaminates:

  • Persistent behavior: users learned a habit or workflow that carries into the new test.

  • Persistent state: cached models, personalization, or saved settings from the old treatment still influence experience.

  • Correlated buckets: if assignments aren't re-drawn, the same population carries its prior exposure into the new split.

• Guardrails:

  1. Insert a washout / cool-down period so lingering effects decay before the next test starts.

  2. Re-randomize with a new salt so buckets are independent of the prior experiment.

  3. Reset or version any persistent state (models, caches) tied to the old treatment.

  4. Check an A/A test or pre-period balance to confirm arms start equivalent.

• Detection: Watch for pre-experiment metric divergence between arms; if they aren't equal before treatment, suspect carryover.

All three compare variants, but differ in how many things change: A/B tests one change against control, A/B/n tests several independent alternatives of one element, and multivariate (factorial) tests multiple elements at once to measure both their individual effects and their interactions.

• A/B testing:

  • One variant vs. control; simplest, most traffic-efficient, clearest to interpret.

  • Use for a single clear hypothesis ("does this new checkout beat the old one?").

• A/B/n testing:

  • One element, many alternatives (e.g. 4 headlines); picks the best of several.

  • Costs: traffic splits across n arms, and more comparisons inflate false positives (needs multiple-comparison correction).

• Multivariate / factorial testing:

  • Varies several factors simultaneously (button color x headline x image) across all combinations.

  • Strength: reveals interaction effects and main effects together; efficient vs. running many separate tests.

  • Costs: combinations explode (2x2x2 = 8 cells), demanding lots of traffic, and results are harder to interpret.

• Trade-off summary:

  • Traffic need and complexity rise A/B < A/B/n < multivariate; interpretability falls in the same order.

  • Choose the simplest design that answers the question: only reach for multivariate when interactions genuinely matter and traffic allows.