Experiment Design

0%
Coding
Theory
Quiz

    Randomization Assignment

    • Deterministic A/B Variant Assignment

      Junior

    • Deterministic Hash Bucketing

      Junior

    • Arbitrary Two-Arm Traffic Split

      Junior

    • Seeded Random Assignment

      Junior

    • Vectorized A/B Bucket Assignment

      Junior

    • Cluster Randomization Assignment

      Mid

    • Weighted Variant Allocation

      Mid

    • Salt Reuse Overlap Fraction

      Mid

    • Session-Level Randomization Assignment

      Mid

    • Global Holdout Over A/B Test

      Mid

    • Realized Weighted Allocation Counts

      Mid

    • Permuted-Block Randomization

      Mid

    • Two-Step Variant Assignment

      Senior

    • Cluster Rollup to Randomization Unit

      Senior

    • Single-Hash Bucket Assignment (SRM Fix)

      Senior

    • Orthogonal Overlapping Experiment Layers

      Senior

    • Stratified Blocked Assignment

      Senior

    • Exclusion Group Assignment

      Senior

    Sample Size Power Mde

    • Arcsine Two-Proportion Sample Size

      Junior

    • Experiment Duration in Days

      Junior

    • Power for Two-Proportion Test

      Junior

    • Cohen's d With Pooled SD

      Junior

    • Cohen's h Effect Size

      Junior

    • Minimum Detectable Effect via Arcsine

      Mid

    • Pooled Two-Proportion Sample Size

      Mid

    • MDE for Two-Sample t-Test

      Mid

    • Beta By Alpha Trade-off

      Mid

    • Unpooled Two-Proportion Sample Size

      Mid

    • One- vs Two-Sided Sample Size

      Mid

    • Unequal Allocation Sample Size

      Mid

    • Sample Size for a Ratio Metric

      Senior

    • Numeric MDE From Fixed Traffic

      Senior

    • End-to-End A/B Experiment Planner

      Senior

    • Power Curve Over Relative Lifts

      Senior

    • Pooled Two-Proportion Power With statsmodels

      Senior

    Metric Definition Aggregation

    • Pooled CTR by Variant

      Junior

    • Triggered Conversion Rate by Variant

      Junior

    • User-Level Conversion Rate by Variant

      Junior

    • Latency Guardrail Regression Check

      Junior

    • Metric Over Fixed Analysis Window

      Junior

    • Pooled CTR vs Mean User CTR

      Mid

    • Winsorize the Upper Tail

      Mid

    • Winsorization Variance Reduction

      Mid

    • Trigger Dilution: All vs Triggered Effect

      Mid

    • Revenue Per User By Variant

      Mid

    • Guardrail Error-Rate Check

      Mid

    • Daily Metric Time Series by Variant

      Mid

    • Delta-Method Ratio Metric Variance

      Senior

    • Shared-Threshold Winsorization by Variant

      Senior

    • Exposure-Relative Windowed Metric

      Senior

    • Multi-Guardrail Non-Inferiority Evaluation

      Senior

    • Composite OEC Score From Metrics

      Senior

    Experiment Analysis Lift

    • Relative Lift Calculation

      Junior

    • Absolute Lift of Conversion Rate

      Junior

    • Reject H0 From Confidence Interval

      Junior

    • Conversion Rates Per Arm

      Junior

    • Unpooled SE of Rate Difference

      Mid

    • Pooled vs Unpooled Standard Errors

      Mid

    • CI for Difference in Proportions

      Mid

    • Welch's T-Test Metric Readout

      Mid

    • Practically Significant Confidence Interval

      Mid

    • Per-Variant Conversion Rate

      Mid

    • Welch Confidence Interval for Mean Difference

      Mid

    • Two-Proportion A/B Test Readout

      Senior

    • Delta-Method SE for a Ratio Metric

      Senior

    • Delta-Method Ratio Metric Readout

      Senior

    • Delta-Method CI for Relative Lift

      Senior

    • Log-Transform CI for Relative Lift

      Senior

    • Continuous Metric Experiment Readout

      Senior

    Validity Checks Srm

    • SRM Chi-Square Goodness-of-Fit P-Value

      Junior

    • SRM Alarm Decision

      Junior

    • SRM Check For Unequal Split

      Junior

    • SRM Chi-Square Goodness-of-Fit Statistic

      Junior

    • SRM Report From Named Arms

      Mid

    • Covariate Balance via SMD

      Mid

    • Multi-Arm SRM Goodness-of-Fit Check

      Mid

    • Per-Day SRM Alarm Detection

      Mid

    • Flag Imbalanced Covariates by SMD

      Mid

    • Filter Bots Before Mean Readout

      Mid

    • Deduplicate Exposure Logs by First Touch

      Mid

    • A/A Test False-Positive Rate

      Senior

    • Detect Novelty Effect From Daily Lift

      Senior

    • Detect Contaminated Users in Exposure Log

      Senior

    • SRM Goodness-of-Fit vs Yates Contingency

      Senior

    • A/A P-Value Uniformity Check

      Senior

    • Unexposed Assigned Users Anti-Join

      Senior

    Variance Reduction Stratification

    • CUPED Theta Coefficient

      Junior

    • CUPED Adjusted-Metric Transform

      Junior

    • CUPED Expected Variance Reduction

      Junior

    • ANCOVA Covariate-Adjusted Treatment Effect

      Mid

    • CUPED-Adjusted Treatment Effect

      Mid

    • End-to-End CUPED Metric Adjustment

      Mid

    • Choose Best CUPED Covariate

      Mid

    • CUPED Effective Sample-Size Multiplier

      Mid

    • Post-Stratified ATE Estimator

      Mid

    • Paired Difference Mean and SE

      Mid

    • CUPED Effect And Standard Error

      Senior

    • CUPED Large-Sample Readout

      Senior

    • ANCOVA Treatment Effect and SE

      Senior

    • Seeded CUPED Variance-Reduction Simulation

      Senior

    • ANCOVA vs Unadjusted SE Reduction

      Senior

    • Post-Stratified ATE and Standard Error

      Senior

    • CUPED Theta by Scope

      Senior

    Sequential Testing Peeking

    • Pocock Alpha-Spending Function

      Junior

    • Bonferroni Peeking Boundary

      Junior

    • Apply Group-Sequential Stopping Boundary

      Junior

    • Simulating Peeking Type-I Inflation

      Mid

    • O'Brien-Fleming Alpha-Spending Function

      Mid

    • mSPRT Likelihood-Ratio Statistic

      Mid

    • Incremental Alpha Spent Per Look

      Mid

    • Pocock Constant-Boundary Stop

      Mid

    • O'Brien-Fleming Alpha Spent at Sample Sizes

      Mid

    • Nominal Per-Look Z Cutoffs

      Mid

    • Naive vs Bonferroni Peeking FPR

      Senior

    • Always-Valid p-Value Sequence via mSPRT

      Senior

    • Confidence Sequence Stopping Rule

      Senior

    • Alpha-Spending Nominal Z-Boundaries

      Senior

    • mSPRT Always-Valid Stopping Rule

      Senior

    • Normal-Mixture Confidence Sequence

      Senior

    • Group-Sequential Boundary Crossing Readout

      Senior

    Multiple Testing Multi Variant Designs

    • Bonferroni-Corrected P-Values

      Junior

    • Family-Wise Error Rate for m Tests

      Junior

    • Count Significant P-Values With Bonferroni

      Junior

    • Benjamini-Hochberg FDR Correction

      Mid

    • Compare Multiple-Testing Corrections

      Mid

    • Benjamini-Hochberg Step-Up Reject Rule

      Mid

    • Holm-Bonferroni Reject Decisions

      Mid

    • A/B/n Readout With Holm Correction

      Mid

    • BH-Correct a Named P-Value Series

      Mid

    • Corrected Alpha Thresholds From multipletests

      Mid

    • Hand-Rolled Benjamini-Hochberg FDR

      Senior

    • Hand-Rolled Holm P-Value Adjustment

      Senior

    • 2x2 Factorial Interaction Effect

      Senior

    • Seeded FWER Inflation Simulation

      Senior

    • Benjamini-Yekutieli FDR by Hand

      Senior

    • Staged Ramp Rollout Gate

      Senior