Fundamentals & Motivation
Explain the problem that variance reduction solves in online controlled experiments and how high variance impacts statistical power, sensitivity, and required sample size or runtime.
Explain the fundamental relationship between a metric's variance, statistical power, and the required sample size for an A/B test.
Explain the core benefits of using variance reduction techniques like CUPED from both a business and statistical perspective.
What are common sources of variance in product metrics during an online controlled experiment, and how do they impact the precision of treatment effect estimates?
How does high variance in a metric impact the minimum detectable effect (MDE) and the overall sensitivity of an A/B test?
Why is reducing the variance of a treatment effect estimate considered more efficient than simply increasing sample size to achieve higher sensitivity?
What does it mean for a variance reduction technique to reduce variance without introducing bias, and why is unbiasedness a non-negotiable constraint?
Why does a lower-variance metric count as a more sensitive metric, and how does that guide metric choice for variance reduction?
Cuped Concepts & Mechanics
Describe the intuition behind CUPED. How does it achieve variance reduction without introducing bias in the treatment effect estimate?
What is CUPED, and when would you use it in a product experiment? Explain how it changes the analysis, what assumptions make it useful, and when it may not help much.
Walk through the mechanics of how CUPED adjusts the observed metric in an A/B test, and the role of pre-existing data and the predicted baseline.
What are the key best practices for implementing CUPED, including metric selection, historical data, and data quality, and when might CUPED not be effective?
In a randomized experiment, how does CUPED differ from a simple difference-in-means estimator, and what specific advantage does it offer?
How much variance reduction is realistically achievable with CUPED in practice, and what factors determine the typical range?
Cuped Covariate Selection
What are common choices for the covariate in CUPED, and why is the pre-period value of the same metric often the strongest covariate?
Why is it crucial that the covariate used in CUPED be measured pre-treatment and be unaffected by the treatment, and what breaks if this assumption is violated?
Explain the relationship between the correlation coefficient between the covariate and the outcome and the amount of variance reduction achieved by CUPED. What level of correlation is generally considered worthwhile?
What are the practical considerations for choosing the appropriate length of the pre-period window for CUPED?
How would you extend CUPED to use multiple pre-experiment covariates, and what does that change about estimating the adjustment?
Practical Application & Tradeoffs
When and how would you use stratification, CUPED, or covariate adjustment to reduce variance in live online experiments?
How would you handle new users or users with very low pre-period activity who lack sufficient historical data for CUPED?
How would you A/B-test a ranking change involving low-traffic listings without burning months of statistical power?
Describe a scenario where CUPED might deliver almost no variance reduction, and how would you diagnose it?
What is the impact of variance reduction techniques like CUPED or post-stratification on required sample size or experiment runtime, especially with unbalanced client sizes or low sample sizes?
What engineering and data-infrastructure costs or complexities are associated with implementing and maintaining variance reduction techniques in an experimentation platform?
Under what circumstances might the effort and complexity of implementing variance reduction not be worthwhile for an A/B test?
When you combine multiple variance reduction techniques such as CUPED and stratification, why do you tend to see diminishing returns?
How does CUPED interact with a triggered analysis, where only a subset of users are exposed to the treatment?
Stratification Techniques
What is stratified randomization, and how does it help reduce variance in A/B experiments?
What are the potential risks or downsides of creating too many or too thin strata in a stratification approach?
Why do design-time variance reduction levers differ from analysis-time levers, and what are the practical tradeoffs of each?
Describe the difference between stratified randomization at design-time and post-stratification at analysis-time as variance reduction techniques, and when would you use each?
Explain how post-stratification can be applied as an analysis-time variance reduction technique, particularly with heavy-tailed metrics.
Heavy Tails & Metric Transformation
How can metric transformation, such as a logarithmic function, help reduce variance for skewed or heavy-tailed metrics?
Why do heavy tails and outlier whale users significantly reduce the sensitivity and statistical power of A/B tests?
How does binarising a metric or using a capped-count version of it serve as a variance-reduction strategy, and what does it cost?
Explain how capping, winsorizing, or trimming can be used for variance reduction in heavy-tailed metrics, and discuss the potential bias these methods introduce.
How do heavy-tailed metrics or outlier 'whale' users affect the sensitivity of an A/B test, and what techniques can address this for variance reduction along with their tradeoffs?
Regression Adjustment & Ancova
How does regression adjustment such as ANCOVA or Lin's estimator reduce variance in online controlled experiments, and what are its advantages or disadvantages compared to CUPED?
When using regression adjustment for variance reduction, what considerations are important regarding the treatment coefficient and the validity of standard errors?
Explain the relationship and equivalence between ANCOVA and CUPED as methods for covariate adjustment in randomized experiments.
What is Lin's estimator, how does adding treatment-covariate interactions differ from plain regression adjustment, and what problem does it address?
Can regression adjustment ever hurt in a randomized experiment, and what did Freedman's critique say about naive adjustment?
Validation Communication & Ratio Metrics
How would you sanity-check a variance-reduced analysis by confirming the point estimate is essentially unchanged while the confidence interval narrows?
How would you explain a variance-reduced experiment readout to non-technical stakeholders who are used to a plain difference-in-means result?
How would you validate the correct implementation and effectiveness of a variance reduction technique like CUPED using an A/A test?
Describe how the Delta Method is used to correctly estimate the standard error and variance of ratio metrics, and why this matters for accurate inference.
How would you implement CUPED for ratio metrics, and what statistical method is used to estimate the variance of such metrics?
When is the bootstrap preferable to the delta method for obtaining the standard error of a CUPED-adjusted or ratio metric estimate?
Cuped Mathematical Derivation
What is the mathematical form of the CUPED-adjusted metric, and how is the optimal coefficient theta typically estimated?
Explain the mathematical derivation of CUPED and why the estimator remains unbiased despite the adjustment.
Derive the optimal value for the coefficient theta in CUPED, and explain why this value minimizes the variance of the adjusted estimator.
Why is the CUPED coefficient theta typically estimated on data pooled across both the treatment and control arms rather than separately per arm?
Does estimating theta from the observed data introduce any finite-sample bias into CUPED, and why is it negligible in large experiments?
Control Variates & Ml Predicted Covariates
Explain the concept of control variates. How are they related to CUPED, and what are the requirements for a good control variate?
In the context of control variates, why is the variance gain driven by the correlation between the outcome and the control variate, and why must the control variate be unaffected by treatment?
How can machine learning models be used to construct a stronger predicted covariate for variance reduction, as in CUPAC?
How do antithetic variates and importance sampling relate to CUPED and control variates within the broader variance-reduction family?
How would you decide which covariate or combination of covariates gives the best predictive power for CUPAC without overfitting?
Experimental Design & Randomization
How do matched-pair designs reduce variance, and when are they appropriate in online experiments?
Explain the concept of clustered randomization and how it affects the variance of treatment effect estimates, and what strategies account for it.
What is the design effect in clustered randomization, and how does it relate to variance inflation in A/B tests?
Explain interleaving as a variance reduction technique for ranking evaluation, and why is it considered highly sensitive?
Describe switchback designs. How do they work, and in what scenarios are they effective for variance reduction?
What is the mismatch between the analysis unit and the randomization unit, and why does it matter for variance estimation?
How do common random numbers and paired or within-subject designs reduce variance compared to independent randomization?