Foundations & Intuition
What is Maximum Likelihood Estimation (MLE)? Explain its core principle and intuition in layman's terms.
What is the difference between a Probability Density Function (PDF) and a likelihood function? Why is the method called 'maximum likelihood' and not 'maximum probability'?
Why is the log-likelihood function often used instead of the raw likelihood function in practice? What are the mathematical and computational advantages?
Explain the distinction between a parameter, a statistic, an estimator, and an estimate.
Why is the likelihood function not a probability distribution over the parameter, and why doesn't it integrate to one?
Describe what a likelihood surface represents. What does a flat, sharply peaked, or multi-modal likelihood surface indicate about parameter estimation?
Likelihood Construction & Derivation
Derive the maximum likelihood estimator for the parameter p of a Bernoulli or binomial distribution.
Walk through the general step-by-step recipe for deriving a maximum likelihood estimator for a given statistical model.
Given that car speed on a highway follows a normal distribution N(μ, 25) after observing n car speeds, what is the Maximum Likelihood Estimator for μ?
Derive the maximum likelihood estimator for the rate parameter of a Poisson distribution.
Derive the Maximum Likelihood Estimator for the rate parameter lambda of an Exponential distribution.
Derive the maximum likelihood estimators for the mean and variance of a normal distribution.
Why does the likelihood take an i.i.d. product form, and how does the likelihood change when observations are not independent?
How do you verify that a solution to the score equation is actually a maximum rather than a minimum or saddle point?
How does deriving an MLE change when there are multiple parameters, and what roles do the score vector and Hessian play?
Derive the maximum likelihood estimators for the parameters of a uniform distribution, and why is this a special case where standard calculus methods may not apply?
Derive the maximum likelihood estimators for a multinomial distribution and explain how the constraint that probabilities sum to one is handled.
What does concavity of the log-likelihood guarantee about the MLE, and why does it matter for optimization?
Estimator Properties
What does it mean for an estimator to be consistent, and is an MLE always consistent?
Explain the invariance property of maximum likelihood estimators and provide an example of how it works.
Are maximum likelihood estimators generally biased or unbiased in finite samples? Provide an example.
What are some desirable properties of Maximum Likelihood Estimators?
Why is asymptotic unbiasedness a weaker guarantee than unbiasedness, and why do we accept it for MLEs?
When and why is the MLE for the variance of a normal biased, and how does dividing by n minus p relate to it?
What is sufficiency, and how does the factorization theorem let you identify a sufficient statistic?
Explain the concept of minimal sufficiency and how Rao-Blackwell can improve an estimator.
What is a UMVUE and how does it differ from the MLE?
Under what circumstances might an MLE not exist or not be unique, and what does it mean for an MLE to be on the boundary of the parameter space?
Asymptotic Theory & Fisher Information
What is Fisher information? What does it quantify, and how is it formally defined?
Why is Fisher information additive over independent observations, and what is the intuition that information equals likelihood curvature?
Explain the concept of asymptotic normality of the MLE. What does this imply about its sampling distribution for large samples?
Define asymptotic efficiency. How does the MLE achieve it, and what is its relationship to the Cramér-Rao lower bound?
Define the score in the context of maximum likelihood estimation. What information does it provide, and what is its expected value at the true parameter?
How does Fisher information relate to the variance of an efficient estimator?
What are the regularity conditions typically assumed for the asymptotic properties of MLEs to hold, and where might they be violated?
What is the difference between expected Fisher information and observed Fisher information, and when would you use each?
State the two equivalent forms of Fisher information and explain why they are equal under regularity conditions.
What is the delta method, and how is it used to obtain the standard error of a transformed MLE?
Standard Errors & Hypothesis Testing
Explain the core idea behind the Wald test, the likelihood ratio test, and the score test. What quantity does each test statistic evaluate?
What is deviance in the context of likelihood-based inference, and how is it used?
How are standard errors for maximum likelihood estimators typically calculated, and what role does the inverse Fisher or observed information matrix play?
Compare and contrast the Wald, likelihood ratio, and score tests in terms of computational cost, theory, and small-sample or boundary behavior. When would you prefer one over another?
What is Wilks' theorem, and why is it crucial for applying the likelihood ratio test?
What is profile likelihood, and how is it used to handle nuisance parameters or construct likelihood-based confidence intervals?
Explain the difference between likelihood-based confidence intervals and symmetric Wald intervals, and why they might produce different results in small samples.
Numerical Optimization
How can maximum likelihood be scaled to very large datasets, and what is the role of stochastic or mini-batch likelihood maximization?
When a closed-form MLE does not exist, what numerical optimization algorithms are commonly used, and can you explain the intuition behind one such as Newton-Raphson or Fisher scoring?
What is iteratively reweighted least squares and how does it relate to Fisher scoring?
Why can a log-likelihood have multiple local maxima, and what strategies address the local-versus-global optimum problem?
What is the log-sum-exp trick, and why is it needed when maximizing a log-likelihood numerically?
What is a quasi-Newton method like BFGS, and how does it differ from Newton-Raphson for maximizing a log-likelihood?
Em & Latent Variables
Explain the Expectation-Maximization algorithm and what types of problems involving latent variables or missing data it is designed to solve.
Describe the two main steps of the EM algorithm, the E-step and the M-step, and the objective of each.
Explain the role of latent variables in the EM algorithm and give an example of a model where EM is used because of them.
How does the EM algorithm guarantee that the observed-data log-likelihood never decreases with each iteration?
What are the limitations of the EM algorithm regarding convergence to local optima, and what practical considerations such as initialization matter for its success?
How can a likelihood become unbounded, for example a mixture component collapsing onto a single point, and what does that mean for the MLE?
What is the label-switching and identifiability problem in mixture models fit by maximum likelihood?
Bayesian & Regularization
What is Maximum Likelihood Estimation (MLE) and how does it differ from Bayesian inference?
How does placing a prior on a parameter turn likelihood maximization into penalized estimation, and why is a prior mathematically equivalent to a regularization term?
What is Maximum a posteriori (MAP) estimation, and how does it differ from Maximum Likelihood Estimation?
How do L1 (Lasso) and L2 (Ridge) regularization relate to MAP estimation with specific prior distributions?
Regression & Machine Learning Connections
How would you apply MLE for a regression problem?
Set up the likelihood for a linear model with i.i.d. Gaussian errors and show why maximizing it reduces to minimizing the sum of squared residuals.
Explain the concept of negative log-likelihood as a loss function, and give examples of how MSE and cross-entropy correspond to negative log-likelihoods under specific probabilistic assumptions.
How does overfitting manifest in maximum likelihood estimation as the number of parameters grows?
How does maximum likelihood estimation relate to the concept of empirical risk minimization in machine learning?
What is Kullback-Leibler divergence, and how does maximizing likelihood relate to minimizing KL divergence between the model and the true data-generating distribution?
How do Contrastive Divergence and Maximum Likelihood Estimation relate to training generative models, and how are these techniques applied in models like Boltzmann machines?
Alternative & Robust Methods
What are AIC and BIC, how do they use penalized likelihood for model selection, and why can't raw likelihood values be compared directly between models of different complexity?
Why is the MLE asymptotically efficient when a method-of-moments estimator generally is not, and what does that difference cost you in practice?
Discuss the behavior of MLE under model misspecification. What is a quasi-MLE, and what is the purpose of the Huber-White sandwich robust covariance estimator?
What is restricted maximum likelihood (REML) and why does it correct the bias in variance-component estimates?
Explain the ideas of pseudo-likelihood, composite likelihood, and partial likelihood, and when they are used instead of the full likelihood.
How does M-estimation generalize maximum likelihood estimation, and why is the MLE a special case?
What is quasi-likelihood, and how does it let you estimate parameters without fully specifying the distribution?
Practical Challenges & Data Issues
What are some common practical challenges or pitfalls when applying MLE, such as small sample sizes, identifiability, numerical stability, or sensitivity to outliers?
When is maximum likelihood estimation not the appropriate tool for a given problem?
What is the zero-frequency or zero-count problem in MLE, and why does smoothing or a prior help?
How would you sanity-check a fitted MLE to detect problems with the fit or optimization?
How does reparameterizing a model help enforce parameter constraints during maximum likelihood estimation?
How does maximum likelihood behave with very small samples, and what precautions should you take?
How do censored and truncated data affect the formulation of the likelihood function and the resulting MLE?