83 Maximum Likelihood Estimation Interview Questions and Answers (2026)

MLE is a method for estimating a model's parameters by choosing the values that make the observed data most probable under the model. Intuitively: ask which parameter setting would have made this exact data most likely to occur? and pick that one.

• Core principle: Given data and a model family, define the likelihood (probability of the data as a function of the parameters) and maximize it.

• Layman's intuition:

  • You flip a coin 10 times and see 7 heads. Which bias explains that best? A coin with p=0.7 makes 7 heads far more plausible than p=0.1, so MLE picks 0.7.

  • It reverses the usual question: instead of "given the parameter, how likely is the data," it asks "given the data, which parameter is most credible."

• How it works mechanically: Write the likelihood, usually take the log, differentiate, set to zero, and solve for the parameter.

• Why it matters: MLE estimators are consistent, asymptotically efficient, and underlie much of classical statistics and many ML loss functions (cross-entropy, MSE under Gaussian noise).

A PDF treats the parameter as fixed and the data as variable: it tells you how probable different data are. A likelihood flips this: it fixes the observed data and treats the parameter as the variable. It's the same formula viewed as a function of different arguments.

• PDF: f(x | θ) as a function of x: θ is fixed; integrates to 1 over x; it is a genuine probability distribution.

• Likelihood: L(θ) = f(x | θ) as a function of θ: x is fixed (the observed data); θ varies; does NOT integrate to 1 over θ.

• Why "likelihood" not "probability":

  • For continuous data, f(x | θ) is a density, not a probability (the probability of any exact point is zero), so calling it a probability of θ would be doubly wrong.

  • The term (from Fisher) deliberately distinguishes a quantity over parameters from a true probability over outcomes.

We maximize the log-likelihood because the log is monotonic (it has the same maximizer) but turns products into sums, which is far easier to differentiate and far more numerically stable. It changes the optimum's location not at all, only the algebra and arithmetic.

• Products become sums: For i.i.d. data L(θ) = ∏ f(xi|θ), so log L = ∑ log f(xi|θ). Sums are much easier to differentiate term by term.

• Numerical stability: Multiplying many small probabilities underflows to zero in floating point; summing logs avoids this.

• Same maximizer: log is strictly increasing, so argmax L = argmax log L; the estimate is identical.

• Nice algebra with exponential families: Distributions like the normal have exp terms; the log cancels them, leaving simple polynomials to differentiate.

• Connection to information: The negative log-likelihood is the loss minimized in many ML models; its curvature (Fisher information) gives standard errors.

These four terms describe the chain from the true world to a computed number. A parameter is the unknown truth, an estimator is the recipe for guessing it, a statistic is any function of the data, and an estimate is the specific number that recipe produces on your sample.

• Parameter: A fixed, usually unknown property of the population/model (e.g. the true mean μ). Not random.

• Statistic: Any quantity computed purely from the sample (e.g. the sample mean, sample variance). Random, because the sample is random.

• Estimator: A statistic specifically used to approximate a parameter: a rule/function like θ̂ = (1/n)∑ xi. It is a random variable with a sampling distribution.

• Estimate: The realized numeric value of the estimator on a particular dataset (e.g. θ̂ = 4.2). A fixed number, not random.

• One-line memory hook: Estimator = the formula (random); estimate = its output on your data (a number); parameter = what you're chasing (unknown truth); statistic = any function of data.

For Bernoulli/binomial data the MLE of the success probability p is just the observed proportion of successes. We maximize the log-likelihood, whose derivative set to zero yields the sample frequency.

• Setup (Bernoulli, n i.i.d. trials): Likelihood: L(p) = p^k (1-p)^(n-k), where k = ∑ xi is the number of successes.

• Log-likelihood: log L = k log p + (n-k) log(1-p).

• Differentiate and solve:

  • d/dp: k/p - (n-k)/(1-p) = 0, which rearranges to k(1-p) = (n-k)p.

  • Result: p̂ = k/n (the sample proportion of successes).

• Notes:

  • The binomial adds a constant C(n,k) that drops out under differentiation, giving the same estimator.

  • This MLE is unbiased and intuitive: 7 heads in 10 flips gives p̂ = 0.7.

The recipe: write the likelihood as the joint density viewed as a function of the parameter, take the log, differentiate, set to zero, solve, and confirm it's a maximum.

1. Write the likelihood: For i.i.d. data, L(θ) = ∏ f(xᵢ; θ), treating the observed data as fixed and θ as the variable.

2. Take the log-likelihood: ℓ(θ) = ∑ log f(xᵢ; θ): turns products into sums and shares the same maximizer since log is monotonic.

3. Differentiate and form the score equation: Set the score ∂ℓ/∂θ = 0 and solve for θ.

4. Verify it's a maximum: Check the second derivative is negative (or the Hessian is negative definite for multiple parameters).

5. Check the boundary / support: When the max isn't at a stationary point (e.g. parameters defining the support), inspect endpoints directly.

With known variance, the MLE for μ is just the sample mean μ̂ = (1/n) ∑ xᵢ: the variance value (25) doesn't affect the estimator of the mean.

• Log-likelihood: ℓ(μ) = -n/2 · log(2π·25) − (1/(2·25)) ∑ (xᵢ − μ)².

• Differentiate in μ: ∂ℓ/∂μ = (1/25) ∑ (xᵢ − μ); the constant terms drop out.

• Set to zero: ∑ (xᵢ − μ) = 0 ⇒ μ̂ = x̄.

• Second derivative: −n/25 < 0, confirming a maximum.

For Poisson(λ), the MLE is the sample mean λ̂ = (1/n) ∑ xᵢ.

• Likelihood: pmf is e^{−λ} λ^{xᵢ} / xᵢ!, so L(λ) = ∏ e^{−λ} λ^{xᵢ} / xᵢ!.

• Log-likelihood: ℓ(λ) = −nλ + (∑ xᵢ) log λ − ∑ log(xᵢ!).

• Score equation: ∂ℓ/∂λ = −n + (∑ xᵢ)/λ = 0 ⇒ λ̂ = x̄.

• Confirm maximum: ∂²ℓ/∂λ² = −(∑ xᵢ)/λ² < 0.

For Exponential(λ) with density λ e^{−λx}, the MLE is the reciprocal of the sample mean λ̂ = 1/x̄ = n / ∑ xᵢ.

• Likelihood: L(λ) = ∏ λ e^{−λxᵢ} = λⁿ e^{−λ ∑ xᵢ}.

• Log-likelihood: ℓ(λ) = n log λ − λ ∑ xᵢ.

• Score equation: ∂ℓ/∂λ = n/λ − ∑ xᵢ = 0 ⇒ λ̂ = n / ∑ xᵢ.

• Confirm maximum: ∂²ℓ/∂λ² = −n/λ² < 0, so it is a maximum.

MLE picks the single parameter value that makes the observed data most probable, by maximizing the likelihood function. Bayesian inference instead treats the parameter as a random variable, combines a prior with the likelihood, and produces a full posterior distribution rather than one point estimate.

• MLE:

  • Maximize L(θ) = p(data | θ), usually its log for numerical stability.

  • Output is a single point estimate θ̂; uncertainty comes separately (e.g. from the Fisher information / asymptotics).

  • Frequentist: θ is fixed but unknown, data is random.

• Bayesian inference:

  • Applies Bayes' rule: p(θ | data) ∝ p(data | θ) p(θ).

  • Output is a whole posterior; uncertainty is built in (credible intervals, predictive distributions).

  • Requires choosing a prior.

• The bridge:

  • MLE equals MAP with a flat prior, and both MAP and MLE are point summaries; full Bayes keeps the entire distribution.

  • With lots of data the prior washes out and the posterior concentrates near the MLE.

The likelihood is a function of the parameter built from a density over the data, not a density over the parameter. Nothing in its construction forces it to integrate to one across θ, and in general it does not, so it is not a probability distribution over θ.

• It normalizes over the wrong variable: f(x | θ) integrates to 1 over x (for each fixed θ), not over θ.

• No probability structure on θ: In frequentist MLE, θ is a fixed unknown constant, not a random variable, so it has no distribution to integrate.

• Only relative values matter: Likelihoods are meaningful only up to a constant; ratios L(θ1)/L(θ2) carry the information, so absolute scale (and total area) is irrelevant.

• Contrast with Bayes: To get a genuine distribution over θ you multiply by a prior and normalize: posterior ∝ likelihood × prior. That normalization is exactly what the likelihood lacks on its own.

A likelihood surface is the plot of the (log-)likelihood value over the parameter space: height is how well each parameter setting explains the data. Its shape tells you how much information the data carry and how confidently you can estimate parameters.

• What it is: For one parameter it's a curve; for two it's a 3D surface; the MLE sits at the peak.

• Sharply peaked: Data strongly constrain the parameter: high curvature means high Fisher information, small standard errors, a precise estimate.

• Flat: Data are nearly uninformative about the parameter: large uncertainty, possible non-identifiability or too little data; the optimizer may wander.

• Multi-modal: Multiple competing explanations (common with mixtures/latent variables); gradient methods can converge to a local rather than global maximum, so use multiple starts.

• Practical link: The curvature at the peak (the negative Hessian) approximates the inverse covariance of the estimate, so surface shape directly yields confidence intervals.

For i.i.d. normal data the MLEs are the sample mean and the (biased, divide-by-n) sample variance. We write the log-likelihood, differentiate with respect to μ and σ², and set the derivatives to zero.

• Setup: Log-likelihood: log L = -n/2 log(2π) - n/2 log σ² - (1/2σ²) ∑ (xi - μ)².

• Solve for μ: ∂/∂μ gives (1/σ²) ∑ (xi - μ) = 0, so μ̂ = (1/n) ∑ xi (the sample mean).

• Solve for σ²: ∂/∂σ² gives -n/(2σ²) + (1/2σ⁴) ∑ (xi - μ)² = 0, so σ̂² = (1/n) ∑ (xi - μ̂)².

• Key caveat: σ̂² divides by n, not n-1, so it is biased; the unbiased sample variance uses n-1. Bias vanishes as n → ∞.

The product form comes directly from independence: the joint density of independent variables factorizes into a product of marginals. When observations are dependent, you must use the true joint density, typically factored via the chain rule of conditional probabilities.

• Why the product: Independence gives f(x₁,...,xₙ) = ∏ f(xᵢ); identical distribution means the same f(·; θ) for every term.

• Log turns it additive: ℓ(θ) = ∑ log f(xᵢ; θ): makes differentiation and asymptotic theory tractable.

• When not independent:

  • Use the chain rule: f(x₁,...,xₙ) = ∏ f(xᵢ | x₁,...,xᵢ₋₁), so the likelihood involves conditional densities.

  • Examples: time series (AR models) condition on past values; correlated data use a joint form like the multivariate normal with a covariance matrix.

Check the curvature: at a maximum the second derivative of the log-likelihood must be negative (or the Hessian negative definite in multiple dimensions), and ideally the log-likelihood is globally concave so the stationary point is the unique global maximum.

• Second-derivative test: Single parameter: confirm ∂²ℓ/∂θ² < 0 at the solution.

• Multivariate case: The Hessian must be negative definite (all eigenvalues negative); indefinite means a saddle point.

• Global concavity: If ℓ is concave everywhere (true for exponential-family models in natural parameterization), any stationary point is the global max.

• Boundary and endpoint checks: Compare with limits as θ approaches the edges of its domain to rule out larger values on the boundary.

With multiple parameters θ = (θ₁,...,θₖ), the single derivative becomes the score vector of partial derivatives set jointly to zero, and the second-derivative check becomes the Hessian matrix, which must be negative definite at the solution.

• Score vector:

  • The gradient ∇ℓ(θ) of partials ∂ℓ/∂θⱼ; set it to the zero vector to get a system of equations.

  • Solve the equations simultaneously (some may couple the parameters, e.g. mean and variance).

• Hessian matrix:

  • The matrix of second partials ∂²ℓ/∂θⱼ∂θₖ; negative definite confirms a maximum.

  • Its negative, −E[Hessian], is the Fisher information matrix used for asymptotic variances and standard errors.

• When no closed form exists: Solve numerically (Newton-Raphson uses the score and Hessian; Fisher scoring uses expected information).

• Example: for N(μ, σ²), the joint score gives μ̂ = x̄ and σ̂² = (1/n)∑(xᵢ − x̄)².

An estimator is consistent if it converges in probability to the true parameter as the sample size grows: more data drives the estimate arbitrarily close to the truth. MLEs are consistent under standard regularity conditions, but not always.

• Definition: θ̂_n → θ in probability as n → ∞: for any ε, P(|θ̂_n - θ| > ε) → 0.

• When MLE is consistent:

  • Under regularity conditions: correct/identifiable model, parameter interior to the space, smoothness of the likelihood, and support not depending on θ.

  • Intuition: the expected log-likelihood is maximized at the true θ (via KL divergence), and the sample average converges to that expectation.

• When it can fail:

  • Number of parameters grows with n (Neyman-Scott problem: too many nuisance parameters).

  • Misspecified model, non-identifiability, or violated regularity (e.g. boundary/support issues).

The invariance property says that if θ̂ is the MLE of θ, then for any function g the MLE of g(θ) is simply g(θ̂). You can transform the estimate directly rather than re-deriving the likelihood in the new parameterization.

• Statement:

  • MLE of g(θ) equals g applied to the MLE of θ, for any (even non-invertible) function.

  • For non-injective g this uses an induced (profile) likelihood, but the result still holds.

• Why it holds: Maximizing the likelihood over θ and then mapping through g identifies the same maximizing point in the transformed space.

• Example:

  • For N(μ,σ²), the MLE of σ² is σ̂² = Σ(x_i-x̄)²/n; the MLE of the standard deviation σ is just √σ̂².

  • For Bernoulli, if p̂ = x̄, the MLE of the odds p/(1-p) is p̂/(1-p̂).

• Caveat: Invariance is exact for point estimates only; unbiasedness is not preserved under nonlinear g.

MLEs are generally biased in finite samples: consistency guarantees the bias vanishes as n grows, but for small samples a nonzero bias is the norm rather than the exception.

• MLE is not required to be unbiased:

  • The likelihood is optimized for fit, not for zero bias, so finite-sample bias is common.

  • Invariance is part of the reason: the MLE of a nonlinear function is the function of the MLE, and unbiasedness is not preserved under nonlinear transforms.

• Classic example: variance of a normal:

  • The MLE is σ̂² = (1/n) Σ(xᵢ - x̄)², whose expectation is (n-1)/n · σ², so it underestimates by a factor (n-1)/n.

  • The unbiased fix divides by n-1 instead of n.

• Bias shrinks with sample size: MLEs are typically asymptotically unbiased: here bias = -σ²/n → 0.

MLEs are prized because, under regularity conditions, they have excellent large-sample behavior: they are consistent, asymptotically normal, asymptotically efficient, and invariant under reparameterization.

• Consistency: The estimate converges in probability to the true parameter as n → ∞.

• Asymptotic normality: √n(θ̂ - θ) converges to a normal distribution, enabling standard errors and confidence intervals.

• Asymptotic efficiency: It attains the Cramér-Rao lower bound asymptotically: no consistent estimator has smaller asymptotic variance.

• Invariance: If θ̂ is the MLE of θ, then g(θ̂) is the MLE of g(θ).

• Caveat: These are mostly asymptotic; finite-sample bias and sensitivity to model misspecification remain.

Asymptotic unbiasedness only promises the bias disappears in the limit as the sample grows, whereas unbiasedness holds exactly at every finite sample size. We accept the weaker property for MLEs because it comes bundled with consistency and efficiency, and the finite-sample bias is usually small and correctable.

• Different scope of the guarantee:

  • Unbiased: E[θ̂] = θ for all n.

  • Asymptotically unbiased: E[θ̂] → θ as n → ∞; at finite n a gap can exist.

• Why we tolerate it:

  • Unbiasedness alone is not enough: an unbiased estimator can have huge variance. We care about total error (MSE = bias² + variance).

  • MLEs trade a small vanishing bias for minimal asymptotic variance (efficiency).

  • The bias often has a known form (e.g. order 1/n) and can be corrected.

• Practical view: With enough data the distinction becomes negligible; the efficiency and generality of MLE outweigh a shrinking bias.

The normal-variance MLE is biased because it estimates the mean from the same data used to compute the deviations, which "uses up" a degree of freedom and makes the sum of squared residuals systematically too small. Dividing by n - p instead of n (with p estimated mean parameters) restores unbiasedness.

• Source of the bias:

  • The MLE σ̂² = (1/n) Σ(xᵢ - x̄)² measures deviations from the sample mean x̄, which minimizes that sum of squares, so it underestimates the true spread around μ.

  • Result: E[σ̂²] = (n-1)/n · σ².

• The degrees-of-freedom correction:

  • Estimating the mean costs one degree of freedom, so dividing by n-1 gives the unbiased s².

  • Generalization: in a regression with p fitted mean parameters, the unbiased variance estimate divides the residual sum of squares by n - p.

• Reconciliation: If μ were known, dividing by n would already be unbiased; the correction only accounts for parameters estimated from the data.

Fisher information quantifies how much information a sample carries about an unknown parameter: the sharper (more curved) the log-likelihood around its peak, the more precisely the parameter can be estimated. It is defined as the variance of the score, equivalently the expected negative curvature of the log-likelihood.

• Two equivalent definitions:

  • Variance of the score: I(θ) = E[(∂/∂θ log L)²].

  • Expected negative second derivative: I(θ) = -E[∂²/∂θ² log L] (under regularity conditions).

• What it quantifies:

  • Curvature of the log-likelihood: high information = sharply peaked likelihood = precise estimation.

  • It is additive over independent observations: I_n(θ) = n · I₁(θ).

• Why it matters:

  • Its inverse is the Cramér-Rao lower bound and the asymptotic variance of the MLE.

  • In multiparameter models it becomes the Fisher information matrix, whose inverse gives the asymptotic covariance.

Fisher information is additive because independent observations have log-likelihoods that add, and the second derivative (curvature) of a sum is the sum of curvatures. The curvature intuition: a sharply peaked log-likelihood pins down the parameter tightly, so high curvature means high information.

• Additivity:

  • Independence gives ℓ(θ) = Σ ℓᵢ(θ), so −ℓ''(θ) = Σ −ℓᵢ''(θ) and taking expectations gives Iₙ(θ) = n·I₁(θ).

  • Each independent data point contributes its own information; more data means proportionally more information.

• Curvature = information:

  • A steep, narrow peak in the log-likelihood means nearby θ values are much less likely, so the data strongly discriminate the parameter.

  • A flat log-likelihood means many θ values fit almost equally well: little information, large variance.

  • Formally the variance of the MLE ≈ inverse curvature, 1/I(θ).

• Why √n shows up: Since information scales with n, variance scales with 1/n and standard error with 1/√n.

All three test a null hypothesis about parameters but look at different features of the log-likelihood: Wald measures how far the MLE is from the null, the likelihood ratio compares heights of the log-likelihood, and the score checks the slope at the null. Asymptotically they are equivalent and each is chi-squared under the null.

• Wald test:

  • Uses the distance (θ̂ − θ₀) scaled by information: W = (θ̂ − θ₀)² · I(θ̂).

  • Only needs the unrestricted MLE; can behave poorly near boundaries or with strong nonlinearity.

• Likelihood ratio test:

  • Compares log-likelihood heights: LR = 2[ℓ(θ̂) − ℓ(θ₀)].

  • Needs both restricted and unrestricted fits; generally the most reliable of the three.

• Score (Lagrange multiplier) test:

  • Uses the slope of the log-likelihood at the null: S = U(θ₀)² / I(θ₀), where U is the score.

  • Only needs the restricted fit under H₀: useful when the full model is hard to estimate.

• Common thread: Each → χ² with df = number of restrictions; they agree asymptotically but differ in finite samples.

Deviance is a goodness-of-fit measure equal to twice the difference in log-likelihood between the saturated (perfect-fit) model and the fitted model. It generalizes the residual sum of squares to likelihood-based models like GLMs.

• Definition: D = 2(ℓ_saturated − ℓ_model); smaller deviance means a closer fit to the data.

• Uses:

  • Comparing nested models: the difference in deviances is exactly the LR statistic, referenced to a chi-squared.

  • Overall goodness of fit and computing residuals (deviance residuals) in GLMs.

  • Serves as the loss minimized when fitting a GLM (equivalent to maximizing likelihood).

• Caveats: Its absolute value as a chi-squared fit test is only valid under certain conditions (e.g. not for sparse binary data); differences between nested models are more trustworthy.

Expectation-Maximization (EM) is an iterative algorithm for finding maximum-likelihood (or MAP) estimates when the likelihood is hard to maximize directly because some data is unobserved. It repeatedly fills in the missing information using the current parameters, then re-estimates the parameters, and this cycle is much easier than maximizing the messy marginal likelihood in one shot.

• The core problem it solves:

  • The observed-data log-likelihood involves a sum/integral over latent variables (log sum_z p(x,z|θ)), which has no clean closed-form maximizer.

  • If the latent variables were known, the complete-data likelihood would be easy to maximize (often just weighted counts/means).

• Types of problems:

  • Latent-variable models: Gaussian mixtures, hidden Markov models, latent class / topic models.

  • Missing data: estimation when some entries are missing at random.

  • Coarsened / censored / truncated data: e.g. survival data where only a bound is observed.

• Why it is attractive: each iteration reduces a hard marginal maximization to an easy complete-data maximization, and it never decreases the observed log-likelihood.

EM alternates between an E-step that computes the expected complete-data log-likelihood under the current parameters, and an M-step that maximizes that expectation to get updated parameters. The E-step builds a tractable surrogate objective (the Q-function); the M-step optimizes it.

• E-step (Expectation):

  • Using current parameters θ_old, compute the posterior over the latent variables p(z | x, θ_old).

  • Form Q(θ | θ_old) = E_{z|x,θ_old}[ log p(x, z | θ) ], the expected complete-data log-likelihood.

  • Objective: replace unknown latents with their soft assignments (responsibilities) to get a function of θ that is easy to maximize.

• M-step (Maximization):

  • Set θ_new = argmax_θ Q(θ | θ_old).

  • For exponential families this is often a closed-form weighted update (e.g. GMM means/covariances weighted by responsibilities).

  • Objective: improve the parameters given the imputed latent structure.

• Iterate E and M until the log-likelihood (or parameters) converges.

Latent variables are the unobserved quantities that, if we knew them, would make the likelihood easy to write and maximize. EM exists precisely because these variables are hidden: it treats them as expectations in the E-step, turning an intractable marginal likelihood into a tractable complete-data problem.

• Role of latent variables:

  • They encode hidden structure: which cluster a point came from, which hidden state generated an observation, an unobserved feature value.

  • The observed likelihood marginalizes them out (sum/integral over z), which couples parameters and blocks a closed-form MLE.

  • EM's E-step replaces them with their posterior expectations (soft assignments), decoupling the problem so the M-step is simple.

• Canonical example: Gaussian Mixture Model:

  • Latent variable z_i = which component generated point x_i.

  • E-step: compute responsibilities γ_ik = p(z_i = k | x_i, θ).

  • M-step: update each component's weight, mean, and covariance as responsibility-weighted averages.

• Other latent-variable uses: HMMs (hidden states via forward-backward), factor analysis / probabilistic PCA, and mixed models.

A prior adds a term to the objective: maximizing the log-posterior is maximizing log-likelihood plus log-prior, and that log-prior acts exactly as a penalty on parameter values. So a regularizer is mathematically just the negative log of a prior distribution, and MAP estimation is penalized maximum likelihood.

• The derivation:

  • Posterior p(θ|D) ∝ p(D|θ) p(θ).

  • Take logs: log p(θ|D) = log p(D|θ) + log p(θ) + const.

  • Maximizing this = minimizing −log p(D|θ) − log p(θ): a loss plus a penalty.

• Why prior ≡ regularizer:

  • The penalty term −log p(θ) is large where the prior says θ is unlikely, pulling estimates toward high-prior regions.

  • A flat (improper uniform) prior contributes a constant, so MAP collapses back to plain MLE, no regularization.

• Consequence: regularization strength is just prior concentration; a tighter prior means a heavier penalty.

MAP estimation chooses the parameter that maximizes the posterior distribution, i.e. the most probable θ given the data and a prior. It differs from MLE by including a prior term, so it is regularized MLE and reduces to MLE when the prior is uniform.

• Definition:

  • θ̂_MAP = argmax p(θ|D) = argmax [log p(D|θ) + log p(θ)].

  • θ̂_MLE = argmax log p(D|θ): no prior term.

• Key differences:

  • MAP incorporates prior beliefs, so it's more stable with small samples and combats overfitting.

  • MLE is a special case of MAP with a flat prior.

  • Both are point estimates, not full posteriors, so both discard uncertainty (unlike full Bayes).

• Caveats:

  • MAP is not reparameterization-invariant: the mode moves under nonlinear transforms of θ, whereas the MLE's transformation is straightforward.

  • As data grows, the likelihood dominates and MAP converges to MLE.

For regression you specify a probabilistic model for the response given the inputs, write the likelihood of the observed targets, and choose parameters that maximize the (log-)likelihood. The choice of noise/response distribution determines the resulting loss: Gaussian errors give least squares, other distributions give other losses.

• Steps:

  1. Assume a distribution for y | x, e.g. y = xᵀβ + ε with ε ~ N(0, σ²).

  2. Write the joint likelihood of all i.i.d. observations as a product of densities.

  3. Take the log to turn the product into a sum (the log-likelihood).

  4. Maximize w.r.t. parameters: set the gradient to zero or use an optimizer.

• Distribution drives the loss:

  • Gaussian errors → minimizing sum of squared residuals (OLS).

  • Laplace errors → minimizing absolute residuals (robust/median regression).

  • Bernoulli response with logit link → logistic regression via cross-entropy.

  • Poisson response → Poisson regression: the whole GLM family is MLE.

• Add a prior/penalty (L2, L1) to get regularized MAP versions (Ridge, Lasso).

With i.i.d. Gaussian errors the log-likelihood is a constant minus a term proportional to the sum of squared residuals, so maximizing likelihood over the coefficients is identical to minimizing that sum: the OLS solution and the MLE coincide.

• Model:

  • yᵢ = xᵢᵀβ + εᵢ, with εᵢ ~ N(0, σ²) i.i.d.

  • So yᵢ | xᵢ ~ N(xᵢᵀβ, σ²).

• Likelihood and log-likelihood: Product of n Gaussian densities; taking logs gives ℓ(β,σ²) = −(n/2)log(2πσ²) − (1/2σ²) Σ(yᵢ − xᵢᵀβ)².

• Why it reduces to least squares:

  • Only the last term depends on β, and it's negative times the residual sum of squares divided by 2σ².

  • Maximizing ℓ over β = minimizing Σ(yᵢ − xᵢᵀβ)², regardless of σ².

  • Hence β̂_MLE = β̂_OLS = (XᵀX)⁻¹Xᵀy.

• Note: the variance MLE σ̂² = RSS/n is biased; the unbiased estimator divides by n − p.

Negative log-likelihood (NLL) turns MLE into a minimization problem: since maximizing log p(data | θ) equals minimizing its negative, NLL is a natural loss. Choosing the observation noise model then determines the exact loss form: Gaussian noise gives MSE, and a categorical/Bernoulli model gives cross-entropy.

• Why negate the log-likelihood:

  • Logs turn products of per-example densities into sums (numerically stable, easy to differentiate).

  • Optimizers minimize by convention, so we minimize -log L instead of maximizing L.

• MSE from a Gaussian assumption:

  • Assume y = f(x) + ε with ε ~ N(0, σ²), fixed variance.

  • The per-point NLL is (y - f(x))² / (2σ²) + const, so minimizing it is exactly minimizing squared error.

• Cross-entropy from a categorical/Bernoulli assumption:

  • Model outputs class probabilities p(y | x); the likelihood of the true label is that probability.

  • NLL becomes -Σ y_k log p_k, which is the cross-entropy between the one-hot target and the prediction.

• Takeaway: a loss function is a modeling choice, not arbitrary. Picking MSE assumes Gaussian residuals; picking cross-entropy assumes a discrete outcome distribution.

MLE has no built-in complexity control: it fits parameters to maximize the likelihood of the training data. As the parameter count grows, the model increasingly explains noise as if it were signal, so training likelihood keeps rising while generalization degrades.

• Why it happens:

  • Adding parameters can only increase (never decrease) the maximum achievable training likelihood.

  • With enough parameters the model interpolates the sample, driving training NLL toward zero but capturing idiosyncratic noise.

• Symptoms:

  • Large gap between training and validation likelihood.

  • Estimated variances shrink toward zero (e.g. Gaussian MLE variance underestimates), and coefficients blow up.

• Remedies:

  • Add a prior / penalty (MAP, L1/L2), turning pure MLE into regularized estimation.

  • Use penalized-likelihood criteria (AIC, BIC) or cross-validation to choose model size.

• Intuition: MLE trusts the data completely; without regularization or more data, that trust becomes overfitting as capacity rises.

AIC and BIC are model-selection criteria that penalize the maximized log-likelihood by a function of the parameter count, so more complex models must earn their extra fit. You can't compare raw maximized likelihoods across models because adding parameters mechanically increases the likelihood, so the biggest model always wins.

• AIC:

  • AIC = 2k - 2·log L, where k is the number of parameters.

  • Rooted in KL divergence: estimates expected predictive loss on new data. Favors predictive accuracy, penalty is constant per parameter.

• BIC:

  • BIC = k·log(n) - 2·log L, with sample size n.

  • Approximates the Bayesian model evidence; penalty grows with n, so it favors simpler models and is consistent (picks the true model as n → ∞ if it's in the set).

• Why raw likelihood can't be compared:

  • A nested larger model can always match or beat the smaller model's likelihood, so higher likelihood alone doesn't mean better generalization.

  • The penalty term is what makes cross-complexity comparison meaningful; lower AIC/BIC is preferred.

• Note: AIC tends to select richer models (better for prediction); BIC tends to select sparser models (better for identifying true structure).

MLE is asymptotically efficient because it uses the full likelihood, so it attains the Cramér-Rao lower bound (the smallest possible asymptotic variance). Method-of-moments (MoM) only matches a few chosen moments and discards the rest of the distributional information, so it is generally consistent but not efficient, costing you larger standard errors and wider confidence intervals for the same data.

• Why MLE is efficient:

  • Its asymptotic variance equals the inverse Fisher information I(θ)⁻¹, the Cramér-Rao bound.

  • The score function extracts all the information the likelihood provides about θ.

• Why MoM usually isn't:

  • It equates sample moments to theoretical ones; those moment conditions are a lossy summary of the data.

  • Its asymptotic variance is generally larger than I(θ)⁻¹ (equal only in special cases where the moments are sufficient statistics).

• The practical cost:

  • To reach the same precision as MLE, MoM needs more data (lower statistical efficiency).

  • Trade-off: MoM is often simpler, has closed forms, needs no optimization, and can be more robust when the full likelihood is hard or misspecified, so it's a useful starting point or fallback.

MLE has excellent asymptotic properties but can behave badly in finite samples: watch for bias and instability in small samples, non-identifiable parameters, poorly conditioned or multimodal optimization, and extreme sensitivity to outliers or model misspecification.

• Small samples:

  • MLE is only asymptotically unbiased; finite-sample bias can be large (e.g. variance underestimation, separation in logistic regression giving infinite estimates).

  • Remedies: bias correction (Firth), penalization, or a prior.

• Identifiability:

  • If different parameters give the same likelihood, the surface has a ridge and the MLE is not unique (common in mixtures, latent-variable, over-parameterized models).

  • Diagnose with a rank-deficient or near-singular Hessian.

• Numerical stability:

  • Always maximize the log-likelihood, not the likelihood, to avoid underflow.

  • Multimodal surfaces need multiple starting values; flat regions and boundary optima (variance = 0) break Newton-type solvers.

• Sensitivity to outliers and misspecification:

  • An unbounded score (e.g. Gaussian) lets a single extreme point dominate the estimate; robust M-estimators bound this influence.

  • Under misspecification MLE converges to the closest (KL) wrong model, so use sandwich standard errors.

MLE is the wrong tool when its core assumptions fail or when a different criterion better matches the goal: for example when the likelihood is unknown or intractable, when the model is badly misspecified, when data are too scarce, or when prediction/decision loss (not fit to a density) is what you actually care about.

• No reliable likelihood: Intractable normalizing constants or unknown distributions: use quasi/composite likelihood, GMM, or simulation-based (ABC) methods instead.

• Small or sparse data: MLE can be unstable, overfit, or diverge (separation, zero counts); regularized or Bayesian estimation with a prior is more reliable.

• Heavy contamination / outliers: An unbounded influence function makes MLE nonrobust; prefer robust M-estimators.

• Goal is prediction or decisions, not the density: Minimizing a task-specific loss or optimizing predictive performance may beat maximizing likelihood.

• Rich prior information exists: A Bayesian (MAP/posterior) approach coherently incorporates it; pure MLE ignores it.

• Model is known to be misspecified: MLE converges to the KL-closest wrong model; be sure that target is meaningful, and at minimum use robust standard errors.

The zero-frequency problem is that MLE assigns probability zero to any category or event never seen in the training sample, which is almost always overconfident and breaks downstream products of probabilities; smoothing or a prior fixes it by pulling estimates away from the empirical extremes toward nonzero values.

• Why MLE produces zeros:

  • For a multinomial/categorical, the MLE of a category probability is its observed relative frequency, so count 0 gives probability exactly 0.

  • An unseen event then makes a whole likelihood or n-gram product collapse to 0 (or log-likelihood to minus infinity).

• Why it is wrong: Not observing something in a finite sample is not evidence it is impossible; it just reflects limited data.

• How smoothing / a prior helps:

  • Add-one (Laplace) or add-k smoothing adds pseudo-counts so every category gets nonzero mass.

  • This is exactly the posterior mean under a Dirichlet prior: the pseudo-counts are the prior's concentration parameters, so smoothing IS Bayesian regularization.

  • The estimate becomes (count + α) / (N + α·K), shrinking toward the uniform prior when data are sparse and approaching the MLE as N grows.

Sanity-checking an MLE means verifying both that the optimizer converged to a true interior maximum and that the resulting model is statistically sensible: check gradients, curvature, robustness to restarts, and fit diagnostics.

• Confirm the optimizer actually converged:

  • Gradient (score) of the log-likelihood should be near zero at the solution.

  • Check the reported convergence flag, iteration count, and that the log-likelihood stopped increasing meaningfully.

• Verify it's a maximum, not a saddle or boundary:

  • The Hessian of the negative log-likelihood should be positive definite (all positive eigenvalues).

  • If an estimate sits on a constraint boundary (e.g. a variance at 0), the standard asymptotics may not apply.

• Test robustness with multiple starts: Re-run from several random or dispersed initial values; consistent convergence argues against a local optimum on a multimodal surface.

• Sanity-check the estimates and their uncertainty:

  • Are values physically plausible and in-range? Extreme or degenerate values often signal identifiability problems.

  • Huge or NaN standard errors (from a near-singular Hessian) suggest weak identification or collinearity.

• Check the fit against the data:

  • Residual/QQ plots, posterior-predictive style simulations, or comparing fitted vs. empirical moments.

  • Overlay the fitted density/curve on a histogram to catch gross model misspecification.

• Guard against numerical issues: Work in log-space, rescale predictors, and use analytic gradients where possible to avoid ill-conditioning.

Reparameterization replaces a constrained parameter with an unconstrained one via a smooth transform, so the optimizer can search all of the real line while the original parameter stays valid: you optimize freely and map back.

• Map constraints to unconstrained space:

  • Positivity (σ > 0): optimize θ = log σ, recover σ = exp(θ).

  • Probability (0 < p < 1): optimize the logit θ = log(p/(1-p)), recover with the sigmoid.

  • Simplex (probabilities summing to 1): use a softmax over unconstrained logits.

• Why it helps optimization:

  • General-purpose optimizers assume unbounded parameters; without transforms they can step outside the valid region and produce NaNs.

  • It avoids fragile boundary handling and often improves conditioning of the surface.

• Key correctness caveat: invariance:

  • MLE is invariant to reparameterization, so the point estimate maps back exactly: the argmax is preserved.

  • But standard errors must be transformed too (delta method or refit the Hessian in the original parameterization), since curvature changes under the transform.

MLE is only guaranteed to be good asymptotically: in small samples it can be badly biased, have large variance, and even hit boundary or non-existent solutions, so its optimality properties should not be trusted blindly.

• Bias appears in finite samples:

  • MLE is consistent but not generally unbiased: e.g. the MLE of variance divides by n, systematically underestimating σ² for small n.

  • Logistic regression MLE is biased away from zero in small samples.

• High variance and unstable estimates:

  • Standard errors from the inverse Hessian are asymptotic approximations and can be too optimistic when n is small.

  • Wald-based confidence intervals may have poor coverage; prefer likelihood-ratio or profile-likelihood intervals.

• Degenerate or boundary solutions:

  • Separation in logistic regression drives coefficients to ±∞; the MLE effectively doesn't exist.

  • Estimated variances can collapse to zero with too few observations.

• Precautions:

  • Use bias corrections or penalized likelihood (e.g. Firth's method) to tame small-sample bias and separation.

  • Add regularization or a prior (shrinking toward a MAP/Bayesian estimate) to stabilize.

  • Prefer likelihood-ratio intervals and use the bootstrap to assess actual variability rather than relying on asymptotic formulas.

  • Keep the model parsimonious: too many parameters relative to n makes everything worse.

For Uniform(0, θ), the MLE is θ̂ = max(xᵢ): calculus fails because the likelihood is maximized at a boundary, not at a point where the derivative vanishes.

• Write the likelihood: Each density is 1/θ for 0 ≤ xᵢ ≤ θ, so L(θ) = θ⁻ⁿ but only if θ ≥ max(xᵢ), else 0.

• Why calculus breaks down: θ⁻ⁿ is strictly decreasing in θ, so its derivative is never zero; there is no interior stationary point.

• Argue from monotonicity: To make θ⁻ⁿ as large as possible, pick θ as small as allowed; the constraint θ ≥ max(xᵢ) forces θ̂ = max(xᵢ).

• Note the bias: This MLE underestimates θ; the unbiased fix is (n+1)/n · max(xᵢ).

For a multinomial with K categories, the MLE of each cell probability is its observed relative frequency, p_k = n_k / n. The sum-to-one constraint is enforced with a Lagrange multiplier, which turns out to equal the total count.

• Setup: Counts n_1,...,n_K with sum n; log-likelihood ℓ = Σ n_k log p_k (dropping the multinomial coefficient, which is constant in p).

• Handle the constraint with a Lagrangian:

  • Maximize Σ n_k log p_k - λ(Σ p_k - 1), since probabilities must satisfy Σ p_k = 1.

  • Set derivative to zero: n_k / p_k - λ = 0 gives p_k = n_k / λ.

• Solve for λ using the constraint:

  • Sum both sides: Σ p_k = Σ n_k / λ = 1, so λ = Σ n_k = n.

  • Therefore p̂_k = n_k / n, the empirical frequencies.

• Interpretation: The multiplier λ is exactly the normalizer that projects the unconstrained solution back onto the simplex; a zero count gives p̂_k = 0, a boundary estimate.

If the log-likelihood is concave in the parameter, any stationary point (zero gradient) is a global maximum, so the MLE is unique and optimization is well behaved. Strict concavity guarantees a single maximizer.

• Global optimality: For a concave function, no local maximum can be inferior to another: a critical point is the global max, so you never get trapped in a bad local optimum.

• Uniqueness: Strict concavity (negative-definite Hessian) means at most one maximizer, removing ambiguity in the estimate.

• Why it matters for optimization:

  • Gradient-based methods (Newton-Raphson, gradient ascent) converge reliably to the answer, and any solver output is trustworthy.

  • Convex optimization theory applies directly (maximizing a concave ℓ = minimizing a convex -ℓ).

• Common examples:

  • Exponential-family log-likelihoods in the natural parameterization are concave; logistic regression's log-likelihood is concave.

  • Caveat: concavity in one parameterization need not hold after a nonlinear reparameterization, and many models (mixtures, neural nets) have non-concave likelihoods with multiple modes.

A statistic is sufficient for a parameter if it captures all the information the data carry about that parameter: conditional on the statistic, the data's distribution no longer depends on the parameter. The factorization theorem identifies sufficiency by checking whether the likelihood splits into a piece involving the parameter only through the statistic and a piece free of the parameter.

• Definition: T(X) is sufficient for θ if P(X = x | T = t) does not depend on θ; knowing T is as good as knowing the full sample for estimating θ.

• Factorization theorem:

  • T is sufficient iff the joint density factors as f(x|θ) = g(T(x), θ) · h(x), where h is free of θ.

  • Practically: write the likelihood, and whatever function of the data θ must interact with is your sufficient statistic.

• Example: For N(μ, σ²) the sum Σx_i (or the pair with Σx_i²) is sufficient; for Poisson, Σx_i is sufficient.

• Why it matters: MLEs are functions of sufficient statistics, and sufficiency underlies data reduction and Rao-Blackwell improvement.

A minimal sufficient statistic is the coarsest sufficient statistic: it achieves the greatest data reduction while still retaining all information about the parameter. Rao-Blackwell then says conditioning any unbiased estimator on a sufficient statistic yields another unbiased estimator with variance no larger, so improving estimators means projecting onto the sufficient statistic.

• Minimal sufficiency:

  • A sufficient statistic T is minimal if it is a function of every other sufficient statistic: no further reduction is possible without losing information.

  • Test (Lehmann-Scheffé): T(x)=T(y) exactly when the likelihood ratio f(x|θ)/f(y|θ) is constant in θ.

• Rao-Blackwell theorem:

  • Given an unbiased estimator δ(X) and sufficient T, define δ*(T) = E[δ(X) | T].

  • δ* is unbiased and Var(δ*) ≤ Var(δ) by the law of total variance; conditioning removes noise unrelated to θ.

  • It is well defined precisely because conditioning on a sufficient statistic gives a distribution free of θ.

• Connection to UMVUE: Rao-Blackwellizing onto a complete sufficient statistic yields the UMVUE (Lehmann-Scheffé), which is why minimal/complete sufficiency matters.

A UMVUE (Uniformly Minimum Variance Unbiased Estimator) is the unbiased estimator with the smallest variance for every value of the parameter. It differs from the MLE in that the MLE is often biased and is defined by maximizing the likelihood rather than by any variance-optimality among unbiased estimators.

• UMVUE:

  • Unbiased for all θ and minimizes variance uniformly (not just at one θ).

  • Found via Lehmann-Scheffé: an unbiased function of a complete sufficient statistic is the unique UMVUE.

• MLE:

  • Maximizes the likelihood; may be biased (e.g. the ML variance σ̂² = Σ(x_i - x̄)²/n divides by n, not n-1).

  • Not required to be unbiased, so it is not necessarily a UMVUE.

• Key differences:

  • Criterion: unbiasedness + minimum variance vs. maximum likelihood.

  • Small samples: UMVUE has the unbiased-optimality guarantee; MLE may trade bias for lower MSE.

  • Large samples: MLE is asymptotically unbiased and efficient (reaches the Cramér-Rao bound), so the two often coincide in the limit.

An MLE may fail to exist if the likelihood has no attained maximum (e.g. the supremum is at infinity or on an open boundary), and it may be non-unique if the likelihood is flat or multimodal at its peak. A boundary MLE occurs when the maximizing parameter value lies on the edge of the allowed parameter space rather than in its interior.

• Non-existence:

  • Likelihood increases without bound or its supremum is not attained (e.g. unbounded likelihood in some mixture models as a variance → 0).

  • The maximizing value sits at an open boundary the parameter space excludes.

• Non-uniqueness:

  • Flat likelihood: a plateau of parameter values all achieve the max (often from non-identifiability or perfect collinearity).

  • Multiple modes: several separated maxima tie (common in mixtures, and label switching).

• Boundary MLE:

  • The estimate lands on the edge of the space, e.g. p̂ = 0 or p̂ = 1 when a category is never/always observed, or a variance component estimated as 0.

  • Consequences: the gradient need not be zero there, so standard score-equation and asymptotic-normality/Fisher-information arguments break down and inference needs care.

• Practical fixes: Regularization or a prior (penalized/MAP estimation) can restore existence and uniqueness by making the objective strictly concave.

Asymptotic normality says that, under regularity conditions, the (properly scaled) MLE converges in distribution to a normal centered at the true parameter, with variance given by the inverse Fisher information. This means for large samples we can treat the estimator as approximately Gaussian.

• The formal statement:

  • √n(θ̂ - θ₀) →d N(0, I₁(θ₀)⁻¹), where I₁ is the per-observation Fisher information.

  • Equivalently θ̂ ≈ N(θ₀, I_n(θ₀)⁻¹) for large n.

• Practical implications:

  • Standard errors come from the inverse (observed or expected) information matrix.

  • Wald confidence intervals: θ̂ ± z · SE.

  • Justifies Wald tests and much of large-sample inference.

• Caveats: It is an approximation that improves with n; it can be poor in small samples or near parameter-space boundaries.

An estimator is asymptotically efficient if its asymptotic variance equals the smallest possible value, the Cramér-Rao lower bound. The MLE achieves this: its limiting variance is exactly the inverse Fisher information, so no consistent, asymptotically normal estimator does better in large samples.

• Cramér-Rao lower bound (CRLB): For any unbiased estimator, Var(θ̂) ≥ I(θ)⁻¹: Fisher information sets the floor on precision.

• How the MLE attains it:

  • Its asymptotic distribution is N(θ₀, I₁(θ₀)⁻¹), so its asymptotic variance meets the bound.

  • The MLE is thus the benchmark: relative efficiency of other estimators is measured against it.

• Qualifications:

  • This is an asymptotic statement; in finite samples the MLE may not hit the CRLB and may even be biased.

  • Requires regularity conditions (differentiable likelihood, support not depending on θ).

The score is the gradient of the log-likelihood with respect to the parameter. It measures how sensitively the log-likelihood responds to changes in the parameter, and at the true parameter value its expected value is zero.

• Definition:

  • U(θ) = ∂/∂θ log L(θ; x), the derivative of the log-likelihood.

  • Setting U(θ) = 0 gives the likelihood equations solved by the MLE.

• What it tells us: Direction and steepness of improvement in fit: it points toward parameter values that raise the likelihood.

• Expected value at the true θ:

  • E_θ[U(θ)] = 0: the score has mean zero at the true parameter (a regularity result).

  • Its variance is exactly the Fisher information: Var(U(θ)) = I(θ).

Fisher information sets the floor on how small an estimator's variance can be: an efficient estimator attains that floor, so its variance equals the inverse of the Fisher information.

• Cramér-Rao lower bound (CRLB):

  • For any unbiased estimator, Var(θ̂) ≥ 1/I(θ) (scalar) or Cov(θ̂) ⪰ I(θ)⁻¹ (vector).

  • More information (sharper likelihood) means a tighter achievable variance.

• Efficiency: An estimator is efficient if it achieves the CRLB exactly; its variance equals I(θ)⁻¹.

• MLE connection:

  • The MLE is asymptotically efficient: √n(θ̂ − θ) → N(0, I₁(θ)⁻¹), so it attains the bound in the limit.

  • This is why the inverse information matrix is used for standard errors.

The asymptotic consistency, normality, and efficiency of MLEs rely on "regularity conditions" that guarantee the log-likelihood is smooth and that the true parameter behaves nicely: they fail at boundaries, with support that depends on the parameter, or when the model is misspecified.

• Identifiability: Different parameter values give different distributions, so the likelihood has a unique maximizer.

• Parameter interior to the space: The true θ is not on a boundary; otherwise normality breaks (e.g. testing a variance component = 0).

• Support does not depend on θ: Violated by the Uniform(0, θ) model, where the MLE converges at rate n, not √n, and is non-normal.

• Smoothness and differentiability: Log-likelihood is twice differentiable and the score has mean zero.

• Interchange of derivative and integral: Allows the two Fisher information identities to coincide; requires dominated, well-behaved integrands.

• Finite, positive-definite information: I(θ) exists and is invertible.

• Where they break: Boundary problems, mixture models (label switching, non-identifiability), heavy-tailed scores, and misspecification (then use a sandwich covariance).

Expected Fisher information is the average curvature of the log-likelihood over the data distribution; observed information is the actual curvature at the observed sample. Expected requires taking an expectation over the model, while observed is computed directly from your data at the MLE.

• Expected Fisher information:

  • I(θ) = E[−∂²ℓ/∂θ²]: expectation taken over the data.

  • A function of θ only, often needs an analytic expectation or integral.

• Observed Fisher information:

  • J(θ̂) = −∂²ℓ/∂θ² evaluated at the MLE using the actual data.

  • No expectation required: it is the negative Hessian your optimizer already computes.

• When to use each:

  • Use observed information in practice: readily available and Efron & Hinkley argue it gives better conditional standard errors.

  • Use expected information when a clean closed form exists or for theoretical design (e.g. optimal experimental design).

  • They agree asymptotically: J(θ̂)/n → I₁(θ).

Fisher information can be written as the variance of the score or as the expected negative second derivative of the log-likelihood; the two are equal because, under regularity, the score has mean zero and derivatives can be interchanged with integration.

• Form 1: variance of the score: I(θ) = E[(∂ℓ/∂θ)²] = Var(score), since the score has zero mean.

• Form 2: expected curvature: I(θ) = −E[∂²ℓ/∂θ²].

• Why they are equal:

  • Start from ∫ f(x;θ) dx = 1 and differentiate twice under the integral sign.

  • The first derivative gives E[score] = 0.

  • Differentiating again produces E[∂²ℓ/∂θ²] + E[(∂ℓ/∂θ)²] = 0, which rearranges to the identity.

• Regularity caveat: The interchange of derivative and integral must be valid (support independent of θ, smoothness); otherwise the two forms can differ.

The delta method approximates the variance of a smooth function of an estimator using a first-order Taylor expansion: the transformed estimator is asymptotically normal with variance scaled by the squared derivative of the transformation.

• Core result (scalar):

  • If √n(θ̂ − θ) → N(0, σ²), then √n(g(θ̂) − g(θ)) → N(0, g'(θ)²σ²).

  • So SE(g(θ̂)) ≈ |g'(θ̂)|·SE(θ̂).

• Multivariate form: Var(g(θ̂)) ≈ ∇g(θ̂)ᵀ · Cov(θ̂) · ∇g(θ̂), using the gradient and the estimated covariance.

• Typical use: Getting a standard error for an odds ratio, a ratio of parameters, or a probability derived from logit coefficients.

• Caveats:

  • Requires g differentiable with nonzero derivative; if g'(θ)=0 a second-order expansion is needed.

  • It is an asymptotic, linear approximation: poor for small samples or highly nonlinear g.

Standard errors for MLEs come from the asymptotic covariance, which is the inverse of the Fisher (or observed) information matrix; the diagonal square roots of that inverse are the standard errors.

• Asymptotic distribution: θ̂ ≈ N(θ, I(θ)⁻¹), so Cov(θ̂) ≈ I(θ)⁻¹.

• Computing it in practice:

  • Take the observed information J(θ̂) = −Hessian of ℓ at the MLE (optimizers return this).

  • Invert it: J(θ̂)⁻¹ estimates the covariance matrix.

  • SE(θ̂ⱼ) = √[J(θ̂)⁻¹]ⱼⱼ, the square root of the j-th diagonal entry.

• Role of the inverse information:

  • Inversion converts curvature into variance: high curvature (large information) yields small variance.

  • Off-diagonal entries capture correlation between parameter estimates.

• Robust alternative: Under misspecification use the sandwich estimator J⁻¹ · Var(score) · J⁻¹.

All three test the same null hypothesis and are asymptotically equivalent (each converges to the same chi-squared distribution), but they differ in what they fit, what they cost, and how they behave near boundaries or in small samples.

• Likelihood ratio (LR) test:

  • Compares log-likelihoods of the full and restricted models: LR = 2(ℓ_full − ℓ_restricted).

  • Cost: requires fitting BOTH models, but is invariant to reparameterization and usually the most reliable in small samples.

• Wald test:

  • Uses only the full-model fit: distance of the estimate from the null, scaled by the estimated variance.

  • Cost: cheapest (one fit), but NOT invariant to parameterization and unreliable near a boundary or when the estimate is far from the null.

• Score (Lagrange multiplier) test:

  • Uses only the restricted-model fit: evaluates the gradient (score) of the log-likelihood at the null.

  • Cost: fits only the restricted model, ideal when the full model is hard to fit.

• Boundary and small-sample behavior: Wald degrades most (can even reject less when the effect is huge: the Hauck-Donner effect); LR is generally best behaved.

• When to prefer which: LR by default for accuracy; score when the full model is expensive or you only have the null fit; Wald for quick single-parameter checks and standard errors already in hand.

Wilks' theorem states that, under the null and regularity conditions, twice the log-likelihood ratio statistic converges to a chi-squared distribution with degrees of freedom equal to the number of constrained parameters. It is what turns the LR statistic into an actual p-value.

• The statement: 2(ℓ_full − ℓ_restricted) → χ²_k as n → ∞, where k = difference in free parameters.

• Why it matters: Without a known reference distribution, the raw LR number is uninterpretable; Wilks gives a distribution-free (asymptotic) way to compute significance for nested models.

• Conditions and caveats:

  • Requires models to be nested, the true parameter to be interior (not on a boundary), and standard regularity (identifiability, differentiability).

  • On a boundary (e.g. testing a variance = 0) the limit becomes a mixture of chi-squareds, not the naive χ²_k.

  • It is asymptotic: small samples may need simulation or bootstrap calibration.