136 Probability Distributions Interview Questions and Answers (2026)

The Normal (Gaussian) distribution is a continuous, symmetric, bell-shaped distribution fully described by two parameters: the mean μ (center) and standard deviation σ (spread). It is central to statistics because of the Central Limit Theorem: sums/averages of many independent effects tend toward normality.

• Shape and symmetry:

  • Symmetric bell curve where mean = median = mode, all at μ.

  • Tails extend infinitely but decay very fast (exponentially in the square of distance from the mean).

• Parameters:

  • μ shifts the curve left/right; σ controls width (larger σ = flatter, wider).

  • The standard normal has μ=0, σ=1; any normal converts via the z-score z=(x−μ)/σ.

• The 68-95-99.7 rule (empirical rule):

  • ~68% of data lies within ±1σ, ~95% within ±2σ, ~99.7% within ±3σ.

  • Gives a quick sense of how extreme a value is and underpins outlier flags and control charts.

• Why it matters:

  • Central Limit Theorem makes it a good model for aggregated/measurement noise.

  • Caveat: real data can be skewed or heavy-tailed, so don't assume normality without checking.

The 68-95-99.7 rule (the empirical rule) states that for a Normal distribution roughly 68%, 95%, and 99.7% of observations fall within one, two, and three standard deviations of the mean, respectively. It is a fast mental model for how data spreads around the center.

• The three bands:

  • μ ± 1σ contains ~68.3% of the mass.

  • μ ± 2σ contains ~95.4% (~95%).

  • μ ± 3σ contains ~99.7%.

• Significance:

  • Quantifies how unusual a value is: beyond 3σ is rare (~0.3%), a natural outlier threshold.

  • Foundation for confidence intervals (~95% ≈ 1.96σ) and quality-control charts.

• Caveats:

  • Only exact for the Normal distribution; skewed or heavy-tailed data violate it.

  • Chebyshev's inequality gives weaker but distribution-free bounds when normality doesn't hold.

The Poisson distribution is a good example: it models the count of independent events occurring in a fixed interval of time or space, given a constant average rate.

• Discrete, count-based: Support is 0, 1, 2, ... (non-negative integers).

• Single parameter λ: λ is both the mean and the variance, so spread grows with the rate.

• Key assumptions: Events are independent and occur at a constant average rate.

• Real-world uses: Calls per hour to a call center, arrivals per minute, defects per unit.

• Relationships: Limit of the Binomial when n is large and p small; the gaps between Poisson events follow an Exponential distribution.

The standard normal distribution is a Normal distribution with mean 0 and standard deviation 1, and a Z-score expresses any value as the number of standard deviations it lies from its mean, which maps any Normal variable onto that standard scale.

• Standard normal: N(0, 1): A fixed reference curve so any Normal can be compared through a single table or function.

• Z-score formula: z = (x - μ) / σ: how far x is from the mean in standard-deviation units.

• Why standardize: Convert probabilities and percentiles for any Normal into the standard normal; compare variables on different scales.

• Interpretation via the empirical rule: About 68%, 95%, 99.7% of values fall within z of ±1, ±2, ±3; |z| > 3 is an unusual observation.

• Common uses: Outlier detection, feature scaling in ML, and computing p-values.

Use the Normal distribution when a quantity results from the sum of many small independent effects, is roughly symmetric and unimodal, and has no hard bounds or heavy tails: it is a natural model for measurement error and aggregated phenomena.

• When it fits:

  • Data clusters symmetrically around a central value with rapidly thinning tails.

  • Quantity is an average or sum of many independent contributions (CLT applies).

• Real-world applications:

  • Human heights and biological measurements.

  • Measurement and instrument errors.

  • Modeling residuals in linear regression and process variation in quality control.

• In statistics and ML: Sampling distributions of estimators, Gaussian noise assumptions, and priors.

• When NOT to use it: Skewed or strictly positive data (incomes, waiting times), heavy-tailed data, or bounded counts: prefer Log-normal, Exponential, Pareto, or Poisson.

A Bernoulli distribution models a single yes/no trial, while a Binomial distribution models the number of successes across many independent Bernoulli trials. Bernoulli is the special case of Binomial with one trial.

• Bernoulli:

  • One trial, outcome 0 or 1, single parameter p.

  • Mean p, variance p(1-p).

• Binomial:

  • Counts successes in n independent trials each with probability p; parameters n and p.

  • Mean np, variance np(1-p).

• The relationship: A Binomial is the sum of n i.i.d. Bernoulli variables; Bernoulli(p) = Binomial(1, p).

• Quick example: One coin flip is Bernoulli; the number of heads in 10 flips is Binomial.

A distribution describes how probability is spread over the possible values of a random variable: which outcomes are possible and how likely each is. Discrete distributions assign probabilities to countable outcomes via a probability mass function (PMF).

• What a distribution specifies:

  • For discrete variables: a PMF giving P(X = x), with all probabilities summing to 1.

  • For continuous variables: a probability density function whose area integrates to 1.

• Bernoulli: Single trial with two outcomes; parameter p (e.g. one coin flip).

• Binomial: Number of successes in n independent Bernoulli trials (e.g. heads in 10 flips).

• Poisson: Count of events in a fixed interval at a constant average rate lambda (e.g. calls per hour).

• Others worth knowing: Geometric (trials until first success) and Negative Binomial (trials until r successes).

The Binomial distribution gives the probability of getting exactly k successes in n independent trials, where each trial succeeds with the same probability p. It is the right model for counting successes among a fixed number of identical, independent yes/no experiments.

• Parameters:

  • n: the fixed number of trials.

  • p: the success probability per trial (constant across trials).

• PMF and moments:

  • P(X=k) = C(n,k) p^k (1-p)^(n-k).

  • Mean np, variance np(1-p).

• When it's appropriate (the assumptions):

  • Fixed n, exactly two outcomes per trial, constant p, and independent trials.

  • If sampling without replacement from a small population, independence breaks (use Hypergeometric); if p varies, it doesn't apply.

• Useful limits: Large n with moderate p approximates Normal; large n with small p approximates Poisson with lambda = np.

The Poisson distribution models the number of events that occur in a fixed interval of time or space when events happen independently at a constant average rate. It is characterized by a single rate parameter and describes rare-event counts.

• Parameter:

  • lambda: the expected number of events per interval (both the mean and the variance).

  • PMF: P(X=k) = e^(-lambda) lambda^k / k!.

• Key property: Mean equals variance equals lambda; if data show variance much larger than the mean, it's overdispersed (consider Negative Binomial).

• Assumptions (the Poisson process): Events occur independently, at a constant rate, and cannot be simultaneous.

• What it models: Calls to a call center per hour, arrivals in a queue, typos per page, mutations per genome region, website hits per minute.

• Connections: It's the limit of a Binomial with large n and small p (lambda = np); waiting times between Poisson events are Exponential.

The Bernoulli distribution is the simplest discrete distribution: a single trial with exactly two outcomes, success (1) with probability p and failure (0) with probability 1-p.

• Parameter and support: One parameter p in [0,1]; the random variable takes only 0 or 1.

• Moments: Mean = p, variance = p(1-p) (maximized at p=0.5).

• Real-world scenarios: A single coin flip, one click/no-click on an ad, one email spam/not-spam, one patient responds/doesn't.

• Building block: A sum of n independent Bernoulli(p) trials is a Binomial(n, p); it also underlies logistic regression's likelihood.

The discrete Uniform distribution assigns equal probability to each of a finite set of outcomes: if there are n possible values, each has probability 1/n. It models a situation with no reason to favor any outcome.

• PMF: For integers a..b, P(X=k) = 1/(b-a+1); the classic example is a fair die (1/6 each).

• Moments (on a..b): Mean = (a+b)/2 (the midpoint); variance = ((b-a+1)^2 - 1)/12.

• Characteristics: Symmetric, flat PMF; it is the maximum-entropy distribution over a finite set (no outcome preferred).

• Applications: Fair dice and coins, random index selection, shuffling, and as a baseline/prior when outcomes are equally likely.

In a Poisson distribution the single parameter lambda equals both the mean and the variance, so the standard deviation is the square root of the mean. Spread grows only with the square root of the rate.

• The key identity: Mean = variance = lambda, therefore standard deviation = sqrt(lambda).

• Practical implication: Relative variability (coefficient of variation = 1/sqrt(lambda)) shrinks as the rate rises, so high-count processes look proportionally more stable.

• Example scenario: A call center receives on average 100 calls/hour: mean = 100, standard deviation = 10, so a typical hour lands roughly 90-110.

• Diagnostic use: If observed variance far exceeds the mean, the Poisson assumption fails (overdispersion), suggesting a Negative Binomial instead.

A probability distribution is a mathematical rule that assigns probabilities to the possible outcomes of a random variable, telling you how likely each value (or range of values) is. It fully characterizes the random variable's behavior.

• A random variable maps outcomes to numbers: The distribution then specifies how probability mass or density is spread across those numbers.

• It must satisfy two axioms:

  • Probabilities are non-negative.

  • They sum (discrete) or integrate (continuous) to 1 over all outcomes.

• Ways to describe it:

  • PMF/PDF: probability of each value or density at each point.

  • CDF: cumulative probability up to a value.

• It summarizes everything you need: center, spread, shape, and the likelihood of any event.

The distinction is about the set of values a variable can take: countable, uncountable, or a combination of both.

• Discrete random variable:

  • Takes countable values (finite or countably infinite); described by a PMF.

  • Examples: number of heads in 10 coin flips, count of customers per hour (Poisson).

• Continuous random variable:

  • Takes uncountably many values in an interval; described by a PDF where P(X = x) = 0.

  • Examples: height, temperature, time until failure (exponential).

• Mixed random variable:

  • Has both a discrete part (point masses) and a continuous part.

  • Example: rainfall amount: a lump of probability at exactly 0 (dry days) plus a continuous density for positive amounts.

Probability distributions describe how likely different outcomes are, and in machine learning they are the language for modeling uncertainty in data, parameters, and predictions. Most models are, explicitly or implicitly, assumptions about a distribution.

• They model the data-generating process: Assuming a distribution (e.g. Gaussian noise) lets you write a likelihood and fit parameters.

• They underpin core algorithms:

  • Naive Bayes, GMMs, and generative models are built directly on distributions.

  • Loss functions like cross-entropy come from maximizing likelihood.

• They quantify uncertainty: Bayesian methods use priors and posteriors; predictive distributions give confidence, not just point estimates.

• They guide practical choices: Weight initialization, dropout, sampling, and data augmentation all rely on distributions.

These four concepts build on one another: a random variable numerically encodes outcomes, a probability distribution assigns probabilities to them, and the PDF and CDF are two functions that specify that distribution.

• Random variable: A function mapping outcomes of a random experiment to real numbers (e.g. sum of two dice).

• Probability distribution: The assignment of probabilities across all values the random variable can take.

• PDF (or PMF): PMF gives P(X = x) for discrete variables; PDF gives density f(x) for continuous variables where probability is the area under the curve over an interval.

• CDF: F(x) = P(X ≤ x): non-decreasing, ranges from 0 to 1, and is the integral (or sum) of the PDF/PMF.

A uniform distribution assigns equal probability to every outcome in its range, meaning no value is more likely than any other. It comes in discrete and continuous forms.

• Discrete uniform: Finite outcomes each with probability 1/n (e.g. a fair die: each face has 1/6).

• Continuous uniform on [a, b]:

  • Constant density f(x) = 1/(b − a) inside the interval, 0 outside.

  • Mean = (a + b)/2, variance = (b − a)²/12.

• Uses: Models complete ignorance (a non-informative prior) and is the basis of random number generation and inverse-transform sampling.

The difference is whether the random variable's possible values are countable or fill a continuum: discrete distributions use a PMF assigning probability to each value, while continuous distributions use a PDF where probability is area under a curve.

• Discrete distributions:

  • Countable outcomes; PMF gives P(X = x) directly, and probabilities sum to 1.

  • Examples: Binomial, Poisson, Geometric.

• Continuous distributions:

  • Uncountable outcomes; PDF gives density, and probability of an interval is an integral, with density integrating to 1.

  • P(X = x) = 0 for any single point; only ranges have positive probability.

  • Examples: Normal, Exponential, Uniform.

• Shared structure: Both have a CDF, and both define expectation and variance (via sums vs. integrals).

The Law of Large Numbers (LLN) says that as you collect more i.i.d. samples, their sample mean converges to the true expected value: averages stabilize even when individual outcomes are random.

• Convergence of averages: For i.i.d. draws with finite mean μ, the sample mean approaches μ as n grows.

• Two versions:

  • Weak LLN: the sample mean converges in probability (large deviations become arbitrarily unlikely).

  • Strong LLN: the sample mean converges almost surely (the sequence itself settles to μ with probability 1).

• Key assumptions:

  • Finite mean is required; heavy-tailed distributions like Cauchy have no mean and the average never settles.

  • Classic form assumes independence and identical distribution, though variants relax these.

• What it is NOT:

  • No "gambler's fallacy": past outcomes don't self-correct. Averages converge because new data dilutes early deviations, not because the process compensates.

  • LLN is about the mean converging; the CLT describes the distribution of fluctuations around it.

The support of a distribution is the set of values where the density or mass is nonzero: the outcomes the model considers possible. It matters because a model can never produce data outside its support, so the support must match your data's true range.

• Definition: The smallest closed set carrying all the probability; e.g. Normal is all of ℝ, Poisson is {0,1,2,...}, Beta is (0,1).

• Why it constrains modeling:

  • A distribution assigns zero probability outside its support, so fitting a Normal to strictly positive or bounded data leaks probability into impossible regions.

  • That leak causes nonsensical predictions (negative counts, values above a known cap) and biased tail estimates.

• Matching support to bounded data:

  • Data in [0,1] (proportions): use Beta.

  • Positive-only data: use Gamma, Log-Normal, or Exponential.

  • Bounded on [a,b]: use a scaled Beta or a truncated distribution.

• Practical alternative: Transform the data to an unbounded scale (e.g. logit for (0,1), log for positives) and model that, so the transformed support matches.

A PMF gives the probability of each exact value of a discrete random variable, while a PDF describes the relative density of a continuous variable, where probability comes from integrating over an interval, not from any single point.

• PMF (discrete):

  • p(x) = P(X = x), a genuine probability for each outcome.

  • Each value is in [0,1] and the values sum to 1 over the support.

  • Examples: Binomial, Poisson, Geometric.

• PDF (continuous):

  • f(x) is a density; probability over [a,b] is the integral of f from a to b.

  • f(x) ≥ 0 and integrates to 1, but f(x) can exceed 1 (it's density, not probability).

  • Examples: Normal, Exponential, Beta.

• Key differences:

  • Summation vs integration: PMF sums to 1, PDF integrates to 1.

  • Point probability: nonzero for a PMF, always zero for a continuous PDF.

  • Range of values: PMF outputs are true probabilities capped at 1; PDF outputs are unbounded densities.

The CDF, F(x) = P(X ≤ x), gives the accumulated probability up to x. It exists for every distribution (discrete, continuous, or mixed) and fully characterizes the distribution.

• Definition: F(x) = P(X ≤ x); for continuous X it's the integral of the PDF up to x, for discrete X the cumulative sum of the PMF.

• Essential properties:

  1. Non-decreasing: if a ≤ b then F(a) ≤ F(b).

  2. Limits: F(x) → 0 as x → -∞ and F(x) → 1 as x → +∞.

  3. Right-continuous: F(x) = lim of F(y) as y decreases to x (jumps occur at discrete mass points).

  4. Bounded in [0,1] for all x.

• Why it's useful:

  • Interval probabilities: P(a < X ≤ b) = F(b) - F(a).

  • Its inverse (quantile function) drives percentiles and inverse-transform sampling.

A PMF gives the probability that a discrete random variable takes each specific value, and it must assign non-negative probabilities that sum to exactly 1 over the support.

• Non-negativity: P(X = x) >= 0 for every value x: a probability can never be negative.

• Normalization (sums to 1): The sum of P(X = x) over all x in the support equals 1: total probability is exhausted.

• Defined on a countable support: Applies to discrete outcomes (finite or countably infinite); values outside the support have probability 0.

• Value gives an actual probability: Unlike a PDF, P(X = x) is itself a real probability, so each value is bounded in [0, 1].

• Links to the CDF: The CDF is the running sum of the PMF: F(x) = sum of P(X = k) for k <= x.

A function is a valid PMF if and only if every value it returns is non-negative and the values sum to exactly 1 across the entire support: these two conditions are what guarantee it defines a coherent probability model.

• Check non-negativity: Verify P(X = x) >= 0 for all x; any negative value disqualifies it because probabilities cannot be negative.

• Check that it sums to 1:

  • Sum over all support points; if it converges to 1 the total probability is conserved.

  • If it sums to a finite constant c (not 1), you can often normalize by dividing by c.

• Why these conditions matter: They are the Kolmogorov axioms applied to discrete outcomes: probabilities are non-negative and the certain event has probability 1.

• Example check: For P(X = k) = c * (1/2)^k on k = 1, 2, 3, ...: the sum is c, so c = 1 makes it valid.

Positive skewness means the distribution has a longer tail on the right (toward large values), while negative skewness means a longer tail on the left (toward small values). The tail is what pulls the mean past the median.

• Positive (right) skew:

  • Bulk of data on the left, sparse large outliers on the right.

  • Mean > median: extreme high values inflate the mean.

  • Examples: income, house prices, insurance claims.

• Negative (left) skew:

  • Bulk of data on the right, sparse small outliers on the left.

  • Mean < median: extreme low values deflate the mean.

  • Examples: exam scores with a ceiling, age at death.

• Rule of thumb: The mean is pulled in the direction of the longer tail; the median stays closer to the bulk, so it's the more robust center under skew.

For a right-skewed (positively skewed) distribution the typical ordering is mode < median < mean, because the long right tail pulls the mean upward more than the median.

• Mode is smallest: It sits at the tallest peak, which is on the left where most mass concentrates.

• Median is in the middle: It's the 50% point, shifted right of the mode but resistant to extreme tail values.

• Mean is largest: The mean is sensitive to magnitude: large values in the long right tail drag it the furthest right.

• Intuition and caveat:

  • "The tail pulls the mean." Right tail → mean pulled right.

  • This ordering is a strong rule of thumb, not a theorem: some contrived or discrete/multimodal distributions violate it.

The expected value is the probability-weighted average of a random variable: the long-run mean you'd converge to if you sampled it many times. It's the distribution's center of mass.

• Definition:

  • Discrete: E[X] = Σ x · P(X = x).

  • Continuous: E[X] = ∫ x · f(x) dx.

• Key uses:

  • Summarizes a distribution with a single representative number.

  • Basis for other quantities: variance is E[(X-μ)^2], and moments are all expectations.

  • Decision-making: comparing expected payoffs/losses under uncertainty.

• Useful properties:

  • Linearity: E[aX + bY] = aE[X] + bE[Y], true even when X and Y are dependent.

  • Law of the unconscious statistician: E[g(X)] = Σ g(x)P(x) without needing the distribution of g(X).

• Caveat: the expected value need not be a possible outcome (e.g., 2.5 children), and some distributions (Cauchy) have no finite expectation.

For a continuous Uniform distribution on [a, b], the expected value is the midpoint (a + b)/2. With a = -3 and b = 7, E(X) = (-3 + 7)/2 = 2.

• Why the midpoint:

  • The density is flat (constant at 1/(b-a)), so the distribution is symmetric about its center.

  • By symmetry the center of mass is exactly halfway between the endpoints.

• Verification by integration: E[X] = ∫_{-3}^{7} x · (1/10) dx = (1/10) · [x^2/2]_{-3}^{7} = (1/10)(49/2 - 9/2) = (1/10)(20) = 2.

The mean sets the location (center) of the bell curve, and the standard deviation sets its spread (width and height); together they fully define a normal distribution.

• Mean (μ) shifts location: Changing μ slides the whole curve left or right along the x-axis without changing its shape; it is the center, median, and mode.

• Standard deviation (σ) controls spread: Larger σ makes the curve wider and flatter; smaller σ makes it taller and narrower (total area stays 1).

• The empirical (68-95-99.7) rule: About 68% of mass lies within μ ± σ, 95% within μ ± 2σ, and 99.7% within μ ± 3σ.

• Standardization: Any normal can be converted to the standard normal via z = (x − μ)/σ, which centers at 0 with spread 1.

These are the three workhorse distributions: the Normal is continuous and symmetric, the Binomial and Poisson are discrete count distributions, and each fits a distinct kind of data-generating mechanism.

• Normal:

  • Continuous, symmetric bell curve defined by mean μ and standard deviation σ; unbounded support.

  • Arises via the central limit theorem; used for measurements, errors, and as an approximation for other distributions.

• Binomial:

  • Discrete, parameters n (trials) and p (success probability); mean np, variance np(1−p).

  • Used for the number of successes in a fixed set of independent trials (defect counts, conversion counts).

• Poisson:

  • Discrete, single parameter λ; mean equals variance equals λ.

  • Used for counts of rare, independent events per interval (arrivals, typos per page, calls per hour).

• How they relate: Binomial → Poisson as n large and p small (λ = np); both Binomial and Poisson → Normal for large counts, enabling normal approximations.

A uniform distribution assigns equal probability across a fixed range so every value is equally likely, while a normal (Gaussian) distribution is bell-shaped and concentrates probability around a central mean with tails thinning symmetrically outward.

• Shape:

  • Uniform: flat, constant density between bounds a and b; zero outside.

  • Normal: symmetric bell defined by mean and standard deviation, with density peaking at the mean.

• Support: Uniform is bounded on [a, b]; Normal spans the entire real line.

• Parameters: Uniform: min and max; Normal: mean and variance.

• Typical use: Uniform models complete ignorance or equal likelihood (random draws, priors); Normal models measurement noise and sums of many small effects via the CLT.

Use the mean when the distribution is roughly symmetric and every value should count proportionally; use the median when the data are skewed or contain outliers, because it reflects the typical case without being dragged by extreme values.

• Prefer the mean when: Data are symmetric with no severe outliers, and you need a measure that respects totals (mean × n = sum), e.g. average revenue per user.

• Prefer the median when: Data are skewed or heavy-tailed (latency, income, home prices); the median is robust to outliers and better describes the typical user.

• Performance metrics angle: For latency SLAs, mean hides tail pain: report the median plus high percentiles (p95, p99) rather than the mean alone.

• Rule of thumb: If mean and median diverge sharply, the distribution is skewed and the median is usually the more honest 'typical' value.

A Log-normal distribution describes a variable whose logarithm is normally distributed. It is defined only for positive values, is right-skewed, and arises whenever a quantity is the product of many independent positive factors (multiplicative growth).

• Definition:

  • If ln(X) ~ Normal(μ, σ²), then X is log-normal.

  • Support is (0, ∞), so it never produces negatives.

• Shape:

  • Right-skewed with a long upper tail; mean > median > mode.

  • Taking logs of the data recovers a symmetric bell curve.

• Why multiplicative processes produce it: A product of many independent factors becomes a sum of logs, which the CLT drives toward normal, so the original is log-normal.

• Common scenarios:

  • Income and wealth distributions, stock prices and asset returns (geometric growth).

  • File sizes, city populations, biological measurements, time-to-failure, and reaction times.

The Exponential distribution is a continuous distribution modeling the waiting time until the next event in a Poisson process, where events occur at a constant average rate λ. Its defining trait is memorylessness: the future wait doesn't depend on how long you've already waited.

• Definition:

  • PDF f(x)=λe^(−λx) for x ≥ 0; mean 1/λ, variance 1/λ².

  • Right-skewed, always positive, monotonically decreasing density.

• Memoryless property:

  • P(X > s+t | X > s) = P(X > t): having waited s already gives no information about the remaining wait.

  • It is the only continuous distribution with this property (the geometric is its discrete analog).

• Relationship to Poisson: If counts per interval are Poisson(λ), the gaps between events are Exponential(λ).

• Common applications:

  • Time between arrivals (customers, requests), inter-failure times, radioactive decay.

  • Queueing theory and reliability models with a constant hazard rate.

• Caveat: Constant hazard is often unrealistic (things age/wear); use Weibull or Gamma when failure rate changes over time.

The Gamma distribution is a continuous, positive, right-skewed distribution modeling the waiting time until the k-th event in a Poisson process. It generalizes the Exponential and, for integer shape, is the Erlang distribution.

• Parameters:

  • Shape k (or α) and rate λ (or scale θ=1/λ); mean k/λ, variance k/λ².

  • Support (0, ∞); skew decreases as k grows (approaches Normal).

• Relationship to Exponential:

  • A Gamma is the sum of k independent Exponential(λ) variables.

  • With k=1 the Gamma reduces exactly to the Exponential.

• Relationship to Erlang:

  • When the shape k is a positive integer, the Gamma is called the Erlang distribution (time to the k-th arrival).

  • Gamma allows non-integer k, making it the continuous generalization.

• Uses: Waiting/queueing times, rainfall amounts, insurance claims; also a conjugate prior for the precision/rate in Bayesian models.

The Beta distribution is a continuous distribution defined on the interval [0, 1], which makes it a natural model for probabilities, proportions, and rates. Its two shape parameters α and β let it take a wide variety of shapes on that interval.

• Parameters and shape:

  • α and β act like counts of successes and failures; mean α/(α+β).

  • α=β=1 is uniform; α=β is symmetric; unequal values skew it; large values concentrate it.

• Why it models probabilities: Bounded on [0,1], it represents uncertainty ABOUT a probability rather than an outcome.

• Bayesian conjugacy:

  • It is the conjugate prior for the Binomial/Bernoulli parameter p.

  • Posterior update is trivial: start with Beta(α, β), observe s successes and f failures, get Beta(α+s, β+f).

• Uses: Estimating conversion/click rates, A/B testing, Thompson sampling in bandits, and any proportion between 0 and 1.

The Weibull distribution is a flexible continuous distribution for positive values, widely used in reliability and survival analysis because its shape parameter lets the failure (hazard) rate increase, decrease, or stay constant over time. It generalizes the Exponential.

• Parameters: Shape k (or β) controls the hazard behavior; scale λ sets the characteristic time.

• What the shape parameter signifies:

  • k < 1: decreasing hazard, early/infant-mortality failures.

  • k = 1: constant hazard, reduces exactly to the Exponential (random failures).

  • k > 1: increasing hazard, wear-out/aging failures (e.g. k≈3.4 approximates a Normal).

• When to use it:

  • Product lifetime and time-to-failure modeling, maintenance scheduling, survival analysis.

  • Wind-speed modeling and other positive, skewed data where the hazard shape matters.

• Why prefer it over Exponential: It drops the unrealistic constant-hazard assumption, capturing aging or early-failure patterns via a single extra parameter.

The Normal (Gaussian) distribution is fundamental largely because of the Central Limit Theorem: sums and averages of many independent effects tend toward it regardless of the underlying distribution, so it appears everywhere in nature and in statistics.

• Central Limit Theorem (CLT):

  • The mean of many independent random variables is approximately Normal, so sampling distributions of estimators are Normal even when raw data isn't.

  • This underpins confidence intervals and hypothesis tests.

• Fully described by two parameters: Mean and variance completely specify it, making it simple to fit and reason about.

• Mathematical convenience:

  • Closed under linear combinations: sums of Normals are Normal.

  • It is the maximum-entropy distribution for a fixed mean and variance (least assumptions given those constraints).

• Ties to core methods: Least-squares regression, Gaussian error models, and many ML priors assume Normality.

• Caveat: Real data with heavy tails or skew is not Normal; blindly assuming it underestimates extreme events.

The Student's t-distribution is a bell-shaped, symmetric distribution like the Normal but with heavier tails, used for inference about a mean when the population variance is unknown and estimated from a small sample.

• Governed by degrees of freedom (df): Lower df means heavier tails (more uncertainty); as df grows the t-distribution converges to the standard Normal.

• Why the heavier tails: With small samples we estimate the standard deviation, adding extra variability that fattens the tails and widens intervals.

• Primary use: the t-test: Confidence intervals and hypothesis tests for a mean when the sample is small and σ is unknown.

• Origin: Arises as the ratio of a standard Normal to the square root of a scaled chi-squared variable.

• Also used in robust modeling: Its heavy tails make it a good likelihood for data with outliers.

The Pareto distribution is a heavy-tailed, power-law distribution where a small number of observations account for most of the total: a large fraction of the mass sits in the tail. The Pareto principle (the '80/20 rule') is an informal consequence of this shape.

• Power-law form: Probability of exceeding x falls off polynomially like x^(-α), far slower than the Normal's exponential decay, so extreme values are relatively common.

• Shape parameter α: Smaller α means heavier tails; if α ≤ 2 the variance is infinite, if α ≤ 1 even the mean is infinite.

• The Pareto principle: Roughly 80% of effects come from 20% of causes: a specific split that emerges for a particular α, not a law that always holds exactly.

• Real-world examples: Wealth and income distribution, city sizes, word frequencies, and file sizes on the web.

• Practical implication: Means and standard deviations can be misleading or unstable; focus on the tail and use robust or rank-based summaries.

The Laplace (double exponential) distribution is symmetric like the Normal but has a sharp peak at its center and heavier, exponential tails: it looks like two exponential distributions placed back to back around the mean.

• Shape: A pointed peak (density has a cusp at the location) with tails that decay exponentially, heavier than the Normal's.

• Parameters: A location (μ) and a scale (b); density ∝ exp(-|x - μ| / b), driven by absolute deviation rather than squared deviation.

• When to use it:

  • Data concentrated near a central value but with occasional large deviations (outliers), where the Normal's thin tails underfit.

  • Modeling errors in finance and signal processing.

• Connections in ML: A Laplace prior on coefficients yields L1 (Lasso) regularization; minimizing absolute error corresponds to a Laplace likelihood and estimates the median.

• Differential privacy: Adding Laplace noise is the classic mechanism for privacy-preserving query answers.

The logistic distribution is a symmetric, bell-shaped distribution very similar to the Normal but with slightly heavier tails and a simple closed-form cumulative distribution function (the logistic/sigmoid function), which makes it especially convenient in modeling.

• Shape vs the Normal: Symmetric and unimodal like the Normal, but with higher kurtosis: more probability in the tails and a slightly sharper peak.

• Closed-form CDF: Its CDF is the sigmoid 1 / (1 + e^(-(x-μ)/s)), unlike the Normal whose CDF has no elementary form.

• Parameters: A location (μ) and a scale (s); the mean equals μ and the variance is (π²/3)s².

• Why it matters:

  • It underlies logistic regression: the log-odds are linear because the latent error is assumed logistic, giving the sigmoid link.

  • Also used in growth models and item response theory.

The Chi-square distribution is the distribution of a sum of squared independent standard Normal variables. It is a special case of the Gamma distribution and is indexed by its degrees of freedom.

• Definition:

  • If Z_1,...,Z_k are i.i.d. N(0,1), then sum(Z_i^2) is Chi-square with k degrees of freedom.

  • Support is nonnegative, right-skewed, with mean k and variance 2k; it approaches Normal as k grows.

• Relationship to other distributions: It is Gamma(k/2, 2); a ratio of two scaled Chi-squares gives the F distribution; a Normal over a scaled Chi-square gives the t distribution.

• Typical uses:

  • Goodness-of-fit and independence tests (the chi-square test on contingency tables).

  • Sampling distribution of the sample variance from a Normal population.

  • Likelihood-ratio tests, where -2 log(likelihood ratio) is asymptotically Chi-square.

Memorylessness means the probability of waiting an additional amount of time is independent of how long you have already waited: the process "forgets" its history. Only two distributions have this property: the Exponential (continuous) and the Geometric (discrete).

• Formal statement:

  • For waiting time X: P(X > s + t | X > s) = P(X > t) for all s, t >= 0.

  • The conditional distribution of remaining wait is identical to the original distribution.

• Exponential (continuous): Models waiting time until an event in a Poisson process; a component that has run for hours is "as good as new".

• Geometric (discrete): Models the number of Bernoulli trials until the first success; past failures don't change the chance of success on the next trial.

• Why it matters / intuition: Memorylessness implies a constant hazard rate: aging or wear-out contradicts it (that's when you reach for Weibull or Gamma instead).

The Beta distribution lives on the interval [0,1] and can take many shapes, so it is a natural distribution over an unknown probability. It is also the conjugate prior for the Bernoulli/Binomial likelihood, which makes Bayesian updating trivial.

• Support matches a probability:

  • Defined on [0,1], exactly the range of a probability p.

  • Two shape parameters alpha and beta flex it from uniform to bell-shaped to U-shaped to skewed.

• Conjugacy with Binomial:

  • Prior Beta(alpha, beta) plus s successes and f failures gives posterior Beta(alpha+s, beta+f).

  • Updating is just counting: add successes to alpha, failures to beta.

• Interpretable parameters:

  • You can read alpha and beta as pseudo-counts of prior successes and failures.

  • Mean is alpha/(alpha+beta); larger alpha+beta means more concentration (more prior confidence).

• Typical use: Estimating conversion rates, click-through rates, or any success probability, and Thompson sampling in bandits.

The Geometric distribution models the number of independent Bernoulli trials until the first success, each trial having success probability p. It is the discrete analogue of the exponential distribution and is the only discrete distribution that is memoryless.

• Two common conventions:

  • Number of trials until first success (support 1,2,3,...): mean = 1/p.

  • Number of failures before first success (support 0,1,2,...): mean = (1-p)/p. Always state which you use.

• PMF and variance: P(X=k) = (1-p)^(k-1) p (trials convention); variance = (1-p)/p^2.

• Memoryless property: P(X > m+n | X > m) = P(X > n): past failures don't change the future. The process has no aging.

• Applications: Trials until first defective item, attempts until first success, number of at-bats until first hit, retries until a request succeeds.

Both count successes in a fixed number of draws, but the Binomial assumes sampling with replacement (or an infinite population) so trials are independent with constant p, while the Hypergeometric assumes sampling without replacement from a finite population, so each draw changes the remaining probabilities.

• Binomial:

  • Parameters n (draws) and p (constant success probability); trials independent.

  • Mean = np, variance = np(1-p).

• Hypergeometric:

  • Parameters N (population), K (successes in population), n (draws); probability shifts each draw.

  • Mean = n*K/N; variance includes a finite population correction factor (N-n)/(N-1) making it smaller than the Binomial's.

• When they converge: As N grows large relative to n, removing items barely changes proportions, so Hypergeometric approaches Binomial with p = K/N.

• Typical uses: Binomial: coin flips, conversion rates. Hypergeometric: drawing cards, quality inspection of a finite lot, capture-recapture.

The Multinomial distribution generalizes the Binomial from two outcomes to k mutually exclusive outcomes: it gives the probability of seeing each category a certain number of times across n independent trials, where category i occurs with probability p_i and the probabilities sum to 1.

• Setup: n trials, k categories with probabilities p_1..p_k summing to 1; the outcome is a count vector (x_1..x_k) with sum n.

• Relation to Binomial: With k=2 it reduces exactly to the Binomial. Each individual category count, viewed alone, is marginally Binomial(n, p_i).

• Moments: E[x_i] = n*p_i; counts are negatively correlated since they must total n (more of one means fewer of others).

• Applications: Rolling a die n times, word counts in text (bag-of-words), survey responses across categories, softmax outputs in classification.

The Negative Binomial distribution models the number of trials (or failures) needed to achieve a fixed number of successes k in repeated independent Bernoulli trials with success probability p. It generalizes the Geometric, which is just the special case k=1.

• Interpretation: Keep running trials until the k-th success occurs; the random variable counts how long that takes.

• Conventions: Counting total trials: support k, k+1, ...; counting failures before k-th success: support 0,1,2,.... State which you mean.

• Moments (failures convention): Mean = k(1-p)/p, variance = k(1-p)/p^2; variance always exceeds the mean.

• Two faces of the same distribution: As a waiting-time model (integer k) and, generalized to real k as a Poisson-Gamma mixture, as the overdispersed count model.

• Applications: Number of sales calls to close k deals, trials to get k defective units, and modeling overdispersed counts like insurance claims.

A sampling distribution is the probability distribution of a statistic (like the sample mean) computed over all possible samples of a given size from a population. It describes how that statistic varies from sample to sample, which is what makes inference possible.

• Key idea: It's a distribution of a statistic, not of raw data; its spread (the standard error) shrinks as sample size grows.

• Common types:

  • Sampling distribution of the mean: by the Central Limit Theorem, approximately normal for large n.

  • Sampling distribution of the proportion: approximately normal for large n.

  • Sampling distribution of the variance: related to the chi-square distribution.

  • Difference of means / t-distribution: used when population variance is unknown.

• Why it matters: It's the foundation of confidence intervals and hypothesis testing.

A probability distribution describes the outcomes of a single random variable (often the raw data), while a sampling distribution describes the outcomes of a statistic computed from repeated samples. The sampling distribution is a special, derived distribution used for inference.

• What each describes:

  • Probability distribution: the likelihood of values for a variable or population (e.g. heights of individuals).

  • Sampling distribution: the likelihood of values for a statistic like the sample mean across all possible samples of size n.

• Spread: The population's spread is fixed; the sampling distribution's spread (standard error) shrinks as n grows.

• Shape: By the Central Limit Theorem, the sampling distribution of the mean tends to normal even if the population is not.

• Purpose: Probability distributions model data; sampling distributions enable confidence intervals and hypothesis tests.

A location-scale family is a set of distributions generated from one standard shape by shifting (location) and stretching (scale): every member has the form (X - μ)/σ following the same base distribution. Fitting reduces to estimating just μ and σ.

• Formal definition:

  • If Z has a standard density f, then X = μ + σZ (σ > 0) has density (1/σ)·f((x - μ)/σ). The whole family is these shifts and scalings.

  • μ moves the distribution along the axis; σ stretches or compresses it while preserving shape.

• Common members:

  • Normal: μ and σ are literally the mean and standard deviation.

  • Also Uniform, Logistic, Laplace, Cauchy, and Student-t (for fixed degrees of freedom).

• Notable non-members:

  • Gamma, Log-Normal, and Poisson are not location-scale: changing parameters alters the shape (e.g. skewness), not just position and spread.

  • Some, like Exponential, are scale families only (no free location parameter).

• Why it's useful: Standardization (z-scores) and results proven for the standard member transfer to all members, simplifying inference and simulation.

For a continuous variable the probability of any single exact value is zero, so f(x) can't be a probability. Instead f(x) is a density: probability only emerges when you multiply by (integrate over) an interval width.

• Single points carry no probability:

  • P(X = x) = integral from x to x of f = 0 for any continuous distribution.

  • There are infinitely many possible values, so each individual one must have measure zero.

• Density has units of probability per unit x:

  • P(x < X < x + dx) ≈ f(x)·dx: only the product is a probability.

  • Because it's per-unit, f(x) can exceed 1 (e.g. a narrow Uniform on [0, 0.5] has f = 2).

• Correct interpretation: f(x) tells you relative likelihood: where f is larger, values are more concentrated, but you always need an interval to get an actual probability.

The CDF accumulates probability, so you reverse the accumulation: differentiate it for a continuous variable, or take successive differences (jump sizes) for a discrete one.

• Continuous case: differentiate: f(x) = dF(x)/dx wherever F is differentiable, since F is the integral of f.

• Discrete case: take differences: p(x) = F(x) - F(x⁻), the size of the jump in the step function at x.

• Practical checks:

  • A jump in F signals a discrete mass point; smooth increases signal continuous density.

  • Confirm the recovered function is valid (non-negative, integrates or sums to 1).

A function is a valid PDF if it is non-negative everywhere and integrates to exactly 1 over its support. Those two conditions are necessary and sufficient.

• Non-negativity: f(x) ≥ 0 for all x; densities can't be negative (though they may exceed 1).

• Normalization:

  • The integral of f over its entire support equals 1.

  • If the integral is a finite constant c ≠ 1, you can normalize by dividing by c to make a valid PDF.

• Watch-outs:

  • The integral must converge; heavy tails can make it diverge, disqualifying the function.

  • Define f as 0 outside the intended support and check the integral only where f is nonzero.

  • Measurability is a technical requirement but is satisfied by any function you'd meet in practice.

The survival function S(x) = P(X > x) = 1 - F(x) gives the probability that the variable exceeds a value, and you prefer it whenever the interesting behavior lives in the upper tail: reliability, time-to-event, and extreme-value analysis.

• Definition: S(x) = 1 - F(x); it decreases from 1 to 0 as x increases, mirroring the CDF.

• When to prefer it over the CDF:

  • Survival/reliability analysis: S(t) is literally the probability of surviving beyond time t.

  • Heavy-tail work: tail probabilities and power-law decay read directly off S(x), and log-log plots of S(x) reveal tail behavior.

  • Numerical accuracy: for extreme x, S(x) avoids catastrophic cancellation from computing 1 - F(x) when F(x) is near 1.

• Related quantities: The hazard rate h(x) = f(x) / S(x) is defined through the survival function, central to survival models.

The quantile function is the inverse of the CDF: given a probability p in [0, 1], it returns the value x such that P(X <= x) = p. You use it when you need thresholds, percentiles, or to generate random samples, rather than evaluating probabilities.

• Definition: Q(p) = F^{-1}(p) = inf{ x : F(x) >= p }; the inf form handles flat spots and discrete jumps.

• When you use it:

  • Percentiles and medians: the median is Q(0.5), quartiles are Q(0.25) and Q(0.75).

  • Risk thresholds: Value-at-Risk is a quantile of a loss distribution.

  • Confidence/critical values: z and t cutoffs come from quantile functions.

• Inverse-transform sampling: If U is uniform on [0, 1], then Q(U) has the target distribution: the standard way to simulate from a known CDF.

• PDF/CDF vs quantile: use the PDF/CDF to go from a value to a probability; use the quantile function to go from a probability to a value.

Skewness measures the asymmetry of a distribution around its mean: zero means symmetric, positive means a longer right tail, negative means a longer left tail. The direction of the tail pulls the mean away from the median and mode.

• Definition: The standardized third central moment: E[((X - mu)/sigma)^3]; it's dimensionless.

• What the sign tells you about shape:

  • Zero: symmetric (e.g. Normal); tails balanced on both sides.

  • Positive (right-skewed): a long thin tail toward large values (e.g. income, waiting times).

  • Negative (left-skewed): a long thin tail toward small values.

• Mean-median-mode relationship:

  • Right-skewed: mode < median < mean, because the extreme high values drag the mean up.

  • Left-skewed: mean < median < mode.

  • Symmetric and unimodal: mean = median = mode (approximately).

• Practical note: skew makes the mean a poor central summary; the median is more robust under skew.

Skewness and kurtosis are the third and fourth standardized moments: skewness measures asymmetry (which way the distribution leans), and kurtosis measures tailedness (how much of the variance comes from extreme outliers versus moderate deviations).

• Skewness (third moment):

  • E[((X - mu)/sigma)^3]; sign gives tail direction, magnitude gives degree of asymmetry.

  • Positive = long right tail, negative = long left tail, 0 = symmetric.

• Kurtosis (fourth moment):

  • E[((X - mu)/sigma)^4]; describes tail heaviness and peakedness driven by outliers.

  • Normal has kurtosis 3; excess kurtosis = kurtosis - 3 rescales so Normal = 0.

  • Leptokurtic (excess > 0): heavy tails, more extreme outliers. Platykurtic (excess < 0): thin tails.

• How they differ: Skewness is about direction/asymmetry; kurtosis is about the weight of the tails, and is symmetric (insensitive to left vs right).

• Practical use: Both are diagnostics for departure from normality: high excess kurtosis warns of outlier-driven risk, nonzero skew warns that the mean misrepresents the center.

These three terms classify distributions by kurtosis, which measures tail heaviness and peakedness relative to the normal distribution.

• Leptokurtic (kurtosis > 3, excess > 0):

  • Heavy tails and a sharper peak: more probability mass in the extremes, so outliers are more likely.

  • Examples: Student's t, Laplace, Cauchy.

• Mesokurtic (kurtosis = 3, excess = 0): The reference case: tail behavior like the normal distribution.

• Platykurtic (kurtosis < 3, excess < 0):

  • Light/thin tails and a flatter peak: fewer extreme outliers.

  • Example: uniform distribution.

• Key nuance: Kurtosis is driven mainly by the tails, not literally the height of the peak, despite the intuitive picture.

Multimodality means a distribution has more than one local peak (mode), which usually signals that the data is a blend of distinct subpopulations or was generated by more than one underlying process.

• Definition: Two peaks is bimodal, more than two is multimodal; a single peak is unimodal.

• What it suggests:

  • A hidden categorical variable: e.g. heights pooling men and women, or response times of two device types.

  • Mixed generating mechanisms or regimes (different states of a system).

• Why it matters:

  • Summary statistics mislead: the mean can fall in a low-density valley between peaks, representing no typical case.

  • Suggests you should segment the data or fit a mixture model rather than a single unimodal distribution.

• Caution: Apparent extra peaks can be sampling noise or a bad histogram bin width; verify with a smoother estimate (KDE) before concluding.

A mixture distribution is formed by combining two or more component distributions with weights that sum to 1: you draw from one component chosen at random according to those weights.

• Definition:

  • The density is a weighted sum: f(x) = Σ wᵢ fᵢ(x), with wᵢ ≥ 0 and Σ wᵢ = 1.

  • The weights wᵢ are the mixing proportions (probabilities of each component).

• Generative view: First sample a latent label i ~ Categorical(w), then sample x from component fᵢ.

• Why useful:

  • Models heterogeneous populations and can produce multimodal or heavy-tailed shapes from simple components.

  • Classic example: a Gaussian Mixture Model, fit via the EM algorithm.

• Note: A mixture of two Gaussians is not itself Gaussian, even though its components are.

Kurtosis is the standardized fourth moment of a distribution; it primarily measures how heavy the tails are (propensity for outliers) relative to the normal distribution.

• Formula: Kurtosis = E[(X − μ)⁴] / σ⁴: the fourth central moment normalized by variance squared.

• What it tells you:

  • High kurtosis: heavy tails, frequent extreme values, high risk of outliers.

  • Low kurtosis: thin tails, values cluster near the mean with few extremes.

• Reference point: The normal distribution has kurtosis 3; comparisons are usually made against this baseline.

• Common misconception: It is about tails, not peakedness: the fourth power weights extreme deviations heavily, so tails dominate the value.

A mixed random variable has a distribution that is part discrete and part continuous: it places positive probability mass on specific points (atoms) while spreading the remaining probability continuously over a range.

• Structure:

  • Its CDF has jumps (from the discrete atoms) plus smoothly increasing segments (from the continuous part).

  • It cannot be fully described by a PMF or a PDF alone; you need both, or the CDF.

• Classic example: insurance/rainfall/censoring:

  • Daily rainfall: a lump of probability at exactly 0 (dry days) plus a continuous distribution over positive amounts.

  • Insurance claims: mass at 0 (no claim) plus a continuous distribution of claim sizes.

• Another common source: censoring/truncation: A sensor capped at max M: continuous below M but a spike of mass at M for all readings that hit the ceiling.

• Computing expectations: Sum the discrete part (Σ x·P(X=x)) and add the continuous part (∫ x f(x) dx).

The Central Limit Theorem (CLT) says that the sum or mean of many independent, identically distributed random variables with finite variance is approximately normally distributed, regardless of the original distribution's shape, as sample size grows.

• Formal statement:

  • For i.i.d. X with mean μ and variance σ², the standardized mean (X̄ − μ)/(σ/√n) converges in distribution to a standard normal as n → ∞.

  • The sample mean's spread shrinks like σ/√n (the standard error).

• Why it matters: It explains why the normal distribution appears everywhere: many real quantities are sums of small independent effects.

• Real-life applications:

  • Polling and surveys: sample proportions are treated as normal to build margins of error.

  • Quality control: averages of measurements form control charts assuming normality.

  • A/B testing: difference in means is normal, enabling z/t tests and confidence intervals.

  • Finance: aggregated returns modeled as normal (with caveats about heavy tails).

• Caveat: Requires finite variance and enough samples; heavy skew or fat tails slow convergence and can make n = 30 insufficient.

The classic CLT requires the observations to be independent, identically distributed, and drawn from a distribution with finite mean and variance, plus a sufficiently large sample. Relaxing these needs stronger conditions or different theorems.

• Independence: Samples must not be correlated; strong dependence breaks the result (weaker mixing conditions exist for dependent data).

• Identically distributed: Classic CLT assumes a common distribution; the Lindeberg/Lyapunov versions allow non-identical variables under conditions that no single term dominates.

• Finite variance: The critical condition: with infinite variance (e.g. Cauchy), the standard CLT does not hold and a generalized version with stable laws applies instead.

• Sufficient sample size: n ≈ 30 is a rough rule; heavily skewed or fat-tailed data need much larger n for the approximation to be good.

Continuity correction adjusts the boundaries of a discrete probability by ±0.5 when you approximate it with a continuous distribution, because a discrete value like X = 5 occupies the whole interval from 4.5 to 5.5 under the continuous curve.

• Why it's needed: A continuous distribution assigns zero probability to a single point, so mapping discrete integers directly loses the width of each bar.

• How to apply it:

  • P(X ≤ k) → P(Y ≤ k + 0.5).

  • P(X ≥ k) → P(Y ≥ k − 0.5).

  • P(X = k) → P(k − 0.5 ≤ Y ≤ k + 0.5).

• Impact: Most important for small-to-moderate n where the 0.5 is a meaningful fraction of the spread; its effect shrinks as n grows.

The binomial counts successes in n independent trials, and by the CLT that count is a sum of n independent Bernoulli variables, so for large n it becomes approximately normal with mean np and variance np(1−p).

• The link:

  • Binomial(n, p) = sum of n i.i.d. Bernoulli(p), so the CLT directly gives normality as n grows.

  • Matched moments: mean np, variance np(1−p).

• Conditions for the approximation:

  • np ≥ 10 and n(1−p) ≥ 10 so the distribution isn't skewed against a boundary.

  • Best when p is near 0.5 (symmetric); extreme p needs larger n.

• Use continuity correction: Apply ±0.5 adjustments since binomial is discrete and normal is continuous.

• Alternative: When p is tiny and n large (np small), use the Poisson approximation instead.

The CLT lets a data scientist treat sample means and aggregates as normally distributed even without knowing the underlying distribution, which is the foundation for inference, hypothesis testing, and estimating uncertainty.

• Confidence intervals and standard errors: Build intervals around means using σ/√n and normal quantiles, regardless of the raw data's shape.

• Hypothesis testing: z and t tests, A/B test comparisons, and p-values all rest on the sampling distribution of the mean being normal.

• Simplifying complex models: Approximate sums of many small random effects (measurement error, aggregated metrics) as normal.

• Justifying resampling and Monte Carlo: Averages of simulated draws converge to normal, giving reliable error bars.

• Caution: Check for heavy tails, dependence, or too-small n before trusting normality.

Expectation is the long-run average value of a random variable and variance measures its spread around that mean. Take the Poisson distribution: both its mean and variance equal its rate parameter λ, which can be derived from its PMF or, more elegantly, its moment generating function.

• Definitions:

  • E[X] = Σ x·P(x) for discrete variables (integral for continuous).

  • Var(X) = E[X²] − (E[X])².

• Poisson PMF: P(X = k) = e^(−λ) λ^k / k! for k = 0, 1, 2, ...

• Deriving the mean: Summing k·P(k) and shifting the index collapses the series back to the Poisson total (which is 1), leaving E[X] = λ.

• Deriving the variance: Compute E[X(X−1)] = λ², so E[X²] = λ² + λ, then Var(X) = λ² + λ − λ² = λ.

• Interpretation: Mean = variance = λ is a signature of the Poisson; if real data has variance far exceeding the mean, it is overdispersed and Poisson is a poor fit.

Central moments measure the distribution's spread and shape around its own mean, defined as E[(X - μ)^n] for the nth moment.

1. Zeroth moment: always 1: E[(X - μ)^0] = E[1] = 1, it just normalizes the distribution.

2. First central moment: always 0: E[(X - μ)] = 0 by definition of the mean; the raw first moment (not central) is the mean itself.

3. Second central moment: variance: E[(X - μ)^2] = σ^2, the average squared deviation, measuring spread.

4. Third central moment: relates to skewness: E[(X - μ)^3]; standardized (divided by σ^3) it gives skewness, measuring asymmetry.

5. Fourth central moment: relates to kurtosis: E[(X - μ)^4]; standardized (divided by σ^4) it gives kurtosis, measuring tail heaviness/peakedness.

Distribution parameters fall into three functional families: location (where it sits), scale (how spread out), and shape (its form). Understanding which is which lets you reason about how a distribution transforms.

• Location parameter:

  • Shifts the distribution along the x-axis without changing its spread or shape.

  • Example: μ in the Normal distribution; adding a constant shifts location.

• Scale parameter:

  • Stretches or compresses the distribution; larger scale means more spread.

  • Example: σ in the Normal, or the rate/scale in the Exponential distribution.

• Shape parameter:

  • Changes the fundamental form (skewness, tail behavior, peakedness), not just position or width.

  • Example: the shape parameter k of the Gamma or Weibull distributions; degrees of freedom in a t-distribution.

• Note: some parameters play multiple roles: For the Poisson, λ affects both mean and variance; not every distribution cleanly separates the three.

The Moment Generating Function (MGF) is M_X(t) = E[e^{tX}], a transform of the distribution that, when it exists, uniquely characterizes it and encodes all its moments.

• Definition: M_X(t) = E[e^{tX}], defined for t in a neighborhood of 0 where the expectation is finite.

• Primary purposes:

  1. Generating moments: successive derivatives at t = 0 give the raw moments.

  2. Uniqueness: if two variables have the same MGF (in an interval around 0), they have the same distribution.

  3. Sums of independent variables: MGFs multiply, making convolutions easy.

• Caveat: The MGF may not exist (heavy-tailed distributions like Cauchy have no finite MGF), which is why the characteristic function is often preferred.

A Q-Q (quantile-quantile) plot compares the quantiles of your data against the quantiles of a theoretical distribution: if the data follows that distribution, the points fall on a straight line.

• Construction:

  • Sort the data (these are the empirical/sample quantiles) on one axis.

  • Compute the corresponding theoretical quantiles: for order statistic i of n, take the inverse CDF at a plotting position like (i - 0.5)/n.

  • Plot sample quantiles against theoretical quantiles.

• Interpretation:

  • Points on (or near) the reference line: data is consistent with the distribution.

  • A straight line at any slope/intercept still indicates a good fit for location-scale families (normal, etc.), since those parameters only shift and stretch the line.

  • Systematic deviations (curves, S-shapes, off-line tails) reveal how the data departs.

• Why it's useful: it is a visual, distribution-general diagnostic that highlights tail behavior and skew far better than a single test statistic, though it is subjective and less reliable for small samples.

The shape of the departure from the reference line tells you specifically how the data differs from the theoretical distribution: tails, skew, or multimodality each leave a characteristic signature.

• Straight line: good fit; data matches the theoretical distribution.

• S-shape (ends bend opposite ways):

  • Points below the line at the left and above at the right: heavier tails than theoretical (leptokurtic, e.g. data heavier than normal).

  • Opposite S (above on left, below on right): lighter/shorter tails than theoretical.

• Convex or concave curve (one-sided bow): Consistent upward or downward curvature indicates skewness: the data is right- or left-skewed relative to the theoretical distribution.

• Off-line points only at the extremes: outliers or a single heavy tail.

• Distinct steps, gaps, or plateaus: discreteness, rounding, or a mixture/multimodal structure.

The empirical CDF (ECDF) is the nonparametric estimate of the distribution function built directly from data: at any value x it is the fraction of observations less than or equal to x. It is the data-driven baseline against which theoretical distributions are compared.

• Definition: F_n(x) = (number of observations ≤ x) / n: a right-continuous step function jumping by 1/n at each data point.

• Properties:

  • An unbiased, consistent estimator of the true CDF.

  • By the Glivenko-Cantelli theorem it converges uniformly to the true CDF as n grows.

• Role in assessing fit:

  • Overlaying the ECDF and a fitted theoretical CDF shows discrepancies directly.

  • It underlies CDF-based tests: K-S uses the max gap between ECDF and theoretical CDF, Anderson-Darling a tail-weighted version.

  • Assumption-free (no binning or smoothing), unlike a histogram or KDE.

Kernel density estimation (KDE) is a nonparametric way to estimate a continuous probability density from data by placing a smooth 'bump' (kernel) at each observation and summing them, producing a smooth curve without assuming a specific distributional form.

• How it works:

  • Each data point contributes a kernel (often Gaussian) centered on it; the estimate is the average of all kernels.

  • The bandwidth h controls the width of each kernel and thus the smoothness.

• Bandwidth is the key choice:

  • Too small: undersmoothed, noisy, spurious bumps (high variance).

  • Too large: oversmoothed, washes out real structure (high bias).

  • Rules like Silverman's or cross-validation pick a balance.

• Why it helps:

  • Reveals shape features (modes, skew, multimodality) more smoothly than a histogram, which is sensitive to bin edges and width.

  • Useful for comparing an empirical shape against a candidate parametric density.

• Caveats: boundary bias near hard limits (e.g. non-negative data), and it struggles in high dimensions.

You estimate a distribution from data either parametrically (assume a family and estimate its parameters) or nonparametrically (let the data define the shape), then validate the fit. The right approach depends on how much you know about the data-generating process.

• Explore first: Plot a histogram, ECDF, or KDE; check summary stats (skewness, kurtosis, support) to propose candidate families.

• Parametric estimation:

  • Choose a family (normal, gamma, Poisson, ...) motivated by the domain.

  • Fit parameters via MLE (usual choice) or Method of Moments; Bayesian estimation gives a posterior over parameters.

• Nonparametric estimation: KDE or the ECDF when no family fits well; more flexible but needs more data.

• Validate the fit:

  • Q-Q/P-P plots for visual diagnosis; goodness-of-fit tests (K-S, Anderson-Darling, Shapiro-Wilk, chi-square).

  • Compare competing models with AIC/BIC to balance fit and complexity.

A P-P (probability-probability) plot compares the empirical cumulative probabilities against the theoretical cumulative probabilities: like a Q-Q plot, a good fit yields a straight diagonal line, but it plots probabilities rather than quantiles, so it emphasizes different regions.

• What is plotted:

  • P-P: empirical CDF value vs. theoretical CDF value at each point, both on [0,1].

  • Q-Q: sample quantiles vs. theoretical quantiles, on the data's scale.

• Where each is sensitive:

  • P-P is most sensitive in the center of the distribution (where probability mass is dense).

  • Q-Q is most sensitive in the tails, making it better for detecting outliers and heavy tails.

• Scale and parameters:

  • Q-Q handles location-scale differences gracefully (still a straight line, different slope/intercept).

  • P-P is bounded to the unit square and depends on correctly specified parameters.

• Practical takeaway: use Q-Q for tail behavior and general distributional checks (more common); use P-P when comparing fit in the body of the distribution.

Assuming X and Y are independent, Z = 2X − Y is normal with mean 5 and variance 20, i.e. Z ~ N(5, 20).

• Mean adds linearly: E[Z] = 2·E[X] − E[Y] = 2·3 − 1 = 5.

• Variance uses squared coefficients: Var(Z) = 2²·Var(X) + (−1)²·Var(Y) = 4·4 + 1·4 = 20 (independence means no covariance term).

• Result:

  • Z ~ N(5, 20), so its standard deviation is √20 ≈ 4.47.

  • Note the coefficient is squared for variance but not for the mean: this is the most common mistake.

The t-distribution's heavier tails come from the uncertainty in estimating the standard deviation from a sample; as degrees of freedom grow, that estimate stabilizes and the extra variability vanishes, so the t converges to the standard normal.

• Where the extra uncertainty comes from: The t statistic divides by a sample standard deviation, itself a random quantity; this added randomness fattens the tails relative to the normal.

• Effect of increasing df:

  • With more data the sample variance estimates the true variance more precisely (its sampling variability shrinks toward zero).

  • The denominator becomes effectively a constant, which is exactly the case that defines the standard normal.

• Practical takeaway: By roughly df ≥ 30 the two are nearly indistinguishable; in the limit df → ∞ they are identical.

Choose by the nature of the outcome: binomial for counts of successes in fixed trials, Poisson for counts of rare events over an interval, normal for continuous quantities driven by many small additive effects, and exponential for waiting times between events.

• Binomial:

  • Fixed number n of independent yes/no trials with constant probability p.

  • Example: number of conversions out of 1,000 website visitors.

• Poisson:

  • Count of events in a fixed time/space window at a constant average rate λ.

  • Example: number of customer support calls arriving per hour.

• Normal:

  • Continuous measurements shaped by many small independent influences (central limit theorem).

  • Example: heights of adults, measurement errors.

• Exponential:

  • Time until the next event in a Poisson process; memoryless.

  • Example: time between successive arrivals of those support calls.

• Connection: Binomial with large n and small p approximates Poisson; the Poisson event counts and exponential inter-arrival times describe the same process.

Match the distribution to the data-generating process: identify the variable's support (discrete vs continuous, bounded vs unbounded), its shape, and the mechanism that produced it, then pick the family whose assumptions fit.

• Start with the support and type: Counts of events: Poisson or Negative Binomial; binary outcomes: Bernoulli/Binomial; continuous unbounded: Normal; positive-only continuous: Log-normal, Gamma, Exponential; proportions on (0,1): Beta.

• Ask about the generating mechanism: Time between independent events: Exponential; number of successes in fixed trials: Binomial; sum of many small independent effects: Normal (CLT justification).

• Inspect the empirical shape: Skew, heavy tails, multimodality, and mean-variance relationship all narrow the candidate set (e.g. variance far above mean in counts suggests overdispersion).

• Validate the fit: Use Q-Q plots, histograms, and goodness-of-fit tests; compare candidates with AIC/BIC or log-likelihood.

• Prefer the simplest defensible choice: Favor interpretability and known properties over an exotic fit unless the data clearly demand it.

Apply a log or variance-stabilizing transform when the data are strongly right-skewed or the variance grows with the mean, so that after transformation the distribution is more symmetric and the variance more constant, which better satisfies model assumptions.

• When a log transform helps:

  • Positive, right-skewed, multiplicative data (income, prices, counts of large magnitude); it compresses large values and pulls in the tail.

  • When effects are relative/proportional: log turns multiplicative relationships into additive ones, easing linear modeling.

• Other variance-stabilizing transforms: Square root for Poisson-like counts; arcsine or logit for proportions; Box-Cox to select an optimal power automatically.

• Why it matters: Stabilizes variance (homoscedasticity), improves approximate normality, and reduces leverage of outliers.

• Caveats: log(0) is undefined: add a small constant or use log1p; interpretation shifts to the transformed scale and back-transforming the mean introduces bias (Jensen's inequality). Often a GLM with the right family is cleaner than transforming.

All three share the Naive Bayes conditional-independence assumption but differ in the P(x | y) likelihood they assume for each feature, which dictates the kind of data each fits.

• Gaussian NB: continuous features:

  • Assumes each feature is normally distributed within a class, estimating a per-class mean and variance.

  • Use for real-valued measurements (e.g. sensor readings).

• Multinomial NB: counts / frequencies:

  • Assumes features are counts drawn from a multinomial distribution over categories.

  • Classic for text (bag-of-words term counts, TF-IDF).

• Bernoulli NB: binary presence/absence:

  • Assumes each feature is a 0/1 Bernoulli variable, explicitly modeling non-occurrence too.

  • Good for short text where word presence matters more than count.

• Key contrast: Multinomial ignores zeros' penalty; Bernoulli penalizes absent features directly, so they can differ sharply on sparse data.

Joint, marginal, and conditional distributions are three views of a multivariable probability model: the joint describes all variables together, the marginal describes one (or a subset) ignoring the rest, and the conditional describes one given fixed values of another.

• Joint: P(X, Y): Full probability over all outcomes simultaneously; the source from which the others are derived.

• Marginal: P(X):

  • Obtained by summing/integrating the joint over the other variable: P(X) = Σ_y P(X, y).

  • Answers "what does X do on its own?"

• Conditional: P(X | Y):

  • Defined as P(X, Y) / P(Y), renormalizing to the slice where Y takes a given value.

  • Answers "what does X do once Y is known?"

• Relationships to remember:

  • Chain rule: P(X, Y) = P(X | Y)·P(Y).

  • Independence iff P(X, Y) = P(X)·P(Y), equivalently P(X | Y) = P(X).

Two random variables are independent when knowing one gives no information about the other: their joint distribution factors into the product of marginals, and conditional distributions collapse to the marginals.

• Joint = product of marginals:

  • Independence iff f(x,y) = f_X(x) f_Y(y) for all x, y (or the CDF factors).

  • This is the definitional condition and must hold everywhere, not just at a point.

• Conditional equals marginal: Equivalently f(x|y) = f_X(x): conditioning on Y changes nothing about X.

• Consequences (necessary but not sufficient):

  • Independence implies zero covariance/correlation, and E[XY] = E[X]E[Y].

  • The converse fails: uncorrelated does not imply independent (except jointly Gaussian).

• Factorization test: If the joint density factors into a function of x times a function of y (with a product-form support), they are independent.

A covariance matrix is a symmetric matrix whose entries are the pairwise covariances of a random vector: diagonals are variances, off-diagonals measure how each pair of variables co-varies. It fully summarizes the second-order (linear) relationships among the variables.

• Definition:

  • For vector X, Σ = E[(X - μ)(X - μ)^T], so Σ_ij = Cov(X_i, X_j).

  • Diagonal entries Σ_ii = Var(X_i); off-diagonals capture linear association.

• Key properties:

  • Symmetric and positive semi-definite (variances of any linear combination are non-negative).

  • Sign of off-diagonals: positive means move together, negative means opposite.

• Relation to correlation: Normalize by standard deviations to get the correlation matrix: ρ_ij = Σ_ij / (σ_i σ_j), making scale-free comparisons.

• Uses:

  • Its eigenvectors/eigenvalues give principal component directions (PCA).

  • It parameterizes the multivariate normal and drives portfolio risk, whitening, and Mahalanobis distance.

• Limitation: Only captures linear dependence; nonlinear relationships can be missed.

These three are nested by construction: a Bernoulli is a single yes/no trial, a Binomial counts successes across many independent Bernoulli trials, and a Poisson is the limiting case of a Binomial when trials become many and rare, counting events over a continuous interval.

• Bernoulli(p):

  • A single trial: outcome 1 with probability p, 0 otherwise.

  • Mean p, variance p(1-p).

• Binomial(n, p):

  • Sum of n independent Bernoulli(p) trials; counts successes.

  • Mean np, variance np(1-p); n=1 recovers Bernoulli.

• Poisson(λ):

  • Counts events in a fixed interval when they occur independently at rate λ.

  • Mean = variance = λ.

• The limiting link:

  • As n → ∞ and p → 0 with np = λ fixed, Binomial(n, p) → Poisson(λ).

  • Hence Poisson approximates the Binomial for many rare trials (law of rare events).

A variable is Log-normal exactly when its natural logarithm is Normal: exponentiating a Normal gives a Log-normal, and taking logs reverses it.

• Definition:

  • If Y ~ N(μ,σ²), then X = e^Y is Log-normal with parameters μ, σ.

  • Conversely ln(X) ~ N(μ,σ²).

• Key features:

  • Support is strictly positive, and the distribution is right-skewed.

  • μ and σ are the mean and sd of the log, not of X itself: E[X] = e^(μ+σ²/2).

• Why it arises: Products of many independent positive factors become Log-normal by the multiplicative CLT (sums of logs are Normal), e.g. asset prices, incomes.

The sum of k independent squared standard normals follows a chi-squared distribution with k degrees of freedom, χ²(k). This is the definitional origin of the chi-squared family.

• Statement: If Z₁,...,Z_k are iid N(0,1), then Σ Zᵢ² ~ χ²(k).

• Why a single Z² is χ²(1): Apply the change-of-variables to Y = Z²; the density works out to a Gamma(1/2, 1/2), which is exactly χ²(1).

• Why the sum is χ²(k):

  • Chi-squared is a Gamma with shape k/2, rate 1/2; Gammas with the same rate are additive in shape, so summing k independent χ²(1) gives χ²(k).

  • Equivalently, moment generating functions multiply and the exponents add.

• Consequence: Mean = k, variance = 2k; this underlies variance estimation and goodness-of-fit tests.

Inverse Transform Sampling draws a uniform U on [0,1] and returns X = F⁻¹(U), the value at which the CDF reaches U. Because F⁻¹ maps uniform mass onto the target distribution, X has the desired CDF F.

• Algorithm:

  1. Sample U ~ Uniform(0,1).

  2. Return X = F⁻¹(U), the inverse CDF (quantile function).

• Why it works: Direct consequence of the Probability Integral Transform: P(X ≤ x) = P(U ≤ F(x)) = F(x).

• Discrete case: Use the generalized inverse: return the smallest x with F(x) ≥ U (equivalent to walking the cumulative probabilities).

• Strengths and limits:

  • Exact and simple when F⁻¹ is closed-form (Exponential: X = -ln(1-U)/λ).

  • Impractical when F⁻¹ has no closed form (e.g. the Normal), motivating Box-Muller or rejection methods.

The Cauchy distribution is a symmetric, bell-shaped distribution that looks superficially like a Normal but has extremely heavy tails. Its defining oddity is that it has no finite mean and no finite variance: the tails are so fat that these moments simply don't exist.

• Shape:

  • Symmetric about a location parameter x₀ with a scale parameter γ; peaked center, but far heavier tails than the Normal.

  • Its center is described by the median and mode (both x₀), not the mean.

• No mean or variance:

  • The integral defining the mean does not converge; variance is likewise undefined.

  • Consequence: the sample mean does NOT converge, so the Law of Large Numbers and CLT fail for it.

• Notable properties:

  • The ratio of two independent standard normals is Cauchy.

  • An average of Cauchy samples is again Cauchy (stable distribution), no tighter than one observation.

• Uses: Models resonance in physics and heavy-tailed, outlier-prone data; a cautionary example of when standard estimators break.

The Gumbel distribution is one of the extreme-value distributions: it models the distribution of the maximum (or minimum) of a large number of samples. It answers questions about how big the biggest observation tends to be.

• Models extremes, not averages:

  • Where the Normal describes typical values via the CLT, extreme-value theory describes the limiting distribution of the maximum of many i.i.d. draws.

  • Gumbel is the limit when the underlying tail decays exponentially (e.g. Normal, Exponential parents).

• Shape and parameters:

  • Has a location parameter and a scale parameter; the density is skewed with a heavier right tail (for the maximum version).

  • CDF is a clean double exponential: F(x) = exp(-exp(-(x-mu)/beta)).

• Typical uses:

  • Hydrology and engineering: flood levels, maximum rainfall, wind loads, the "100-year event".

  • Reliability and finance: worst-case losses, breaking strengths.

  • Machine learning: the Gumbel-max trick for sampling from a categorical distribution via argmax of Gumbel-perturbed logits.

• Part of a family: The generalized extreme value (GEV) distribution unifies Gumbel, Fréchet (heavy tails), and Weibull (bounded tails) as one parametric family.

Overdispersion means the observed variance of count data exceeds its mean. The Poisson forces variance to equal the mean, so it underestimates spread; the Negative Binomial adds a dispersion parameter that lets variance exceed the mean, giving a better fit.

• The Poisson constraint: Poisson has mean = variance = lambda. Real counts (unobserved heterogeneity, clustering) often vary more than this.

• Consequences of ignoring it: Standard errors are too small, so p-values and confidence intervals are overly optimistic (false significance).

• Why Negative Binomial helps:

  • It arises as a Poisson whose rate is itself Gamma-distributed (a Poisson-Gamma mixture), capturing rate variation across units.

  • Variance = mu + mu^2/r; as the dispersion size r grows large it collapses back to the Poisson.

• Diagnosis: Check the ratio of variance to mean, or Pearson deviance / residual df > 1; use a dispersion test before switching.

Heavy (fat) tails mean the probability of extreme values decays slowly, so rare, large events happen far more often than a normal distribution would predict. They matter because averages, variances, and risk estimates can be dominated or destabilized by these outliers.

• What heavy tails are:

  • The tail of the survival function decays slower than exponential (often as a power law, x^{-alpha}).

  • Examples: Cauchy, Pareto, Student-t with low degrees of freedom, log-normal.

• Light tails for contrast: Decay exponentially or faster; extremes are effectively negligible (Normal, Exponential).

• Why they matter in practice:

  • Moments can be infinite/undefined: e.g. Cauchy has no finite mean or variance, so sample means don't stabilize.

  • Risk underestimation: normal-based models drastically underprice extreme losses (finance, insurance, network traffic).

  • A single observation can dominate the sum, so the law of large numbers and CLT may converge slowly or not apply.

• How to detect: High kurtosis, straight lines on log-log survival plots, or heavy-tailed QQ plots.

Heavy-tailed distributions put so much mass in the extremes that key moments (mean, variance) can be undefined, which invalidates the assumptions the Law of Large Numbers and Central Limit Theorem rely on.

• The mean may not exist: For the Cauchy distribution the integral defining the expectation does not converge, so there is no finite population mean to estimate.

• Sample averages don't stabilize: The sample mean of Cauchy draws is itself Cauchy-distributed: averaging more data does not shrink its spread, so the Law of Large Numbers fails.

• The CLT requires finite variance: The classical CLT assumes finite mean and variance; Cauchy has neither, so scaled sums do not converge to a normal distribution.

• Practical consequence:

  • A single extreme observation can dominate the average, so mean/variance-based reasoning (confidence intervals, z-tests) becomes misleading.

  • Use robust statistics (median, trimmed mean, MAD) instead.

• Caveat: Some heavy-tailed distributions have a finite mean but infinite variance (e.g. certain Pareto): the mean is fine but the CLT still breaks or converges to a stable law.

The fourth central moment is the raw, unstandardized average of (X − μ)⁴, while excess kurtosis is the standardized fourth moment minus 3, making it a scale-free, normal-referenced measure of tail heaviness.

• Fourth central moment: μ₄ = E[(X − μ)⁴]: Always non-negative, but its magnitude depends on the units and scale of X, so raw values aren't comparable across distributions.

• Kurtosis: μ₄ / σ⁴: Dividing by variance-squared removes scale, giving a dimensionless number; equals 3 for the normal.

• Excess kurtosis: (μ₄ / σ⁴) − 3:

  • Subtracting 3 centers the normal at 0 for easy interpretation.

  • Positive → heavier tails than normal (leptokurtic); negative → lighter tails (platykurtic).

• Why standardize: You want to compare tail behavior across distributions regardless of their spread or units, which the raw moment can't do.

A distribution has infinite or undefined variance when the integral (or sum) defining E[(X-μ)²] does not converge to a finite number, typically because the tails are so heavy that extreme values dominate. This breaks many standard tools that silently assume finite second moments.

• What causes it:

  • Heavy (power-law) tails where the probability of extreme values decays too slowly, so the variance integral diverges.

  • Undefined (not just infinite) when even the mean fails to converge, e.g. the Cauchy distribution has no defined mean or variance.

• Classic examples:

  • Cauchy: mean and variance both undefined.

  • Pareto with shape α ≤ 2: infinite variance; α ≤ 1 gives infinite mean too.

• Practical problems:

  • The Central Limit Theorem in its classic form does not apply, so sample means don't shrink to a normal at the usual rate.

  • Sample variance and standard error don't converge: they keep jumping as more data arrives, so confidence intervals are unreliable.

  • Risk metrics (VaR, expected shortfall) and least-squares estimators can be dominated by rare extremes and badly underestimate risk.

• What to do instead: Use robust statistics (median, MAD, quantiles) and heavy-tailed models; consider generalized CLT with stable distributions.

Use the normal approximation when the discrete distribution's shape becomes roughly symmetric and bell-shaped, which happens when the expected count of events is large enough. It simplifies probability calculations and enables normal-based inference, but fails when the distribution is still skewed near a boundary.

• Binomial → Normal:

  • Valid when both np ≥ 10 and n(1−p) ≥ 10 (some texts use 5), giving mean np and variance np(1−p).

  • Fails when p is near 0 or 1 with small n: the distribution is skewed and bunched near a boundary.

• Poisson → Normal:

  • Valid when λ is large (roughly λ ≥ 10), with mean λ and variance λ.

  • Fails for small λ where the distribution is right-skewed.

• When to prefer other approximations: For Binomial with large n and small p (np moderate), use the Poisson approximation instead of normal.

• Always apply a continuity correction: Since you're approximating a discrete variable with a continuous one, adjust by ±0.5 to keep accuracy.

Differentiating the MGF at t = 0 yields moments, and because the MGF of a sum of independent variables is the product of their MGFs, it turns convolutions into simple multiplication.

• Deriving moments:

  • The nth raw moment is the nth derivative evaluated at zero: E[X^n] = M_X^{(n)}(0).

  • Intuition: expanding e^{tX} as a power series, the coefficient of t^n/n! is E[X^n].

• Sums of independent variables:

  • If X and Y are independent, M_{X+Y}(t) = M_X(t) · M_Y(t), since E[e^{t(X+Y)}] = E[e^{tX}]E[e^{tY}].

  • This proves closure properties: e.g., the sum of independent Normals is Normal, sum of independent Poissons is Poisson.

Both transform a distribution to encode its moments and uniquely identify it, but the MGF uses a real exponential E[e^{tX}] and can diverge, whereas the characteristic function uses a complex exponential E[e^{itX}] and always exists.

• Definitions:

  • MGF: M_X(t) = E[e^{tX}], real-valued.

  • Characteristic function: φ_X(t) = E[e^{itX}], complex-valued (essentially the Fourier transform of the density).

• Why the characteristic function always exists:

  • Since |e^{itX}| = 1 for all real t, the expectation is always finite (bounded by 1).

  • The MGF requires e^{tX} to be integrable, which fails for heavy-tailed distributions where large values grow the exponential without bound.

• Practical implication:

  • For distributions like the Cauchy or log-normal the MGF doesn't exist, so proofs (e.g., the general CLT via convergence of transforms) rely on the characteristic function.

  • When the MGF exists, both give the same information and the MGF is often simpler to manipulate.