80 Probability & Bayes' Theorem Interview Questions and Answers (2026)

A random variable is a function that maps outcomes of a random experiment to numbers, letting us describe uncertainty quantitatively and compute probabilities, expectations, and variances.

• Definition:

  • A map from the sample space to real numbers (e.g. X = number of heads in 3 coin flips).

  • Notation: capital letters for the variable (X), lowercase for a realized value (x).

• Two main types:

  • Discrete: countable outcomes, described by a PMF P(X = x).

  • Continuous: values over an interval, described by a PDF f(x).

• How you use them:

  • Compute probabilities of events, e.g. P(X > 2).

  • Summarize with expectation E[X] and variance Var(X).

  • Transform them: functions of random variables (Y = g(X)) are themselves random variables.

Discrete distributions place probability on a countable set of specific values, while continuous distributions spread probability over a continuum where any single point has probability zero.

• Discrete:

  • Described by a PMF; each value has a positive mass and masses sum to 1.

  • Examples: Bernoulli, Binomial, Poisson.

• Continuous:

  • Described by a PDF; probability comes from area under the curve over an interval, integrating to 1.

  • P(X = x) = 0 for any exact point; only ranges have positive probability.

  • Examples: Normal, Exponential, Uniform.

• Key operational difference:

  • Discrete uses summation over values; continuous uses integration over intervals.

  • Both share a CDF, which is a valid unifying tool for either type.

A probability density function (PDF) describes the relative likelihood of a continuous random variable across its range; probabilities are obtained by integrating the PDF over an interval, not by reading a single point.

• What it represents:

  • f(x) is a density, not a probability: it can exceed 1, but the total area under it equals 1.

  • P(a ≤ X ≤ b) = integral of f(x) from a to b.

• Key properties:

  • f(x) ≥ 0 everywhere.

  • Integral over the whole domain = 1.

  • P(X = c) = 0 for any single point c.

• Relation to the CDF: The PDF is the derivative of the CDF; the CDF is the integral of the PDF.

The cumulative distribution function (CDF), F(x) = P(X ≤ x), gives the accumulated probability up to a value and works uniformly for both discrete and continuous variables.

• What it tells you:

  • Probability that the variable falls at or below a threshold.

  • Interval probabilities: P(a < X ≤ b) = F(b) - F(a).

• Properties:

  • Non-decreasing, right-continuous.

  • Limits: approaches 0 as x → -∞ and 1 as x → +∞.

• Why it is useful:

  • Unifies discrete (step function) and continuous (smooth) cases.

  • Its inverse (quantile function) drives percentiles and inverse-transform sampling.

A probability distribution is simply a description of how likely each possible outcome is: it tells you where results tend to land and how spread out they are.

• Everyday analogy:

  • Roll a die: each of six faces has an equal 1-in-6 chance, that is the whole distribution.

  • Heights of adults: most cluster near an average, fewer are very tall or very short, forming a bell shape.

• What it captures:

  • The set of possible outcomes.

  • How the total certainty (100%) is shared among them.

• Why it matters: It lets you answer questions like: what is typical, what is rare, and what to expect on average.

A PDF (probability density function) describes the relative likelihood at each point of a continuous random variable, while a CDF (cumulative distribution function) gives the probability of being at or below a value. The CDF is the integral of the PDF, and the PDF is the derivative of the CDF.

• PDF: f(x):

  • Density, not probability: f(x) can exceed 1, and P(X = x) = 0 for any single point.

  • Probability comes from area: P(a ≤ X ≤ b) = integral of f(x) from a to b; total area = 1.

  • Discrete analog is the PMF (probability mass function), where each point has real probability.

• CDF: F(x) = P(X ≤ x):

  • Monotonically non-decreasing, ranges from 0 to 1.

  • Works for both continuous and discrete variables (step function in the discrete case).

• Relationship:

  • F(x) = integral of f(t) from -infinity to x; conversely f(x) = dF/dx.

  • P(a ≤ X ≤ b) = F(b) - F(a).

Two events are independent if the occurrence of one does not change the probability of the other; they are dependent if it does. Formally, independence means P(A ∩ B) = P(A)·P(B).

• Independent events:

  • P(A | B) = P(A): knowing B tells you nothing about A.

  • Example: two separate coin flips.

• Dependent events:

  • P(A | B) ≠ P(A): the joint uses the general multiplication rule P(A ∩ B) = P(A)·P(B | A).

  • Example: drawing two cards without replacement (the first draw changes the second's odds).

• Common pitfalls:

  • Independent is not the same as mutually exclusive: mutually exclusive events (disjoint) are actually dependent, since one occurring forces the other to 0.

  • Pairwise independence does not imply mutual independence across three or more events.

Probability is the mathematical measure of uncertainty, assigning numbers between 0 and 1 to how likely events are. In machine learning it is the language for modeling noise, making predictions, and reasoning about confidence.

• Two interpretations:

  • Frequentist: long-run frequency of an event over repeated trials.

  • Bayesian: a degree of belief that updates as evidence arrives.

• Where it appears in ML:

  • Models output distributions: classifiers produce P(class | features), not just labels.

  • Training via likelihood: maximum likelihood and cross-entropy loss are grounded in probability.

  • Generative models (Naive Bayes, GMMs, VAEs) model the joint or data distribution directly.

  • Bayesian inference expresses uncertainty over parameters and predictions.

• Why it matters: It quantifies confidence, handles noisy/incomplete data, and provides principled objectives to optimize.

The sample space is the set of all possible outcomes of a random experiment, and an event is any subset of that sample space: a collection of outcomes we care about.

• Sample space (often denoted Ω or S):

  • Every possible outcome exactly once; outcomes are mutually exclusive and collectively exhaustive.

  • Example: a die roll has Ω = {1, 2, 3, 4, 5, 6}.

  • Can be finite, countably infinite, or continuous (e.g. all reals in [0, 1]).

• Event:

  • A subset of Ω; occurs if the observed outcome lies in it.

  • Example: 'even number' = {2, 4, 6}.

  • Special cases: the whole Ω (certain), the empty set (impossible), and single-outcome elementary events.

These three rules let you combine and manipulate probabilities: the addition rule handles 'or', the multiplication rule handles 'and', and the complement rule handles 'not'.

• Addition rule (union, 'or'):

  • General: P(A ∪ B) = P(A) + P(B) - P(A ∩ B).

  • If A and B are mutually exclusive, the overlap is 0, so P(A ∪ B) = P(A) + P(B).

• Multiplication rule (intersection, 'and'):

  • General: P(A ∩ B) = P(A)·P(B | A).

  • If independent, P(A ∩ B) = P(A)·P(B).

• Complement rule ('not'): P(Aᶜ) = 1 - P(A); useful when 'at least one' is easier computed as 1 minus 'none'.

The Kolmogorov axioms are the three foundational rules that any valid probability measure must satisfy, defined over a sample space and a sigma-algebra of events. They make probability rigorous by grounding it in measure theory.

• 1. Non-negativity: For any event A, P(A) ≥ 0.

• 2. Normalization: P(Ω) = 1: the probability of the entire sample space is 1.

• 3. Countable additivity: For mutually exclusive (disjoint) events A₁, A₂, ..., P(∪ Aᵢ) = Σ P(Aᵢ).

• Consequences derived from them:

  • P(∅) = 0, P(Aᶜ) = 1 - P(A), and 0 ≤ P(A) ≤ 1.

  • The inclusion-exclusion addition rule follows directly.

Mutually exclusive events cannot happen together; independent events can happen together but neither affects the other's probability. They are distinct, often opposite, ideas.

• Mutually exclusive (disjoint):

  • Both cannot occur at once: P(A ∩ B) = 0.

  • Knowing one happened tells you the other did NOT, so they are actually dependent (unless one has probability 0).

• Independent:

  • Joint equals product: P(A ∩ B) = P(A)P(B).

  • Knowing one occurred leaves the other's probability unchanged: P(A|B) = P(A).

• Key contrast: Two events with positive probability cannot be both mutually exclusive and independent: if disjoint, one occurring forces the other out.

• Example: One die roll: 'even' and 'odd' are mutually exclusive; two separate rolls are independent.

Computing 'at least one' directly means summing many overlapping cases; it's far easier to compute the probability of NONE and subtract from 1, since 'at least one' and 'none' are complements.

• Core rule: P(at least one) = 1 − P(none).

• Independent trials: If each trial fails with probability q, then P(at least one success) = 1 − qⁿ.

• Why it's easier: 'None' is a single intersection (all trials miss), avoiding inclusion-exclusion over all subsets.

• Example: Rolling a die 4 times, P(at least one six) = 1 − (5/6)⁴ ≈ 0.518.

The normal (Gaussian) distribution is a continuous, symmetric, bell-shaped distribution fully described by two parameters: its mean (center) and variance (spread).

• Shape and symmetry: Symmetric about the mean, so mean = median = mode; it is unimodal with thin tails.

• Two parameters: N(μ, σ²): μ shifts the center, σ controls width; standard normal is μ=0, σ=1.

• Empirical (68-95-99.7) rule: About 68%, 95%, and 99.7% of mass lies within 1, 2, and 3 standard deviations of the mean.

• Useful properties:

  • A linear combination of independent normals is normal; standardization via z = (x-μ)/σ maps any normal to standard normal.

  • It arises naturally as the limiting distribution of sums (Central Limit Theorem) and is the maximum-entropy distribution for a fixed mean and variance.

A z-score expresses a value as the number of standard deviations it lies from the mean; standardization rescales any normal to the standard normal, and the 68-95-99.7 rule states how much probability falls within 1, 2, and 3 standard deviations.

• Z-score:

  • z = (x - μ) / σ: a z of +2 means the value is 2 SDs above the mean.

  • Makes different scales comparable and lets you look up probabilities in a standard normal table.

• Standardization: Transforms X ~ N(μ, σ²) into Z ~ N(0, 1), mean 0 and SD 1.

• 68-95-99.7 rule (empirical rule):

  • ≈68% of values within ±1σ, ≈95% within ±2σ, ≈99.7% within ±3σ.

  • Quick sanity check: a value beyond 3σ is very rare (about 0.3%), useful for spotting outliers.

Use a uniform distribution when every outcome in a range is equally likely and you have no reason to favor any region: the continuous version on [a, b] has mean (a+b)/2 and variance (b-a)²/12.

• When to model with it:

  • Equal likelihood over an interval: rounding error, a random arrival time within a known window, a non-informative prior.

  • Discrete version for equally likely finite outcomes (a fair die).

• Mean and variance (continuous):

  • Mean is the midpoint (a+b)/2 by symmetry.

  • Variance (b-a)²/12 grows with the square of the width.

• Useful role: The Uniform(0,1) is the basis for generating other distributions via the inverse-CDF method.

Expected value is the probability-weighted average of a random variable's outcomes: the long-run mean you'd get by repeating the experiment many times. It's the center of gravity of the distribution.

• Definition: Discrete: E[X] = Σ x·P(x); continuous: E[X] = ∫ x·f(x) dx.

• Key properties:

  • Linearity: E[aX + bY] = aE[X] + bE[Y], true even when X and Y are dependent.

  • Law of the unconscious statistician: E[g(X)] = Σ g(x)·P(x).

• How it's used:

  • Decision-making: compare expected payoffs of bets, pricing, and strategies.

  • Building block for variance Var(X) = E[X²] - (E[X])² and for estimators.

• Caveat: The mean isn't always "typical": skewed distributions or heavy tails can make E[X] unrepresentative (or even infinite).

No: the expected value is a long-run average, not necessarily a value the random variable can ever take. It is what the mean of many independent samples converges to, per the law of large numbers.

• It can be unobservable: A fair die has E[X] = 3.5, which no face shows.

• It is a balancing point: The center of mass of the distribution, not the most likely outcome (that is the mode).

• Interpretation: Average many repetitions and the sample mean approaches E[X]: that is the sense in which it is 'expected'.

• Caveat: For skewed or heavy-tailed distributions the mean can be a poor description of a typical outcome; the median may be more representative.

A permutation counts arrangements where order matters; a combination counts selections where order does not. A combination is a permutation with the internal orderings divided out.

• Permutation: order matters:

  • P(n,k) = n! / (n−k)!.

  • Use for rankings, seat arrangements, PIN codes, race finishes.

• Combination: order doesn't matter:

  • C(n,k) = n! / [k!(n−k)!].

  • Use for choosing a committee, a hand of cards, a subset of balls.

• The link between them: C(n,k) = P(n,k) / k!: divide by k! because each unordered set corresponds to k! orderings.

• Quick test: Ask: would swapping two chosen items give a different outcome? Yes → permutation; No → combination.

The multiplication (counting) principle says that if a process happens in independent stages, the total number of outcomes is the product of the number of choices at each stage.

• Statement: If step 1 has n1 outcomes, step 2 has n2, ..., step k has nk, the whole has n1 × n2 × ... × nk outcomes.

• Condition: The count at each stage must not depend on which specific earlier choices were made (the number of options, not the options themselves, stays fixed).

• Examples:

  • A 3-digit code with digits 0-9: 10 × 10 × 10 = 1000.

  • Rolling two dice: 6 × 6 = 36 outcomes.

• Why it underpins other formulas: Permutations n×(n−1)×...×(n−k+1) come directly from applying it to shrinking option sets.

A PMF gives actual probabilities of specific values for a discrete variable, while a PDF gives densities for a continuous variable where probability only arises from integrating over an interval.

• PMF (discrete):

  • P(X = x) is a real probability between 0 and 1.

  • Values sum to 1 over all possible x.

• PDF (continuous):

  • f(x) is a density that can exceed 1; only areas are probabilities.

  • Integrates to 1 over the domain; P(X = x) = 0 at any point.

• Shared idea:

  • Both define the distribution and both derive the same CDF (sum vs integral).

  • Mnemonic: PMF = mass at points, PDF = density over ranges.

Probability distributions model the uncertainty in data and predictions; machine learning relies on them to describe data-generating processes, define loss functions, and quantify confidence.

• What they are: Mathematical rules assigning probabilities (or densities) to outcomes, e.g. Normal, Bernoulli, Categorical, Poisson.

• Why they matter in ML:

  1. Modeling assumptions: linear regression assumes Gaussian noise; classification uses Bernoulli/Categorical outputs.

  2. Loss functions: maximizing likelihood under an assumed distribution gives MSE (Gaussian) or cross-entropy (Categorical).

  3. Uncertainty: Bayesian methods represent parameters and predictions as distributions, not point values.

  4. Generative models: VAEs, diffusion, and GANs learn to sample from a data distribution.

• Practical payoff: Choosing the right distribution improves fit, calibration, and honest confidence estimates.

Most real distributions are non-normal: they can be uniform, skewed, heavy-tailed, or multimodal, meaning they lack the symmetric bell shape of a Gaussian. A uniform distribution is a clean example where every outcome in a range is equally likely.

• Ways a distribution departs from normal:

  • Skewness: asymmetric, like income (long right tail).

  • Heavy tails: extreme events more likely than Gaussian predicts (financial returns).

  • Multimodal: multiple peaks (heights of a mixed male/female population).

  • Bounded/flat: uniform, where density is constant over an interval.

• Uniform example:

  • Rolling a fair die or a random number in [0, 1]: no value is favored, so the shape is a flat rectangle, not a bell.

  • Its PDF is constant, f(x) = 1/(b - a) on [a, b], with mean at the midpoint.

• Why it matters: Assuming normality when data is skewed or heavy-tailed leads to underestimated risk and poor models; always check the actual shape.

The Law of Large Numbers states that as you take more independent samples, the sample average converges to the true expected value (population mean). It is why empirical frequencies stabilize toward true probabilities over many trials.

• Two forms:

  • Weak LLN: the sample mean converges in probability to the mean (large deviations become arbitrarily unlikely).

  • Strong LLN: the sample mean converges almost surely (with probability 1) to the mean.

• What it does and doesn't say:

  • It's about the average, not sums: it doesn't imply outcomes 'balance out' (that's the gambler's fallacy).

  • It gives convergence but not the rate: the Central Limit Theorem describes the fluctuations around the mean.

• Why it matters: Justifies Monte Carlo estimation and using sample statistics to approximate population quantities.

Inclusion-exclusion corrects for double counting when you add probabilities of overlapping events: add the singles, subtract the pairwise overlaps, add back the triples, and so on.

• Two events: P(A ∪ B) = P(A) + P(B) − P(A ∩ B): subtract the overlap counted twice.

• Three events: P(A ∪ B ∪ C) = P(A)+P(B)+P(C) − P(A∩B) − P(A∩C) − P(B∩C) + P(A∩B∩C).

• General pattern: Alternate signs by intersection size: add odd-sized intersections, subtract even-sized ones.

• Special case: If events are mutually exclusive, all intersections vanish and it reduces to simple addition.

Both describe behavior of sample averages as n grows, but they answer different questions: the Law of Large Numbers says WHERE the average goes, while the Central Limit Theorem describes the SHAPE of its fluctuations around that point.

• Law of Large Numbers (LLN):

  • The sample mean converges to the true mean μ as n → ∞.

  • It's a statement about convergence to a single value, not about distribution shape.

• Central Limit Theorem (CLT):

  • The distribution of the standardized sample mean approaches a normal distribution, regardless of the original distribution's shape.

  • Fluctuations shrink at rate σ/√n, which is what powers confidence intervals.

• How they complement each other: LLN tells you the average is reliable; CLT tells you how uncertain it still is and lets you quantify that uncertainty.

Joint, marginal, and conditional are three views of the same probability distribution: joint is the probability of events together, marginal is one variable ignoring the others, and conditional is one variable given another is fixed.

• Joint: P(A ∩ B): probability both occur; the full cell of a two-way table.

• Marginal:

  • P(A): probability of one event alone, obtained by summing the joint over the other variable (P(A) = Σ_b P(A, b)).

  • Named for the row/column margins of the table.

• Conditional: P(A|B) = P(A ∩ B) / P(B): rescales the joint to the world where B is known true.

• The link: Chain rule ties them together: P(A ∩ B) = P(A|B)P(B).

P(A|B) and P(B|A) divide the same joint probability by different denominators, so they are equal only when the two events have equal marginal probabilities. Confusing them is the classic base-rate fallacy.

• Different definitions: P(A|B) = P(A∩B)/P(B) while P(B|A) = P(A∩B)/P(A): same numerator, different base.

• Bayes' link: P(A|B) = P(B|A)P(A)/P(B): they match only if P(A) = P(B).

• Intuition: P(rain | clouds) is not P(clouds | rain): almost all rain has clouds, but most cloudy moments have no rain.

• Why it matters: Medical tests: P(positive | disease) can be high while P(disease | positive) is low when the disease is rare.

The law of total probability computes the overall probability of an event by splitting the sample space into disjoint pieces (a partition), finding the event's probability within each piece, and combining them weighted by each piece's probability.

• The partition: Events B₁,...,Bₙ that are mutually exclusive and collectively exhaustive (cover the whole space, no overlap).

• The formula: P(A) = Σᵢ P(A | Bᵢ) P(Bᵢ): a weighted average of conditional probabilities.

• How to use it: When P(A) is hard directly but easy given each scenario Bᵢ, condition on the scenarios and reassemble.

• Relation to Bayes: It supplies the denominator P(A) in Bayes' theorem.

• Example: Two factories make bulbs at known rates and defect rates; total defect probability = sum over factories of (defect rate × production share).

The chain rule decomposes the joint probability of several events into a product of conditional probabilities, letting you build a joint distribution from sequential conditionals.

• Two-event form: P(A,B) = P(A)·P(B|A) = P(B)·P(A|B), directly from the definition of conditional probability.

• General form:

  • P(X1,...,Xn) = P(X1)·P(X2|X1)·P(X3|X1,X2)···P(Xn|X1,...,Xn-1).

  • Each factor conditions on all variables that came before it in the chosen ordering.

• Ordering is free: Any permutation of the variables gives a valid factorization; the total is always equal.

• Why it matters:

  • Foundation of Bayesian networks and autoregressive models: independence assumptions drop terms from each conditional, shrinking the number of parameters.

  • With full independence it collapses to P(X1)·P(X2)···P(Xn) (the naive Bayes simplification).

Use a Binomial distribution to model the number of successes in a fixed number of independent trials, each with the same success probability p (e.g. heads in n coin flips).

• When it applies (the BINS conditions): Fixed number of trials n, Binary outcome per trial, Independent trials, and a Stable probability p.

• PMF: P(X=k) = C(n,k)·p^k·(1-p)^(n-k) for k = 0,...,n.

• Mean and variance: Mean = n·p; Variance = n·p·(1-p), maximized at p=0.5.

• Relationships: A sum of n independent Bernoulli(p) trials; approaches normal for large n and Poisson when n is large and p small.

In machine learning the Binomial distribution models counts of successes across repeated binary events, which makes it the natural likelihood for evaluating and comparing classifiers on independent test cases.

• Modeling correct predictions: The number of correct classifications out of n test examples is Binomial, so accuracy has a known sampling distribution.

• Confidence intervals and significance: Gives error bars on accuracy and powers A/B tests and significance tests on conversion or click rates.

• Likelihood foundation: Logistic regression's log-likelihood is the Binomial/Bernoulli likelihood; the Beta distribution is its conjugate prior in Bayesian estimation of a rate.

• Sampling and simulation: Used to model counts in bootstrap resampling and to reason about class imbalance in sampled batches.

The Binomial counts successes in a fixed number of trials, while the Poisson counts the number of events in a fixed interval of time or space when there is no natural upper bound. Poisson is the limit of the Binomial as trials grow large and success probability grows small.

• Support and setup:

  • Binomial: k in 0,...,n with a fixed n and probability p per trial.

  • Poisson: k in 0,1,2,... with no upper bound, driven by a rate λ (events per interval).

• Mean vs variance:

  • Binomial: mean n·p, variance n·p·(1-p), so variance < mean.

  • Poisson: mean = variance = λ (a diagnostic; real overdispersed data may violate this).

• Limiting connection: As n→∞ and p→0 with n·p = λ held constant, the Binomial converges to Poisson(λ).

• When to use each: Binomial for a known number of independent yes/no trials; Poisson for rare event counts (arrivals, defects, clicks per hour).

The Bernoulli distribution models a single binary trial with success probability p, and it is the atomic building block for binary classification: every yes/no label is a Bernoulli outcome.

• Definition: X in {0,1} with P(X=1)=p; mean = p, variance = p·(1-p).

• Likelihood for classification: Logistic regression models P(y=1|x) as a Bernoulli parameter; its loss is the negative Bernoulli log-likelihood, i.e. binary cross-entropy.

• Relationship to other distributions: A sum of n independent Bernoulli(p) is Binomial(n,p); the categorical distribution generalizes it to more than two outcomes.

• Other ML uses: Models binary features in Bernoulli naive Bayes and the on/off masks in dropout regularization.

Each fits a different question: binomial counts successes in a fixed number of trials, Poisson counts events in a continuous interval, and normal models continuous quantities that cluster symmetrically around a mean.

• Binomial:

  • Fixed number n of independent trials, each success with constant probability p; counts total successes.

  • Example: number of heads in 20 coin flips, conversions out of 100 visitors.

• Poisson:

  • Counts rare events over time/space at a constant average rate, no fixed upper bound.

  • Example: emails per hour, defects per meter of wire.

• Normal:

  • Continuous, symmetric bell shape; arises when many small independent effects add up (Central Limit Theorem).

  • Example: heights, measurement errors, sample means.

• How they connect: Binomial → Poisson when n is large and p small; binomial/Poisson → normal when counts are large.

If each trial succeeds independently with probability p, the number of trials until the first success is geometric, and its expected value is 1/p.

• Direct formula: E[X] = 1/p: e.g. a fair die showing a 6 takes 1/(1/6) = 6 rolls on average.

• Why (self-referencing argument): Either you succeed now (prob p, 1 trial) or you fail (prob 1-p) and start over: E = p·1 + (1-p)(1+E), solving gives E = 1/p.

• Assumptions: Trials must be independent with constant p; if not, this formula breaks down.

The geometric distribution models the number of independent Bernoulli trials up to and including the first success, with mean 1/p and variance (1-p)/p².

• When it arises:

  • Repeated independent trials with fixed success probability p, counting how many attempts until the first success.

  • Example: number of sales calls until the first sale, coin flips until first head.

• PMF: P(X=k) = (1-p)^{k-1} · p for k = 1, 2, 3, ...

• Mean and variance: Mean 1/p; variance (1-p)/p². Smaller p means longer waits and higher variability.

• Key property: Memoryless: past failures don't change the expected remaining trials (the discrete analog of the exponential).

Probability and likelihood use the same formula but treat different things as fixed. Probability treats parameters as fixed and asks about data; likelihood treats data as fixed and asks about parameters.

• Probability: P(data | parameters): Parameters θ are known/fixed; you ask how probable an outcome is. Over all outcomes it integrates/sums to 1.

• Likelihood: L(parameters | data): Data is observed/fixed; you vary θ to ask which parameters make the observed data most plausible. It is NOT a probability distribution over θ and need not integrate to 1.

• Same expression, different variable: Numerically L(θ | x) = P(x | θ), but as a function of θ (likelihood) versus a function of x (probability).

• Why it matters: Maximum likelihood estimation picks the θ that maximizes L(θ | data); Bayes combines likelihood with a prior to get a posterior.

Expectation is the probability-weighted average of a random variable, variance is the expected squared deviation from that average, and covariance measures how two variables move together.

• Expectation E[X]: The mean: Σ x·P(x) (discrete) or ∫ x·f(x) dx (continuous). The distribution's center of mass.

• Variance Var(X): Var(X) = E[(X - μ)^2] = E[X^2] - (E[X])^2: average squared spread. Its square root is the standard deviation, in the original units.

• Covariance Cov(X,Y):

  • Cov(X,Y) = E[(X-μ_X)(Y-μ_Y)] = E[XY] - E[X]E[Y]: positive if they tend to rise together, negative if one rises as the other falls, 0 if uncorrelated.

  • Note Cov(X,X) = Var(X).

Linearity of expectation says the expectation of a sum equals the sum of expectations, and constants factor out: E[aX + bY + c] = a·E[X] + b·E[Y] + c. Crucially, it holds even when the variables are dependent.

• No independence required: E[X + Y] = E[X] + E[Y] always, unlike variance which needs independence (or a covariance term).

• Scaling and shifting: Constants pull out and additive constants pass through: E[aX + b] = a·E[X] + b.

• Why it's powerful: Decompose a hard random variable into a sum of simple indicator variables, then add their expectations: it solves many counting problems easily.

• Example: Expected number of heads in n coin flips: sum of n indicators each with expectation p, so E = np.

Variance measures squared spread, so it is always non-negative, scales by the square of a constant, ignores additive shifts, and for sums it adds a covariance term that vanishes only when variables are uncorrelated.

• Non-negativity: Var(X) ≥ 0, and equals 0 only if X is constant.

• Scaling and shifting: Var(aX + b) = a^2·Var(X): adding a constant b doesn't change spread; multiplying scales by a^2.

• Sum of two variables:

  • Var(X + Y) = Var(X) + Var(Y) + 2·Cov(X,Y).

  • If independent (or uncorrelated), the covariance term drops: Var(X + Y) = Var(X) + Var(Y).

• Difference of variables: Var(X - Y) = Var(X) + Var(Y) - 2·Cov(X,Y): for independent variables variances still add, they do not subtract.

• Units: Variance is in squared units; take the square root for standard deviation in original units.

Both measure how two variables move together, but covariance is unbounded and unit-dependent, while correlation is covariance normalized into a dimensionless number between -1 and 1. Correlation is essentially standardized covariance.

• Covariance: Cov(X,Y) = E[(X-μ_X)(Y-μ_Y)]: sign tells direction, but magnitude depends on the units and scales of X and Y, so it is hard to compare across pairs.

• Correlation: ρ = Cov(X,Y) / (σ_X·σ_Y): bounded in [-1, 1], unitless, so its magnitude directly reflects strength of the linear relationship.

• Interpreting values: ±1 is perfect linear relationship, 0 is no linear relationship.

• Shared caveats: Both capture only linear association: a strong nonlinear relationship can give correlation near 0. And zero correlation does not imply independence (except for jointly normal variables).

Compute the expected value: sum each outcome's payoff times its probability. A positive EV means the bet is favorable on average, but EV alone is not the whole story: variance, bankroll, and whether the game is repeated all matter.

• Compute EV:

  • EV = Σ (probability × payoff), using net gain/loss including the cost to play.

  • EV > 0 favorable, EV = 0 fair, EV < 0 unfavorable.

• Why EV isn't enough:

  • Variance/risk: a positive-EV bet with tiny win probability can still ruin you in the short run.

  • One-shot vs repeated: EV is a long-run average, most reliable when the bet is repeated many times (law of large numbers).

  • Utility, not dollars: people are risk-averse; the utility of money is nonlinear (see the St. Petersburg paradox).

• Practical framing: For repeated favorable bets, sizing matters (e.g. the Kelly criterion) to avoid ruin.

Bayes' Theorem tells you how to update the probability of a hypothesis after observing evidence, by combining prior belief with how well the hypothesis predicts that evidence. It is the backbone of probabilistic classifiers like the Naive Bayes spam filter.

• The rule: P(H | E) = P(E | H) P(H) / P(E): posterior ∝ likelihood × prior.

• Spam detection (Naive Bayes):

  • P(spam | words) ∝ P(spam) × Π P(word | spam), assuming words are conditionally independent given the class.

  • Compare P(spam | words) vs P(ham | words) and pick the larger.

• Other ML uses:

  • Bayesian inference: update model parameters as data arrives.

  • Interpreting classifier outputs and medical/diagnostic tests (base-rate matters).

• Key insight: A rare condition (low prior) means even an accurate test can yield mostly false positives: the base rate dominates.

You start with a prior (belief before data), multiply by the likelihood (how probable the observed data is under each hypothesis), and normalize to get the posterior (updated belief). Today's posterior becomes tomorrow's prior, which makes the process naturally sequential.

• The three pieces:

  1. Prior P(H): what you believe before seeing the new evidence.

  2. Likelihood P(E | H): how well each hypothesis explains the observed data.

  3. Posterior P(H | E): the updated belief after combining the two.

• The update: posterior ∝ likelihood × prior, then divide by the evidence P(E) so probabilities sum to 1.

• Sequential updating: Feed the posterior in as the prior for the next observation; order doesn't change the final result.

• Behavior: Strong prior + weak data: belief barely moves. Weak prior + strong data: data dominates.

Bayes' Theorem is P(H | E) = P(E | H) P(H) / P(E). Each term has a name and role: it converts the prior belief into a posterior belief by weighting it with the likelihood and normalizing by the evidence.

• P(H | E) — posterior: Updated probability of the hypothesis after seeing the evidence; the quantity you want.

• P(E | H) — likelihood: How probable the observed evidence is if the hypothesis is true.

• P(H) — prior: Belief in the hypothesis before observing the evidence.

• P(E) — evidence (marginal likelihood):

  • Total probability of the evidence across all hypotheses: P(E) = Σ P(E | H_i) P(H_i).

  • Acts as the normalizing constant so the posterior sums to 1.

• Compact form: posterior ∝ likelihood × prior: the evidence just rescales, so it can be ignored when only comparing hypotheses.

The posterior is the component proportional to likelihood × prior. Bayes' Theorem writes P(H | E) = P(E | H) P(H) / P(E), and since the evidence P(E) is a constant that doesn't depend on the hypothesis, dropping it leaves posterior ∝ likelihood × prior.

• Why proportional, not equal: The evidence P(E) is the same for every hypothesis, so it only normalizes; it doesn't change the ranking.

• Why it's useful:

  • When choosing the most probable hypothesis (MAP estimate), you can skip computing P(E) entirely.

  • P(E) is often the hardest part (an integral/sum over all hypotheses), so ignoring it simplifies inference and underlies methods like MCMC.

The base-rate fallacy is the tendency to judge how likely a hypothesis is from the evidence alone while ignoring its prior probability (the base rate): this inflates rare-event conclusions because a small false-positive rate applied to a huge negative population can swamp the true positives.

• What it is: People focus on P(evidence | hypothesis) and neglect P(hypothesis), the prior.

• Why ignoring the prior misleads:

  • For a rare condition, the pool of true cases is tiny, so even a small false-positive rate on the large healthy pool produces many false alarms.

  • The posterior depends on both terms; dropping the prior treats a rare and a common hypothesis identically.

• Classic example: a 99% accurate test for a disease with 0.1% prevalence:

  • Out of 100,000 people: ~100 sick (99 test positive) vs ~99,900 healthy (~999 false positives).

  • P(sick | positive) ≈ 99 / (99 + 999) ≈ 9%, not 99%.

• Takeaway: Always combine likelihood with the base rate; the posterior is proportional to prior × likelihood.

Bayes' theorem falls straight out of the definition of conditional probability by writing the joint probability P(A∩B) two different ways and equating them.

1. Definition of conditional probability: P(A|B) = P(A∩B) / P(B) and P(B|A) = P(A∩B) / P(A).

2. Express the joint two ways: P(A∩B) = P(A|B)·P(B) = P(B|A)·P(A).

3. Equate and solve: P(A|B) = P(B|A)·P(A) / P(B).

4. Expand the denominator (law of total probability): P(B) = P(B|A)·P(A) + P(B|¬A)·P(¬A), giving a fully computable form.

Treat every such problem the same way: nail down the sample space, decide whether order and replacement matter, then compute favorable outcomes over total outcomes (or use conditional/complement tricks when direct counting is messy).

• Define the experiment precisely: List what one outcome looks like and whether all outcomes are equally likely.

• Ask three framing questions:

  1. Does order matter? (permutation vs combination)

  2. With or without replacement?

  3. Are items distinguishable? (labeled dice vs identical balls)

• Pick the counting tool: Multiplication principle for staged choices, combinations for unordered selection, permutations for ordered.

• Use shortcuts when brute force is hard:

  • Complement: P(at least one) = 1 − P(none).

  • Symmetry, indicator/linearity of expectation, and conditioning on the first event.

• Sanity-check: Probabilities lie in [0,1]; try a tiny case (2 cards, 1 die) to verify the formula.

With replacement, each draw is returned so the pool stays the same and draws are independent; without replacement, the pool shrinks each time so draws are dependent and counts differ.

• With replacement:

  • Pool size is constant; ordered draws of k from n give n^k outcomes.

  • Draws are independent, so probabilities multiply unchanged (e.g. P(two aces) = (4/52)²).

• Without replacement:

  • Pool shrinks; ordered draws give n×(n−1)×...×(n−k+1) = P(n,k), unordered give C(n,k).

  • Draws are dependent: condition on prior draws (e.g. P(two aces) = (4/52)·(3/51)).

• How to tell which applies: If the item can repeat, it's with replacement; if each item is used at most once, it's without.

• Effect on variance: Without replacement reduces variability (finite-population correction); with replacement matches the binomial model.

The birthday problem asks how many people you need in a room for a 50% chance that two share a birthday: the surprising answer is just 23. Intuition fails because we instinctively count comparisons against ourselves, but the probability grows with the number of pairs, which explodes combinatorially.

• Compute the complement:

  • It is easier to find P(no shared birthday) and subtract from 1.

  • P(no match) = 365/365 × 364/365 × ... × (365-n+1)/365; for n=23 this is about 0.493, so P(match) ≈ 0.507.

• Pairs, not people, drive it:

  • With n people there are n(n-1)/2 pairs; 23 people give 253 pairs, each a chance to match.

  • The intuition trap: you imagine matching your own birthday (needs ~253 people for 50%), not any-to-any matching.

• Why it feels wrong: We think linearly in n, but the number of chances grows quadratically, so probability rises far faster than expected.

You want P(disease | negative), the false negative risk after a negative result. Apply Bayes' theorem using prevalence as the prior and the test's miss rate (1 - sensitivity) as the likelihood of a negative among the sick.

• Define the terms:

  • Prevalence P(D) = prior probability of disease.

  • Sensitivity = P(+ | D), so P(- | D) = 1 - sensitivity.

  • Specificity = P(- | not D).

• The formula: P(D | -) = [P(- | D)·P(D)] / [P(- | D)·P(D) + P(- | not D)·P(not D)].

• Worked example:

  • Prevalence 1%, sensitivity 99%, specificity 95%.

  • Numerator: 0.01 × 0.01 = 0.0001. Denominator: 0.0001 + (0.95 × 0.99) = 0.0001 + 0.9405 = 0.9406.

  • P(D | -) ≈ 0.0001/0.9406 ≈ 0.011%, so a negative is very reassuring here.

Conditional probability P(A | B) is the probability of A given that B has occurred, computed as P(A and B) / P(B). In real scenarios like disease testing, you rarely have the joint directly, so you combine known conditional rates with base rates using the law of total probability and Bayes' theorem.

• Start from the definition: P(A | B) = P(A ∩ B) / P(B): restrict the world to B, then measure A within it.

• Break the denominator with total probability: P(positive) = P(+ | D)P(D) + P(+ | not D)P(not D), summing true and false positives.

• Invert with Bayes when you have the wrong direction: Tests give you P(+ | D); you want P(D | +), so flip via Bayes using the prior P(D).

• Practical tip: use natural frequencies: Imagining a concrete population (e.g. out of 10,000 people) makes true/false positives countable and avoids base-rate mistakes.

Given an observed outcome (say a sequence of flips), you use Bayes' theorem to update from a prior belief about the bias to a posterior. You compare the likelihood of the data under 'biased' versus 'fair' hypotheses, weighted by your prior, rather than reading probability straight off the data.

• Set up hypotheses and prior:

  • E.g. H_fair (p=0.5) vs H_biased (p=some value, or a distribution over p).

  • Assign priors P(H_fair), P(H_biased) reflecting initial belief.

• Compute the likelihood of the data: For k heads in n flips, likelihood = p^k (1-p)^(n-k) under each hypothesis.

• Apply Bayes: P(biased | data) = [L(data | biased)·P(biased)] / [L(data|biased)P(biased) + L(data|fair)P(fair)].

• Continuous version:

  • Put a Beta prior on p; after k heads and (n-k) tails the posterior is Beta(α+k, β+n-k), a clean conjugate update.

  • You can then read P(p > 0.5) or a credible interval to judge bias.

This is a classic Bayes' theorem application: you know each machine's share of production and its defect rate, and you want P(machine | defective). You reverse the conditioning by weighting each machine's defect contribution against the total defect rate.

• Identify the pieces:

  • P(M_i) = fraction of items made by machine i (the prior).

  • P(defect | M_i) = machine i's defect rate (the likelihood).

• Total probability of a defect: P(defect) = Σ P(defect | M_i)·P(M_i) over all machines.

• Bayes' theorem: P(M_i | defect) = [P(defect | M_i)·P(M_i)] / P(defect).

• Example:

  • Machine A makes 60% at 1% defect, B makes 40% at 3% defect.

  • P(defect)=0.6×0.01 + 0.4×0.03 = 0.006 + 0.012 = 0.018.

  • P(A | defect) = 0.006/0.018 = 33%, even though A makes most items.

Applying Bayes' theorem, the probability an email flagged as spam is truly spam is about 85%. The 90% of legitimate emails generate false positives that dilute the flagged pool, so the answer is well below the 99% detection rate.

• Known values: P(spam) = 0.10, P(flag | spam) = 0.99, P(flag | not spam) = 0.02.

• Total probability of a flag: P(flag) = 0.99×0.10 + 0.02×0.90 = 0.099 + 0.018 = 0.117.

• Apply Bayes: P(spam | flag) = 0.099 / 0.117 ≈ 0.846, so about 85%.

• Interpretation: Roughly 1 in 7 flagged emails is legitimate; lowering the false positive rate matters more than raising sensitivity here.

These three quantities describe how a diagnostic test performs, but they answer different questions: sensitivity and specificity are properties of the test itself, while positive predictive value depends on how common the disease is.

• Sensitivity: P(test+ | disease): Among truly diseased people, the fraction the test flags. High sensitivity means few false negatives.

• Specificity: P(test- | no disease): Among healthy people, the fraction correctly cleared. High specificity means few false positives.

• Positive predictive value (PPV): P(disease | test+):

  • The quantity a patient actually cares about: given a positive result, how likely you truly have the disease.

  • Computed via Bayes' theorem, so it depends on prevalence (the prior).

• Key intuition: the base-rate effect: For a rare disease, even a very sensitive/specific test can have low PPV because false positives swamp true positives.

In the Monty Hall problem you pick one of three doors (one hides a car), the host who knows the contents opens a different door revealing a goat, then offers you a switch. You should switch: switching wins with probability 2/3, staying wins only 1/3.

• The setup carries information: The host's choice is not random: he always opens a goat door and never your door, so his action is conditioned on where the car is.

• Reasoning by the initial pick:

  • Your first door is right with probability 1/3 and wrong with probability 2/3.

  • If your first pick was wrong (2/3 of the time), the host is forced to reveal the other goat, so the remaining door hides the car. Switching wins exactly in these cases.

• Bayesian view: The 1/3 prior on your door does not change when the host opens a door; the eliminated 1/3 collapses onto the other unopened door, giving it 2/3.

• Common trap: Assuming two remaining doors are 50/50: that only holds if the host opened a door at random, which he does not.

Throw random points into the unit square and count how many land inside the quarter circle: the fraction inside approximates the ratio of areas, which is pi/4, so multiplying by 4 estimates pi.

• The geometry:

  • A quarter circle of radius 1 has area pi/4; the enclosing unit square has area 1.

  • A uniformly random point lands inside the circle with probability pi/4.

• The estimator:

  • Sample x, y uniformly in [0,1], count a hit when x^2 + y^2 <= 1.

  • pi ~= 4 * (hits / N).

• Accuracy: Error shrinks like 1/sqrt(N), so it converges slowly (many samples for a few decimal places).