83 Hypothesis Testing Interview Questions and Answers (2026)

Hypothesis testing is a formal procedure for deciding whether sample data provide enough evidence to support a claim about a population, while controlling the chance of being fooled by random variation.

• The problem it solves:

  • A sample rarely matches the population exactly, so an observed effect could be real or just sampling noise.

  • Testing gives a rule to separate signal from noise with a known error rate.

• How it works:

  • State a null and an alternative, compute a test statistic, and assess how surprising the data are if the null were true.

  • If the data are sufficiently unlikely under the null, reject it.

• Why it matters: It converts a subjective judgment ('this looks different') into a reproducible decision with quantified uncertainty.

Descriptive statistics summarize the data you actually have; inferential statistics use that sample to draw conclusions about a larger population you didn't fully observe.

• Descriptive:

  • Measures of center and spread (mean, median, standard deviation), plus charts and tables.

  • No uncertainty or generalization: it just describes what's in front of you.

• Inferential:

  • Uses probability to generalize from sample to population: hypothesis tests, confidence intervals, regression.

  • Always carries quantified uncertainty because it extrapolates beyond observed data.

• Relationship: Descriptive summaries (like the sample mean) are the inputs that inferential methods build on.

A population is the entire set of units you want to know about; a sample is the subset you actually measure and use to make inferences about that population.

• Population:

  • The complete group of interest (e.g., all customers, all voters).

  • Described by parameters (fixed but usually unknown), e.g., μ and σ.

• Sample:

  • A subset drawn from the population, ideally randomly to avoid bias.

  • Described by statistics (computed from data), e.g., x̄ and s, which estimate the parameters.

• Why the distinction matters: Hypothesis testing makes claims about population parameters using sample statistics, and a non-representative sample invalidates that inference.

The standard error is the standard deviation of a sample statistic's sampling distribution: it measures how much the statistic would vary from sample to sample. It's the denominator that scales a test statistic, turning a raw difference into 'how many standard errors from the null.'

• Definition:

  • For the sample mean, SE = σ/√n (or s/√n when σ is estimated).

  • It shrinks as n grows, reflecting more precise estimates from larger samples.

• Role in the test statistic:

  • A test statistic is (estimate − null value) / SE.

  • Dividing by SE standardizes the effect, so it can be compared to a known reference distribution (z or t).

• Intuition: A big observed difference isn't impressive if the statistic is noisy; SE tells you the noise scale.

The null hypothesis (H0) states there is no effect or no difference, and the alternative (H1 or Ha) states there is one. H0 is the 'status quo' because testing is built to challenge a default assumption: we assume nothing is happening until data give us reason to think otherwise.

• Null hypothesis (H0):

  • A specific, testable statement of 'no effect' (e.g., μ = μ0, two means equal).

  • Must be precise so we can compute the distribution of the test statistic under it.

• Alternative hypothesis (H1/Ha): The claim we're gathering evidence for (e.g., μ ≠ μ0, or a directional >, <).

• Why H0 is the default:

  • It gives an exact reference model to compute probabilities against; 'an effect exists' is usually too vague to model directly.

  • It embodies scientific conservatism: the burden of proof lies on showing an effect, protecting against false discoveries.

The null hypothesis is framed as 'no effect' because hypothesis testing is built to disprove a default skeptical position: we assume nothing is happening and require the data to provide strong evidence against that before we claim an effect exists.

• It sets a testable baseline:

  • 'No difference' gives a specific value (e.g. difference = 0) that defines an exact sampling distribution to compute probabilities from.

  • An 'effect exists' statement is vague (which effect? how large?) and can't anchor a single distribution.

• It reflects the burden of proof:

  • Like a courtroom presumption of innocence: we don't credit an effect until evidence forces us to.

  • This guards against false positives (Type I errors), which we control at level alpha.

• It aligns with falsifiability: We can never 'prove' the null, only fail to reject it; framing the skeptical claim as the null makes rejection the informative outcome.

A good hypothesis is specific, testable, and falsifiable, stated in terms of a measurable population parameter before seeing the data. The null and alternative should be mutually exclusive and jointly cover the possibilities of interest.

• Qualities of a good hypothesis:

  • Testable and falsifiable: it can be contradicted by data.

  • Specific: refers to a defined parameter (mean, proportion, difference) not a vague notion.

  • Stated a priori: defined before data collection to avoid bias.

  • About the population, not the sample: e.g. the population mean, not the observed average.

• Framing the null (H0): The status-quo / no-effect claim, always containing equality: =, ≤, or ≥.

• Framing the alternative (H1):

  • The research claim you seek evidence for; uses ≠, >, or <.

  • Two-sided (≠) if direction is unknown; one-sided (>, <) if theory predicts a direction.

Hypothesis testing follows a fixed recipe: state the hypotheses, fix the risk you'll tolerate, compute a test statistic from the data, and compare it against that risk threshold to decide.

1. State H0 and H1: Define the parameter and whether the test is one- or two-sided.

2. Choose the significance level alpha: Commonly 0.05; this is the accepted probability of a Type I error.

3. Select the appropriate test and check assumptions: e.g. z-test, t-test, chi-square; verify normality, independence, variance conditions.

4. Compute the test statistic: Standardize the observed data under the assumption H0 is true.

5. Find the p-value or critical value: Locate the rejection region for alpha.

6. Make a decision and interpret: Reject or fail to reject H0, then state the conclusion in the problem's context.

Both approaches lead to the same decision: reject H0 when the evidence is more extreme than your threshold. With p-values you compare to alpha; with critical values you compare the test statistic to a cutoff.

• p-value rule:

  • If p ≤ alpha: reject H0 (result is statistically significant).

  • If p > alpha: fail to reject H0.

• Critical value rule:

  • If the test statistic falls in the rejection region (beyond the critical value): reject H0.

  • Otherwise: fail to reject H0.

• Interpretation cautions:

  • The p-value is P(data this extreme or more | H0 true), not the probability H0 is true.

  • 'Fail to reject' is not 'accept': absence of evidence isn't evidence of absence.

The critical value method defines a rejection region on the test-statistic scale before computing anything, then checks whether the observed statistic lands inside it.

1. State H0, H1, and the significance level alpha.

2. Identify the test statistic's distribution under H0: e.g. standard normal for a z-test, t with df for a t-test.

3. Find the critical value(s) for alpha:

   • Two-sided splits alpha into both tails (alpha/2 each); one-sided puts all of alpha in one tail.

   • Read from a table or quantile function, defining the rejection region.

4. Compute the observed test statistic from the sample.

5. Compare and decide: If it falls in the rejection region, reject H0; otherwise fail to reject.

You reject the null when the test statistic is more extreme than the critical value (i.e. it falls in the rejection region); if it does not, you fail to reject. The exact comparison depends on whether the test is one- or two-sided.

• Two-sided test: Reject H0 if |statistic| > critical value.

• Right-tailed test: Reject H0 if statistic > critical value.

• Left-tailed test: Reject H0 if statistic < -critical value.

• Worked example:

  • z = 2.30, two-sided critical value at alpha = 0.05 is 1.96. Since 2.30 > 1.96, reject H0.

  • If instead z = 1.50, then 1.50 < 1.96, so fail to reject H0.

There's no single universal test: the right one depends on the data type and comparison, but the most common choices for "is this observed difference significant?" are a t-test for two means, ANOVA for three or more means, a z-test for proportions, and chi-square for categorical counts.

• Difference in means:

  • Two groups: two-sample t-test (Welch's if variances differ).

  • 3+ groups: ANOVA, then post-hoc tests.

• Difference in proportions: Two-proportion z-test (common in A/B tests of conversion rates).

• Difference in categorical distributions: Chi-square test of independence on the contingency table.

• Non-normal / ordinal data: Nonparametric alternatives: Mann-Whitney U (unpaired), Wilcoxon signed-rank (paired).

• Decision rule: Set a significance level (e.g. α = 0.05), compute the p-value, and reject the null (call the difference significant) when p < α.

A one-sample t-test checks whether the mean of a single sample differs significantly from a known or hypothesized population value, when the population standard deviation is unknown and estimated from the sample.

• Hypotheses: H0: population mean equals the hypothesized value (μ = μ₀); H1: it differs (two-sided) or is greater/less (one-sided).

• Test statistic:

  • t = (x̄ − μ₀) / (s / √n), compared to a t-distribution with n − 1 degrees of freedom.

  • Uses sample standard deviation s, which is why it's a t rather than a z.

• Assumptions: Independent observations, and the sample mean is approximately normal (true if data is normal, or n is large by the CLT).

• Example: Testing whether the average battery life of a production batch differs from the advertised 10 hours.

All three compare observed data against what a null hypothesis predicts, but they differ by what they test and which distribution the statistic follows: z and t test means, F tests variances (and ratios of variances, as in ANOVA).

• Z-test:

  • Tests means/proportions when population variance is known or n is large; statistic follows the standard normal.

  • Common for large-sample proportion tests.

• T-test:

  • Tests means when population variance is unknown and estimated from the sample; follows the t-distribution.

  • Variants: one-sample, two-sample (independent), and paired.

• F-test:

  • Tests ratios of variances; follows the F-distribution.

  • Used to compare two variances and, in ANOVA, to test whether several group means are equal by comparing between-group to within-group variance.

Choose based on how many categorical variables you have and what you're comparing: goodness-of-fit checks one variable against an expected distribution, while the test of independence checks whether two variables are associated.

• Goodness-of-fit:

  • One categorical variable; compares observed category counts to a theoretical/expected distribution.

  • Example: do dice roll frequencies match a uniform distribution?

  • Degrees of freedom = k − 1.

• Test of independence:

  • Two categorical variables in a contingency table; tests whether they are associated.

  • Example: is smoking status related to disease occurrence?

  • Degrees of freedom = (r − 1)(c − 1).

• Shared mechanics: Both use χ² = Σ (O − E)² / E and assume expected counts are large enough (commonly ≥ 5).

A Type I error is rejecting a true null hypothesis (a false positive), and a Type II error is failing to reject a false null (a false negative). Their probabilities are α and β respectively, and there's a trade-off between them.

• Type I error (α):

  • Concluding an effect exists when it doesn't; controlled by the significance level.

  • Example: a medical test says a healthy patient has a disease, triggering needless treatment.

• Type II error (β):

  • Missing a real effect; power is 1 − β.

  • Example: the test clears a patient who actually has the disease, so treatment is missed.

• Trade-off:

  • Lowering α (stricter threshold) tends to raise β; increasing sample size reduces both.

  • Which is worse depends on context: a false positive vs. a missed diagnosis have different real-world costs.

A Type I error is rejecting a true null hypothesis (a false positive). Its probability is the significance level, α, which you set before the test.

• Probability = α: Choosing α = 0.05 means a 5% chance of a false positive when the null is actually true.

• Controlled by setting α upfront: Lowering α (e.g. to 0.01) makes false positives rarer but reduces power.

• Multiple comparisons inflate it: Running many tests raises the family-wise error rate; control it with corrections like Bonferroni or FDR (Benjamini-Hochberg).

A Type II error is failing to reject a false null hypothesis (a false negative): you miss a real effect. Its probability is β, and power equals 1 - β.

• β is not directly set: Unlike α, β is not chosen directly; it depends on effect size, sample size, α, and variability.

• Common target: Studies often aim for β = 0.20, i.e. 80% power, as a conventional adequacy threshold.

• Relationship to power: Anything that raises power (bigger n, larger effect, less noise) lowers β.

A p-value is the probability of observing a test statistic at least as extreme as the one obtained, assuming the null hypothesis is true.

• Precise definition:

  • It is a conditional probability: P(data at least as extreme as observed | H0 true).

  • "As extreme" is defined relative to the alternative hypothesis and the direction of the test.

• What it measures: How compatible the observed data are with the null hypothesis: small values indicate the data are surprising under H0.

• What it is NOT: Not the probability that H0 is true, and not the probability the result happened by chance.

The significance level alpha is a threshold, chosen before the test, that represents the probability of rejecting a true null hypothesis (a Type I error) you are willing to tolerate.

• What it represents:

  • The maximum acceptable rate of false positives: P(reject H0 | H0 true).

  • Common conventional values are 0.05, 0.01, and 0.10, but the choice depends on the cost of a false positive.

• Its role in decision-making:

  • It defines the rejection region and serves as the cutoff the p-value is compared against.

  • Must be fixed in advance to keep the Type I error rate valid.

• Trade-off: Lowering alpha reduces Type I errors but increases Type II errors (missing a real effect), all else equal.

A result is statistically significant when the p-value is less than or equal to your pre-chosen significance level alpha, meaning the observed data would be sufficiently unlikely if the null hypothesis were true.

• The mechanical check:

  • Compare p to alpha; if p ≤ alpha, declare significance.

  • Equivalently, check if the test statistic falls in the rejection region, or if the confidence interval excludes the null value.

• Statistical vs. practical significance:

  • Significance says an effect is unlikely due to chance, not that it is large or important.

  • Always report the effect size and confidence interval alongside the p-value.

• Caveat: Large samples can make tiny, meaningless effects significant; small samples can hide real effects.

A p-value measures how compatible your data are with the null hypothesis: a low p-value means the data are surprising under H0 (evidence against it), while a high p-value means the data are consistent with H0 (no evidence against it).

• Low p-value (e.g., p < 0.05):

  • Such extreme data would rarely occur if H0 were true, so you reject H0 in favor of the alternative.

  • It signals evidence against H0, not proof that the alternative is true.

• High p-value:

  • The data are consistent with H0, so you fail to reject it.

  • This is not proof that H0 is true: absence of evidence is not evidence of absence.

• Correct framing: Always interpret the p-value conditionally: it assumes H0 is true and quantifies the data, not the hypothesis.

A p-value < 0.05 generally indicates a statistically significant result at the 5% level: the observed data would occur less than 5% of the time if the null hypothesis were true, so you reject H0.

• What it conventionally means:

  • The evidence against H0 is strong enough to cross the common alpha = 0.05 threshold.

  • It implies at most a 5% Type I error rate under repeated testing.

• What it does NOT mean: Not a 95% chance the alternative is true, and not proof of a large or meaningful effect.

• Caveats:

  • 0.05 is an arbitrary convention; the right threshold depends on context and cost of errors.

  • Multiple comparisons inflate false positives, so adjust (e.g., Bonferroni) when running many tests.

No. A p-value of 0.03 provides evidence against the null hypothesis at the 5% level, but it never proves any hypothesis: hypothesis testing offers probabilistic evidence, not proof.

• Why it's not proof:

  • p = 0.03 only says such data are unlikely (3%) if H0 were true; it says nothing directly about whether your alternative is correct.

  • There remains a real chance of a Type I error (rejecting a true null).

• What you can actually claim: You can say the result is statistically significant at alpha = 0.05 and you reject H0.

• What to check before believing it:

  • Effect size and confidence interval (is it practically meaningful?).

  • Whether assumptions hold, the sample is adequate, and the finding replicates.

A test statistic is a single number computed from your sample that summarizes how far the data sit from what the null hypothesis predicts, scaled by the variability you'd expect by chance. It converts raw data into a value whose distribution under the null is known, so you can judge how surprising the result is.

• General form: Often (estimate − null value) / (standard error): a signal-to-noise ratio.

• Why it matters:

  • It has a known reference distribution under the null (z, t, chi-square, F), which lets you compute p-values or compare to critical values.

  • Larger magnitude means the data are more inconsistent with the null.

• Examples: z and t statistics for means; chi-square for counts/variance; F for ratios of variances.

Many test statistics have the form (observed effect minus null value) divided by a standard error, so the value literally counts how many standard errors your data sit from what the null predicts. Bigger magnitude means more surprising under the null.

• The general shape: statistic = (estimate − null value) / (standard error): numerator is the raw distance, denominator is the noise scale.

• Why standardize:

  • Dividing by the SE removes units and sample-size effects, so a value of 2 means roughly the same 'surprise' across problems.

  • It maps the result onto a known reference distribution (normal, t, chi-square) where extreme values are rare under the null.

• Reading the value:

  • Near 0: data are consistent with the null. Large |value|: far in the tail, so unlikely if the null were true.

  • It grows with a bigger effect OR a smaller SE (more data, less variance), which is why huge samples make tiny effects significant.

The Central Limit Theorem (CLT) states that the sampling distribution of the sample mean approaches a normal distribution as sample size grows, regardless of the population's shape. This is what lets us use normal-based tests even on non-normal data.

• What it says:

  • For i.i.d. data with finite variance, the distribution of the sample mean tends to Normal with mean equal to the population mean and standard deviation equal to the standard error.

  • Convergence is faster for symmetric populations; skewed data need larger n (a common rule of thumb is n ≥ 30).

• Why it enables testing:

  • Most test statistics (z, t) rely on the estimator being approximately normal, which the CLT guarantees at reasonable sample sizes.

  • It lets us compute p-values and confidence intervals without knowing the true population distribution.

• Caveat: It applies to the sampling distribution of a statistic, not to the raw data themselves.

We aim to reject the null because the logic mirrors proof by contradiction: we provisionally assume H0 is true, then show the observed data would be very unlikely under that assumption, which is evidence against it. We can disprove a specific claim but can't positively prove one.

• The contradiction logic:

  • Assume H0 holds, derive what the data should look like, and check how far reality deviates.

  • If deviation is extreme (small p-value), the assumption is implausible, so we reject it.

• Why not 'accept' the null:

  • Absence of evidence isn't evidence of absence: failing to find an effect could just mean low power or a small sample.

  • Many parameter values close to the null could also be consistent with the data, so we can't single out H0 as true.

• Asymmetry of proof: One strongly contradicting observation can refute H0; no amount of consistent data can confirm it exactly.

'Failing to reject the null' means the data didn't provide strong enough evidence against H0, not that H0 is true. It's a statement about insufficient evidence, not proof of no effect.

• What it actually means:

  • The observed data are reasonably compatible with H0, so we don't have grounds to overturn it.

  • The effect could still be real but too small to detect with this sample.

• Why it's not 'accepting' H0:

  • Low statistical power or small n can produce non-significance even when a true effect exists (Type II error).

  • A whole range of alternative values would also be consistent with the same data, so H0 isn't uniquely supported.

• Correct framing: Say 'insufficient evidence to reject H0,' and note that a confidence interval or power analysis better conveys what effect sizes remain plausible.

A simple hypothesis fully specifies the distribution (a single exact parameter value), while a composite hypothesis covers a range of values and so does not pin down one distribution.

• Simple hypothesis:

  • Specifies the parameter exactly, e.g. mu = 100.

  • Determines a single, fully known sampling distribution, making the test easier to analyze.

• Composite hypothesis:

  • Specifies a set of values, e.g. mu > 100 or mu ≠ 100.

  • Corresponds to many possible distributions, so power varies with the true value.

• Why it matters:

  • Most real alternatives are composite; the null is often simple (or a boundary of a composite).

  • The Neyman-Pearson lemma gives the most powerful test only for simple-vs-simple cases.

They are two sides of the same decision rule and always agree; they differ in what scale you compare on and how much information the result conveys.

• Critical value method:

  • Compares on the test-statistic scale: is the statistic beyond the cutoff?

  • Threshold is fixed by alpha before seeing data; classic for hand calculation with tables.

• p-value method:

  • Compares on the probability scale: is p ≤ alpha?

  • Quantifies the strength of evidence, not just pass/fail; standard in software output.

• Key relationship:

  • They give identical conclusions: the statistic exceeds the critical value exactly when p ≤ alpha.

  • The p-value tells you how close the call was; the critical value alone does not.

Both compare means, but the choice hinges on whether you know the population variance and, relatedly, on sample size: use a Z-test when the population standard deviation is known (or n is large), and a T-test when you must estimate it from the sample.

• Z-test:

  • Use when the population variance/SD is known, or the sample is large (rule of thumb n > 30) so the estimate is stable.

  • Test statistic uses the normal distribution; assumes the sampling distribution of the mean is approximately normal (via CLT or a normal population).

• T-test:

  • Use when the population variance is unknown (the usual real-world case) and estimated from the sample.

  • Uses the t-distribution with n-1 degrees of freedom, which has heavier tails to account for the extra uncertainty from estimating the SD.

  • Especially important for small samples; assumes the underlying data is approximately normal.

• Practical note: As n grows, the t-distribution converges to the normal, so Z and T give nearly identical results for large samples. When in doubt with unknown variance, use the T-test.

The Chi-square test compares observed counts to counts expected under a null hypothesis, using categorical (frequency) data. It has two main flavors: goodness-of-fit (does one categorical variable match an expected distribution?) and test of independence (are two categorical variables associated?).

• Goodness-of-fit: One categorical variable: tests whether observed frequencies match a hypothesized distribution (e.g. is a die fair?).

• Test of independence:

  • Two categorical variables in a contingency table: tests whether they are related (e.g. is treatment associated with outcome?).

  • Expected count per cell = (row total × column total) / grand total.

• Data it works with: Counts/frequencies of categories, not means or proportions directly, and never percentages.

• Key assumptions:

  • Independent observations (each unit counted once).

  • Expected count in each cell is sufficiently large (commonly ≥ 5); otherwise use Fisher's exact test.

  • Statistic Σ(O-E)²/E follows a chi-square distribution with the appropriate degrees of freedom.

An F-test compares two variances by taking their ratio, which follows an F-distribution under the null that the variances are equal. It underlies variance comparisons, ANOVA, and overall regression significance.

• Core idea:

  • Statistic F = s₁²/s₂² (ratio of sample variances); if variances are equal it's near 1.

  • The F-distribution is defined by two degrees-of-freedom parameters (numerator and denominator).

• Common uses:

  • Comparing two population variances (e.g. checking the equal-variance assumption before a t-test).

  • ANOVA: compares between-group variance to within-group variance to test if several group means differ.

  • Regression: tests whether a set of predictors jointly explains significant variance.

• Assumptions: Independent samples and (especially for the variance ratio test) normally distributed data; the F-test is sensitive to non-normality.

All three test means, but differ in how many groups and whether observations are paired: one-sample compares a mean to a known value, two-sample independent compares means of two separate groups, and paired compares two measurements on the same units.

• One-sample t-test: Compares a single sample mean to a hypothesized/known value (e.g. is average delivery time = 30 min?).

• Two-sample independent t-test:

  • Compares means of two unrelated groups (e.g. control vs. treatment).

  • Assumes equal variances (Student's) or unequal (Welch's).

• Paired t-test:

  • Compares two related measurements on the same subjects (e.g. before vs. after); tests whether the mean difference is zero.

  • By analyzing differences it removes between-subject variability, giving more power.

• Shared assumptions: Approximately normal data (or the relevant differences), independent observations (independence between groups for the two-sample; within-pair dependence is expected for paired), and continuous outcome measured on an interval/ratio scale.

Use a t-test to compare means when the population standard deviation is unknown and estimated from the sample, typically with small-to-moderate samples. It uses the t-distribution to account for that added uncertainty.

• When to use: Comparing a sample mean to a value, two group means, or paired measurements, with a continuous outcome and unknown population variance.

• Key assumptions:

  • Normality: the data (or sampling distribution of the mean) is approximately normal; matters most for small n.

  • Independence: observations are independent (except intentional within-pair pairing).

  • Scale: outcome is continuous (interval/ratio).

  • Homogeneity of variance for the standard two-sample version; use Welch's if variances differ.

• Robustness: Fairly robust to mild non-normality for larger samples (CLT); for heavy skew or ordinal data prefer a nonparametric test like Mann-Whitney.

Match the test to the question by asking a short series of questions: what is the goal (compare, associate, predict?), what type of data do you have, how many groups, are they paired, and do the parametric assumptions hold. Those answers narrow you to a specific test.

• Identify the goal: Compare means, compare proportions/counts, test association, or model a relationship.

• Classify the variables: Continuous vs. categorical outcome; continuous means favor t-tests/ANOVA, categorical counts favor chi-square.

• Count and structure the groups: One sample, two groups, or 3+ (ANOVA); independent vs. paired/repeated measures.

• Check assumptions: Normality, equal variance, independence, sample size. If violated, use a nonparametric alternative (Mann-Whitney, Wilcoxon, Kruskal-Wallis).

• Example mapping: Two group means, normal data: two-sample t-test. Two categorical variables: chi-square. 3+ group means: ANOVA. Association of two continuous variables: correlation/regression.

A p-value quantifies the evidence against a null hypothesis for a single test, while a confidence interval gives a range of plausible values for the parameter itself. They are related but answer different questions.

• P-value:

  • Probability of observing data at least as extreme as yours, assuming H0 is true.

  • A single number tied to a specific null; small p means the data is unlikely under H0.

• Confidence interval:

  • A range that, under repeated sampling, contains the true parameter (1 − α)% of the time.

  • Conveys both direction and magnitude/precision of the effect, not just significance.

• How they connect: For a two-sided test at level α, the result is significant exactly when the hypothesized value falls outside the (1 − α) CI.

• Why the CI is often preferred: It shows effect size and uncertainty; a p-value alone can flag significance without revealing whether the effect is practically meaningful.

Use a one-proportion test comparing the sample proportion to the hypothesized value. For a reasonably large sample, the statistic is a z-score based on the normal approximation to the binomial; for small samples use an exact binomial test.

• Hypotheses: H0: p = p₀; H1: p ≠ p₀ (or one-sided).

• Test statistic (z):

  • z = (p̂ − p₀) / √(p₀(1 − p₀)/n), where p̂ is the observed proportion.

  • Note the standard error uses p₀ (the null value), not p̂.

• When to use which:

  • Normal approximation is valid when np₀ and n(1 − p₀) are both ≥ about 10.

  • Otherwise use an exact binomial test.

• Example: Is a coin's observed 62% heads in 100 flips different from the fair value 0.5?

Effect size measures the magnitude of a difference or relationship, independent of sample size. It matters because statistical significance only tells you an effect is unlikely to be zero, not whether it's large enough to care about.

• What it captures:

  • Cohen's d expresses a mean difference in standard-deviation units: d = (x̄₁ − x̄₂) / s_pooled.

  • Rough benchmarks: 0.2 small, 0.5 medium, 0.8 large (context-dependent).

  • Other examples: correlation r, odds ratios, η².

• Why it complements significance:

  • With huge samples, tiny, practically meaningless effects become statistically significant.

  • With small samples, a real, large effect may fail to reach significance.

• Practical roles:

  • Drives power/sample-size calculations and enables meta-analysis comparisons across studies.

  • Answers "how big?" while the p-value answers "is it likely real?"

Statistical power (1 − β) is the probability that a test correctly detects a true effect, i.e., rejects the null when it's actually false. It's central to designing experiments that can actually find the effects they seek.

• Definition: Probability of avoiding a Type II error; conventionally researchers target 0.80 or higher.

• Drivers of power:

  • Sample size: larger n increases power.

  • Effect size: bigger true effects are easier to detect.

  • Significance level α: a more lenient α raises power (at the cost of more false positives).

  • Lower variance/noise in the data increases power.

• Why it matters:

  • Design: a prospective power analysis sets the sample size needed to detect a meaningful effect.

  • Interpretation: an underpowered non-significant result is inconclusive, not proof of "no effect."

Power is the probability of correctly rejecting a false null hypothesis (1 - β). It is governed by four interrelated factors: effect size, sample size, significance level, and variability.

• Effect size: The magnitude of the true difference or relationship. Larger effects are easier to detect, so bigger effect size increases power.

• Sample size (n): More observations shrink the standard error and sharpen estimates, so larger n increases power. This is the factor you most directly control.

• Significance level (α): A more lenient threshold (e.g. 0.10 vs 0.01) makes rejection easier, so raising α increases power, but at the cost of more Type I errors.

• Variability (σ): More noise in the data widens sampling distributions and lowers power, so reducing variability (better measurement, control, matched designs) increases power.

Statistical significance says an effect is unlikely due to chance; practical significance says the effect is large enough to matter in the real world. They answer different questions and can disagree.

• Statistical significance: Driven by the p-value: is the result distinguishable from noise? Heavily influenced by sample size.

• Practical significance: Driven by effect size and context: is the difference big enough to change a decision or justify a cost?

• Significant but not practical: With a huge sample, a trivial difference (e.g. 0.1% conversion lift) can be statistically significant yet not worth implementing.

• Practical but not significant: A meaningfully large effect in a small, underpowered study may fail to reach significance due to wide confidence intervals.

• Takeaway: Always report effect sizes and confidence intervals alongside p-values, not the p-value alone.

For a fixed sample size, α and β move in opposite directions: making it harder to commit a false positive makes false negatives more likely, and vice versa. You manage the trade-off primarily by increasing sample size, which relaxes the tension.

• The tension:

  • Lowering α tightens the rejection region, so you reject less often, raising β and lowering power.

  • Raising α to catch more true effects increases the false positive rate.

• Managing it:

  • Increase sample size: reduces β without changing α, the cleanest lever.

  • Reduce variability through better design and measurement.

  • Set α based on which error is costlier in context, not by reflex.

The right balance between the two errors depends on their real-world consequences, so you should weigh which mistake is more expensive before fixing α and power.

• Type I error (false positive): Acting on an effect that isn't real: approving an ineffective drug, launching a useless feature, convicting an innocent person.

• Type II error (false negative): Missing a real effect: rejecting a beneficial treatment, failing to detect fraud or a real safety hazard.

• Context sets the priority: Medical screening often tolerates more false positives to avoid missing disease; a costly product launch may demand a very low false positive rate.

Increasing sample size leaves the Type I error rate (α) unchanged but lowers the Type II error rate (β) and raises power, because larger samples produce more precise estimates.

• Type I error (α): Unaffected: α is fixed by you regardless of n. Larger samples don't change the false positive rate you set.

• Type II error (β): Decreases: a smaller standard error narrows the sampling distributions, so a false null is rejected more often.

• Power (1 - β): Increases: this is why n is the main lever for hitting a target power.

• Caveat: Very large samples make even trivial effects significant, so distinguish statistical from practical significance.

You compare the p-value to a pre-chosen significance level alpha: if p ≤ alpha you reject the null hypothesis, otherwise you fail to reject it. The difference between one- and two-tailed tests lies in how the extreme region (and thus the p-value) is computed.

• The decision rule:

  • If p ≤ alpha: reject H0 (result is statistically significant).

  • If p > alpha: fail to reject H0 (insufficient evidence).

• Two-tailed test:

  • Alternative is "not equal"; extreme results in either direction count, so alpha is split across both tails (alpha/2 each).

  • The p-value counts probability in both tails.

• One-tailed test:

  • Alternative specifies a direction (greater than or less than); all of alpha is in one tail.

  • Easier to reach significance in that direction, but you gain no power to detect an effect in the opposite direction.

• Key point: choose alpha and tail direction before seeing the data to avoid biasing the decision.

The p-value is one of the most misinterpreted statistics: it is a probability about the data assuming H0, not a probability about the hypotheses themselves. Nearly every common error conflates the two.

• "It's the probability H0 is true": Wrong: p is P(data | H0), not P(H0 | data). Getting the latter requires Bayesian priors.

• "It's the probability the result is due to chance": Wrong: it already assumes chance-only (H0) and asks how extreme the data are under that assumption.

• "1 minus p is the probability the alternative is true": Wrong: the p-value says nothing directly about the probability of the alternative.

• "A non-significant p proves H0": Wrong: failing to reject is not accepting; it may just reflect low power.

• "p measures effect size or importance": Wrong: a small p can come from a trivial effect with a large sample. Report effect sizes separately.

α = 0.05 is a historical convention (popularized by Fisher) that sets a 5% tolerance for false positives; it is a balance point, not a law of nature, and different fields adjust it based on how costly Type I versus Type II errors are.

• Why 0.05 caught on:

  • Fisher suggested it as a reasonable cutoff (~1 in 20), and it became entrenched by tradition and convenience rather than deep theory.

  • It roughly corresponds to about 2 standard deviations in a normal distribution.

• α controls the Type I error rate: It is the probability of rejecting a true null (false positive) you are willing to accept.

• Lowering α (e.g. 0.01):

  • Fewer false positives, so stronger evidence required: good when a false discovery is costly (clinical trials, particle physics uses far stricter thresholds).

  • Trade-off: lower power, higher Type II error (β), so real effects are missed more often; needs larger samples to compensate.

• Raising α (e.g. 0.10):

  • More power to detect effects: useful in exploratory or pilot studies where missing a real effect is worse than a false alarm.

  • Trade-off: more false positives.

• Key principle: Choose α before seeing the data, based on the relative cost of the two error types, not to make a result significant.

"At least as extreme" means all outcomes that deviate from what the null predicts by as much as, or more than, the observed data, in the direction(s) the alternative cares about. The p-value is the total probability of that whole set of outcomes, assuming the null is true.

• "Extreme" is measured relative to the null: Farther from the null's expected value = more extreme = stronger evidence against the null.

• Direction depends on the alternative:

  • One-tailed test: only deviations in the specified direction count (e.g. only large positive statistics).

  • Two-tailed test: deviations in both directions count, so you include both tails.

• Why "at least as extreme," not just the observed value: For continuous data any single exact outcome has probability ~0, so we ask how likely the observed result or anything more surprising is.

• Example: Observed z = 2.1 in a two-tailed test: the p-value is P(|Z| ≥ 2.1), i.e. the area beyond +2.1 plus the area beyond -2.1.

A critical value is the threshold on the test statistic's null distribution that marks off the most extreme α proportion of outcomes; the rejection region is the set of statistic values beyond that threshold. If your observed statistic falls in the rejection region, you reject the null.

• Critical value: Found from the null distribution and α (e.g. z = 1.96 for a two-tailed test at α = 0.05).

• Rejection region: The tail(s) of the distribution containing total probability α: one tail for one-sided tests, split across both tails for two-sided.

• The decision rule:

  • Statistic in rejection region (|stat| > critical value): reject H0; otherwise fail to reject.

  • Equivalent to the p-value approach: statistic beyond the critical value is exactly the same as p ≤ α.

• Interpretation: The rejection region is defined so that, if the null is true, you land in it only α of the time, which is the Type I error rate.

Degrees of freedom (df) are the number of values in a calculation that are free to vary once any constraints (like estimated parameters) are imposed. They matter because they determine the exact shape of reference distributions like t, chi-square, and F, and thus the critical values and p-values.

• Core idea:

  • Each parameter you estimate from the data uses up one piece of information, reducing the freedom left.

  • Example: with n data points and a sample mean estimated, only n − 1 residuals are free (they must sum to zero), so the sample variance uses n − 1 df.

• In the t-test: One-sample: df = n − 1. Lower df gives fatter tails (more uncertainty in the estimated standard deviation); as df grows the t approaches the normal.

• In the chi-square test: Goodness-of-fit: df = categories − 1 − (parameters estimated). Contingency table: df = (rows − 1)(cols − 1).

• Why it's not just cosmetic: Using the wrong df picks the wrong reference distribution, giving incorrect p-values and biased conclusions.

The F-statistic is a ratio of two variances (mean squares), each following a chi-square distribution under the null. It's used to test whether groups differ or whether a set of model terms explains meaningful variation, by comparing explained variance to unexplained (error) variance.

• General form: F = (variance explained by the effect) / (variance from residual/error), i.e. a ratio of mean squares.

• Behavior under the null:

  • If there's no real effect, both numerator and denominator estimate the same variance, so F is near 1.

  • A large F means the effect explains more variation than chance, giving evidence against the null.

• Its distribution: Follows an F-distribution with two df parameters (numerator and denominator), always non-negative and right-skewed.

• Common uses: ANOVA (comparing means across groups), overall significance of a regression model, and comparing nested models.

Classical parametric tests assume a distributional form (often normality), independent observations, and for many tests equal variances; violations mainly distort the Type I error rate and the validity of the p-value.

• t-test / z-test:

  • Independent observations; approximate normality of the sampling distribution of the mean; for two-sample t, equal variances (unless using Welch's).

  • z-test additionally assumes a known population variance (or very large n).

• Chi-square (goodness-of-fit / independence): Independent counts, mutually exclusive categories, and adequate expected counts (commonly all expected ≥ 5).

• What violations do:

  • Non-normality: minor issues shrink with large n by the CLT, but heavy skew or small samples inflate or deflate error rates.

  • Unequal variances: biases the standard t-test; use Welch's t-test instead.

  • Dependence: the most damaging: understates the true SE, giving falsely small p-values and inflated Type I error.

Parametric tests assume the data follow a specified distribution and test its parameters; non-parametric tests make weaker distributional assumptions (often using ranks), trading some power for robustness. Choose non-parametric when those assumptions clearly fail.

• Parametric:

  • Assume a form (e.g. normal) and estimate parameters like the mean; more powerful when assumptions hold.

  • Examples: t-test, ANOVA, z-test.

• Non-parametric:

  • Distribution-free, often rank-based, testing medians or distributions rather than means.

  • Examples: Mann-Whitney U (vs two-sample t), Wilcoxon signed-rank (vs paired t), Kruskal-Wallis (vs ANOVA).

• When to switch to non-parametric:

  • Small samples with clear non-normality, heavy skew or strong outliers, or ordinal data where means aren't meaningful.

  • Cost: usually less power when parametric assumptions actually hold.

The chi-square approximation relies on expected (not observed) counts being large enough: a common rule is all expected counts ≥ 5, with no more than ~20% below 5 and none below 1. When cells are too sparse, the chi-square distribution is a poor approximation and the p-value is unreliable.

• The conditions:

  • Expected count = (row total × column total) / grand total for each cell.

  • Cochran's rule: all expected ≥ 5 (or at least 80% of cells ≥ 5 and all ≥ 1).

• Fixes for low expected counts:

  • Combine (collapse) sparse categories into meaningful larger groups.

  • Collect more data to raise expected counts.

  • Use an exact test: Fisher's exact test (especially 2×2) or a Monte Carlo / simulated p-value for larger tables.

• Note: The condition is about expected counts, not observed: a cell can have 0 observed and still be fine if its expected is large.

Independence underlies the formula for the standard error: if observations are correlated, each new data point carries less new information than the math assumes, so the SE is underestimated. That inflates the test statistic and produces artificially small p-values and too many false positives.

• Why it matters:

  • Standard errors assume n independent pieces of information; positive correlation means the effective sample size is smaller than n.

  • Result: SE too small, statistic too large, Type I error rate far above the nominal alpha.

• Common ways it's violated:

  • Repeated measures on the same subject, clustered data (students in classes), and time series with autocorrelation.

  • Pseudoreplication: treating correlated measurements as if independent.

• Remedies:

  • Model the structure: paired tests, mixed-effects/hierarchical models, cluster-robust standard errors, or time-series methods.

  • Fix it in design: proper randomization and independent sampling units.

Non-normality is often not fatal, and you have a ladder of options: lean on the CLT with larger samples, transform the data, switch to a non-parametric or resampling test, or use a robust model. The right choice depends on sample size and how the assumption fails.

• Rely on the Central Limit Theorem: With large n, the sampling distribution of the mean is approximately normal, so t-tests are fairly robust; mild non-normality is usually fine.

• Transform the data: Log, square-root, or Box-Cox can pull skewed data toward normality (note it changes the scale of interpretation).

• Use a non-parametric test: Mann-Whitney U, Wilcoxon signed-rank, or Kruskal-Wallis make no normality assumption.

• Resampling methods: Bootstrap confidence intervals or permutation tests build the null distribution empirically.

• Caveat: Check whether normality even matters here: outliers or the wrong test may be the real issue.

Exploratory analysis (EDA) searches the data openly to generate hypotheses; confirmatory testing evaluates one pre-specified hypothesis with a valid p-value. The distinction matters because a p-value only means what it claims when the test was decided before looking at the data.

• Exploratory data analysis:

  • Flexible, iterative: plot, slice, and probe to find patterns and form questions.

  • Many implicit comparisons, so any p-values are descriptive, not inferential.

• Confirmatory hypothesis testing:

  • Hypothesis, outcome, and analysis fixed in advance; ideally on fresh data.

  • Controls Type I error and yields an interpretable p-value.

• Why the distinction is critical:

  • Using the same data to both discover and "confirm" a hypothesis is circular and inflates false positives.

  • Best practice: explore on one split, confirm on another (or in a new study).

A p-value only tells you whether an effect is distinguishable from the null, not how big or how practically meaningful it is. Effect sizes and confidence intervals convey magnitude and precision, which is what actually informs decisions.

• What the p-value lacks:

  • It conflates effect magnitude with sample size: a tiny effect can be highly significant with large n.

  • Significance ≠ importance.

• What effect size adds:

  • Quantifies the magnitude on an interpretable scale (mean difference, Cohen's d, odds ratio).

  • Lets you judge practical/clinical relevance and compare across studies (meta-analysis).

• What the confidence interval adds:

  • Shows precision: a wide interval signals uncertainty even if 'significant'.

  • Displays the full range of plausible effect values, including whether they are all practically trivial or large.

  • Contains the test result: if the null value is outside the interval, the test is significant.

A hypothesis test gives a decision about whether data are compatible with the null; a confidence interval gives a range of parameter values compatible with the data. Both must be interpreted carefully as statements about long-run procedures, not about the probability of a specific hypothesis.

• Interpreting a hypothesis test:

  • A small p-value means the observed data (or more extreme) would be unlikely if the null were true; you reject the null.

  • Failing to reject is not proof the null is true: absence of evidence isn't evidence of absence.

  • The p-value is NOT the probability the null is true.

• Interpreting a confidence interval:

  • A 95% CI means: if we repeated the procedure many times, ~95% of such intervals would contain the true parameter.

  • It's not a 95% probability the parameter lies in this specific interval (frequentist view).

  • Width reflects precision; position relative to the null reflects significance.

• Always pair statistical significance with practical significance and context.