83 Linear Regression Interview Questions and Answers (2026)

Linear regression is the natural baseline: it is fast, interpretable, needs little data, and tells you quickly whether a simple relationship exists before you reach for heavier machinery.

• Interpretability: Each coefficient has a direct meaning: expected change in the target per unit change in a predictor.

• Cheap and stable: Closed-form or fast convergence, few hyperparameters, low variance, works with small datasets.

• A benchmark: It sets a performance floor: a complex model must beat linear regression to justify its cost and opacity.

• Diagnostic value: Residual plots reveal nonlinearity, heteroscedasticity, or bad features, guiding what to try next.

• Occam's razor: If a simple model is good enough, prefer it: easier to deploy, debug, and explain.

Both fit a straight-line (linear-in-parameters) relationship; the difference is simply the number of predictors: Simple Linear Regression uses one, Multiple Linear Regression uses two or more.

• Simple Linear Regression:

  • One predictor: y = β0 + β1·x + ε.

  • Fits a line in 2D; easy to visualize with a scatter plot.

• Multiple Linear Regression:

  • Several predictors: y = β0 + β1·x1 + ... + βk·xk + ε.

  • Fits a hyperplane; each coefficient is the effect of its predictor holding the others fixed.

• New concerns with multiple predictors: Multicollinearity, the need for adjusted R², and harder visualization all appear only in the multiple case.

The error term ε captures everything that affects the target but is not explained by the predictors: it makes the model honest about the fact that real relationships are not exact.

• What it represents:

  • Omitted variables not included as predictors.

  • Inherent randomness and measurement error in y.

  • Any misspecification (the true form isn't perfectly linear).

• Model form: y = β0 + β1·x + ε, where β0 + β1·x is the systematic part and ε is the random part.

• Assumptions on it matter: Classical assumptions (zero mean, constant variance, independence, often normality) are what make coefficient estimates unbiased and inference valid.

• Distinguish ε from the residual: ε is the true unobservable error; the residual is its estimate from the fitted model.

The difference is the type of target: regression predicts a continuous numeric value, while classification predicts a discrete category or label.

• Regression:

  • Output is a real number (price, temperature, demand).

  • Evaluated with error metrics like MSE, RMSE, MAE, R².

• Classification:

  • Output is a class (spam/not spam, digit 0-9).

  • Evaluated with accuracy, precision, recall, F1, AUC.

• They can blur together: A model may output a continuous score (probability) that you threshold into a class, as logistic regression does.

• Why not use linear regression for classes: It can predict values outside [0,1] and treats categories as ordered numbers, which is why logistic regression is preferred.

Linear regression is simple and interpretable, but that simplicity is also its weakness: it assumes a linear, additive relationship and is sensitive to violations of its assumptions.

• Assumes linearity: Cannot capture curved or interaction effects unless you engineer such features manually.

• Sensitive to outliers: Least squares squares the errors, so a few extreme points can dominate the fit.

• Multicollinearity: Correlated predictors inflate coefficient variance and make them unstable and hard to interpret.

• Assumption dependence: Relies on constant variance, independent errors, and (for inference) normal errors; heteroscedasticity or autocorrelation break standard errors.

• Limited expressiveness: Underfits genuinely complex, nonlinear data where tree ensembles or neural nets excel.

• Needs preprocessing: Categorical encoding, scaling (for regularized variants), and handling of missing values are on you.

The linearity assumption states that the expected value of the response is a linear function of the predictors' coefficients: the relationship between each predictor and the mean of y is a straight line (holding others fixed). It concerns linearity in the parameters, not necessarily the raw variables.

• What it actually requires:

  • A one-unit change in a predictor shifts the mean of the response by a constant amount, regardless of the predictor's level.

  • Linear in coefficients: y = b0 + b1*x + b2*x^2 is still "linear" because polynomial terms enter linearly.

• How to recognise violation:

  • Residuals vs fitted plot shows curvature (a U or inverted-U pattern) instead of a flat, random band.

  • Partial/component-plus-residual plots reveal a nonlinear shape for a specific predictor.

  • Systematic runs of positive then negative residuals across the range of x.

• Fixes: Transform predictors or response (log, square root), add polynomial or spline terms, or switch to a nonlinear/GAM model.

A coefficient is the estimated change in the mean of the response for a one-unit increase in that predictor, while every other predictor in the model stays fixed. "Holding all else constant" is the partial effect: it isolates one predictor's contribution after removing the linear influence of the others.

• Interpretation template:

  • "A one-unit increase in x is associated with a b-unit change in y, holding the other predictors constant."

  • Units matter: the coefficient's meaning is tied to x's units and y's units.

• What "holding constant" means in practice:

  • Statistically it means partialling out: the coefficient reflects the effect on the part of x not explained by the other predictors.

  • It is a mathematical adjustment, not a real experiment: you are not physically fixing the other variables.

• Caveats:

  • With high multicollinearity, "holding others constant" becomes unstable because predictors move together in the data.

  • "Associated with," not "causes," unless the design supports causal claims.

The intercept is the predicted mean of the response when all predictors equal zero. It anchors the regression line vertically. It only has practical meaning when zero is a plausible, meaningful value for every predictor; otherwise it is just a mathematical offset.

• Interpretation:

  • "When all x's are zero, the expected y is b0."

  • It ensures the residuals average to zero and the line passes through the mean point.

• When it has no practical meaning:

  • Predicting weight from height: height = 0 is impossible, so the intercept (weight at zero height) is not interpretable.

  • Any model where zero lies far outside the observed data range.

• Making it meaningful: Center predictors: after centering, the intercept becomes the predicted y at the mean of the predictors, which is interpretable.

A dummy-coded categorical variable is 0/1, and its coefficient is the difference in the mean response between that category and the omitted reference category, holding other predictors constant. Interpretation is always relative to whichever level you left out.

• The mechanics:

  • For a k-level category you create k-1 dummies; the excluded level is the reference (baseline).

  • The intercept represents the reference group's mean (when other predictors are zero).

• Interpretation:

  • "Being in this category changes the expected y by b relative to the reference category."

  • A positive coefficient means that group's mean is higher than the reference; negative means lower.

• Practical notes:

  • Comparisons between two non-reference groups require differencing their coefficients.

  • Choosing a meaningful reference level (a common or control group) makes coefficients easier to explain.

The OLS objective function is the sum (or mean) of squared residuals: we pick the coefficients that make the model's predictions collectively as close as possible to the observed responses in a squared-distance sense.

• What it measures: Total squared vertical distance between actual y and predicted Xβ.

• Properties:

  • Convex in β, so any stationary point is the global minimum.

  • Differentiable, giving a closed-form minimizer via the Normal Equation.

• Interpretation: Minimizing it is equivalent to maximum likelihood under the assumption of Gaussian, constant-variance errors.

R-squared is the proportion of variance in the outcome explained by the model, computed as 1 - RSS/TSS. It ranges from 0 to 1 for OLS with an intercept.

• What it represents:

  • Fraction of total variation captured by the fit vs. the mean-only baseline.

  • R-squared = 0 means no better than predicting the mean; 1 means perfect fit.

• Limitations:

  • Never decreases when you add predictors, so it rewards complexity (use adjusted R-squared).

  • Says nothing about bias, correct functional form, or assumption violations.

  • Is unitless, so it doesn't tell you prediction error in the outcome's units.

  • Can be misleadingly high with trending data and high in absolute yet useless for prediction.

RMSE (root mean squared error) is the square root of the average squared residual: it summarizes typical prediction error in the same units as the outcome, giving heavier weight to large errors.

• What it tells you:

  • The magnitude of typical prediction error; lower is better.

  • Because errors are squared then rooted, it is sensitive to outliers.

• Units: Same units as the target (predict price in dollars, RMSE is in dollars), which makes it interpretable.

• Comparisons: Compare against the outcome's scale/standard deviation, or against MAE to detect outlier influence.

A coefficient's p-value is the probability of seeing an estimate at least this far from zero if the true coefficient were actually zero, given the model's assumptions. It measures evidence against the null of no effect, holding the other predictors fixed.

• What the null hypothesis is: That this predictor's true slope is zero: it adds nothing once the others are in the model.

• How it is computed: From the t-statistic = coefficient / standard error, compared to a t-distribution.

• What a small p-value means: Strong evidence the effect is not zero; it does NOT tell you the effect is large or that the model is correct.

• Common misreadings to avoid:

  • It is not the probability the null is true, and "not significant" does not prove the coefficient is zero, only that you lack evidence.

  • It is conditional on the other variables in the model, so it can shift when you add or drop predictors.

The linear refers to the parameters (coefficients), not the predictors: the model must be linear in the β terms, but the predictors themselves can be transformed however you like.

• Linear in parameters:

  • Each parameter enters as a simple multiplier that is added up: no parameter appears inside a power, product, or nonlinear function.

  • Legal: y = β0 + β1·x + β2·x². Not linear: y = β0 + x^β1.

• Predictors can be nonlinear: You may feed in log(x), x², or interactions, and the model is still linear regression.

• Why it matters: Linearity in parameters is what gives a closed-form least-squares solution and makes the geometry (projection onto a column space) work.

A deterministic relationship is exact (the same input always gives the same output), while a statistical relationship is probabilistic: the target varies around a trend, so regression models the average behavior plus random scatter.

• Deterministic: A physical formula like C = 5/9·(F − 32): no error term, points fall exactly on the line.

• Statistical:

  • For a given x, y follows a distribution; we model its mean: E[y|x] = β0 + β1·x.

  • Individual points scatter because of unmeasured factors and randomness.

• Why regression is statistical: Real data has noise and omitted causes, so we predict a trend, not an exact value, which is exactly why the error term exists.

Polynomial regression is linear because it is still linear in its parameters: raising a predictor to a power creates a new feature, but the coefficients on those features are combined additively, which is all linear regression requires.

• The key distinction: Linearity is judged in the β terms, not in x. y = β0 + β1·x + β2·x² is linear in β0, β1, β2.

• It's just feature engineering: Treat x², x³ as extra columns; the same least-squares math applies unchanged.

• Consequences: You get the closed-form solution and standard inference, while modeling a curved trend.

• Contrast: A model like y = β0·e^(β1·x) is nonlinear because a parameter sits inside a nonlinear function.

Prefer linear regression when interpretability, small data, or a genuinely near-linear signal matters more than squeezing out maximum predictive accuracy.

• You need interpretability and inference: Coefficients give a signed, quantified effect per unit of each feature, plus standard errors, p-values, and confidence intervals: tree ensembles don't offer this cleanly.

• The relationship is approximately linear: If the true signal is linear (or made linear via transforms/interactions), a flexible model just adds variance without reducing bias.

• Small or noisy datasets: Low-variance, high-bias models generalize better when data is scarce; ensembles overfit.

• Operational simplicity: Fast to train, trivial to deploy, easy to explain to stakeholders and regulators (credit scoring, clinical settings).

• Need for extrapolation: Trees can't predict beyond the range of the training data; a linear model extends smoothly (assuming linearity holds).

• Rule of thumb: reach for boosting/forests when there are complex nonlinearities and interactions, plenty of data, and prediction accuracy is the sole goal.

OLS relies on linearity, exogeneity (zero-mean errors given X), independence of errors, homoskedasticity, and (for exact inference) normal errors. The single most critical one for unbiased coefficients is exogeneity.

• Linearity: The outcome is a linear function of the parameters; misspecification biases estimates.

• Exogeneity: E[ε | X] = 0: Errors are uncorrelated with predictors. This is what makes coefficients unbiased; violate it (omitted variables, reverse causality) and estimates are biased.

• Independence of errors: Observations' errors aren't correlated with each other; matters for correct standard errors.

• Homoskedasticity: Constant error variance; needed for efficient estimates and valid standard errors, not for unbiasedness.

• Normality of errors: Only needed for exact small-sample t/F inference; not required for unbiasedness.

• No perfect multicollinearity: Technically required just to estimate unique coefficients at all.

Homoskedasticity means the variance of the errors is constant across all levels of the predictors. Heteroskedasticity (non-constant variance) doesn't bias your coefficients, but it breaks your standard errors, making inference unreliable.

• Definition: Var(ε | X) is the same for every observation; residuals fan out or funnel when it's violated.

• Coefficients stay unbiased: OLS point estimates remain unbiased and consistent under heteroskedasticity.

• What actually breaks:

  • Standard errors are wrong, so t-stats, p-values, and confidence intervals are invalid: you may declare significance that isn't real.

  • OLS is no longer BLUE (best/most efficient linear unbiased estimator).

• Fixes: Use robust (White/Huber-White) standard errors, weighted least squares, or transform the outcome (e.g. log).

No: linear regression assumes neither the predictors nor the raw outcome are normally distributed. The only distributional assumption is on the errors (residuals), and even that is only needed for certain inference.

• Predictors need no distribution: X can be skewed, binary, or categorical (dummy-coded); OLS treats them as fixed/known.

• The outcome need not be marginally normal: A skewed Y is fine as long as the errors around the fitted line behave well.

• The assumption is about errors, conditionally: It's the conditional distribution of Y given X (i.e. the residuals) that is assumed normal, and only for exact t/F tests in small samples.

• Common confusion: People test normality of raw Y; you should instead inspect residuals via a Q-Q plot.

Independence of errors assumes each observation's residual carries no information about any other's. Serial correlation (common in time series) violates this: it leaves coefficients unbiased but makes standard errors wrong, usually understated, so you overstate significance.

• What it means: Cov(ε_i, ε_j) = 0 for all i ≠ j; errors are uncorrelated across observations.

• When it breaks: Time series (today's error relates to yesterday's), or clustered/grouped data (students in a school).

• Consequences:

  • Point estimates stay unbiased but inefficient (OLS no longer BLUE).

  • Standard errors are biased (typically too small), inflating t-stats and giving false confidence.

• Detect and fix:

  • Diagnose with a Durbin-Watson test or residual-vs-time plot.

  • Use Newey-West (HAC) standard errors, clustered standard errors, or model the dynamics (e.g. add lags, GLS/ARIMA).

An interaction term (the product of two predictors) lets the effect of one predictor depend on the value of another. When present, the coefficients on the individual predictors are no longer overall main effects: they become conditional effects evaluated where the other variable equals zero.

• What it captures:

  • The slope of x on y changes as z changes, e.g. a drug's effect differs by age.

  • Modeled as y = b0 + b1*x + b2*z + b3*(x*z).

• How interpretation shifts:

  • b1 is the effect of x only when z = 0, not its average effect.

  • The full effect of x is b1 + b3*z, so it varies with z.

  • b3 tells you how much x's slope changes per one-unit increase in z.

• Practical tips:

  • Centering predictors makes the main-effect coefficients interpretable at the mean rather than at zero.

  • Keep both main effects in the model when their interaction is included (respect hierarchy).

An unstandardized coefficient is in the raw units of the data (change in y per one unit of x), while a standardized (beta) coefficient is what you get after converting all variables to z-scores, expressing the effect in standard deviations. Use unstandardized for interpretation in real units, and standardized to compare relative importance across predictors on different scales.

• Unstandardized coefficient:

  • "y changes by b units per one-unit change in x."

  • Best for practical, real-world interpretation and prediction (dollars, kilograms).

• Standardized (beta) coefficient:

  • "y changes by beta standard deviations per one-SD change in x."

  • Removes units so predictors measured on different scales can be compared for relative strength.

• When to prefer which:

  • Communicating a concrete effect to stakeholders: unstandardized.

  • Ranking which predictors matter most within one model: standardized.

  • Caveat: standardized coefficients are sample-dependent (they scale with SDs) and can mislead with dummy variables.

The intercept sets the baseline level of the response and allows the regression line to shift up or down so it best fits the data. Forcing it to zero (regression through the origin) is appropriate only when theory guarantees that y must be zero when all predictors are zero.

• Role of the intercept:

  • Provides the vertical anchor so the slope terms are not forced to absorb the baseline.

  • Its presence keeps estimates unbiased; omitting it wrongly can distort the slopes.

• When forcing it to zero is justified:

  • Physical necessity: zero input must give zero output, e.g. distance = speed x time (zero time, zero distance).

  • Proportional relationships grounded in theory, not just to "simplify" the model.

• Cautions:

  • Dropping the intercept without justification biases the slope and inflates R-squared misleadingly (it is computed differently through the origin).

  • When in doubt, keep the intercept: if it is truly near zero, the model will estimate it as small anyway.

Centering and standardizing don't change the fit or predictions, but they rescale and reposition predictors so the intercept and slopes take on more interpretable meanings.

• Centering (subtract the mean from each predictor):

  • The intercept becomes the predicted response at the mean of the predictors, instead of at zero (which may be meaningless, e.g. zero height).

  • Slopes are unchanged: a one-unit change still has the same effect.

  • Reduces collinearity between main effects and interaction/polynomial terms.

• Standardizing (subtract mean, divide by SD):

  • Coefficients become the change in response per one-standard-deviation change in the predictor, so they are comparable across predictors on different units.

  • Intercept is the predicted response when all predictors are at their mean.

  • Essential for fair penalization in regularized models (Ridge/Lasso).

• What does not change: Predictions, residuals, R-squared, and p-values for the slopes stay identical: it's a reparameterization, not a different model.

The Least Squares criterion chooses the coefficients that minimize the sum of squared residuals (the vertical gaps between observed and predicted values). We square rather than take absolute values mainly because squaring gives a smooth, differentiable objective with a unique closed-form solution.

• The criterion: Minimize the sum over all observations of (observed minus predicted) squared.

• Why squared instead of absolute:

  • Differentiability: the squared loss is smooth everywhere, so setting the gradient to zero yields the Normal Equation and a unique solution; absolute error has non-differentiable kinks.

  • Statistical link: minimizing squared error equals the maximum likelihood estimate under Gaussian errors.

  • Squaring penalizes large errors more heavily, pulling the fit toward reducing big misses.

• The trade-off: That same heavy penalty makes OLS sensitive to outliers; minimizing absolute error (LAD regression) is more robust but harder to solve and gives the conditional median rather than mean.

When the model includes an intercept, the OLS fitting conditions force the residuals to sum to zero, and that algebraically guarantees the line passes through the point (mean of x, mean of y).

• The intercept's first-order condition: Setting the derivative of the squared-error loss with respect to the intercept to zero requires the residuals to sum to zero.

• From that, the algebra: Averaging the equation ŷ = b₀ + b₁x over all points gives ȳ = b₀ + b₁x̄, so the mean point lies exactly on the line.

• Caveat: This holds only when an intercept term is fit; regression through the origin need not pass through the means.

The Normal Equation is the closed-form solution for OLS coefficients, obtained by setting the gradient of the squared-error loss to zero. It directly yields the best-fit coefficients without iteration.

• Where it comes from:

  • Differentiate ‖y − Xβ‖², set it to zero, giving XᵀXβ = Xᵀy; solving for β yields the formula.

  • Geometrically it's the orthogonality condition: residuals perpendicular to the column space of X.

• Requirements and cautions:

  • XᵀX must be invertible, which fails under perfect multicollinearity or when features exceed observations.

  • In practice solvers use QR or SVD (or a pseudo-inverse) rather than explicitly inverting, for numerical stability.

• Advantages: No learning rate, no iterations, exact solution: ideal when the number of features is modest.

Prefer an iterative solver (e.g. gradient descent) when the closed-form Normal Equation becomes too expensive or unstable, typically because of scale or ill-conditioning.

• Large number of features: The Normal Equation needs inverting an XᵀX matrix that costs roughly O(p³); with many features this is prohibitive, while gradient descent scales far better.

• Very large datasets that don't fit in memory: Stochastic/mini-batch gradient descent streams data in chunks, so you never materialize the full matrix product.

• Ill-conditioned or singular XᵀX: When features are collinear the inverse is numerically unstable; iterative methods (with regularization) are more robust.

• Online / streaming settings: Iterative updates let you refit incrementally as new data arrives.

• When the closed form wins: Small-to-medium p with data in memory: the Normal Equation is exact, has no learning rate to tune, and needs no convergence checks.

Because the intercept's own first-order condition (its normal equation) forces the sum of residuals to zero: the intercept is chosen precisely so that positive and negative errors cancel.

• The intercept column is all ones: OLS sets the gradient of the loss w.r.t. each coefficient to zero, which means each feature column is orthogonal to the residuals: Xᵀe = 0.

• Applied to the ones column: The condition for that column is 1ᵀe = Σeᵢ = 0, i.e. residuals sum to exactly zero.

• Consequence: The mean residual is zero, and the regression plane passes through the point of means (ȳ, x̄).

• Caveat: This holds only when an intercept is included; a no-intercept (through-origin) model has no such guarantee.

The design matrix X is the n×p table of predictor values (one row per observation, one column per feature, usually including a ones column for the intercept). It must be full column rank so the coefficients are uniquely identified.

• What it represents: Each row is a data point's feature vector; the model prediction is the matrix product Xβ.

• Full rank means independent columns: No feature is an exact linear combination of others; otherwise XᵀX is singular and cannot be inverted.

• Why it matters for fitting: The closed form β = (XᵀX)⁻¹Xᵀy requires that inverse to exist; rank deficiency gives infinitely many solutions.

• Common causes of rank deficiency: Duplicate/perfectly collinear features, the dummy-variable trap (all one-hot categories plus an intercept), or p > n.

R-squared is the fraction of variance in the outcome explained by the model, but it never decreases when you add features. Adjusted R-squared penalizes for the number of predictors, so it only rises when a new feature improves the fit more than chance would.

• R-squared:

  • Defined as 1 − SSres/SStot; measures goodness of fit on a 0–1 scale.

  • Monotonically non-decreasing as predictors are added, even useless ones, so it rewards complexity.

• Adjusted R-squared:

  • Adjusts by degrees of freedom: 1 − (1 − R²)(n − 1)/(n − p − 1).

  • Can decrease if an added feature doesn't earn its degree of freedom, discouraging overfitting.

• Why adjusted is better for comparison: It lets you compare models with different numbers of predictors fairly, unlike plain R² which always favors the bigger model.

• Caveat: Both are in-sample metrics; for real generalization use cross-validation or a held-out test set.

Prefer MAE when you want an error metric that is robust to outliers and interpretable in the outcome's units, and when large errors shouldn't be penalized disproportionately.

• Robustness to outliers: RMSE squares errors, so a few large residuals dominate; MAE weights all errors linearly and is far less sensitive.

• Interpretability: MAE is the average absolute miss in original units (e.g. dollars), which is intuitive to stakeholders.

• What each targets: MAE is minimized by the conditional median, RMSE by the conditional mean; choose based on which central tendency matters.

• When RMSE is preferable instead: When large errors are genuinely costly and should be punished more, or when you want a metric consistent with the squared-error loss OLS actually optimizes.

R-squared is the proportion of variance in the outcome explained by the model, 1 − SSres/SStot. Adding a random noise variable will (weakly) increase R-squared, essentially never decreasing it, even though the feature carries no real information.

• Why it goes up: OLS can always set the noise coefficient to zero, so at worst the fit is unchanged; in practice random noise correlates slightly with the residuals by chance, so the coefficient becomes nonzero and SSres drops a bit.

• It never decreases: More parameters give more freedom to reduce training error, so R² is monotonically non-decreasing in the number of predictors.

• The lesson: A higher R² doesn't mean a better model; this is exactly why Adjusted R² (which penalizes added parameters) or out-of-sample validation is needed.

• Contrast with adjusted: Adjusted R² would typically fall for a pure-noise feature, correctly signaling it adds no value.

Yes. A high R-squared only measures how much variance is explained on the fitted data, not whether the model is correct, honest, or generalizable.

• Overfitting: Adding many predictors (or high-degree polynomials) drives training R-squared toward 1 while test performance collapses.

• Violated assumptions hidden behind a big number: A high R-squared can coexist with curved residuals, heteroscedasticity, or autocorrelation that invalidate inference.

• Spurious / non-stationary relationships: Two unrelated trending time series can regress with R-squared > 0.9 purely from shared trend.

• Concrete scenario: Fit a 10-degree polynomial to 12 points: R-squared near 1, but predictions on new points are wild.

• Bottom line: pair R-squared with residual diagnostics and out-of-sample validation.

Because R-squared mechanically rises (or stays flat) as you add predictors, so it always favors the more complex model even when the extra variables add noise, not signal.

• Monotonic with predictors: Any new feature can only keep RSS the same or lower, so training R-squared never penalizes complexity.

• In-sample, not out-of-sample: It reflects fit on the data used to train, which overfitting inflates; it doesn't measure generalization.

• Ignores parsimony and interpretability: Two models can have equal R-squared but very different complexity and stability.

• Better selection tools: Adjusted R-squared, AIC/BIC, and cross-validated error penalize complexity or test generalization.

The model with the higher RMSE is likely more affected by outliers, because RMSE squares errors and so is dominated by a few large residuals, while R-squared can stay similar.

• RMSE penalizes large errors quadratically: A handful of big misses inflate RMSE far more than many small ones.

• Why R-squared can look similar: R-squared is a ratio relative to total variance; if the outcome variance is also large, the explained fraction can stay comparable.

• Diagnostic follow-up:

  • Compare RMSE to MAE: a large RMSE/MAE gap signals outlier-driven error.

  • Inspect residual plots for a few extreme points.

The residual standard error (RSE) estimates the typical size of the residuals in the outcome's own units: roughly the standard deviation of the errors around the regression line. Unlike R-squared, it's an absolute measure of fit, not a proportion.

• Definition:

  • RSE = sqrt(RSS / (n - p - 1)), where p is the number of predictors.

  • The denominator uses degrees of freedom, so RSE corrects for model size.

• How it differs from R-squared:

  • RSE is in the units of y (e.g. dollars), so it answers 'how far off, typically?'

  • R-squared is unitless and relative, answering 'what fraction of variance is explained?'

  • A model can have high R-squared yet a large RSE if the outcome itself varies a lot.

• Relation to RMSE: RSE is essentially RMSE adjusted for degrees of freedom rather than divided by n.

Total variation in y decomposes into the part the model explains plus the part it leaves unexplained: TSS = ESS + RSS. This identity holds exactly for OLS with an intercept.

• TSS (total sum of squares): Sum of (y_i - ȳ)²: total variability of the outcome around its mean.

• ESS (explained sum of squares): Sum of (ŷ_i - ȳ)²: variability captured by the fitted values.

• RSS (residual sum of squares): Sum of (y_i - ŷ_i)²: leftover variability the model misses.

• Why they add up:

  • OLS makes residuals orthogonal to fitted values, so the cross term vanishes and TSS splits cleanly.

  • This gives R-squared = ESS/TSS = 1 - RSS/TSS.

• Caveat: the identity relies on an intercept; without one it can break.

A confidence interval for a coefficient captures uncertainty about a fixed population parameter (the true slope), while a prediction interval captures uncertainty about a single future observation of y. The prediction interval is wider because it includes both estimation uncertainty and the irreducible noise of an individual outcome.

• Confidence interval (coefficient or mean response):

  • Range of plausible values for a parameter like β, or for the mean of y at given x.

  • Reflects only sampling/estimation error, which shrinks as n grows.

• Prediction interval (new observation):

  • Range for an actual new y at given x, not its average.

  • Adds the error variance σ² on top of the estimation uncertainty.

• Which is wider and why:

  • The prediction interval is always wider: its variance ≈ estimation variance + σ².

  • Even with infinite data (estimation error → 0), the prediction interval stays wide because individual noise remains, while a confidence interval collapses toward zero width.

The F-statistic tests the model as a whole: whether at least one predictor has a nonzero coefficient, while individual t-tests only judge one coefficient at a time holding the others fixed.

• Joint vs. individual hypotheses:

  • The F-test's null is that all slopes are zero simultaneously; a t-test's null is that one specific slope is zero.

  • A small F p-value says the model explains more variance than chance, even if no single t-test looks strong.

• Handles multicollinearity: Correlated predictors can inflate standard errors, making every t-test insignificant while the overall F is highly significant.

• Guards against multiple-testing false positives: With many predictors, some t-tests may look significant by chance; the F-test gives one honest overall check.

• Interpretation rule of thumb: Significant F but no significant t's often signals correlated predictors, not a useless model.

Statistical significance asks whether an effect is distinguishable from zero given noise; practical significance asks whether the effect is large enough to matter in the real world. They are independent questions.

• Statistical significance: Driven by the p-value and sample size: with enough data, even a trivially tiny coefficient becomes significant.

• Practical significance: Judged by the coefficient's magnitude in domain units: does a $0.02 lift per customer justify the cost of the change?

• The four combinations:

  • Significant + large: act on it. Significant + tiny: real but may not be worth acting on.

  • Not significant + large: possibly important but underpowered, gather more data. Not significant + tiny: ignore.

• Best practice: Report effect sizes and confidence intervals, not just p-values, so magnitude and uncertainty are both visible.

The standard error of a coefficient measures how much that estimate would bounce around if you re-collected the data; it is the estimated standard deviation of the coefficient's sampling distribution. Smaller means a more precise, more trustworthy estimate.

• What it drives: It feeds the t-statistic and confidence interval: a larger SE means wider intervals and weaker significance.

• Factors that make it smaller (more precise):

  • More data (larger n).

  • Lower residual variance (the model fits well, little unexplained noise).

  • More spread in the predictor: a wide range of x values pins down the slope better.

• Factors that make it larger (less precise):

  • Multicollinearity: correlated predictors inflate SEs (quantified by the variance inflation factor, VIF).

  • High noise, small samples, or a predictor with little variation.

Read it top-down as three layers: overall model fit, then individual coefficients, then diagnostics on the residuals and assumptions. Each layer answers a different question before you trust the numbers.

1. Overall fit:

   • R-squared / adjusted R-squared: fraction of variance explained; adjusted penalizes extra predictors.

   • F-statistic and its p-value: is the model as a whole better than no predictors?

2. Coefficients:

   • Sign and magnitude: does the direction make domain sense, and is the effect size practically meaningful?

   • Standard error, t-stat, p-value, and confidence interval: precision and significance of each estimate.

3. Diagnostics: Residual normality (Jarque-Bera, Omnibus), autocorrelation (Durbin-Watson), and condition number for multicollinearity.

4. Synthesis: Cross-check: a strong F with all t's insignificant hints at collinearity; a high R-squared with bad residual plots means the model is misspecified.

Extrapolation means predicting outside the range of X values the model was trained on, where you have no data to justify that the fitted relationship still holds. The linear form is only an approximation over the observed range, so beyond it predictions can be wildly wrong with false confidence.

• Why it is dangerous:

  • A line fit locally may capture a curve's slope in the middle but diverge sharply at the ends where the true relationship bends or saturates.

  • No data means no way to detect that the assumed linearity has broken down.

• What happens numerically:

  • Prediction intervals widen as you move from the mean of X, so uncertainty grows even under the model.

  • You can get physically impossible outputs (negative counts, probabilities above 1, absurd magnitudes).

• How to handle it:

  • Report the training range and flag out-of-range inputs; treat far predictions as speculative.

  • Use domain knowledge or nonlinear/constrained models if you must project beyond the data.

Linear regression models a continuous, unbounded response with a straight line, but a binary target is a probability bounded in [0,1] with non-normal errors, so the linear model's core assumptions and outputs break down.

• Unbounded predictions: A line extends to -inf and +inf, so predicted 'probabilities' fall below 0 or above 1, which is meaningless.

• Violated error assumptions: With a 0/1 target the residuals are neither normal nor homoscedastic (variance depends on the mean: p(1-p)), so OLS inference (t-tests, CIs) is invalid.

• Wrong link between predictors and outcome: Probability responds to inputs in an S-shaped way, not linearly; a straight line poorly fits the tails.

• The fix: Logistic regression applies a sigmoid (logit link) so outputs stay in (0,1) and errors follow a Bernoulli model.

Keep the model linear in the parameters but transform the features so the relationship becomes linear: a linear model can fit curves as long as the coefficients enter linearly.

• Feature transformations: Add polynomial terms (x^2, x^3), logs, square roots, or interactions to capture curvature.

• Basis expansions: Splines or piecewise-linear terms model bends flexibly while staying linear in the fitted weights.

• Transform the target: A log(y) transform linearizes multiplicative/exponential growth.

• Key distinction: 'Linear' refers to being linear in parameters, not in the raw inputs, so curved fits are fully allowed.

• Guard against overfitting: High-degree polynomials swing wildly; add regularization or use cross-validation to pick complexity.

OLS minimizes squared errors, so it punishes large residuals disproportionately and gets pulled strongly toward outliers; MAE minimizes absolute errors and treats each point more evenly, making it far more robust.

• OLS squares the residuals:

  • A point twice as far contributes four times the loss, so a single extreme point can tilt the whole line to reduce its error.

  • The OLS fit follows the conditional mean, which is sensitive to extremes.

• MAE (L1) grows linearly:

  • Each residual's influence is proportional to its size, not its square, so outliers have bounded pull.

  • The MAE fit tracks the conditional median, which is robust to outliers.

• Trade-offs:

  • OLS has a closed-form solution, nice statistical theory, and is efficient under Gaussian errors.

  • MAE has no simple derivative at zero and needs iterative/LP solvers, but resists contamination.

• Middle ground: Huber loss is quadratic near zero and linear in the tails, combining efficiency and robustness.

An outlier is a point with an unusual residual (far from the fitted line in y), while an influential point is one whose removal noticeably changes the fitted coefficients: influence comes from combining an unusual x-position (leverage) with an unusual y-value.

• Outlier: Large residual, but if it sits near the center of the x-range it may barely move the line.

• Leverage: An extreme x-value has high leverage: it has the potential to pull the line toward itself.

• Influential point:

  • High leverage AND off-trend, so it swings the slope/intercept substantially; removing it changes conclusions.

  • A point can be a big outlier yet not influential (central x), or have high leverage yet not influential (on-trend).

• How to detect: Use Cook's distance, leverage (hat) values, and studentized residuals rather than eyeballing residuals alone.

• Why it matters for OLS: Because OLS squares residuals, a single influential point can dominate the fit and mislead every coefficient.

Overfitting is when the model learns noise in the training data instead of the true signal: it fits the training set almost perfectly but generalizes poorly to new data. In linear regression it appears when the model has too much flexibility relative to the data (too many predictors or a too-high polynomial degree).

• How it manifests:

  • Training R-squared is very high while test/validation error is much worse: a large train-test gap.

  • Coefficients become large and unstable, swinging wildly with small data changes.

  • High-degree polynomials wiggle to hit each point and oscillate badly between/beyond them.

• Why too many predictors hurt: Each added feature can only raise training R-squared, so the model fits idiosyncratic noise; with p near n it can fit almost anything.

• Bias-variance view: Extra complexity lowers bias but raises variance; overfitting is the high-variance extreme.

• Remedies: Cross-validation to choose complexity, regularization (ridge, lasso), feature selection, use adjusted R-squared or AIC/BIC, and get more data.

The error term is the unobservable deviation of the true outcome from the population regression line; the residual is its observable, estimated counterpart: the gap between the actual value and the fitted value from your sample.

• Error term (u_i):

  • Defined by the population model: y_i = Xβ + u_i.

  • Unobservable because the true parameters β are unknown.

  • Captures omitted factors, measurement error, and inherent randomness.

• Residual (ê_i):

  • Defined by the fitted model: ê_i = y_i - ŷ_i.

  • Observable, computed from estimated coefficients β̂.

  • An estimate of the unobserved error; used to check assumptions and estimate σ².

• Relationship:

  • Residuals are the sample analog of errors; OLS chooses β̂ to minimize the sum of squared residuals.

  • OLS residuals sum to zero (with an intercept), a mechanical property; errors carry no such guarantee.

The population model describes the true, unknown data-generating relationship with fixed parameters and an error term; the fitted model is our sample-based estimate of it, with computed coefficients and residuals in place of the unknowns.

• Population (true) model:

  • Written y = β₀ + β₁x + u.

  • Parameters β are fixed constants we never observe.

  • Includes the unobservable error u.

• Fitted (estimated) model:

  • Written ŷ = β̂₀ + β̂₁x.

  • Coefficients β̂ are computed from a specific sample and vary sample to sample.

  • Produces observable residuals ê instead of errors.

• The bridge between them:

  • β̂ is a random variable that estimates the fixed β; unbiasedness means E[β̂] = β.

  • Statistical inference (standard errors, tests) quantifies how far β̂ might be from β.

A regression coefficient measures how the outcome moves with a predictor holding other included variables fixed, but that is a statistical association: calling it causal requires strong assumptions that observational data usually cannot guarantee.

• What the coefficient actually is:

  • A conditional correlation: the average difference in y per unit of x, net of other regressors.

  • It reflects patterns in the data, not necessarily a mechanism.

• Why it usually isn't causal:

  • Omitted variable bias: an unmeasured confounder drives both x and y.

  • Reverse causality: y may influence x.

  • Selection bias and measurement error also break the causal interpretation.

• What causality would require:

  • Exogeneity of the treatment (x unrelated to the error), typically via randomization or a credible identification strategy.

  • Tools like RCTs, instrumental variables, difference-in-differences, or controlling for all confounders.

The dummy variable trap is the perfect multicollinearity that arises when you one-hot encode a categorical variable into as many dummies as categories and also keep an intercept: the dummies sum to a column of ones, making the design matrix non-invertible. You avoid it by dropping one category as a reference.

• Why it breaks the model:

  • With k dummies plus an intercept, the dummies are an exact linear combination of the intercept.

  • This violates the no-perfect-multicollinearity assumption, so (XᵀX) is singular and coefficients are not uniquely estimable.

• How to avoid it:

  • Use k - 1 dummies, dropping one level as the baseline (pd.get_dummies(drop_first=True)).

  • Coefficients are then interpreted relative to that reference category.

  • Alternatively, keep all k dummies but drop the intercept (equivalent, different interpretation).

• Notes: Regularized models (ridge/lasso) and tree-based models tolerate full one-hot encoding, so the trap is mainly an OLS concern.

Logging the dependent variable turns coefficients into approximate proportional (percentage) effects on y rather than additive effects in the original units.

• Level-log becomes log-level (log y on level x):

  • A one-unit increase in x changes y by approximately 100 * beta percent (for small beta).

  • Exact version: multiply y by exp(beta), so percent change = 100 * (exp(beta) - 1).

• Why the approximation works:

  • The derivative of log y is the relative change in y, so beta measures a proportional response.

  • The ~beta approximation is fine for small coefficients (say |beta| < 0.1); use the exp form for large ones.

• Practical reasons to log y:

  • Tames right-skewed, multiplicative outcomes (wages, prices, counts of money) and can stabilize non-constant variance.

  • y must be strictly positive; predictions live on the log scale and back-transforming the mean needs a correction (naive exp() underestimates).

Choose a log-log model (log y on log x) when you believe the relationship is multiplicative/constant-elasticity and both variables are positive and skewed, so a percent change in x drives a percent change in y.

• You want an elasticity: The coefficient reads directly as %ΔY / %ΔX, which is exactly what economics of demand, supply, and production often care about.

• The true relationship looks like a power law: A model y = a * x^b becomes linear after logging both sides: log y = log a + b log x.

• Diagnostic and distributional fit: Both variables span several orders of magnitude or are right-skewed; logging linearizes the scatter and stabilizes variance.

• Requirements and cautions:

  • Both x and y must be strictly positive (zeros break the log).

  • Constant elasticity is an assumption; if the true elasticity varies, the log-log fit is a simplification.

In a level-log model (y in levels, x logged), the coefficient tells you the change in y in its own units for a one-percent change in x, specifically about beta / 100 per 1% increase.

• The rule of thumb:

  • A 1% increase in x raises y by approximately beta / 100 units.

  • A doubling of x (100% increase) raises y by about beta * ln(2) ≈ 0.693 * beta units.

• Where it comes from: Since dy = beta * d(log x) and d(log x) is the fractional change in x, a 0.01 fractional change gives 0.01 * beta.

• Interpretation contrast: Outcome effect is additive (in units), driver effect is proportional (in percent), so this is a 'percent-in, level-out' model.

Reach for Ridge or Lasso when plain OLS is unstable or overfits: high-dimensional data, correlated predictors, or when you want built-in variable selection, accepting a little bias to sharply reduce variance.

• Many features or p near/above n: OLS overfits or has no unique solution; the penalty regularizes and makes estimation well-posed.

• Multicollinearity: Ridge (L2) shrinks correlated coefficients together, stabilizing them and cutting variance.

• You want sparsity / feature selection: Lasso (L1) drives some coefficients exactly to zero, producing a simpler, more interpretable model.

• Bias-variance tradeoff: The penalty lambda trades a small bias for lower variance, usually improving out-of-sample error; tune it by cross-validation.

• Notes:

  • Standardize predictors first, since penalties are scale-sensitive.

  • Elastic Net blends L1 and L2 when you have groups of correlated features and still want selection.

You need enough observations per predictor to estimate coefficients reliably: a common rule of thumb is at least 10 to 20 observations per predictor. As the number of predictors p approaches the sample size n, the model overfits, and when p exceeds n the ordinary least squares solution isn't even unique.

• Rules of thumb:

  • Aim for roughly 10-20 observations per estimated parameter (include intercept and dummy levels).

  • More predictors need more data to keep standard errors and variance in check.

• What happens as p grows relative to n:

  • Overfitting: the model fits noise, so in-sample R-squared looks great but out-of-sample error is poor.

  • High-variance, unstable coefficients with wide confidence intervals, worsened by multicollinearity.

  • Degrees of freedom for error, n - p - 1, shrink, weakening tests and error estimates.

• When p >= n (high-dimensional):

  • OLS has no unique solution: the design matrix X'X is singular and cannot be inverted.

  • When p = n the model fits perfectly (residuals zero) but is meaningless.

• Remedies:

  • Collect more data, or reduce dimensionality (drop variables, PCA).

  • Use regularization (ridge, LASSO) that works even when p > n.

Normality of errors is only needed to get exact small-sample inference (valid t-tests, F-tests, and confidence intervals). In large samples, the Central Limit Theorem makes the coefficient estimators approximately normal regardless of the error distribution, so the assumption becomes largely irrelevant.

• Not needed for estimation: Coefficients are unbiased (and efficient among linear estimators) without normality, per Gauss-Markov.

• Needed for exact small-sample inference: With few observations, valid t and F distributions rely on the errors actually being normal.

• How the CLT relaxes it:

  • The OLS estimator is a weighted sum/average of the errors; as n grows, that sum tends to normal even if individual errors aren't.

  • So standard errors and p-values stay approximately valid asymptotically.

• Caveats:

  • Very heavy tails or strong skew slow convergence, so 'large enough n' depends on the data.

  • The CLT rescues inference on coefficients, not prediction intervals for individual points, which still lean on error normality.

Exogeneity is the assumption that the error term has zero mean conditional on the predictors (E[ε | X] = 0), meaning no predictor carries information about what's left in the error. When violated, part of what the error captures gets wrongly attributed to a correlated predictor, biasing its coefficient.

• Formal statement: E[ε | X] = 0, equivalently the predictors are uncorrelated with the error.

• Why violation biases coefficients: OLS cannot separate the predictor's true effect from the correlated error component, so the estimate absorbs both.

• Common causes:

  • Omitted variable bias: a left-out factor drives both X and Y.

  • Reverse causality/simultaneity: Y also affects X.

  • Measurement error in a predictor.

• Remedies: Add the omitted controls, use instrumental variables, fixed effects, or a proper causal design.