106 Logistic Regression Interview Questions and Answers (2026)

Logistic regression is a supervised classification algorithm that models the probability that an input belongs to a class by passing a linear combination of features through the sigmoid (logistic) function.

• Core idea:

  • It computes a linear score z = w·x + b then squashes it to a probability in (0, 1) via the sigmoid.

  • A threshold (usually 0.5) converts the probability into a class label.

• What it learns:

  • Weights are fit by maximizing likelihood (equivalently, minimizing log loss / cross-entropy).

  • There is no closed-form solution, so it uses gradient-based optimization.

• Interpretability: Each coefficient is the change in log-odds per unit change in a feature, so exp(w) is an odds ratio.

Linear regression predicts a continuous number by fitting a straight line, while logistic regression predicts a class probability by fitting a sigmoid to the log-odds. They differ in output, link function, and loss.

• Output:

  • Linear: an unbounded real value.

  • Logistic: a probability in (0, 1), then a class label.

• Model form:

  • Linear predicts the target directly as w·x + b.

  • Logistic predicts the log-odds as w·x + b and applies the sigmoid.

• Loss function:

  • Linear minimizes mean squared error (least squares).

  • Logistic minimizes log loss (cross-entropy), fit via maximum likelihood.

• Solution: Linear has a closed-form (normal equation); logistic requires iterative optimization.

It is called "regression" because, under the hood, it regresses a continuous quantity (the log-odds of the positive class) on the features as a linear function: the classification step is just thresholding that regressed value.

• It fits a linear model to log-odds: The equation log(p/(1-p)) = w·x + b is a genuine linear regression on the logit scale.

• It belongs to the GLM family: Logistic regression is a generalized linear model with a logit link, sharing the statistical machinery of regression.

• Output is continuous before thresholding: It predicts a continuous probability; the class label is a downstream decision, not the model's raw output.

• Historical naming: The term comes from its statistical roots, before ML reframed it primarily as a classifier.

The type depends on the target variable: binary for two classes, multinomial for several unordered classes, and ordinal for classes with a natural order.

• Binary logistic regression: Two outcomes (spam vs not spam); uses a single sigmoid.

• Multinomial logistic regression:

  • Three or more unordered classes (cat, dog, bird); uses the softmax function.

  • Can also be approximated with a one-vs-rest scheme of binary models.

• Ordinal logistic regression: Three or more ordered classes (low, medium, high); models cumulative probabilities and preserves rank.

Logistic regression is a common baseline because it is fast, simple, interpretable, and requires little tuning, giving a solid reference point before trying more complex models.

• Simplicity and speed: Trains quickly on large data with few hyperparameters, so it is cheap to stand up first.

• Interpretability: Coefficients map to odds ratios, making it easy to explain and to sanity-check features.

• Sets a performance floor: If a heavy model can't beat it, that model likely isn't worth the complexity.

• Robust and well-calibrated: Outputs meaningful probabilities and resists overfitting, especially with regularization.

Regression predicts a continuous quantity, while classification predicts a discrete category: the difference is clearest when you consider the same house data used for two different targets.

• Regression example: Predict a house's selling price (e.g. $342,000) from size, location, and age: the output is a number on a continuous scale.

• Classification example: Predict whether that house will sell within 30 days (yes/no) from the same features: the output is a category.

• Key contrast:

  • Regression is measured with errors like RMSE; classification with metrics like accuracy or F1.

  • Same inputs, different target type, drives the choice of algorithm.

Linear regression is a poor fit for classification because its unbounded output isn't a valid probability, its squared-error loss is the wrong objective for classes, and its predictions are sensitive to the exact issues classification doesn't care about.

• Outputs aren't probabilities: Predictions can be below 0 or above 1, which can't be interpreted as class probabilities.

• Sensitive to outliers: Extreme points shift the fitted line and move the effective decision boundary, hurting classification.

• Wrong loss: Squared error penalizes confident correct predictions and doesn't match the discrete nature of labels; log loss is the right objective.

• Logistic regression fixes this: The sigmoid bounds output to (0, 1) and cross-entropy gives a proper probabilistic classifier.

Yes: logistic regression is a supervised learning algorithm because it trains on labeled data, learning a mapping from input features to known output classes.

• Requires labels: Each training example includes both features and the true class, which the model uses to fit its weights.

• Primarily a classifier: It predicts categorical outcomes, though it outputs probabilities along the way.

• Not unsupervised: Unlike clustering or dimensionality reduction, it cannot learn without target labels.

Logistic regression predicts the probability that an observation belongs to a class, so the dependent variable is categorical (typically binary), even though the model outputs a continuous probability internally.

• Target is categorical: The label is a class (e.g. spam/not-spam, churn/no-churn), not a continuous measurement.

• Output is a probability: The model produces a number in [0, 1], then applies a threshold to assign a class.

• Contrast with linear regression: Linear regression predicts a numerical outcome; logistic regression is a classifier despite the name.

The classic (binary) logistic regression models the log-odds of a single event happening versus not happening, so it is built around a two-outcome response; a plain binary target keeps that odds interpretation valid.

• The math assumes two outcomes: It estimates P(y=1) and treats P(y=0) = 1 - P(y=1), which only partitions into two classes.

• The link is the logit of one event: Coefficients express the log-odds of the positive class, an interpretation that needs a clear success/failure split.

• Multi-class is an extension, not the base case: More than two categories requires multinomial (softmax) or one-vs-rest schemes layered on top of the binary core.

Both are generalized linear models: they compute the same linear combination of inputs, w·x + b, and estimate weights that describe how each feature moves the outcome; they differ mainly in what they do with that linear score.

• Shared linear predictor: Each feature gets a coefficient and the model sums them into a single linear score.

• Similar interpretation of coefficients: A weight's sign and magnitude tell you the direction and strength of a feature's effect in both.

• Key difference: Linear regression outputs the linear score directly; logistic regression passes it through the sigmoid to bound it to a probability.

The predicted probability p always lies strictly between 0 and 1 (open interval), because the sigmoid squashes any real-valued linear score into that range.

• Range is (0, 1): The sigmoid asymptotically approaches 0 and 1 but never reaches them for finite inputs.

• Why this matters: It guarantees a valid probability, unlike linear regression which could predict values below 0 or above 1.

• From probability to class: A threshold (default 0.5) converts p into a class label.

Logistic regression does not require a linear relationship between the predictors and the response itself, nor does it require normally distributed residuals or homoscedasticity, the assumptions that linear regression relies on.

• Not required:

  • Linearity between predictors and the raw outcome (it only needs linearity in the log-odds).

  • Normally distributed errors or a normally distributed dependent variable.

  • Homoscedasticity (constant error variance).

• Still required: Independence of observations, little multicollinearity, and a linear relationship between predictors and the logit.

Interpretability is a core strength because each coefficient has a direct, quantifiable meaning: it maps to how a feature changes the odds of the outcome, so you can explain and defend every prediction.

• Coefficients are odds ratios: Exponentiating a weight, exp(w), gives the multiplicative change in odds per unit increase in that feature.

• Direction and importance are readable: Sign shows whether a feature raises or lowers the probability; magnitude shows how much.

• Regulatory and trust value: In credit, healthcare, and other regulated fields, being able to justify decisions is often mandatory, favoring transparent models over black boxes.

The sigmoid (logistic) function converts the model's unbounded linear score into a probability in (0, 1), giving logistic regression its S-shaped mapping from features to class probability.

• Purpose:

  • Squash the linear predictor z = w·x + b (which ranges over all reals) into a valid probability.

  • It is the inverse of the logit link, so log-odds are linear in the features.

• Mathematical form: σ(z) = 1 / (1 + e^(-z)), giving 0.5 at z = 0 and saturating toward 0 and 1 at the extremes.

• Useful property: Its derivative is σ(z)(1 - σ(z)), which makes gradient-based optimization clean.

A logit of 0 maps to a probability of exactly 0.5, the point of maximum uncertainty and the location of the decision boundary.

• The math: sigmoid(0) = 1/(1 + e^0) = 1/(1+1) = 0.5.

• Interpretation:

  • Odds are e^0 = 1, i.e. the event is equally likely as not.

  • This is exactly where the model sits on its decision boundary.

It means the model estimates a 75% chance that this particular email belongs to the positive (spam) class given its features. It is a confidence/likelihood, not a guarantee, and it becomes a hard label only after applying a threshold.

• What it is:

  • An estimate of P(spam | features) = 0.75 for this email.

  • In odds terms: 0.75 / 0.25 = 3, so spam is estimated three times as likely as not-spam.

• What it is not: Not a statement that 75% of the email is spam, and not automatically the final decision.

• Turning it into a decision:

  • With the default 0.5 threshold it is labeled spam; you can raise the threshold to reduce false positives.

  • Frequentist reading: among many emails scored 0.75, roughly 75% are truly spam if the model is well calibrated.

Logistic regression learns a set of weights (and a bias) that define a linear decision boundary in feature space, tuned so the sigmoid of the weighted sum best matches the observed class labels.

• The parameters:

  • A weight vector w (one coefficient per feature) and an intercept b.

  • Each weight is the change in log-odds per unit change in that feature.

• What it optimizes:

  • It finds w, b that maximize the likelihood of the labels (equivalently, minimize log loss).

  • There is no closed form; it's solved iteratively (gradient descent, Newton's method).

• What it does NOT learn: It does not memorize data; it learns a linear separating surface: w·x + b = 0.

Not by itself: h(x)=0.55 is an estimated probability, not a decision. With the standard 0.5 threshold you would predict "recommend," but the output only says the model assigns a 55% chance to that class.

• Interpretation:

  • h(x) = P(y=1 | x) ≈ 0.55, a calibrated-ish confidence, not a hard label.

  • The prediction depends on the chosen threshold: 0.5 gives "recommend," but 0.55 is barely above it (low confidence).

• Threshold is a business choice:

  • If false positives are costly, raise the threshold; if false negatives are costly, lower it.

  • Tune it using the ROC/precision-recall trade-off, not always 0.5.

Regularization adds a penalty on coefficient magnitude to the logistic loss, discouraging overly large weights so the model generalizes better instead of overfitting the training data.

• What it changes:

  • The objective becomes log-loss + λ·penalty(β), trading a bit of training fit for smaller, more stable coefficients.

  • Bias-variance tradeoff: adds bias, reduces variance.

• Why it helps:

  • Controls overfitting with many or correlated features.

  • Tames coefficients under quasi-separation, where unpenalized weights blow up to infinity.

• The strength knob:

  • Controlled by λ (or C = 1/λ in scikit-learn); tuned by cross-validation.

  • The intercept is normally left unpenalized.

Logistic regression is a linear model sensitive to feature scale, encoding, and correlation, so preprocessing focuses on clean numeric inputs on comparable scales.

• Handle missing values: Impute (mean/median or model-based) since the model can't accept NaNs.

• Scale numeric features: Use StandardScaler so regularization penalizes all coefficients fairly and gradient solvers converge faster.

• Encode categoricals: One-hot for low cardinality; target/frequency encoding or hashing for high cardinality.

• Address multicollinearity: Drop or combine highly correlated predictors; it destabilizes coefficients and their interpretation.

• Other steps:

  • Treat outliers, consider transforms for skew, and add interaction/nonlinear terms since the boundary is linear in the features.

  • Handle class imbalance via class_weight or resampling.

• Fit scaling/encoding inside a Pipeline on train folds only to avoid leakage.

Logistic regression models the probability of a class as a sigmoid of a linear combination of features, fits the weights by minimizing cross-entropy loss, then thresholds the probability to produce a class.

• Linear score: Compute z = w·x + b, a weighted sum of features.

• Squash to probability: Apply the sigmoid: p = 1 / (1 + e^-z), mapping the score to (0, 1).

• Define the loss: Use log loss (negative log-likelihood), which is convex, so there's a single global optimum.

• Optimize the weights: Minimize the loss with gradient-based solvers, often with an L1/L2 penalty added.

• Predict and threshold: Output predict_proba, then classify at a cutoff (default 0.5, tuned for the business metric).

Logistic regression is a simple, interpretable, well-calibrated linear classifier that trains fast and resists overfitting, but its linear decision boundary limits it on complex data and it is sensitive to outliers and correlated features.

• Advantages:

  • Interpretable: coefficients map to log-odds / odds ratios.

  • Outputs calibrated probabilities, not just labels.

  • Fast to train and predict; scales well and is a strong baseline.

  • Regularization (L1/L2) controls overfitting and does feature selection.

  • Convex loss, so training reaches a global optimum.

• Disadvantages:

  • Assumes linearity in the log-odds; misses interactions/non-linearity unless engineered.

  • Sensitive to outliers and multicollinearity (unstable coefficients).

  • Can underperform on complex, high-dimensional patterns vs trees/nets.

  • Suffers from perfect separation and needs decent feature scaling.

The probability is the real output because it carries the model's confidence and lets you choose a decision threshold to fit your costs; the hard 0/1 label is just a downstream choice that discards that information.

• Probability conveys confidence: 0.51 and 0.99 both map to class 1, but they represent very different certainty; the label hides that.

• Threshold is a business decision: You can lower the cutoff to catch more positives (higher recall) or raise it for precision, tuning to the cost of errors.

• Enables ranking and calibration: Probabilities let you rank cases, feed downstream expected-value calculations, and evaluate with metrics like AUC and log-loss.

A decision boundary is the surface in feature space where the model is equally uncertain between the two classes: predicted probability = 0.5. In standard logistic regression it is linear because that 0.5 point corresponds to the linear predictor being zero.

• Definition: The set of points where the model switches its predicted class, i.e. where P(y=1|x) = 0.5.

• Why it is linear:

  • The sigmoid maps the linear predictor z = w·x + b to probability, and sigmoid(z) = 0.5 exactly when z = 0.

  • So the boundary is w·x + b = 0, which is a hyperplane (a line in 2D, a plane in 3D).

• Linear in features, not in probability: The probability output is nonlinear (S-shaped), but the boundary itself is a linear function of the inputs.

• Making it nonlinear: Add polynomial or interaction terms; the boundary is then linear in the expanded feature space but curved in the original space.

Odds are the ratio of the probability of an event to its complement; log-odds (the logit) are the natural log of the odds and are what logistic regression models linearly; the odds ratio describes how the odds change multiplicatively for a one-unit change in a feature.

• Odds:

  • odds = p / (1 - p): e.g. p = 0.8 gives odds of 4 (event four times as likely as not).

  • Range is 0 to infinity.

• Log-odds (logit):

  • logit(p) = ln(p / (1 - p)), ranging over all real numbers.

  • Logistic regression assumes this is linear: ln(p/(1-p)) = w·x + b.

• Odds ratio:

  • Because coefficients act on log-odds, exp(w_j) is the multiplicative change in odds per unit increase in feature j (holding others fixed).

  • An odds ratio > 1 raises the odds; < 1 lowers them; = 1 means no effect.

Sigmoid maps a single scalar to a probability for binary (or independent multi-label) problems; softmax maps a vector of scores to a probability distribution across mutually exclusive classes that sums to 1. Softmax is the multiclass generalization of sigmoid.

• Sigmoid:

  • sigmoid(z) = 1 / (1 + e^-z), one output per unit, each independent.

  • Used for binary classification or multi-label tasks (an example can belong to several classes).

• Softmax:

  • softmax(z_i) = e^z_i / Σ e^z_j, outputs are coupled and sum to 1.

  • Used for multiclass, single-label classification (classes mutually exclusive).

• Relationship: Softmax with two classes reduces to sigmoid on the score difference.

The logistic function is the pure mathematical sigmoid that maps any real number to (0, 1); the logistic model is the full regression that first computes a linear predictor from the features and then passes it through that function to produce a class probability.

• Logistic function output:

  • Input: any real number z; output: a value in (0, 1) via 1/(1+e^-z).

  • It is just the squashing transform, agnostic to where z came from.

• Logistic model output:

  • Input: the feature vector x; it computes z = w·x + b then applies the logistic function.

  • Output: the estimated probability P(y=1|x), learned from data.

• In short: The function is the tool; the model is the function plus learned linear predictor applied to real inputs.

A probability near 0 or 1 means the logit has a large magnitude: strongly negative for probabilities near 0 and strongly positive for probabilities near 1, because the sigmoid saturates at its extremes.

• Probability near 1: logit → +∞; e.g. z = 5 gives about 0.993.

• Probability near 0: logit → -∞; e.g. z = -5 gives about 0.007.

• Why (saturation):

  • The sigmoid flattens at both ends, so large changes in z barely move the probability once it is near 0 or 1.

  • Practical caution: very large logits can signal overconfidence or separable data (unbounded weights), a sign to check regularization.

Logistic regression models the probability of the positive class by passing a linear combination of features through the sigmoid; equivalently, it models the log-odds as a linear function of the features.

• Linear predictor: z = w·x + b, a weighted sum of features plus bias.

• Logistic transformation: P(y=1|x) = σ(z) = 1 / (1 + e^-(w·x + b)), squashing z into (0, 1).

• Logit form: ln( p / (1 - p) ) = w·x + b: the inverse of the sigmoid, showing the model is linear in log-odds.

• Training: Weights are fit by maximizing likelihood (minimizing binary cross-entropy / log loss), typically via gradient descent.

Softmax constrains the class probabilities to each lie in (0,1) and to sum to exactly 1, so they form a valid probability distribution over the mutually exclusive classes.

• The softmax transform:

  • Each class k gets a score z_k = w_k·x + b_k, then P(y=k) = exp(z_k) / Σ_j exp(z_j).

  • Exponentiation makes every numerator positive; dividing by the sum normalizes to 1.

• Consequences:

  • Probabilities are coupled: raising one lowers others (a simplex constraint).

  • Only differences of z_k matter, so the model is over-parameterized (one class can be fixed as reference).

Not exactly: the linear scores in multinomial logistic regression correspond to log-odds relative to a reference (or up to an additive constant), not an absolute log-odds per category. Softmax works with relative logits, so only differences between classes are identified.

• What is actually modeled:

  • The log-ratio between two classes is linear: log(P(y=k)/P(y=j)) = (w_k - w_j)·x + (b_k - b_j).

  • Pick a reference class j; then each w_k gives the log-odds of k versus that reference.

• Why not absolute log-odds:

  • Adding a constant to every z_k leaves the softmax probabilities unchanged, so absolute logits aren't identifiable.

  • In the binary case this collapses to the familiar single log-odds log(p/(1-p)).

Because the model is linear on the log-odds scale: log-odds is a straight-line function of the features. The sigmoid only appears when you map that linear quantity back to a probability, and that mapping is what bends the curve into an S-shape.

• Two scales, two shapes:

  • Log-odds: log(p/(1-p)) = w·x + b, perfectly linear.

  • Probability: p = 1/(1+exp(-(w·x+b))), a non-linear squashing into (0,1).

• Why the curve bends: The sigmoid compresses extreme linear values toward 0 and 1, so equal changes in w·x move probability more near the middle than at the tails.

• Why it matters: Coefficients are interpreted as effects on log-odds (constant), not on probability (varies with x).

Log loss is the negative log-likelihood of the Bernoulli model behind logistic regression, so minimizing it is maximum likelihood estimation. It also yields a convex objective with well-behaved gradients, whereas MSE on sigmoid outputs is non-convex and produces weak gradients where the model is most wrong.

• Why BCE is the right loss:

  • It is the MLE objective: -[y log p + (1-y) log(1-p)].

  • Convex in the weights, so gradient descent reaches the global optimum.

  • It penalizes confident wrong predictions heavily (loss → ∞ as p → wrong extreme).

  • Clean gradient: (p - y)·x, the sigmoid derivative cancels.

• Why MSE is unsuitable:

  • MSE with a sigmoid is non-convex, so optimization can stall in local minima.

  • Its gradient carries a σ'(z) factor that vanishes in saturation, so very wrong predictions learn slowly.

  • It assumes Gaussian errors, which is wrong for binary 0/1 outcomes.

Maximum Likelihood Estimation (MLE) chooses the coefficients that make the observed labels most probable under the model. In logistic regression the model outputs a probability via the sigmoid, and MLE finds the weights that maximize the likelihood of the training data.

• The model defines a probability: p = sigmoid(wᵀx + b) is the predicted probability that y = 1.

• Likelihood of the data: Each label is Bernoulli, so the likelihood is the product of pᵢ^yᵢ (1-pᵢ)^(1-yᵢ) over all samples.

• Log-likelihood for tractability: Take the log to turn the product into a sum; maximizing it is equivalent and numerically stable.

• Estimation: The MLE coefficients are the ones that maximize this log-likelihood (equivalently, minimize log-loss), found via iterative optimization.

Logistic regression is trained by iteratively adjusting the weights to maximize the log-likelihood (minimize log-loss). Unlike OLS, the gradient equations are nonlinear in the parameters, so you cannot solve them algebraically and must use an iterative optimizer.

• How it is trained:

  • Start with initial weights, compute predicted probabilities, then update weights along the gradient of log-loss until convergence.

  • Common solvers: gradient descent, Newton-Raphson / IRLS, L-BFGS.

• Why no closed form:

  • OLS minimizes a quadratic, so its gradient is linear in w and solves in one step: w = (XᵀX)⁻¹Xᵀy.

  • The sigmoid makes the gradient Σ(yᵢ - σ(wᵀxᵢ))xᵢ nonlinear in w, so ∇ℓ = 0 has no algebraic solution.

• Good news: The log-loss is convex, so iterative methods reliably reach the global optimum.

Log-loss (negative log-likelihood) for logistic regression is convex in the weights because it is a sum of convex functions of a linear combination of the parameters. Convexity means any local minimum is the global minimum, so gradient-based optimizers reliably find the best fit.

• Why it's convex:

  • The per-example loss is -y·log(σ(wᵀx)) - (1-y)·log(1-σ(wᵀx)), which is convex in w.

  • Its Hessian is XᵀSX with S a diagonal matrix of σ(1-σ) ≥ 0, so it is positive semi-definite everywhere.

• Why convexity matters for fitting:

  • No spurious local minima: gradient descent, Newton, or IRLS converge to the unique global optimum (unique if data isn't perfectly separable and no collinearity).

  • Makes optimization predictable and initialization-insensitive.

• Contrast: using squared error with the sigmoid would give a non-convex surface with local minima, which is why we use log-loss.

• Caveat: perfect separation drives weights to infinity (MLE doesn't exist); regularization restores a finite unique solution.

Logistic regression assumes the log-odds are a linear function of the predictors, observations are independent, and there's no severe multicollinearity. Crucially, it drops several assumptions linear regression makes about the error term because it models a Bernoulli outcome, not a Gaussian one.

• What it assumes:

  • Linearity in the logit: log(p/(1-p)) = β₀ + βᵀx (linear in predictors, not in p).

  • Independent observations.

  • Little multicollinearity among predictors.

  • Adequate sample size (MLE is asymptotic; needs enough events per predictor).

  • Correct outcome coding (binary/Bernoulli).

• What it does NOT assume (unlike linear regression):

  • No normally distributed residuals: errors follow a binomial distribution.

  • No homoscedasticity: variance is p(1-p), inherently non-constant.

  • No linear relationship between predictors and the raw outcome: linearity is on the logit scale.

Diagnostics for logistic regression fall into three buckets: goodness-of-fit tests, residual/influence analysis, and discrimination/calibration metrics. Together they tell you whether the model fits, which points distort it, and how well it predicts.

• Goodness-of-fit:

  • Hosmer-Lemeshow test: compares observed vs predicted in probability deciles.

  • Deviance and Pearson chi-square residual statistics.

  • Pseudo-R² (McFadden, Nagelkerke) for a rough fit summary.

• Residuals and influence:

  • Deviance and Pearson residuals to spot poorly fit observations.

  • Leverage, Cook's distance, and DFBETAs for influential points.

• Multicollinearity: VIF on the predictors.

• Discrimination and calibration:

  • ROC-AUC and the confusion matrix for classification performance.

  • Calibration plots / Brier score to check that predicted probabilities match observed frequencies.

• Linearity check: Box-Tidwell test or binned log-odds plots for continuous predictors.

Logistic regression is linear in the logit, but you can model non-linear relationships by adding transformed features: interaction terms let one predictor's effect depend on another, and polynomial/spline terms let a single predictor's effect curve. The model stays linear in its parameters, so fitting is unchanged.

• Interaction terms:

  • Add a product x₁·x₂ so the slope of one variable shifts with another (e.g. age effect differs by sex).

  • Captures effect modification the additive model can't.

• Polynomial and spline terms:

  • Add x², x³, or spline basis functions to bend the log-odds curve.

  • Splines are usually safer than high-degree polynomials, which oscillate at the edges.

• Practical cautions:

  • These expand the feature space, so use regularization and cross-validation to avoid overfitting.

  • Center/standardize inputs to reduce collinearity between a term and its powers.

  • Interpretation gets harder: coefficients are conditional, so explain effects via marginal predictions or plots.

No. Multinomial (and binary) logistic regression drops the classic linear-regression assumptions of normality, linearity of the outcome, and homoscedasticity, because it models a probability through a link function rather than a continuous conditional mean with additive Gaussian error.

• No normality of residuals: The outcome is categorical; errors follow a binomial/multinomial distribution, not a normal one.

• No homoscedasticity: Variance of the response depends on the predicted probability p(1-p), so it is inherently non-constant.

• Linearity is required, but on the logit scale: Predictors are assumed linear in the log-odds, not linear in the probability or the raw outcome.

• What it does assume: Independence of observations, correct link/functional form, little multicollinearity, and (for multinomial) independence of irrelevant alternatives (IIA).

The events-per-variable (EPV) rule of thumb says you need roughly 10 events of the rarer outcome class per predictor (coefficient) estimated. It matters because logistic regression's maximum-likelihood estimates become unstable, biased, or fail to converge when the minority class is too small relative to model complexity.

• How to count EPV:

  • Use the count of the less frequent outcome, not total N. With 40 events and 4 predictors, EPV = 10.

  • Each dummy level and interaction counts as a parameter.

• Why small samples hurt:

  • MLE coefficients are biased away from zero in small samples (odds ratios overstated).

  • Wide confidence intervals and unstable estimates that swing with resampling.

  • Complete/quasi-complete separation becomes likely, driving coefficients to infinity.

• Caveats:

  • 10 is a guideline, not a law; some work suggests 5-9 can suffice and others recommend 20+.

  • When events are scarce, use penalized methods like Firth logistic regression instead of shrinking the model blindly.

Logistic regression handles any number of classes: binary out of the box, and multiclass through two strategies, One-vs-Rest (OvR) which wraps multiple binary models, or Multinomial (softmax) which models all classes jointly.

• Binary (2 classes): The base case: a single sigmoid outputs P(class=1).

• One-vs-Rest (K classes):

  • Train K independent binary classifiers, each separating one class from all others.

  • Predict by picking the class with the highest score; probabilities aren't guaranteed to sum to 1.

• Multinomial (softmax):

  • A single model with one weight vector per class; softmax converts scores into a proper probability distribution over all classes.

  • Coefficients are estimated jointly, so probabilities are internally consistent (sum to 1).

For three classes you need three binary classifiers in one-vs-all: in general K classes require K classifiers, one per class.

• One classifier per class: Classifier A: class A vs {B, C}; Classifier B: class B vs {A, C}; Classifier C: class C vs {A, B}.

• Prediction: Each model outputs a confidence/probability for its class; pick the class whose classifier is most confident (argmax).

• Caveats:

  • Scores come from separately trained models, so they aren't calibrated against each other and don't sum to 1.

  • Each model trains on the full dataset, which can create class imbalance (1 class vs the rest).

Multinomial logistic regression (softmax regression) generalizes binary logistic regression to K classes with a single model that assigns each class its own linear score and normalizes those scores into one probability distribution.

• One weight vector per class: Each class k has parameters that produce a raw score (logit) z_k = w_k·x + b_k.

• Softmax normalization: P(y=k|x) = exp(z_k) / Σ_j exp(z_j), guaranteeing probabilities are positive and sum to 1.

• Joint training: Fit by minimizing cross-entropy (categorical log-loss) across all classes at once, so classes compete directly.

• Identifiability: Softmax is over-parameterized (one class can be a reference); regularization or fixing a baseline resolves this.

OvR is simpler and more modular but produces uncalibrated, non-comparable scores; multinomial softmax gives coherent joint probabilities at the cost of a single, less flexible model. Pick based on whether you need proper probabilities and how the classes relate.

• One-vs-Rest:

  • Pros: trains K independent models (parallelizable), each swappable, works with any binary classifier.

  • Cons: scores aren't calibrated across models and don't sum to 1; each model faces imbalance (one vs rest).

• Multinomial (softmax):

  • Pros: one jointly optimized model, well-calibrated probabilities that sum to 1, classes compete directly.

  • Cons: less modular, assumes a shared linear structure, and can be costlier/less convenient to fit for very many classes.

• Rule of thumb: Need trustworthy multiclass probabilities: prefer multinomial; need flexibility or per-class tuning: OvR.

The threshold is a business decision, not a statistical constant: 0.5 only makes sense if false positives and false negatives cost the same and classes are balanced. You choose a threshold by mapping probabilities to the trade-off (or cost) that matters for your problem.

• Why 0.5 is arbitrary: It assumes equal error costs and a balanced base rate; with imbalance or asymmetric costs it's usually wrong.

• Selection methods:

  • Cost-based: choose the threshold minimizing expected cost given false-positive/false-negative costs.

  • Metric-based: maximize F1 or Youden's J, or read off the precision-recall / ROC curve for a target recall or precision.

  • Constraint-based: pick the threshold that meets an operational limit (e.g. only flag top 5% for review).

• Important caveats:

  • Tune the threshold on validation data, not test, to avoid leakage.

  • Calibrate probabilities first if you rely on their absolute values (e.g. Platt scaling, isotonic).

Evaluate on two levels: how well the model ranks/scores (threshold-independent) and how well it classifies at a chosen threshold, plus whether its probabilities are trustworthy. Always match the metric to the problem, especially under class imbalance.

• Threshold-independent (ranking):

  • ROC-AUC: probability a random positive ranks above a random negative; can look optimistic under heavy imbalance.

  • PR-AUC (average precision): more informative when positives are rare.

• Threshold-dependent (classification): Confusion matrix, precision, recall, F1; accuracy alone is misleading on imbalanced data.

• Probability quality:

  • Log-loss / Brier score measure calibration and confidence, not just correctness.

  • A calibration curve checks whether predicted probabilities match observed frequencies.

• Validation discipline: Use cross-validation and a held-out test set; inspect coefficients/odds ratios for sanity and interpretability.

To predict churn, frame it as binary classification (churn = 1), fit logistic regression to output each customer's churn probability, then validate that the model both discriminates well and generalizes, choosing an operating threshold aligned to retention costs.

• Framing and features:

  • Define the churn label and observation window; engineer features (tenure, usage, complaints, billing).

  • Handle scaling and encoding so regularized coefficients are comparable.

• Modeling:

  • Fit logistic regression; interpret odds ratios to explain what drives churn.

  • Address imbalance via class weights or resampling if churners are rare.

• Validation steps:

  • Split (ideally time-based) to mimic predicting future churn; use cross-validation for stability.

  • Evaluate with PR-AUC/ROC-AUC and recall on churners; check calibration if probabilities feed decisions.

  • Select a threshold from the cost of contacting a customer vs the value of retaining one.

  • Monitor for drift after deployment; retrain as behavior changes.

The threshold only converts predicted probabilities into class labels: it trades precision against recall (and sensitivity against specificity) without changing the model itself.

• Lower threshold: More positives predicted: higher recall/sensitivity, lower precision, more false positives.

• Higher threshold: Fewer positives: higher precision, lower recall, more false negatives.

• What does not change: The coefficients, the probability ranking, and threshold-free metrics like AUC are unaffected.

• How to choose it: Pick based on the cost of each error type, not the default 0.5: use an ROC or precision-recall curve to select the operating point.

Accuracy hides where the model fails and is easily gamed by imbalanced data: a model can score high while being useless for the class you care about.

• Imbalance trap: With 95% negatives, always predicting the majority class gives 95% accuracy but zero recall on positives.

• Treats all errors equally: A false negative (missed fraud) and false positive can have very different costs; accuracy ignores this.

• Ignores probability quality: Logistic regression outputs probabilities; accuracy discards the confidence information a threshold-free view keeps.

• Better alternatives: Precision, recall, F1, ROC-AUC, PR-AUC, and log-loss depending on goals.

Each coefficient is the change in the log-odds of the positive class per one-unit increase in that feature, holding others fixed; exponentiating it gives the odds ratio, which is easier to interpret.

• On the log-odds scale: A coefficient β means the log-odds change by β per unit increase in x; the sign tells direction.

• Odds ratio = exp(β):

  • exp(β) > 1: odds multiply up (increases likelihood); exp(β) < 1: odds decrease; = 1 means no effect.

  • Example: exp(β)=1.5 means each unit raises the odds by 50%.

• Key nuance: It affects odds, not probability directly: the probability change depends on the starting point (nonlinear).

• Scaling matters: Coefficient magnitude depends on the feature's units/scale, so standardize before comparing importances.

The intercept is the log-odds of the positive class when all features equal zero; exponentiating it gives the baseline odds, and passing it through the sigmoid gives the baseline probability.

• Baseline meaning: It sets the model's prediction at the reference point where every predictor is 0.

• Convert to probability: Use 1 / (1 + exp(-β0)) to get the baseline probability.

• Often not meaningful literally: If all-zero features are impossible (e.g. age = 0), the intercept is just a fitting constant; centering/standardizing features makes it interpretable as the value at the mean.

Because logistic regression is linear in the log-odds, not in the probability: the sigmoid maps the linear predictor onto an S-shaped curve, so the same one-unit change moves probability a lot near the middle and very little near the extremes.

• Linearity lives on the log-odds scale:

  • A one-unit change in x changes the log-odds by a constant β, and multiplies the odds by exp(β) (constant).

  • But probability is p = 1/(1+e^-z), a nonlinear transform of that log-odds.

• The sigmoid is S-shaped:

  • Its slope p(1-p)·β is largest at p = 0.5 and shrinks toward 0 as p approaches 0 or 1.

  • So the marginal effect on probability depends on where you currently sit on the curve.

• Practical consequence: Report odds ratios (constant) or marginal effects evaluated at specific values, and don't claim a fixed probability change per unit.

A k-level categorical variable is encoded as k-1 dummies, and each dummy's coefficient is interpreted relative to the omitted reference category; including all k dummies plus an intercept causes the dummy variable trap (perfect collinearity).

• Reference category sets the baseline:

  • Each coefficient is the log-odds difference between that level and the reference, so exp(β) is the odds ratio versus the reference.

  • Changing the reference changes the numbers and their significance, but not the model's fit or predictions.

• The dummy variable trap:

  • Encoding all k levels while keeping an intercept makes the dummies sum to a constant, so they are perfectly collinear and coefficients are not identifiable.

  • Fix: drop one level (drop_first=True) or drop the intercept.

• Choosing a reference wisely:

  • Pick a meaningful, well-populated baseline; tiny reference groups give wide, unstable intervals.

  • Under L1/L2 regularization, one-hot encoding all levels (no drop) is sometimes preferred since the penalty resolves the redundancy.

Build the interval on the coefficient (log-odds) scale where the sampling distribution is approximately normal, then exponentiate the endpoints: an odds ratio interval that excludes 1 indicates a statistically significant effect.

• Compute on the log-odds scale:

  • Wald interval: β ± z·SE(β) (z=1.96 for 95%).

  • Exponentiate: OR interval is [exp(β - z·SE), exp(β + z·SE)].

• Why not build it directly on OR: The OR distribution is skewed; normality holds for β, so the exponentiated interval is asymmetric around the point OR (correctly so).

• Interpretation:

  • OR > 1 raises the odds, OR < 1 lowers them; an interval spanning 1 means the effect is not significant.

  • For small samples or separation, prefer profile-likelihood or bootstrap intervals over the Wald approximation.

Scaling matters because logistic regression's regularization penalty is applied to raw coefficient magnitudes, so features on larger numeric scales get penalized more heavily and unfairly, distorting the fit; it also speeds and stabilizes gradient-based optimization.

• Regularization is scale-sensitive:

  • A feature measured in small units needs a large coefficient for the same effect, which the penalty then shrinks disproportionately.

  • Standardizing (mean 0, unit variance) puts all coefficients on a comparable footing so the penalty treats features fairly.

• Optimization benefits: Similar scales give well-conditioned gradients, so solvers converge faster and more reliably.

• Without regularization: Unpenalized logistic regression is invariant to linear rescaling of the fit itself (coefficients just rescale), but scaling still helps convergence.

• Practical note: Fit the scaler on training data only and apply it to test/production data to avoid leakage.

L1 (Lasso) penalizes the sum of absolute coefficients and produces sparse solutions, while L2 (Ridge) penalizes the sum of squared coefficients and shrinks weights smoothly toward zero without eliminating them.

• L1 penalty λΣ|β|:

  • Its diamond-shaped constraint has corners on the axes, so the optimum often lands with some coefficients exactly 0.

  • Acts as built-in feature selection; good for high-dimensional, sparse problems.

• L2 penalty λΣβ²:

  • Smooth, differentiable everywhere; shrinks all coefficients but rarely to exactly 0.

  • Handles correlated predictors gracefully by spreading weight across them.

• Choosing between them:

  • Want a compact, interpretable model: L1; want stability with correlated features: L2.

  • Elastic Net mixes both when you want sparsity plus stability.

L1 regularization (Lasso) is the one that can drive coefficients exactly to zero, effectively performing feature selection.

• Why L1 zeros out weights: The absolute-value penalty is non-differentiable at 0 and its constraint region has corners on the axes, so solutions are pulled onto those axes (zero coefficients).

• Why L2 does not: The squared penalty shrinks coefficients asymptotically toward but never exactly to 0.

• Elastic Net: Combining L1 and L2 also produces sparsity while retaining L2's stability on correlated features.

Penalized logistic regression is used whenever plain maximum-likelihood fitting would overfit or fail to converge: many features, correlated predictors, small samples, or separation in the data.

• High-dimensional data: When features outnumber or rival observations (p ≈ n or p > n), the penalty makes the problem well-posed.

• Multicollinearity: L2 stabilizes coefficients that would otherwise be huge and erratic under correlation.

• Feature selection needed: L1 yields a sparse, interpretable model by dropping irrelevant predictors.

• Complete or quasi-separation: When classes are perfectly separable, unpenalized coefficients diverge to infinity; a penalty keeps them finite.

• General overfitting control: Any time validation performance lags training performance, with λ tuned by cross-validation.

In scikit-learn, C is the inverse of regularization strength: C = 1/lambda. So a small C means strong regularization and a large C means weak regularization, which is the opposite of how people expect a 'strength' knob to behave.

• What C controls:

  • It scales the penalty on coefficient size in the loss: smaller C shrinks weights harder toward zero.

  • Large C lets the model fit the data closely (higher variance, risk of overfitting).

• Why it trips people up:

  • Most textbooks parameterize by lambda (bigger = more regularization); scikit-learn inverts this.

  • People raise C hoping to reduce overfitting, but they're actually weakening the penalty.

• Practical note: tune C on a log scale (e.g. 0.001, 0.01, 0.1, 1, 10) and always scale features first, since the penalty is applied to raw coefficients.

They're not competing models: logistic regression is a model (the hypothesis and loss), while an SGD classifier is an optimization method that can fit that model. SGDClassifier with loss='log_loss' is logistic regression trained by stochastic gradient descent.

• Logistic regression (the model): Defined by the sigmoid/logit link and cross-entropy loss; LogisticRegression solves it with batch solvers (lbfgs, liblinear) to near-exact convergence.

• SGDClassifier (the optimizer):

  • Minimizes a chosen loss one (or a few) samples at a time; changing loss switches the model (log_loss = logistic, hinge = linear SVM).

  • Scales to huge/streaming datasets via partial_fit, but is more sensitive to learning rate and feature scaling.

• Bottom line: same objective, different training regime. Use LogisticRegression for moderate data, SGDClassifier for very large or online settings.

The warning means the solver hit its iteration cap before the gradient reached the convergence tolerance, so the reported coefficients aren't at the optimum. It's usually caused by unscaled features, too few iterations, or near-perfect separation, not a bug.

• Common causes:

  • Unscaled features: wildly different ranges make the loss surface ill-conditioned and slow to descend.

  • Too few iterations: default max_iter is too low for the problem.

  • Perfect/quasi separation: coefficients try to grow to infinity, so the solver never settles.

  • Multicollinearity: correlated features create a flat, ambiguous minimum.

• Fixes:

  • Scale first with StandardScaler: usually the single biggest fix.

  • Raise max_iter (e.g. to 1000+).

  • Increase regularization (lower C) to tame separation and shrink coefficients.

  • Try a solver suited to the data, or remove redundant features.

It reweights the loss so each class contributes proportionally to its inverse frequency, making the minority class effectively as important as the majority: this counters the bias where a model minimizes total error by favoring the dominant class.

• The problem it solves: With imbalanced data, unweighted log-loss is dominated by the majority class, so the decision boundary drifts and minority recall collapses.

• How the weights are computed:

  • sklearn uses n_samples / (n_classes * np.bincount(y)), giving rarer classes larger weights.

  • Each sample's contribution to the loss is scaled by its class weight during optimization.

• Effect on the model:

  • Shifts the intercept/boundary toward better minority detection (higher recall) at the cost of more false positives.

  • Roughly equivalent to oversampling the minority class, but without duplicating data.

• Caveat: It distorts the predicted probabilities relative to true base rates, so recalibrate if you need calibrated outputs.

Improving logistic regression is mostly about feature quality and regularization, since the model itself is simple: give it better inputs, control overfitting, and handle data issues like scaling and imbalance.

• Feature engineering:

  • Add interaction and polynomial terms since the model is linear in log-odds and can't capture them alone.

  • Transform skewed features (log, binning) and encode categoricals sensibly.

• Regularization: Tune the strength C and choose L1 (sparsity/feature selection) vs L2 (shrinkage) via cross-validation.

• Scale the features: Standardize inputs so regularization is fair and gradient-based solvers converge faster.

• Handle class imbalance: Use class_weight='balanced', resampling, or adjust the decision threshold instead of defaulting to 0.5.

• Fix data and multicollinearity: Remove/redundant highly correlated features that make coefficients unstable; clean outliers.

• Tune threshold to the metric: Optimize for AUC, F1, or recall depending on the business cost, not raw accuracy.

Logistic regression is literally a single neuron: it takes a weighted sum of inputs plus a bias, applies a sigmoid activation, and outputs a probability, which is exactly one perceptron unit with a logistic activation.

• The mapping: Weights = coefficients, bias = intercept, activation = sigmoid, loss = binary cross-entropy (same as log-loss).

• Same training machinery: Both trained by gradient descent on the same convex log-loss, so the update rules coincide.

• Why the mental model helps:

  • Softmax regression = a single layer of such neurons (multiclass output).

  • Adding hidden layers with non-linear activations is the natural generalization to deep nets.

  • It demystifies neural nets: you already understand the atomic building block.

Class imbalance biases logistic regression toward the majority class: because the model minimizes overall log-loss, it can achieve low loss by mostly predicting the majority, pushing the intercept and predicted probabilities down and hurting recall on the minority (often the class you actually care about).

• What actually breaks:

  • Coefficients are usually still fine; the problem is the calibrated probabilities cluster low and the default 0.5 threshold rarely flags the minority.

  • Accuracy becomes misleading: a 99%/1% split scores 99% by predicting all-negative.

• Fix the evaluation first:

  • Judge with precision/recall, F1, PR-AUC, or ROC-AUC instead of accuracy.

  • Tune the decision threshold to your cost tradeoff rather than defaulting to 0.5.

• Reweight the loss: Use class weights (e.g. class_weight='balanced') so minority errors cost more; conceptually equivalent to duplicating minority rows.

• Resample the data: Oversample the minority (including SMOTE) or undersample the majority; apply only on the training fold to avoid leakage.

• Adjust the intercept: After weighting or resampling, probabilities are distorted; recalibrate (or correct the intercept back to the true base rate) if you need well-calibrated probabilities.

• Caveat: If the minority is genuinely rare and separable, imbalance may not need fixing at all; measure before intervening.

Saturation is when the sigmoid input is very large in magnitude, pushing its output near 0 or 1 where the curve is almost flat. In that region the gradient nearly vanishes, which slows or destabilizes fitting.

• What saturation is: For large |z|, σ(z) ≈ 0 or 1 and its derivative σ(z)(1-σ(z)) ≈ 0.

• Practical issues:

  • Vanishing gradients: updates become tiny even when predictions are wrong, so learning stalls.

  • Perfectly separable data drives weights toward infinity (no finite MLE), causing overconfident probabilities.

  • Numerical overflow in exp if not handled carefully.

• Mitigations:

  • Feature scaling/standardization to keep z in a reasonable range.

  • L2 regularization to bound weights and prevent runaway saturation.

With two classes, softmax's over-parameterization lets you subtract one class's weights from both. What remains is a single linear score fed through the sigmoid, which is exactly binary logistic regression.

• The reduction:

  • Softmax for class 1: P(y=1) = exp(z_1)/(exp(z_0)+exp(z_1)).

  • Divide numerator and denominator by exp(z_0): P(y=1) = 1/(1+exp(-(z_1-z_0))).

  • Define z = z_1 - z_0 = (w_1-w_0)·x + (b_1-b_0), giving P(y=1) = σ(z).

• Takeaway: Only the difference of the two weight vectors is identifiable, which is the single weight vector of binary logistic regression.

You start from the Bernoulli likelihood, take its log, then differentiate with respect to the weights. The derivative sets up the score equations that an optimizer solves.

1. Model: pᵢ = σ(wᵀxᵢ) = 1/(1+e^(-wᵀxᵢ)).

2. Likelihood: L(w) = ∏ pᵢ^yᵢ (1-pᵢ)^(1-yᵢ).

3. Log-likelihood: ℓ(w) = Σ [ yᵢ log pᵢ + (1-yᵢ) log(1-pᵢ) ].

4. Use the key identity ∂pᵢ/∂w = pᵢ(1-pᵢ)xᵢ; the algebra collapses neatly.

5. Gradient: ∇ℓ(w) = Σ (yᵢ - pᵢ) xᵢ. Setting it to zero gives the score equations.

These equations are nonlinear in w, so there is no closed form: the estimate is obtained numerically (e.g. Newton-Raphson / IRLS).

MLE is the natural fitting principle because logistic regression is a probabilistic (Bernoulli) model, and maximizing its log-likelihood is mathematically identical to minimizing binary cross-entropy: they are the same objective up to a sign and scaling.

• Why MLE: The model outputs probabilities, so choosing weights that make the observed labels most likely is principled and gives statistically consistent estimates.

• The equivalence:

  • Log-likelihood: ℓ = Σ [ yᵢ log pᵢ + (1-yᵢ) log(1-pᵢ) ].

  • Log-loss: BCE = -(1/N) Σ [ yᵢ log pᵢ + (1-yᵢ) log(1-pᵢ) ].

  • So maximizing ℓ = minimizing BCE (negative mean log-likelihood).

• Takeaway: "MLE" is the statistician's framing; "minimizing log-loss" is the ML framing of the exact same optimization.

Yes: because log-loss is smooth and convex, several optimizers beat plain gradient descent, especially by using curvature (second-order) information or being tuned for large or sparse data.

• Second-order methods:

  • Newton-Raphson / IRLS: use the Hessian for very fast quadratic convergence on modest-sized problems.

  • Quasi-Newton like L-BFGS: approximate the Hessian, memory-efficient, a common default (scikit-learn's lbfgs).

• Coordinate descent: Efficient for L1 penalties (liblinear), updating one coordinate at a time.

• Stochastic / mini-batch variants: SGD, saga: scale to huge datasets where full-batch or Hessian methods are too costly.

• Rule of thumb: Small/medium data: L-BFGS or Newton; very large/sparse: SAGA/SGD.

The logistic regression cost (negative log-likelihood / log-loss) is convex, smooth, and differentiable, which makes optimization well-behaved and guaranteed to reach a global minimum.

• Convex: No local minima traps; any optimizer that converges reaches the global optimum. (Squared error with a sigmoid would be non-convex, which is why log-loss is used.)

• Smooth and differentiable: Clean gradient Σ(pᵢ - yᵢ)xᵢ and a positive semidefinite Hessian enable gradient- and Newton-based methods.

• Penalizes confident mistakes heavily: Loss → ∞ as a confident wrong prediction approaches 0/1, encouraging calibrated probabilities.

• Caveats: On perfectly separable data the MLE diverges (weights → ∞); regularization keeps the solution finite and adds strong convexity.

Both minimize log-loss, but gradient descent is a first-order method that follows the gradient, while Newton-Raphson is a second-order method that also uses the Hessian to take better-scaled steps and converge in far fewer iterations.

• Gradient descent:

  • Update: w ← w - η ∇ℓ. Uses only the gradient.

  • Cheap per iteration but needs a learning rate and many steps; scales well to huge data (as SGD).

• Newton-Raphson:

  • Update: w ← w - H⁻¹ ∇ℓ, where H is the Hessian.

  • Quadratic convergence (few iterations), no learning rate; for logistic regression it becomes IRLS.

  • Cost: forming and inverting H is O(d³) per step, expensive in high dimensions.

• Practical middle ground: Quasi-Newton (L-BFGS) approximates the Hessian, getting fast convergence without the full inversion cost.

The gradient of the log-loss with respect to the weights is simply the sum over samples of the prediction error times the input features: it has the same clean form as linear regression's gradient, with the sigmoid "hidden" inside the prediction.

• What the terms mean:

  • (pᵢ - yᵢ) is the residual (predicted probability minus true label).

  • Multiply by xᵢ so each feature is nudged in proportion to error and feature value.

• Why it is elegant:

  • Identical shape to OLS's gradient even though the model is nonlinear: the messy sigmoid derivative cancels neatly against the log-loss.

  • It is intuitive: bigger error means bigger update, and correct confident predictions contribute ~0.

  • It vectorizes cleanly to Xᵀ(p - y), making implementation trivial.

IRLS is Newton's method applied to logistic regression's log-likelihood, rewritten so each step looks like a weighted least squares regression. It's the classic algorithm behind glm() and repeatedly refits a reweighted linear problem until the coefficients converge.

• The core idea:

  • Newton's update w ← w - H⁻¹g can be algebraically arranged into a weighted least squares solve.

  • Each iteration builds a working response z (a linearized target) and weights S = diag(p(1-p)), then solves w = (XᵀSX)⁻¹XᵀSz.

• Why it works well:

  • Uses second-order (curvature) information, so it converges quadratically: very few iterations near the optimum.

  • Convex loss guarantees it heads to the global optimum.

• Relation to fitting:

  • It is the standard MLE fitting method for logistic (and general GLM) regression in statistical software.

  • Cost: forming and inverting XᵀSX is expensive for many features, so large/sparse problems often use gradient methods (SGD, L-BFGS) instead.

Multicollinearity is when predictors are highly correlated, so the model can't separate their individual effects. In logistic regression it inflates standard errors and destabilizes coefficients, hurting interpretation even when overall predictions stay fine.

• What happens to coefficients:

  • Estimates become unstable: large magnitude, unexpected signs, big swings when data changes slightly.

  • Standard errors inflate, so Wald tests lose power and predictors look insignificant despite being predictive.

  • The Hessian XᵀSX becomes near-singular, making the MLE ill-conditioned.

• How to detect it:

  • Variance Inflation Factor (VIF): values above ~5-10 flag trouble.

  • High pairwise correlations or a near-singular design matrix.

• How to address it:

  • Drop or combine redundant features; use domain knowledge.

  • Apply regularization (L2 shrinks correlated coefficients toward each other and stabilizes them).

  • Use dimensionality reduction (PCA) when appropriate.

• Key nuance: multicollinearity harms interpretation and inference, not necessarily predictive accuracy.

You test whether predictors matter using significance tests on coefficients (Wald, likelihood ratio) and compare whole models with information criteria (AIC/BIC). The likelihood ratio test is generally the most reliable; AIC/BIC are for model selection rather than per-coefficient significance.

• Wald test:

  • Tests one coefficient: z = β̂ / SE(β̂), squared is chi-square with 1 df.

  • Fast (single fit) but unreliable with large coefficients or separation (understates significance).

• Likelihood ratio test (LRT):

  • Compares nested models: LR = 2(logL_full - logL_reduced), chi-square with df = number of dropped parameters.

  • Requires fitting both models but is more accurate than Wald, especially in small samples.

• AIC / BIC:

  • For comparing non-nested or many models: AIC = -2logL + 2k, BIC = -2logL + k·log(n).

  • Lower is better; BIC penalizes complexity more, favoring smaller models. These rank models, they don't give p-values.

• Practical rule: use LRT for individual/grouped predictor significance, AIC/BIC for overall model selection.