59 Regularization Interview Questions and Answers (2026)
Every model that ships to production has to generalize, and Regularization is how you make that happen. Interviewers no longer accept "L1 makes things sparse" as an answer. They'll ask why the intercept goes unpenalized, what happens to coefficients as lambda blows up, and how you'd spot an over-regularized model, so walking in shaky gets exposed fast.
This guide gives you 59 questions with tight, interview-ready answers and code where it actually helps. They're ordered Junior → Mid → Senior, starting with the bias-variance fundamentals and building up to regularization paths, structured penalties, and priors. Work through them in order and you'll walk in able to reason through any Regularization question they throw at you.
Q1.Why does adding a penalty to the loss function improve model generalization?
A penalty makes the optimizer trade a little training fit for simpler parameters. Simpler models are less able to memorize noise in the training set, so their variance drops more than their bias rises, and test error falls.
Constrains the hypothesis space:
Minimizing loss + λ·Ω(w) is equivalent to minimizing loss subject to Ω(w) ≤ t. The model can only use a smaller set of functions.
Large weights let a model produce sharp, wiggly functions that fit individual noisy points. Small weights force smoother functions.
Bias-variance trade-off:
Shrinkage adds some bias, but it greatly reduces how much the fitted model changes across different training samples (variance).
When the model is flexible relative to the amount of data, the drop in variance dominates, so expected test error goes down.
Bayesian view: The penalty acts as a prior: L2 corresponds to a Gaussian prior and L1 to a Laplace prior. The result is a MAP estimate that encodes the belief that weights should be small.
Numerical stability: L2 makes XᵀX + λI invertible. This stabilizes estimates when features are collinear or when there are more features than samples (p > n).
Caveat: it only helps when λ is tuned, for example by cross-validation. Too large a λ causes underfitting.
Q2.What signs indicate that a model is over-regularized?
An over-regularized model underfits: it performs poorly on the training data itself, and validation performance looks almost the same because the model is too constrained to capture real signal.
Error patterns:
Training error is high and close to validation error (high bias, small generalization gap).
Lowering λ improves both training and validation scores.
Learning curves plateau early at a poor level, and adding more data doesn't help.
Parameter symptoms:
Coefficients are shrunk close to zero, or (with L1) almost all are exactly zero, including features you know matter.
Predictions collapse toward the mean or the majority class, and predicted probabilities sit near the base rate.
Diagnostics:
On a validation curve over λ, the chosen value lies on the right side of the U, past the minimum.
Residuals show clear structure, such as unmodeled trends or nonlinearity.
In neural networks: very high dropout or weight decay leads to slow or stalled training loss.
Fix: reduce λ or dropout, or remove extra constraints, and re-tune with cross-validation.
Q3.What happens to the model coefficients as the regularization parameter lambda approaches infinity?
As λ → ∞, the penalty dominates the data-fit term, so all penalized coefficients are driven to zero. The model reduces to the intercept alone, which predicts the mean of the target for regression or the base rate for classification.
Ridge (L2): Coefficients shrink smoothly toward zero but are never exactly zero for any finite λ: w = (XᵀX + λI)⁻¹Xᵀy → 0.
Lasso (L1):
Coefficients hit exactly zero one by one as λ grows.
There is a finite threshold, λ_max = max|xⱼᵀy|/n (for centered data), above which every coefficient is zero.
Elastic Net combines both behaviors: some coefficients are set exactly to zero, and all of them shrink.
Bias-variance consequence: Variance goes to zero (the predictions no longer depend on the training sample), while bias is maximal. This is extreme underfitting.
The intercept is usually unpenalized, so it stays at the mean of y rather than going to zero.
Q4.What is the difference between L1 and L2 regularization?
L1 penalizes the sum of absolute weights and produces sparse models, setting some weights exactly to zero. L2 penalizes the sum of squared weights and shrinks all weights smoothly without zeroing them.
Penalty form:
L1 (Lasso): λ Σ|wⱼ|
L2 (Ridge / weight decay): λ Σwⱼ²
Why L1 gives sparsity:
The gradient of the L1 penalty has constant magnitude λ, so it keeps pushing a weight even when the weight is small. It drives weights all the way to 0, and the solution uses soft-thresholding.
The L2 gradient is 2λw, which fades as w shrinks, so weights get small but rarely reach exactly zero.
Geometrically, the L1 constraint region is a diamond with corners on the axes. Loss contours usually touch it first at a corner, where some coordinates are zero.
Correlated features:
L1 tends to pick one feature from a correlated group somewhat arbitrarily, which makes its selection unstable.
L2 spreads weight evenly across the group, which is more stable.
Optimization:
L2 is smooth and has a closed form for linear regression.
L1 is non-differentiable at 0, so it needs coordinate descent, proximal methods, or subgradients.
Priors: L1 corresponds to a Laplace prior and L2 to a Gaussian prior.
When to use each:
L1 when you expect few relevant features or want built-in feature selection.
L2 when many features each contribute a little, or when features are collinear.
Elastic Net when you want both.
Q5.Why should features be standardized before applying an L1 or L2 penalty?
The penalty acts on raw coefficient magnitudes, and those depend on each feature's units. Without standardization, features are penalized unequally for arbitrary reasons, so both the amount of shrinkage and which features get selected depend on the measurement scale.
Scale changes coefficient size:
Measure a feature in meters instead of kilometers and its coefficient becomes 1000× smaller for the same effect.
Unpenalized OLS doesn't care, because predictions are unchanged. A penalty does care.
Unequal treatment:
Features with small numeric ranges need large coefficients, so they are penalized heavily and shrunk or dropped.
Features with large ranges need tiny coefficients, so they are barely penalized.
With L1, this can decide which features survive feature selection.
Standardizing (zero mean, unit variance) puts every coefficient on a common scale, so λ means the same thing for each feature.
It also improves optimization conditioning, for example for coordinate or gradient descent.
Practical notes:
Fit the scaler on training data only, for example inside a Pipeline, to avoid leakage.
Some libraries (like glmnet) standardize internally. scikit-learn's Ridge and Lasso do not.
Q6.What is ridge regression?
Ridge regression is linear least squares with an L2 penalty on the coefficients. It shrinks weights toward zero, trading a little bias for a large reduction in variance, which is especially useful with correlated features or many predictors.
Objective:
min_w ||y − Xw||² + λ||w||², where λ ≥ 0 controls the strength of the penalty.
λ = 0 gives OLS. As λ → ∞, w → 0.
Key properties:
Closed form: w = (XᵀX + λI)⁻¹Xᵀy. It is always unique for λ > 0.
It shrinks coefficients but doesn't set them exactly to zero, so it performs no feature selection (unlike Lasso).
It stabilizes estimates under multicollinearity and works when p > n.
Interpretations:
Constrained: minimize RSS subject to ||w||² ≤ t.
Bayesian: the MAP estimate under a Gaussian prior w ~ N(0, σ²/λ · I).
Practical notes:
Standardize features first: the penalty is scale-dependent.
Don't penalize the intercept (center the data instead).
Choose λ by cross-validation (e.g. RidgeCV in scikit-learn).
Q7.What is lasso regression?
Lasso (Least Absolute Shrinkage and Selection Operator) is linear regression with an L1 penalty on the coefficients, which both shrinks them and sets some exactly to zero, performing automatic feature selection.
Objective:
min_β (1/2n)‖y − Xβ‖² + λ‖β‖₁, where ‖β‖₁ = Σ|β_j|.
λ controls strength: λ = 0 is OLS; large λ drives all coefficients to zero.
Key property: sparsity: The non-differentiable kink of |β_j| at zero lets weak coefficients land exactly at 0, yielding simpler, interpretable models.
Fitting:
No closed form in general; solved by coordinate descent with soft thresholding (e.g. sklearn.linear_model.Lasso, glmnet) or LARS for the full path.
Choose λ by cross-validation (LassoCV).
Practical notes:
Standardize features first: the penalty is scale-dependent.
When p > n, it selects at most n features.
Unstable with correlated features (picks one arbitrarily) and biases large coefficients toward zero; elastic net or relaxed lasso help.
Q8.What is Elastic Net regularization?
Elastic net adds a weighted mix of L1 and L2 penalties to the loss. It keeps lasso's sparsity while adding ridge's stability and its tendency to group correlated features together.
Objective:
Loss + λ[α||β||₁ + (1−α)/2·||β||₂²].
λ sets the overall strength. α sets the mix: α = 1 is lasso, α = 0 is ridge.
In scikit-learn, alpha is λ and l1_ratio is α.
What the L1 part gives: Exact zeros, so the model performs feature selection.
What the L2 part gives:
Strict convexity, so the solution is unique even when p > n or features are collinear.
A grouping effect: highly correlated features get similar coefficients instead of lasso's arbitrary pick-one behavior.
It can select more than n features.
Practical notes:
Standardize features first, because both penalties are scale-sensitive.
The original "naive" elastic net (Zou & Hastie, 2005) shrinks twice, so the authors rescale coefficients by (1 + λ₂).
Fit with coordinate descent along a λ path (glmnet, sklearn.linear_model.ElasticNet).
Q9.Why must the regularization parameter lambda be selected on validation data rather than training data?
lambda be selected on validation data rather than training data?Training error almost always falls as λ falls, because a less constrained model can fit the training data better. Selecting on training data would therefore always pick λ ≈ 0, which is no regularization at all. λ exists to optimize generalization, so it must be chosen using data the fit did not see.
Training error is a biased estimate:
It is optimistic, and the optimism grows with model flexibility (effective degrees of freedom).
Lowering λ raises flexibility, so training error rewards overfitting.
Validation error estimates generalization:
It traces the U-shaped bias–variance curve. Large λ underfits, small λ overfits, and the minimum sits in between.
K-fold cross-validation reduces the noise of a single split.
Keep the test set separate:
Tuning λ on validation data makes validation error optimistic too, because you picked the minimum.
Report performance on an untouched test set, or use nested CV.
Avoid leakage:
Fit preprocessing such as scaling or feature selection inside each training fold.
Scaling matters especially here, because penalties depend on feature scale.
Alternatives: information criteria (AIC, BIC) or generalized CV. These adjust training error for complexity analytically, but they still don't simply minimize raw training error.
Q10.How does a dropout layer behave differently in training mode versus evaluation mode?
In training mode, dropout randomly zeroes each activation with probability p and scales the survivors by 1/(1−p). In evaluation mode, it is the identity: nothing is dropped and nothing is scaled, so outputs are deterministic.
Training mode:
A fresh Bernoulli mask is sampled every forward pass, so each minibatch sees a different thinned network.
With inverted dropout, survivors are multiplied by 1/(1−p) so the expected activation is unchanged.
Gradients flow only through the kept units.
Evaluation mode: The full network is used. Because scaling was already done in training, no correction is needed at inference.
How you switch modes:
PyTorch: model.train() / model.eval(). These set the training flag on every nn.Dropout (and on BatchNorm).
Keras: the training argument. fit() sets it to True, while predict() and evaluate() set it to False.
Common pitfalls:
Forgetting model.eval() gives noisy, non-reproducible predictions and underestimated validation accuracy.
model.eval() does not disable autograd. Wrap inference in torch.no_grad() for that.
Intentional exception: Monte Carlo dropout keeps dropout active at inference and averages several passes to estimate uncertainty.
Q11.How does dropout prevent overfitting in deep neural networks?
Dropout randomly removes units during training, so no unit can rely on specific other units being present. This forces redundant, independently useful features, and it effectively trains an exponential ensemble of weight-sharing subnetworks that are averaged at test time.
Breaks co-adaptation:
Without dropout, units can form fragile combinations that only work together to fit noise in the training set.
With random removal, each feature must be useful in many contexts, which yields more robust, distributed representations.
Implicit ensemble:
A network with n droppable units contains up to 2ⁿ thinned subnetworks, all sharing weights.
Using the full network at test time (with the right scaling) approximates averaging their predictions. Ensembles reduce variance, which is the source of overfitting.
Noise injection:
Multiplicative noise makes it hard to memorize exact training examples.
For linear or GLM models, dropout is roughly equivalent to an adaptive L2 penalty that is scaled by feature variance.
Practical notes:
Typical rates: about 0.5 for large fully connected layers, 0.1–0.3 for transformers and embeddings, and lower or none in conv layers.
Costs: noisier gradients and slower convergence. Training loss sits above validation loss.
It is often redundant alongside heavy BatchNorm and data augmentation in CNNs, and stacking dropout before BN can cause variance shift.
Q12.What is weight decay in neural network training?
Weight decay shrinks every weight toward zero by a small fraction at each update, w ← (1 − ηλ)w − η∇L. This biases the model toward smaller weights, and therefore smoother, less overfit functions.
Relation to L2 regularization:
With plain SGD, adding (λ/2)||w||² to the loss gives exactly the same update, so the two are equivalent.
With adaptive optimizers (Adam), they differ. An L2 gradient gets divided by the per-parameter adaptive scale, so weights with large gradient history are barely decayed.
AdamW decouples the decay: it subtracts ηλw directly, outside the adaptive step. This is the standard choice for transformers.
Why it helps:
Small weights limit how sharply the function can change, which reduces effective capacity and variance.
Bayesian view: a Gaussian prior on the weights.
With BatchNorm or LayerNorm, the weights are scale-invariant. Decay then mainly raises the effective learning rate rather than restricting capacity.
Practice:
Typical λ: about 1e-4 to 5e-4 for SGD on CNNs, and about 0.01–0.1 for AdamW.
Usually exclude biases and normalization gains or offsets from decay.
Q13.What is early stopping?
Early stopping halts training when performance on a held-out validation set stops improving, then keeps the best checkpoint. It regularizes by limiting how far the optimizer can move from its initialization toward fitting noise.
How it works:
Evaluate a validation metric (usually loss) after each epoch or every N steps.
Track the best value seen so far. If it hasn't improved by at least min_delta for patience evaluations, stop.
Restore the best weights rather than the last ones.
Why it regularizes:
Training error keeps falling, but validation error forms a U-shape once the model starts memorizing. Stopping at the minimum picks the bias-variance sweet spot.
For a linear model with quadratic loss trained by gradient descent from zero, stopping after τ steps behaves like L2 regularization with λ ≈ 1/(ητ).
Pros:
Nearly free, and it also saves compute.
It tunes the effective capacity automatically.
Cons:
It consumes validation data.
Noisy metrics need a sensible patience setting.
Repeated checking can overfit the validation set.
It interacts with LR schedules: a loss plateau before an LR drop can trigger a premature stop.
Under double descent, validation loss can improve again later.
Common variant: find the best epoch count, then retrain on train plus validation data for that many epochs.
Q14.Why is the intercept term typically excluded from the regularization penalty?
The intercept only sets the baseline level of predictions; it does not affect model complexity or overfitting. Penalizing it would bias predictions toward zero for no generalization benefit, and it would make the model depend on where the target's origin happens to be.
No complexity contribution:
Overfitting comes from how strongly the output reacts to the inputs, which the slopes control.
The intercept is a constant shift and cannot fit noise in x.
Shift invariance:
If you add a constant c to y, the ideal model should just shift by c.
A penalized intercept would resist that shift, so results would depend on arbitrary choices such as Celsius vs. Kelvin.
Strong-penalty limit:
With an unpenalized intercept, a large λ makes the model predict the mean of y, which is a sensible baseline.
If the intercept were penalized, predictions would collapse to 0.
Implementation:
Center X and y, fit the penalized slopes, then recover the intercept as b = ȳ − x̄ᵀw.
Watch out: some solvers (e.g. liblinear in scikit-learn's LogisticRegression) do penalize the intercept. In neural networks, biases are often excluded from weight decay too.
Q15.Why does adding a regularization penalty introduce bias into parameter estimates?
The penalty pulls estimates toward zero regardless of what the data says, so their expected value no longer equals the true parameter. That bias is the deliberate price paid for lower variance.
The unpenalized estimator is unbiased: Under standard assumptions, OLS satisfies E[ŵ] = w.
Ridge shrinks systematically:
Ridge estimate: ŵ = (XᵀX + λI)⁻¹Xᵀy.
So E[ŵ] = (XᵀX + λI)⁻¹XᵀX·w, which is not w for λ > 0.
With orthonormal features, each coefficient is simply scaled by 1/(1+λ).
Lasso both shrinks and thresholds:
Every nonzero coefficient is reduced by about λ (soft-thresholding).
Small true effects can be set to zero entirely.
Why accept the bias:
MSE = bias² + variance. Shrinkage reduces variance a lot, especially with collinearity or small n.
For a suitable λ, total MSE falls below that of OLS. Ridge provably achieves this for some λ > 0, and Stein's paradox is a related result.
Bayesian framing: the prior centered at zero pulls the estimate away from the pure likelihood maximum.
Consequences:
Penalized coefficients are not suitable for unbiased effect estimation, and standard confidence intervals don't apply.
Remedies include refitting OLS on the features Lasso selected (relaxed lasso) or using debiased lasso methods.
Q16.What signs indicate that a model is under-regularized even with a penalty enabled?
If the model still overfits with a penalty switched on, the penalty is too weak or not acting where it should. The classic sign is a large, persistent gap between training and validation performance.
Performance signals:
Training error is very low while validation or test error is much higher.
Validation loss bottoms out and then rises during training while training loss keeps falling.
Increasing λ improves validation score, which means the chosen λ sits on the left side of the validation curve.
Cross-validation fold scores vary widely.
Parameter signals:
Coefficients are large, possibly with opposite signs on correlated features that cancel each other out.
Coefficients are unstable across bootstrap resamples or random seeds.
With L1, almost no coefficients are zeroed, even ones you'd expect to be noise.
Classifiers produce overconfident, poorly calibrated probabilities.
Why the penalty may be ineffective:
λ is tiny relative to the loss scale. For example, scikit-learn's C is the inverse of λ, so a large C means weak regularization.
Features are unstandardized, so large-scale features escape the penalty.
The model has far too much capacity for the data, such as p ≫ n or a huge network.
In neural networks, the L2 term in Adam is diluted by adaptive scaling, so AdamW is preferred.
Fixes:
Tune λ with cross-validation over a log-scale grid.
Standardize features.
Add complementary regularization such as dropout, early stopping, or data augmentation, or reduce model capacity.
Q17.What is the closed-form solution of ridge regression?
The solution is ŵ = (XᵀX + λI)⁻¹Xᵀy. It comes from setting the gradient of ||y − Xw||² + λ||w||² to zero. Adding λI makes the matrix invertible for any λ > 0.
Derivation:
Gradient: −2Xᵀ(y − Xw) + 2λw = 0
Rearrange: (XᵀX + λI)w = Xᵀy
The Hessian 2(XᵀX + λI) is positive definite, so this is the unique global minimum.
SVD view (with X = UDVᵀ):
ŵ = V · diag(dᵢ / (dᵢ² + λ)) · Uᵀy
Compared with OLS (1/dᵢ), each direction is shrunk by dᵢ² / (dᵢ² + λ). Low-variance directions are shrunk the most.
Practical details:
Center X and y so the intercept isn't penalized: the intercept is ȳ − x̄ᵀŵ.
Use a linear solve or Cholesky rather than an explicit inverse.
When p ≫ n, use the dual form ŵ = Xᵀ(XXᵀ + λI)⁻¹y, which needs an n×n solve instead of p×p.
Q18.Why can ridge regression still produce a unique solution when p > n, whereas ordinary least squares fails?
p > n, whereas ordinary least squares fails?When p > n, XᵀX has rank at most n < p, so it is singular and OLS has infinitely many minimizers. Ridge adds λI, which makes every eigenvalue strictly positive. The objective then becomes strictly convex, so it has exactly one minimizer.
Why OLS fails:
X has a nontrivial null space. Adding any null-space vector to a solution leaves Xw (and the RSS) unchanged.
There are infinitely many interpolating solutions with zero training error, and (XᵀX)⁻¹ doesn't exist.
Why ridge works:
The eigenvalues of XᵀX + λI are dᵢ² + λ, where zero eigenvalues become λ > 0. The matrix is positive definite and invertible.
Equivalently, the penalty λ||w||² is strictly convex. It breaks ties among the equal-RSS solutions by preferring the smallest norm.
Structure of the solution:
The ridge solution lies in the row space of X: ŵ = Xᵀ(XXᵀ + λI)⁻¹y. Any null-space component would only add to the penalty.
As λ → 0⁺ it converges to the minimum-norm solution X⁺y (the pseudoinverse).
Caveat: uniqueness doesn't mean the model is correct. Ridge picks one estimate through bias, and good generalization still depends on choosing λ well.
Q19.How does Ridge regression stabilize coefficient estimates in the presence of multicollinearity?
Multicollinearity makes some eigenvalues of XᵀX nearly zero. OLS divides by them, so coefficient variance explodes along those directions. Ridge adds λ to every eigenvalue, which caps that variance at the cost of a small bias.
The problem in OLS:
Var(ŵ_OLS) = σ²(XᵀX)⁻¹ = σ² Σ vᵢvᵢᵀ / dᵢ². A tiny dᵢ gives a huge variance.
Symptoms: large, opposite-signed coefficients that flip with small data changes.
What ridge does:
Shrinkage per principal direction: dᵢ² / (dᵢ² + λ). High-variance directions are barely touched, while near-degenerate ones are shrunk heavily toward 0.
Variance becomes σ² Σ dᵢ² vᵢvᵢᵀ / (dᵢ² + λ)², which is bounded even as dᵢ → 0.
The condition number improves from d_max²/d_min² to (d_max² + λ)/(d_min² + λ), so the solve is also numerically stable.
Effect on correlated features: Instead of one huge positive and one huge negative weight, ridge spreads similar, moderate weights across the correlated group.
The trade-off:
It introduces bias, but for a suitable λ the total MSE is lower than OLS. A ridge λ that beats OLS in MSE always exists (Hoerl & Kennard).
Pick λ by cross-validation, and standardize features first.
Q20.How does Ridge regression affect the weights of two perfectly collinear features?
For identical (perfectly collinear, same-scale) features, ridge splits the weight equally between them. OLS can't choose among infinitely many splits, but ridge picks the one with the smallest L2 norm.
Why an equal split:
If x₁ = x₂, the fit depends only on w₁ + w₂ = c.
Minimizing w₁² + w₂² under that constraint gives w₁ = w₂ = c/2.
With k identical copies, each gets about 1/k of the weight that a single feature would receive.
Scaled copies (x₂ = a·x₁):
The fit depends on w₁ + a·w₂, and the min-norm solution has (w₁, w₂) ∝ (1, a).
So the split depends on feature scale, which is another reason to standardize.
Total effect:
The combined weight is still shrunk by λ.
Duplicating a feature effectively reduces its penalty, because splitting halves the squared norm cost.
Contrast with Lasso:
Every split with the same sign has the same L1 norm, so the solution is non-unique.
In practice the solver often puts all the weight on one feature, chosen arbitrarily.
Elastic net adds an L2 term to recover ridge's grouping behavior.
Q21.When would Ridge regression be expected to outperform Lasso in predictive accuracy?
Ridge tends to win when the true signal is dense: many predictors each contribute small or moderate effects, especially when they are correlated. Lasso wins when only a few predictors truly matter.
Dense true coefficients:
Lasso zeroes out real but small effects, adding bias.
Ridge keeps all of them, shrinking smoothly.
Bayesian view: a Gaussian prior (ridge) matches many small effects, while a Laplace prior (Lasso) matches a few large ones.
Highly correlated predictors:
Lasso arbitrarily picks one feature from a correlated group. That selection is unstable across samples, which increases variance.
Ridge averages across the group, which reduces noise.
Signal spread over directions, not features: If the truth isn't sparse in the original feature basis (e.g. it is a combination of many features), Lasso's selection doesn't help.
When Lasso wins instead:
A sparse truth with a few large effects, particularly when p ≫ n.
Cases where interpretability or feature selection is needed.
In practice the structure is unknown, so compare by cross-validation, or use elastic net to interpolate between the two.
Q22.How can ridge regression be reproduced as ordinary least squares on an augmented dataset?
Append p pseudo-rows √λ·I to X and p zeros to y. The OLS residual sum of squares on this augmented data equals ||y − Xw||² + λ||w||², so plain least squares returns the ridge solution.
Construction:
X̃ = [X; √λ·I] with shape (n+p)×p, and ỹ = [y; 0] of length n+p.
||ỹ − X̃w||² = ||y − Xw||² + ||0 − √λ·w||² = RSS + λ||w||²
Check via the normal equations:
X̃ᵀX̃ = XᵀX + λI and X̃ᵀỹ = Xᵀy, which gives exactly the ridge equations.
X̃ has full column rank for λ > 0, so OLS is well-defined even when p > n.
Interpretation:
Each pseudo-row is a fake observation saying that coefficient j should be 0, with weight √λ.
This is the Gaussian-prior-as-data view.
Uses and details:
It lets you reuse any OLS solver, including QR-based ones that avoid forming XᵀX (better numerically).
Leave the intercept unpenalized: use a zero in its column of the pseudo-rows, or center the data.
It generalizes to Tikhonov regularization by using √λ·Γ in place of √λ·I.
Q23.Why does lasso introduce systematic shrinkage bias on large true coefficients?
The L1 penalty pulls on every nonzero coefficient with the same constant force λ, regardless of its size, so large true coefficients are shrunk by roughly λ just like small ones; that shrinkage does not fade for strong signals, leaving a persistent bias.
Constant penalty gradient:
For β_j ≠ 0, ∂(λ|β_j|)/∂β_j = λ·sign(β_j): magnitude never decreases as |β_j| grows.
Orthonormal design makes it explicit: β̂_lasso = sign(β̂_OLS)(|β̂_OLS| − λ)₊, so surviving coefficients are all offset by exactly λ.
The tension:
λ must be large enough to zero out noise features, but that same λ is subtracted from strong ones.
Cross-validation often picks a smaller λ to reduce this bias, which then admits extra false-positive features.
Consequently plain lasso lacks the "oracle" property (consistent selection plus unbiased estimates) except under restrictive conditions.
Remedies:
Relaxed lasso / refit OLS on the selected support (lasso for selection, OLS for estimation).
Adaptive lasso: weights w_j = 1/|β̂_init,j| so large coefficients are penalized less.
Non-convex penalties (SCAD, MCP) whose penalty derivative tapers to zero for large |β_j|, leaving big coefficients nearly unbiased.
Q24.Why does L1 regularization produce sparse solutions with exact zeros while L2 does not?
L1's penalty has a kink at zero that exerts a constant pull of size λ no matter how small a coefficient is, so if the loss gradient is weaker than λ the optimum sits exactly at 0. L2's pull is proportional to the coefficient and vanishes near zero, so it shrinks values toward zero but almost never reaches it.
Calculus (subgradient) view:
L1: the subdifferential of λ|β| at 0 is the interval [−λ, λ]. If |∂Loss/∂β_j| ≤ λ at β_j = 0, zero is optimal.
L2: gradient of λβ² is 2λβ, which is 0 at β = 0, so any nonzero loss gradient moves β off zero.
Orthonormal-design closed forms:
Lasso: sign(z)(|z| − λ)₊ (soft thresholding): exactly 0 whenever |z| ≤ λ.
Ridge: z / (1 + λ) (proportional scaling): zero only if z is already zero.
Geometric view:
Constrained form: the L1 ball is a diamond (cross-polytope) with corners on the axes; elliptical loss contours typically first touch it at a corner, where some coordinates are 0.
The L2 ball is round with no corners, so the tangent point generically has all coordinates nonzero.
Practical consequence: lasso performs feature selection; ridge keeps all features but tames their magnitudes (better with many small, correlated effects).
Q25.What is soft thresholding, and how does it produce the lasso solution?
Soft thresholding is the operator S_λ(z) = sign(z)·max(|z| − λ, 0): it moves a value toward zero by λ and sets it to exactly zero if it was within λ of zero. It is the exact solution of the one-dimensional lasso problem, so lasso is solved by applying it coordinate by coordinate (or as a proximal step).
Where it comes from:
S_λ(z) = argmin_b ½(b − z)² + λ|b|, i.e. the proximal operator of the L1 norm.
Subgradient condition: b − z + λ·∂|b| ∋ 0; for |z| ≤ λ this holds at b = 0, otherwise b = z ∓ λ.
Orthonormal design: With XᵀX = I and objective ½‖y − Xβ‖² + λ‖β‖₁, the problem decouples: β̂_j = S_λ(x_jᵀy) (soft-thresholded OLS).
General design: coordinate descent:
Fix all coefficients except β_j; compute partial residual r_j = y − Σ_{k≠j} x_k β_k.
Update β_j = S_λ(x_jᵀ r_j) / (x_jᵀ x_j).
Cycle over features until convergence (convex objective, so this reaches the global optimum).
Proximal gradient (ISTA/FISTA): take a gradient step on the squared loss, then apply S_{tλ} with step size t.
Contrast: ridge's analogous operator is z/(1+λ) (scaling, never zero); hard thresholding (keep z if |z| > λ) corresponds to best-subset/L0.
Q26.Why does lasso pick arbitrarily among highly correlated predictors?
Because the L1 penalty is not strictly convex, it is indifferent to how weight is split among near-duplicate features, and it prefers putting all weight on one (sparsity). Which one wins is then decided by tiny noise differences or algorithm details, not by any real distinction.
Perfectly correlated case:
If x₁ = x₂, any split β₁ + β₂ = c with matching signs gives the same fit and the same penalty |β₁| + |β₂| = |c|.
So there are infinitely many optimal solutions; the solver returns one depending on feature order, initialization, or warm starts.
Highly (not perfectly) correlated case:
The solution becomes unique, but the feature with marginally higher correlation with y in this sample usually takes all the weight while the other is zeroed.
That margin is noise-driven, so selection flips across bootstrap samples or data splits: unstable and misleading for interpretation.
Contrast with ridge: The L2 penalty is strictly convex: for fixed β₁ + β₂, β₁² + β₂² is minimized by an equal split, so ridge spreads weight evenly (grouping effect).
Remedies:
Elastic net: adding an L2 term restores strict convexity and selects correlated groups together.
Group lasso when groups are known; stability selection to assess how reliably features are chosen.
Note: prediction accuracy is often fine; the problem is mainly interpretation and reproducibility of the selected set.
Q27.When would you prefer elastic net over pure lasso?
Prefer elastic net when features are correlated, when p ≫ n, or when you care more about stable selection and prediction than a minimal feature set. Lasso misbehaves in exactly those settings.
Correlated feature groups:
Lasso arbitrarily keeps one feature from a correlated group and drops the rest. Elastic net keeps the group with similar coefficients.
Typical examples: genes in a pathway, lagged or overlapping signals, near-duplicate engineered features.
p ≫ n: Lasso saturates at n selected features. Elastic net has no such limit.
Stability and reproducibility:
Elastic net's selected set varies less across resamples or retrains, which matters for production models and scientific interpretation.
Empirically it often predicts better than lasso under collinearity.
When pure lasso is still better:
Features are roughly independent and the truth is genuinely sparse.
You need the smallest possible model (cost of collecting features, latency, interpretability).
You want one fewer hyperparameter to tune.
You don't have to guess: tune l1_ratio by cross-validation. If CV picks a value near 1, lasso was fine.
Q28.When is Group Lasso preferred over standard Lasso?
Use group lasso when features come in natural groups that should enter or leave the model together. It zeros out whole groups rather than individual coefficients.
Penalty:
λ Σ_g √p_g·||β_g||₂: the unsquared L2 norm of each group, scaled by the square root of the group size.
It acts like L1 across groups, so whole groups become exactly zero.
It acts like L2 within a group, so selected groups are dense.
Typical use cases:
One-hot encoded categoricals.
Basis expansions of one variable: polynomial terms, splines, additive models.
Multi-task or multi-class models, where a feature should be shared across all outputs or dropped from all of them.
Domain groupings, such as gene pathways or sensor channels.
Why standard lasso falls short there:
For categoricals it can keep a few dummy columns of a categorical feature and drop others, which is hard to interpret.
Which dummies survive depends on the encoding (for example, the choice of reference level).
It answers "which columns matter?" when the real question is "which variables matter?"
Caveats:
The groups must be defined in advance and are usually non-overlapping. Overlapping groups need special variants.
Groups are often orthonormalized internally so the penalty is fair across them.
If you also want sparsity inside groups, use the sparse group lasso (L1 + group penalty).
Q29.How would you choose the L1/L2 mixing ratio in elastic net?
Treat the mixing ratio (α, or l1_ratio) as a hyperparameter. Tune it jointly with λ by cross-validation over a small grid, and use prior knowledge about correlation and the sparsity you need to shape that grid and break ties.
Joint cross-validation:
For each α in a grid such as [0.1, 0.5, 0.7, 0.9, 0.95, 0.99, 1], fit the full λ path.
Pick the (α, λ) pair with the best CV error.
Weight the grid toward 1, because the model changes most between 0.9 and 1.
Use priors to set the grid:
Strong correlation, or p ≫ n: lean toward lower α for more L2 and grouping.
Need a small, interpretable model: lean toward high α.
Breaking ties:
CV surfaces are often flat across α, so prefer the simpler model.
Use the 1-SE rule on λ, or compare how stable the selected features are across folds.
Hygiene:
Standardize features.
Reuse the same fold assignments across α values so comparisons are fair.
Use nested CV if you need an unbiased estimate of generalization error.
Very small l1_ratio values (≤ 0.01) are numerically unreliable in scikit-learn's coordinate descent. Use Ridge instead.
Q30.What is a regularization path?
A regularization path is the set of fitted coefficients β̂(λ) traced out as the penalty strength λ sweeps from very large (all coefficients zero) to very small (close to the unpenalized fit). Instead of solving for one λ, you get the whole family of models.
How it's computed:
Start at λ_max, the smallest value that zeroes all coefficients. For lasso with a 1/(2n) loss, it is max_j |x_jᵀ y| / n.
Decrease along a log-spaced grid, typically about 100 values down to λ_max · 1e-3.
Tools: glmnet, sklearn.linear_model.lasso_path, LassoCV.
Shape depends on the penalty:
Lasso: the path is piecewise linear, and LARS computes it exactly. Variables enter, and sometimes leave, at kinks.
Ridge: coefficients shrink smoothly toward zero but never reach it.
Elastic Net: in between the two.
Why it's useful:
Model selection: evaluate CV error at every λ and pick the best (or use the one-SE rule).
Interpretation: the order in which features enter hints at their importance. Erratic paths signal collinearity.
Efficiency: with warm starts, the full path costs about as much as a single fit.
Q31.What is the one-standard-error rule for choosing the regularization strength?
The one-standard-error rule picks the most regularized (simplest) model whose cross-validated error is within one standard error of the minimum CV error. Differences smaller than the noise in the CV estimate aren't meaningful, so you take the parsimonious option.
Procedure:
Compute the mean CV error at each λ over K folds.
Compute the standard error at each λ: SE = sd(fold errors) / sqrt(K).
Find λ_min with the lowest mean error.
Choose the largest λ whose mean error is ≤ error(λ_min) + SE(λ_min).
Rationale:
CV curves are often flat and noisy near the minimum, so λ_min itself is a high-variance choice.
The simpler model has comparable expected error, with fewer features, more stability, and easier interpretation.
In practice: glmnet's cv.glmnet reports both lambda.min and lambda.1se.
Caveats:
It is a heuristic, not a theorem.
Fold errors are correlated because training sets overlap, so the SE is only approximate.
When pure predictive accuracy is the goal, λ_min may be preferable. The rule can slightly underfit.
Q32.What is a max-norm constraint on weight vectors?
A max-norm constraint caps the L2 norm of each unit's incoming weight vector at a constant c. After every gradient step, any vector with ||w||₂ > c is projected back onto the ball by rescaling it to norm c.
Mechanics (projected gradient descent):
Take a normal update, then apply w ← w · min(1, c / ||w||₂).
It applies per unit (per row or column of the weight matrix), not to the whole matrix.
Typical values of c are 3–4.
Compared with L2 or weight decay:
L2 is a soft penalty that always pulls weights toward zero.
Max-norm is a hard constraint. Weights inside the ball are untouched, and only oversized vectors are rescaled.
Why it's useful:
It guarantees weights can't explode, so you can use large learning rates (and momentum) safely.
It pairs well with dropout: the original dropout paper found dropout plus max-norm worked best, because noisy updates plus a high LR can otherwise blow up weights.
It bounds each unit's sensitivity to its input, which limits capacity.
Q33.How does adding Gaussian noise to model inputs during training act as a regularizer?
Adding Gaussian noise to inputs forces the model to give similar outputs for nearby inputs, so it learns a smoother function. In expectation it is equivalent to penalizing the model's sensitivity to its input, and for linear regression it is exactly ridge regression.
Linear case: exact ridge:
With ε ~ N(0, σ²I): E[(y − wᵀ(x+ε))²] = (y − wᵀx)² + σ²||w||².
The cross term vanishes because the noise has zero mean. What remains is an L2 penalty whose strength is the noise variance.
General case: Tikhonov regularization (Bishop, 1995):
For small σ, a Taylor expansion shows the expected loss ≈ clean loss + σ²·||∇ₓf(x)||² (plus higher-order terms).
This penalizes steep input gradients, so small input perturbations can't swing predictions.
Data augmentation view: Each epoch sees new jittered copies of the data. This effectively enlarges the training set and makes exact memorization of points impossible.
Practical notes:
Standardize features first so a single σ is meaningful across them. Too much σ causes underfitting.
Apply it only during training. Keras GaussianNoise is inactive at inference.
Related ideas: denoising autoencoders, noise on weights or activations, and adversarial training (worst-case rather than random noise).
Q34.Why do deep learning frameworks scale activations by 1/(1-p) during training rather than at test time?
1/(1-p) during training rather than at test time?Scaling at training time ("inverted dropout") keeps each unit's expected activation the same as at test time, so inference becomes a plain forward pass. Deployed models never need to know dropout existed, which makes inference simpler, faster and less error-prone.
The expectation argument:
With keep probability 1−p: E[m·a] = (1−p)·a, so without correction train-time activations are smaller than at test time.
Fix A (original paper): multiply the weights by (1−p) at test time.
Fix B (inverted): divide by (1−p) during training, giving E[m·a/(1−p)] = a.
Both match in expectation. Fix B moves the cost to training.
Why frameworks prefer training-time scaling:
Inference is the identity: no extra op, no latency, and nothing extra to export to ONNX or serving runtimes.
The model is self-contained. Inference code needn't track each layer's p, and you can change p or remove dropout without touching deployment.
Activation scales stay consistent between modes, so validating with eval() is directly comparable to training.
Inference runs far more often than training in production, so it should be the cheap path.
Caveat:
The match is exact only for the mean through linear layers. Through nonlinearities, the test-time pass approximates the ensemble average (the "weight scaling inference rule").
Variance also differs between modes, which is why dropout placed before BatchNorm can cause a train/test shift.
Q35.Why can dropout be interpreted as training an ensemble of thinned sub-networks?
Each training step samples a random binary mask over units, which selects one "thinned" sub-network. All 2^n possible sub-networks share the same weights, so training with dropout trains a huge, weight-shared ensemble. At test time, scaling the weights approximates averaging all of their predictions.
Sampling a sub-network:
With n droppable units there are 2^n masks, and each one defines a different architecture.
Each minibatch trains one sampled sub-network for one step. Most sub-networks are never seen at all, but weight sharing lets every update benefit the others.
Comparison to bagging:
Bagging trains independent models on resampled data.
Dropout trains models that share parameters, each on different minibatches. It is an extreme, cheap form of bagging.
Inference approximates the ensemble average:
Weight scaling multiplies weights by the keep probability. Inverted dropout does the equivalent by scaling activations by 1/p during training.
For a single softmax layer this is exactly the normalized geometric mean of all sub-network predictions. For deep nonlinear nets it is a good approximation.
An alternative is MC dropout: keep dropout on at test time and average several stochastic passes. This also gives an uncertainty estimate.
Why this regularizes:
A unit cannot rely on specific other units being present, which discourages co-adaptation.
Each unit is pushed to learn features that are useful on their own, and averaging many models reduces variance.
Q36.How would you choose the dropout rate, and which layers would you apply dropout to?
Treat the dropout rate as a hyperparameter tuned on validation data, starting from standard defaults. Apply dropout where the parameter count is high and overfitting is likely, typically the wide fully connected layers, and avoid it where it disrupts structure or the output.
Typical starting rates (drop probability):
Hidden fully connected layers: about 0.5.
Inputs: about 0.1–0.2, because dropping raw features loses information.
Transformers: about 0.1 on attention weights, residual branches and embeddings.
Convolutional layers: low or none. Prefer spatial dropout (Dropout2d), which drops whole feature maps, because neighbouring pixels are correlated.
How to tune the rate:
Raise it if the train–validation gap is large. Lower it if the model underfits or training is too slow to converge.
Bigger models and smaller datasets usually need more dropout. Very large datasets often need little or none.
Search a small grid (0.1–0.5) together with weight decay, since the two interact.
Where to avoid or adapt dropout:
Not directly before the output layer.
With BatchNorm, dropout placed before BN causes a train/test variance shift. Put dropout after BN, or rely on BN's own regularization effect.
In RNNs, use variational dropout (the same mask at every timestep) or restrict dropout to non-recurrent connections.
API gotcha:
PyTorch nn.Dropout(p) and Keras Dropout(rate) both take the drop probability.
The original paper's p is the retain probability, so check which convention a source uses.
Q37.How do you decide which metric to monitor and how much patience to allow when using early stopping?
Monitor a validation metric that reflects what you actually care about and is stable enough to show real trends. Set patience long enough to ride out noise and learning-rate plateaus, but short enough not to waste compute, and always restore the best checkpoint.
Choosing the metric:
Always use a held-out validation set, never training loss.
Validation loss is smooth and sensitive, so it is a good default. Accuracy is coarse and can plateau while loss still improves, or keep rising after calibration has started to degrade.
If the deployment objective differs (F1, AUC or BLEU for imbalanced or ranking tasks), monitor that metric, or monitor loss and select the final checkpoint by the task metric.
Set the direction correctly: mode="min" for loss, "max" for scores.
Choosing patience:
Noisy metrics (small validation set, small batches) need more patience.
With ReduceLROnPlateau, early-stopping patience must exceed the scheduler's patience, so the model gets a chance to improve after the LR drops.
A common heuristic is 5–10% of the maximum epochs, often 5–20 epochs. On huge datasets, evaluate several times per epoch and count patience in evaluations rather than epochs.
Use min_delta so that tiny, noisy improvements don't reset the counter.
Restore the best weights:
Without restore_best_weights, you end up with the last model, which is patience epochs past the best one.
Patience then only costs compute, not model quality.
Q38.How does applying L1 or L2 regularization affect the interpretation of feature importance from regression coefficients?
Regularized coefficients are deliberately biased toward zero, so their magnitudes are no longer unbiased effect sizes. They depend on λ, on feature scaling and on how the penalty splits credit among correlated features. Treat them as a rough ranking under a specific λ, not as causal or exact importance.
Scaling matters:
The penalty acts on raw coefficient size, so unstandardized features are penalized unequally.
Standardize the features before fitting and comparing coefficients.
L2 (ridge):
Shrinks all coefficients but none to exactly zero.
Correlated features share weight (the grouping effect), so each one looks less important than the group really is.
Rankings are stable, but magnitudes are understated.
L1 (lasso):
Zeros out features, but a zero coefficient does not mean the feature is irrelevant.
Among correlated features, lasso keeps one somewhat arbitrarily and drops the rest.
The selected set can change across bootstrap samples or small changes in λ, so check stability (stability selection, bootstrap selection frequencies).
Elastic net is a compromise between selection and grouping.
Inference caveats:
Standard p-values and confidence intervals are invalid after shrinkage or selection.
Use post-selection inference or the debiased lasso, or refit OLS on the selected features on separate data.
Better practice:
Inspect the full regularization path: the order in which features enter as λ decreases is informative.
Complement coefficients with model-agnostic measures such as permutation importance.
Q39.How is the penalized form of regularization mathematically equivalent to a constrained optimization problem?
Minimizing L(w) + λ·R(w) is the Lagrangian of minimizing L(w) subject to R(w) ≤ t. When the loss and penalty are convex, each λ ≥ 0 corresponds to some budget t (and vice versa) that yields the same solution, with λ acting as the Lagrange multiplier.
The two forms:
Penalized: min_w ||y − Xw||² + λ||w||²
Constrained: min_w ||y − Xw||² s.t. ||w||² ≤ t
Lagrangian / KKT argument:
The Lagrangian of the constrained problem is L(w) + λ(||w||² − t). The −λt term doesn't depend on w, so minimizing over w is exactly the penalized problem.
Penalized → constrained: if w*(λ) solves the penalized problem, set t = ||w*(λ)||². Any w with a smaller loss inside that ball would also have a smaller penalized objective, which is a contradiction.
Constrained → penalized: convexity plus Slater's condition give strong duality, so a multiplier λ ≥ 0 exists that satisfies KKT. That w is then a minimizer of the penalized objective.
Properties of the mapping:
It is monotone: larger λ means a smaller t (tighter budget).
Complementary slackness: if t is at least the norm of the unregularized solution, the constraint is inactive and λ = 0.
The mapping depends on the data, so a given λ doesn't correspond to a fixed t across datasets.
Why it matters: geometry:
The solution is where the loss contours first touch the constraint region.
The L2 region is a ball, so contact happens at generic points (shrinkage, no zeros).
The L1 region is a diamond with corners on the axes, so contact often happens at a corner, which gives sparsity.
Caveat: for non-convex losses (e.g. neural nets), the equivalence can fail in one direction. Some constrained solutions may not be reachable by any λ.
Q40.From an SVD perspective, which directions in feature space does ridge regression shrink most heavily?
Ridge shrinks most heavily along the directions with the smallest singular values of the design matrix: the low-variance principal component directions. High-variance directions are barely touched.
SVD of the (centered) design: X = U D Vᵀ, where the columns of V are principal directions and d_j are singular values (d_j² is proportional to the sample variance along direction j).
Ridge fitted values per direction:
ŷ_ridge = Σ_j u_j · [d_j² / (d_j² + λ)] · u_jᵀ y
OLS uses factor 1 for every direction; ridge multiplies each by the shrinkage factor d_j² / (d_j² + λ), which lies in (0, 1).
Which directions shrink most:
Large d_j (d_j² ≫ λ): factor near 1, little shrinkage.
Small d_j (d_j² ≪ λ): factor near 0, heavy shrinkage.
Why this is sensible: Coefficient variance along direction j scales like σ² / d_j², so low-variance directions are where OLS estimates are noisiest (collinearity). Ridge trades a little bias there for a large variance reduction.
Q41.Why are the effective degrees of freedom of ridge regression strictly decreasing in lambda?
Ridge's effective degrees of freedom are the trace of its hat matrix, df(λ) = Σ_j d_j² / (d_j² + λ), and every term with d_j > 0 strictly decreases as λ grows, so the sum does too.
Where the formula comes from:
Ridge is a linear smoother: ŷ = H_λ y with H_λ = X(XᵀX + λI)⁻¹Xᵀ.
For linear smoothers, df = tr(H_λ); using the SVD X = UDVᵀ, the eigenvalues of H_λ are d_j²/(d_j²+λ).
Monotonicity:
d/dλ [d_j²/(d_j²+λ)] = −d_j² / (d_j²+λ)² < 0 whenever d_j ≠ 0.
A sum of strictly decreasing terms is strictly decreasing (needs at least one nonzero singular value, i.e. X ≠ 0).
Endpoints:
λ = 0: df equals rank(X) (OLS).
λ → ∞: df → 0 (all coefficients shrunk to zero).
Intuition: larger λ shrinks every eigen-direction's contribution, so the fit depends less on y (df also equals Σ Cov(ŷ_i, y_i)/σ²). This makes df a smooth, one-to-one reparameterization of λ, handy for comparing models.
Q42.How does ridge regression compare to principal components regression in how each treats low-variance directions of the data?
Both operate in the principal-component basis and down-weight low-variance directions, but PCR does it with a hard cutoff (keep or discard) while ridge does it with smooth, continuous shrinkage.
PCR: hard thresholding on components:
Regress y on the top k principal components; shrinkage factor is 1 for the k largest-variance directions and 0 for the rest.
Tuning parameter is the discrete k.
Ridge: soft shrinkage on all components:
Every direction is kept but scaled by d_j²/(d_j²+λ), so low-variance directions are shrunk heavily, never fully removed.
Tuning parameter λ is continuous, giving a smoother bias-variance path and often slightly better prediction.
Shared assumption and weakness:
Both assume the response varies mostly along high-variance directions of X.
Neither uses y to choose directions (unlike partial least squares), so if the signal lives in a low-variance direction, both will suppress it.
Interpretability: neither produces sparse coefficients in the original features; PCR's discarded components still mix all features.
Q43.How does L1 regularization behave when the number of features is much larger than the number of samples?
When p ≫ n, lasso still gives a usable, sparse fit where OLS is undefined, but it has hard limits: it selects at most n features, behaves erratically among correlated features, and recovers the true support only under strong conditions.
It stays well-posed:
OLS has infinitely many perfect-fit solutions when p > n. The L1 penalty picks a sparse solution from among them.
The coefficients may not be unique, but the fitted values are unique.
Saturation: at most n non-zero coefficients:
A typical lasso solution (for example, the one found by LARS) has at most min(n, p) active features.
If more than n features are truly relevant, lasso cannot include them all.
Correlated features cause unstable selection:
Among a group of correlated features, lasso tends to pick one somewhat arbitrarily and zero out the others.
Small changes to the data can flip which feature is chosen, so the selected set is fragile across bootstrap samples.
When support recovery works:
You need roughly n ≳ s·log(p) samples, where s is the true number of non-zero coefficients.
Irrelevant features must not be too correlated with relevant ones (the irrepresentable condition).
True coefficients must be large enough to stand out from noise (beta-min).
Bias trade-off:
With many noise features, cross-validation tends to pick a small λ, which lets in false positives.
A larger λ removes the false positives but over-shrinks the real signals.
Common fixes:
Elastic net: removes the n-feature limit and groups correlated features.
Stability selection: keeps features that are chosen consistently across resamples.
Adaptive or relaxed lasso: reduces the shrinkage bias.
Q44.Why does an Lq penalty with q > 1, such as an L3 norm, never produce exact zeros?
q > 1, such as an L3 norm, never produce exact zeros?Q45.What problem with lasso do non-convex penalties like SCAD or MCP attempt to solve?
Q46.What is the conceptual motivation for using the adaptive lasso or relaxed lasso?
Q47.What is the grouping effect in Elastic Net regression?
Q48.Why does naive Elastic Net require a (1 + lambda2) scaling correction to undo double shrinkage?
(1 + lambda2) scaling correction to undo double shrinkage?