74 Support Vector Machines Interview Questions and Answers (2026)

Many hyperplanes can separate the training data, but the one with the largest margin is the most robust and tends to generalize best. It also turns training into a convex problem with a single, well-defined answer, instead of an arbitrary choice among infinitely many separators.

• Any separator is ambiguous:

  • A perceptron stops at whichever separator it reaches first. That boundary may sit right next to some training points.

  • A boundary that hugs the data will misclassify new points that land only slightly away from the training examples.

• Robustness to noise:

  • With a large margin, every training point can be perturbed by up to the margin width without crossing the boundary.

  • The classifier therefore tolerates small measurement noise and shifts in the test data.

• Generalization theory:

  • Margin-based bounds say the capacity of large-margin classifiers scales roughly as R²/ρ² (R is the data radius, ρ the margin) rather than with the input dimension.

  • A larger margin means a simpler effective hypothesis class and a tighter bound on test error. This is why SVMs can work in very high-dimensional and kernel feature spaces.

• Unique, tractable optimum:

  • Maximizing the margin becomes minimizing ½‖w‖² under linear constraints. This is a convex quadratic program, so it has one global solution.

  • The solution depends only on the support vectors, which makes it sparse.

A separating hyperplane is a flat decision surface w·x + b = 0 that puts all points of one class on one side and all points of the other class on the other side. An SVM chooses, among all such hyperplanes, the one with the maximum margin.

• Geometry:

  • In d dimensions it is a (d−1)-dimensional subspace: a line in 2D, a plane in 3D.

  • w is the normal vector and sets the orientation. b shifts the hyperplane away from the origin, which is at distance |b|/‖w‖.

• Separation condition:

  • With labels y ∈ {−1, +1}, the hyperplane separates the data if yᵢ(w·xᵢ + b) > 0 for every training point.

  • A new point is classified by which side it falls on: sign(w·x + b).

• Not unique: If the data is separable, infinitely many hyperplanes work. The SVM picks the one farthest from the nearest points of both classes.

• With kernels: The hyperplane lives in the transformed feature space φ(x). Mapped back to the input space, it can be a nonlinear boundary.

Support vectors are the training points that lie on the margin boundaries or violate them. They are the only points with nonzero dual coefficients, so they alone determine the decision boundary.

• Formal definition:

  • They are the points whose Lagrange multiplier αᵢ > 0.

  • The weight vector is built only from them: w = Σ αᵢ yᵢ xᵢ.

• Where they sit:

  • Hard margin: exactly on the margin, where yᵢ f(xᵢ) = 1.

  • Soft margin: also points inside the margin or misclassified. These have αᵢ = C.

• Why they matter:

  • Removing any other point leaves the model unchanged. Removing a support vector can move the boundary.

  • The model is sparse: prediction cost scales with the number of support vectors, not the dataset size. This matters most with kernels.

  • The leave-one-out error is at most #SV / n, so having few support vectors suggests good generalization.

• In scikit-learn they are exposed via support_vectors_, n_support_ and dual_coef_ (which holds αᵢyᵢ).

A labeled dataset is strictly linearly separable if some hyperplane puts every point strictly on its correct side, with no point lying on the hyperplane: there exist w, b such that yᵢ(w·xᵢ + b) > 0 for all i.

• Equivalent conditions for a finite dataset:

  • Rescaled form: there exist w, b with yᵢ(w·xᵢ + b) ≥ 1 for all i. Since the dataset is finite, the minimum positive value can always be scaled up to 1.

  • Convex hull form: the convex hulls of the two classes do not intersect.

  • SVM form: the hard-margin SVM problem is feasible and has a positive margin.

• Strict vs. weak separability:

  • Weak separability allows ≥ 0, meaning points may lie on the hyperplane.

  • Strict separability guarantees a positive margin exists.

• Things that rule it out:

  • Identical inputs that carry different labels.

  • Classic nonlinear patterns, such as XOR or concentric rings, in the original feature space.

• Things that make it easier:

  • Higher dimension: any n ≤ d+1 points in general position can be shattered by hyperplanes in ℝᵈ, i.e. separated under every labeling.

  • Feature maps or kernels can make data separable in the new space. An RBF kernel can separate any set of distinct points.

• How to check in practice:

  • Solve the linear-programming feasibility problem yᵢ(w·xᵢ + b) ≥ 1.

  • Or train a linear SVM with a very large C and check whether training error is zero.

The decision function is f(x) = w·x + b, or Σ αᵢ yᵢ K(xᵢ, x) + b with kernels. Its sign gives the predicted class. Its magnitude says how far the point is from the boundary, which is a rough measure of confidence.

• Sign gives the class: f(x) > 0 predicts +1, f(x) < 0 predicts −1, and f(x) = 0 means the point is on the hyperplane.

• Magnitude gives distance:

  • For a linear SVM, the geometric distance to the hyperplane is |f(x)| / ‖w‖.

  • |f(x)| ≥ 1 means the point is on or beyond the margin, a confident prediction.

  • |f(x)| < 1 means it falls inside the margin, an uncertain prediction.

• It is not a probability:

  • The scores are uncalibrated and their scale depends on C and the kernel.

  • To get probabilities, apply Platt scaling (SVC(probability=True)) or CalibratedClassifierCV.

• Practical uses:

  • Shift the threshold away from 0 to trade precision against recall.

  • Rank points by score for ROC curves and AUC (decision_function() in scikit-learn).

  • In one-vs-rest multiclass, predict the class whose decision function is largest.

A hard-margin SVM requires every point to be correctly classified outside the margin, so it only works on linearly separable data. A soft-margin SVM adds slack variables that allow violations at a cost weighted by C, trading margin width against training errors.

• Hard margin:

  • Minimise ½‖w‖² subject to yᵢ(w·xᵢ + b) ≥ 1.

  • The problem is infeasible if the classes overlap, and it is very sensitive to outliers and label noise.

• Soft margin:

  • Minimise ½‖w‖² + C Σ ξᵢ subject to yᵢ(w·xᵢ + b) ≥ 1 − ξᵢ and ξᵢ ≥ 0.

  • How to read the slack: 0 < ξᵢ ≤ 1 means the point is inside the margin but still correct; ξᵢ > 1 means it is misclassified.

  • This is equivalent to minimising hinge loss plus L2 regularization.

• Dual view: The only change is a box constraint: 0 ≤ αᵢ ≤ C instead of αᵢ ≥ 0. This caps any single point's influence on the solution.

• Relationship:

  • As C → ∞, the soft margin approaches the hard margin on separable data.

  • In practice the soft margin, with C tuned by cross-validation, is almost always used.

Every point must lie outside the margin on the correct side, with no exceptions. A single outlier near or across the other class can therefore force a drastically narrower or tilted boundary, or make the problem infeasible altogether.

• Constraints are absolute:

  • There is no slack, so no point can be ignored. The solution is dictated by the most extreme points, which are exactly where outliers live.

  • An outlier close to the other class becomes a support vector, and the margin shrinks to whatever gap remains.

• Unbounded influence:

  • In the dual, αᵢ has no upper bound, so one point can carry arbitrarily large weight in w = Σ αᵢyᵢxᵢ.

  • In convex-hull terms, one outlier stretches its class's hull toward the other class.

• Failure modes:

  1. An outlier near the boundary produces a narrow margin and poor generalization.

  2. A mislabeled point inside the other class makes the data non-separable, so no solution exists.

• Fix: use a soft margin:

  • Slack variables plus 0 ≤ αᵢ ≤ C cap each point's influence. Hinge loss grows only linearly in the violation.

  • Points far on the correct side never matter, because only near-boundary points affect the solution.

Increasing C raises the penalty on margin violations. The SVM then prefers fitting the training data over a wide margin, so the margin narrows, bias decreases and variance increases.

• Mechanism:

  • The objective is ½‖w‖² + C Σ ξᵢ. A larger C makes slack expensive relative to ‖w‖, so ‖w‖ grows and the margin 2/‖w‖ shrinks.

  • C acts as an inverse regularization strength, roughly λ ∝ 1/C.

• Large C:

  • Narrow margin, few training errors, and typically fewer support vectors.

  • Low bias, high variance: the model is sensitive to outliers and noise. As C → ∞ it approaches the hard margin.

• Small C:

  • Wide margin, many violations tolerated, more support vectors.

  • High bias, low variance, with a risk of underfitting.

• In practice:

  • Tune C on a log scale with cross-validation.

  • With an RBF kernel, tune it jointly with gamma, since both control flexibility.

The kernel trick replaces every inner product φ(x)·φ(z) with a kernel function K(x, z) that computes the same value directly in input space. The SVM then learns a linear separator in a high- or infinite-dimensional feature space without ever constructing that space.

• Why it works: The dual objective and the decision function f(x) = Σ α_i y_i K(x_i, x) + b use the data only through inner products.

• Example: degree-2 polynomial:

  • K(x, z) = (x·z)² equals an inner product over all pairwise products x_i x_j.

  • Computing it costs O(d) instead of O(d²).

• Common kernels: Linear, polynomial (γ x·z + r)^p, RBF exp(−γ‖x − z‖²) (infinite-dimensional feature space), and sigmoid.

• Result: The boundary is linear in feature space, which corresponds to a non-linear boundary in input space.

• Requirement: K must be symmetric positive semi-definite (Mercer) so that a real feature map exists.

• Costs:

  • The n × n kernel matrix makes training roughly O(n²) to O(n³).

  • Prediction scales with the number of support vectors.

  • Hyperparameters such as γ and C need tuning.

In k(x, z) = exp(-γ‖x − z‖²), gamma sets how far each training point's influence reaches. It is the inverse width of the Gaussian bump (γ = 1/(2σ²)), so it directly controls how smooth or wiggly the decision boundary is.

• Large γ: narrow influence:

  • The kernel decays quickly, so each support vector affects only its immediate neighborhood.

  • The boundary becomes highly flexible and can wrap around individual points, which leads to overfitting and many support vectors.

  • In the extreme, K approaches the identity matrix and the model behaves like memorization.

• Small γ: broad influence:

  • Points far apart still look similar, giving a smooth, nearly linear boundary.

  • If γ is too small, the model underfits.

• Interaction with C:

  • γ controls model complexity through the kernel; C controls the penalty on margin violations.

  • Tune them jointly with a grid search on a log scale (for example 10⁻³ to 10³).

• Practical notes:

  • γ acts on squared distances, so scale your features first.

  • scikit-learn's default gamma="scale" uses 1 / (n_features · X.var()) as a sensible starting point.

In k(x, z) = (γ·xᵀz + r)^d, the degree d sets the highest order of feature interactions the SVM can model. A higher d gives a richer, more curved boundary at the cost of overfitting and numerical problems.

• What d means in feature space:

  • The implicit features are all monomials up to degree d. For example, d = 2 gives xᵢ², xᵢxⱼ, xᵢ, and a constant (when r > 0).

  • The feature dimension is C(n + d, d), which grows combinatorially, but the kernel trick avoids ever computing it.

  • d = 1 with r = 0 reduces to the linear kernel.

• Effect on the model:

  • Higher d allows more complex boundaries and more variance, so it overfits more easily, especially with large C.

  • Kernel values explode when |γxᵀz + r| > 1 and vanish when it is < 1. This makes training ill-conditioned for large d, so scale your features.

  • In practice, d = 2 or 3 is typical.

• Related parameters:

  • r (coef0 in scikit-learn) weights lower-order terms against higher-order ones. With r = 0, only exact degree-d terms remain.

  • In scikit-learn, degree is ignored unless kernel="poly".

The RBF (Gaussian) kernel is k(x, z) = exp(-‖x − z‖² / (2σ²)) = exp(-γ‖x − z‖²), with γ = 1/(2σ²) > 0. It measures similarity as a Gaussian function of the Euclidean distance between two points.

• Key properties:

  • Values lie in (0, 1]: exactly 1 when x = z, approaching 0 as points move apart.

  • It is stationary: it depends only on x − z, so it is translation-invariant but sensitive to feature scaling.

  • It is always positive definite for distinct points, so it is a valid Mercer kernel.

• Infinite-dimensional feature space:

  • Rewrite it as e^{-γ‖x‖²}·e^{-γ‖z‖²}·exp(2γ xᵀz).

  • The Taylor series of exp(2γ xᵀz) contains polynomial kernels of every degree, so φ(x) is infinite-dimensional.

  • It is a universal kernel: it can approximate any continuous decision function given enough data.

• Resulting decision function: f(x) = Σ αᵢyᵢ exp(-γ‖xᵢ − x‖²) + b is a weighted sum of Gaussian bumps centered on the support vectors. γ sets the width of those bumps.

One-vs-Rest (OvR) trains K binary SVMs, each separating one class from all the others, and predicts by the highest score. One-vs-One (OvO) trains K(K−1)/2 SVMs, one per pair of classes, and predicts by majority vote.

• Number and size of problems:

  • OvR trains K classifiers, and each uses all n samples.

  • OvO trains K(K−1)/2 classifiers, and each uses only about 2n/K samples.

  • Kernel SVM training is superlinear in n, so many small OvO problems are often cheaper in total. This is why SVC/LIBSVM use OvO.

• Weaknesses of OvR:

  • Each subproblem is imbalanced (1 class vs K−1 classes).

  • Scores from independently trained classifiers are not calibrated against each other, so argmax can be unreliable.

• Weaknesses of OvO:

  • The number of models grows quadratically in K, so prediction cost and storage grow too.

  • Votes can tie.

  • Each model sees only two classes, so it is never trained on the other classes.

• In sklearn:

  • LinearSVC uses OvR, which is cheap for linear models.

  • SVC uses OvO internally, and decision_function_shape='ovr' only reshapes its output.

  • OneVsRestClassifier and OneVsOneClassifier let you choose explicitly.

• In practice, accuracy is usually similar when the binary classifiers are well tuned. The choice is mostly about compute and scale.

SVMs are distance-based: both the margin and common kernels (RBF, polynomial) depend on dot products or Euclidean distances, so features with large numeric ranges dominate the geometry and the optimizer, regardless of how informative they are.

• The margin is measured in feature space:

  • The margin width is 2/||w||. If one feature ranges 0 to 100,000 and another 0 to 1, the large one dominates distances and the boundary effectively ignores the small one.

  • The regularizer ||w||² penalizes all weights equally, which is only fair if features are on comparable scales.

• Kernels amplify the problem

• Optimization behaves better: Badly scaled features make the problem ill-conditioned, so solvers (liblinear, libsvm) converge slowly or hit iteration limits.

• Practical rules:

  • Use StandardScaler (or MaxAbsScaler for sparse data) inside a Pipeline so the scaler is fit on training folds only (no leakage).

  • Contrast: tree models split on thresholds per feature and are scale-invariant.

The functional margin yᵢ(w·xᵢ + b) is a raw score that changes when w and b are rescaled. The geometric margin divides it by ‖w‖, giving the true signed Euclidean distance to the hyperplane, which does not depend on scale.

• Functional margin: γ̂ᵢ = yᵢ(w·xᵢ + b):

  • A positive value means the point is classified correctly.

  • Scaling (w, b) to (2w, 2b) doubles it, yet the hyperplane is unchanged. So you cannot maximize it directly: it grows without bound.

• Geometric margin: γᵢ = γ̂ᵢ / ‖w‖:

  • It is the actual distance from the point to the hyperplane, signed by whether the point is correct.

  • It is invariant to rescaling, so it is the quantity that really matters.

• Dataset margin: For the whole dataset, take the minimum over all points.

• How the SVM uses both:

  • Use the scaling freedom to fix the minimum functional margin at 1.

  • The geometric margin then equals 1/‖w‖, so maximizing it means minimizing ‖w‖.

At the optimum, KKT complementary slackness forces αᵢ = 0 for every point strictly outside the margin. Because w = Σ αᵢ yᵢ xᵢ, those points add nothing to w or b. Their constraints are inactive, so they do not shape the solution.

• The math:

  1. Complementary slackness requires αᵢ [yᵢ f(xᵢ) − 1 + ξᵢ] = 0.

  2. If yᵢ f(xᵢ) > 1, the bracket is positive, so αᵢ must be 0.

  3. Both w and b, and therefore f(x), are built only from terms with αᵢ > 0.

• The intuition:

  • An inactive constraint is not "pushing" on the solution, much like a wall you are not touching.

  • You can delete these points, or move them around (as long as they stay outside the margin), and retraining gives the same hyperplane.

• Caveats:

  • This holds only for the final solution. If a point moves into the margin, it becomes a support vector and changes the boundary.

  • It comes from the hinge loss, which is exactly zero beyond the margin. Logistic loss is never zero, so in logistic regression every point affects the fit.

Scale w and b so the closest points satisfy yᵢ(w·xᵢ + b) = 1. Their distance to the hyperplane is then 1/‖w‖, so maximizing the margin is the same as minimizing ‖w‖, which is done in practice by minimizing ½‖w‖².

• Derivation:

  1. The distance from x to the hyperplane w·x + b = 0 is |w·x + b| / ‖w‖.

  2. (w, b) and (cw, cb) describe the same hyperplane, so fix the scale with the canonical choice minᵢ yᵢ(w·xᵢ + b) = 1.

  3. The nearest points then lie at distance 1/‖w‖, and the full margin width is 2/‖w‖.

  4. Maximizing 1/‖w‖ is equivalent to minimizing ‖w‖ subject to yᵢ(w·xᵢ + b) ≥ 1.

• Why minimize ½‖w‖² instead of ‖w‖:

  • Squaring is monotone for nonnegative values, so the minimizer is the same.

  • The squared form is smooth and strictly convex, giving a quadratic program with a unique solution.

  • The ½ only makes the gradient cleaner (it becomes w).

• Interpretation:

  • A small ‖w‖ means f(x) changes slowly as x moves, so points must be far from the boundary to reach a score of ±1.

  • This is the same idea as L2 regularization. The soft-margin SVM is literally hinge loss plus an L2 penalty on w.

The hard-margin SVM finds the hyperplane that separates the classes with the widest margin. It solves a convex quadratic program: minimise ½‖w‖² subject to yᵢ(w·xᵢ + b) ≥ 1 for every training point.

• Where the objective comes from:

  1. The distance from a point to the hyperplane is |w·x + b| / ‖w‖.

  2. w and b can be rescaled without moving the hyperplane. Fix the scale (the canonical form) so that the closest points satisfy yᵢ(w·xᵢ + b) = 1.

  3. The margin width is then 2/‖w‖. Maximising it is the same as minimising ‖w‖, and ½‖w‖² is used because it is smooth and differentiable.

• Constraints:

  • yᵢ ∈ {−1, +1}, so a single inequality says each point lies on the correct side, at least on the margin boundary.

  • There are no slack variables, so no violations are allowed. The problem is infeasible if the data are not linearly separable.

• Properties:

  • It is a convex QP with linear constraints, so any local optimum is the global optimum, and the optimal w is unique.

  • Constraints that are active at the optimum (equality holds) belong to the support vectors.

  • Usually it is solved through the Lagrangian dual, which exposes the dot products xᵢ·xⱼ and so enables kernels.

Class weights replace the single penalty C with a per-class penalty C·w_{yᵢ}. Slack on minority-class points becomes more expensive, which pushes the boundary away from the minority class instead of letting the majority dominate.

• Modified objective:

  • Minimise ½‖w‖² + C Σ w_{yᵢ} ξᵢ, or equivalently C₊ Σ_{pos} ξᵢ + C₋ Σ_{neg} ξᵢ.

  • In the dual, the box constraint becomes 0 ≤ αᵢ ≤ C·w_{yᵢ}, so minority points can exert more influence.

• Why it is needed:

  • With equal C, total slack is dominated by the many majority points. The cheapest solution sacrifices minority points and leaves a boundary skewed into the minority region.

  • Weighting equalises each class's total contribution to the loss.

• Choosing weights:

  • Inverse frequency: w_c = n_samples / (n_classes · n_c). This is what class_weight='balanced' does in scikit-learn.

  • You can also treat the weights as hyperparameters tuned against a metric such as F1, recall or PR-AUC rather than accuracy.

• Caveat:

  • Weighting shifts the boundary, but it does not produce calibrated probabilities.

  • Alternatives include threshold tuning on the decision function and resampling.

Slack variables ξ_i ≥ 0 relax the hard-margin constraints so the SVM can still be solved when data is not linearly separable or is noisy. Each one measures how far a point violates the margin, and the objective penalizes the total violation through C.

• The formulation:

  • min ½‖w‖² + C Σ ξ_i subject to y_i(w·x_i + b) ≥ 1 − ξ_i and ξ_i ≥ 0.

  • Without slack, there is no feasible solution for non-separable data, and a single outlier can force a very narrow margin.

• How to read the value of ξ_i:

  • ξ_i = 0: the point is on or outside the margin and on the correct side.

  • 0 < ξ_i ≤ 1: the point is inside the margin but still correctly classified.

  • ξ_i > 1: the point is misclassified, so Σ ξ_i upper-bounds the number of training errors.

• The trade-off with C:

  • Large C: slack is expensive, which gives a narrow margin and few violations (risk of overfitting).

  • Small C: slack is cheap, which gives a wider margin that tolerates more violations (more regularization).

• Connections:

  • At the optimum, ξ_i = max(0, 1 − y_i f(x_i)), which is exactly the hinge loss.

  • In the dual, slack shows up as the box constraint 0 ≤ α_i ≤ C. Points with ξ_i > 0 have α_i = C.

Increasing C generally reduces the number of support vectors. A larger penalty on margin violations shrinks the margin, so fewer points end up on or inside it. The relationship is a tendency, not a strict monotonic law.

• Why small C gives many support vectors:

  • Violations are cheap, so the optimizer picks a wide margin.

  • Many points fall inside that margin. Each of them has α_i > 0 (often α_i = C), which makes it a support vector.

• Why large C gives fewer support vectors:

  • Violations are expensive, so the margin narrows to fit the training data.

  • Fewer points lie inside the margin.

  • As C → ∞, the model approaches the hard-margin SVM, where only points exactly on the margin are support vectors (when the data is separable).

• Caveats:

  • On overlapping or noisy data, many points stay as support vectors even at high C, because some violations are unavoidable.

  • The count can fluctuate locally, and kernel parameters (for example RBF gamma) interact with C.

• Practical implications:

  • Fewer support vectors means faster prediction, since kernel SVM cost scales with the number of support vectors.

  • However, it often also means a more overfit boundary.

Eliminate the slack variables: at the optimum each ξ_i equals the hinge loss of its point. The constrained QP then becomes an unconstrained regularized empirical risk minimization problem.

• The derivation:

  1. Start from the primal: min ½‖w‖² + C Σ ξ_i subject to ξ_i ≥ 1 − y_i(w·x_i + b) and ξ_i ≥ 0.

  2. For fixed w, b, the smallest feasible slack is ξ_i = max(0, 1 − y_i(w·x_i + b)).

  3. Substituting gives: min_{w,b} ½‖w‖² + C Σ max(0, 1 − y_i(w·x_i + b)).

• Equivalent regularized-loss form:

  • Divide by nC to get λ‖w‖² + (1/n) Σ hinge_i, where λ = 1/(2nC).

  • This reads as data-fit (hinge loss) plus L2 regularization (which corresponds to margin maximization).

• Why this view is useful:

  • It places the SVM alongside logistic regression and ridge regression: same structure, different loss.

  • It enables subgradient or SGD solvers (for example Pegasos and SGDClassifier(loss="hinge")) that scale to large linear problems.

With margin m = y·f(x), standard hinge is max(0, 1 − m) (L1-SVM) and squared hinge is max(0, 1 − m)² (L2-SVM). Squaring makes the loss smooth but penalizes large violations much more heavily.

• Smoothness:

  • Hinge has a kink at m = 1, so it is not differentiable there and needs subgradients.

  • Squared hinge is differentiable everywhere, with gradient −2(1 − m)·y·x when m < 1.

  • This enables smooth optimizers (Newton, L-BFGS, coordinate descent), and it is why LinearSVC defaults to loss='squared_hinge'.

• How violations are penalized:

  • Hinge grows linearly. Its gradient is bounded, so an outlier's influence is capped.

  • Squared hinge grows quadratically, so it is more sensitive to outliers and mislabeled points.

  • It is more lenient on small violations, where (1 − m)² < 1 − m.

• Effect on the dual:

  • Hinge corresponds to the slack penalty C Σ ξ_i and the box constraint 0 ≤ α_i ≤ C.

  • Squared hinge corresponds to C Σ ξ_i². It removes the upper bound on α_i and adds 1/(2C) to the kernel diagonal, effectively a ridge on the Gram matrix.

• What they share:

  • Both are convex.

  • Both give zero loss for m ≥ 1, so both still produce sparse solutions in support vectors.

The 0-1 loss is what we ultimately care about, but it is non-convex and discontinuous, and minimizing it is NP-hard. Hinge loss is a convex upper bound (surrogate) on 0-1 loss. It keeps the problem tractable while still leading to good classifiers.

• Problems with 0-1 loss:

  • Its gradient is zero almost everywhere and undefined at the boundary, which gives no signal for optimization.

  • It is non-convex, so it has many local optima. Exact minimization over linear classifiers is NP-hard.

• Benefits of hinge loss:

  • Convex, so together with ½‖w‖² it yields a QP with a global optimum and a clean dual (which enables kernels).

  • Upper bound: max(0, 1 − m) ≥ 1[m ≤ 0], so driving hinge risk down also controls training error.

  • Enforces a margin by penalizing points that are correct but within the margin (0 < m < 1). This links to max-margin generalization bounds.

  • Classification-calibrated (Bartlett, Jordan, McAuliffe; Zhang): minimizing expected hinge loss recovers the Bayes-optimal sign.

• Hinge versus other surrogates:

  • Unlike the logistic loss, hinge is exactly zero for m ≥ 1.

  • This produces sparse support vectors.

  • The cost is that it gives no calibrated probabilities.

For L(w) = max(0, 1 − y·w·x), the subgradient is −y·x when the margin y·w·x < 1 and 0 when it is > 1. It is zero for confidently correct points because the loss is flat (identically zero) in a whole neighborhood of w.

• The three cases:

  • y·w·x < 1 (violates the margin or is misclassified): the loss is the linear piece 1 − y·w·x, with gradient −y·x.

  • y·w·x > 1: the loss is constant 0, with gradient 0.

  • y·w·x = 1 (the kink): the subdifferential is the set {−t·y·x : t ∈ [0, 1]}, and any element is a valid subgradient.

• Why zero outside the margin:

  • A point with margin strictly above 1 stays above 1 under small changes to w, so its loss does not change.

  • It therefore exerts no pull on the boundary. Only margin and violating points (the support vectors) shape w.

  • This mirrors the dual, where such points have α_i = 0.

• The full regularized objective:

  • Add λw from (λ/2)‖w‖².

  • Correct points therefore still see weight shrinkage, just no data term.

Setting the Lagrangian's gradient with respect to w to zero gives w = Σ α_i y_i x_i. The optimal weight vector is a linear combination of training points. Every place w appeared (‖w‖² and w·x_i) then becomes a sum of dot products between samples.

• The derivation:

  1. Write the Lagrangian: L = ½‖w‖² − Σ α_i [y_i(w·x_i + b) − 1].

  2. ∂L/∂w = 0 gives w = Σ α_i y_i x_i.

  3. ∂L/∂b = 0 gives Σ α_i y_i = 0, which involves no features.

  4. Substituting gives the dual: max Σ α_i − ½ Σ_i Σ_j α_i α_j y_i y_j (x_i·x_j).

• Prediction also uses only inner products:

  • The decision function is f(x) = Σ α_i y_i (x_i·x) + b.

  • Only support vectors (α_i > 0) contribute.

• Consequences:

  • Kernel trick: replace x_i·x_j with K(x_i, x_j) = φ(x_i)·φ(x_j). This trains in a high- or infinite-dimensional feature space without ever computing φ.

  • The data is summarized by the n × n Gram matrix, so cost scales with the number of samples rather than the dimension. That is good for high d and costly for large n.

  • This is a special case of the representer theorem: L2-regularized losses have solutions in the span of the training data.

The dual expresses the problem entirely through inner products between training points. That enables the kernel trick, makes the number of variables equal to the number of samples rather than features, and gives a QP with simple constraints and sparse solutions. Strong duality guarantees it has the same optimum as the primal.

• Kernel trick: The dual objective and the decision function depend on data only through x_i · x_j, so you can swap in K(x_i, x_j) and train in very high- or infinite-dimensional feature spaces without ever computing φ(x).

• Dimensionality of the optimization: The primal optimizes over w, which lives in the feature space and may be infinite-dimensional. The dual optimizes n multipliers α_i, one per sample.

• Simpler constraints:

  • The constraints reduce to box constraints 0 ≤ α_i ≤ C plus one equality Σ α_i y_i = 0.

  • This structure is what decomposition methods such as SMO (used by libsvm) exploit: they update two αs at a time analytically.

• Interpretability and sparsity: Most α_i = 0. The non-zero ones identify the support vectors, and only those are needed at prediction time.

• Why it's legitimate: The problem is convex with affine constraints, so strong duality holds. The duality gap is zero, and KKT conditions link the two solutions (for example w = Σ α_i y_i φ(x_i)).

• Caveat: the dual is not always better: For linear SVMs with very large n, primal solvers (liblinear, SGD on hinge loss) scale better than building an n × n kernel matrix.

The KKT complementary-slackness conditions sort the training points into three groups by the value of α_i:

• α_i = 0 means the point lies outside the margin (not a support vector).

• 0 < α_i < C means the point lies exactly on the margin (a free support vector).

• α_i = C means the point is inside the margin or misclassified (a bounded support vector).

• Underlying conditions: α_i [y_i f(x_i) − 1 + ξ_i] = 0 and (C − α_i) ξ_i = 0.

• α_i = 0:

  • The point satisfies y_i f(x_i) ≥ 1: it is correctly classified and on or beyond the margin.

  • It has no influence on w. Removing it and retraining gives the same model.

• 0 < α_i < C:

  • Here α_i < C forces ξ_i = 0, and α_i > 0 forces y_i f(x_i) = 1.

  • These points sit exactly on the margin boundary, which is why they are used to compute the intercept b.

• α_i = C:

  • Here ξ_i ≥ 0 and y_i f(x_i) = 1 − ξ_i ≤ 1.

  • The sub-cases follow from the slack value:

  • 0 < ξ_i ≤ 1: inside the margin but still correctly classified.

  • ξ_i > 1: misclassified.

• Edge case: A point can lie exactly on the margin and still have α_i = 0 (a degenerate case). The implications run from α to position, not always the other way.

• Effect of C: A smaller C gives a wider margin, so more points hit the bound and the model has more support vectors.

The dual solution gives no value for b directly. You recover it from the KKT conditions: any free support vector (0 < α_i < C) lies exactly on the margin, so y_i f(x_i) = 1 can be solved for b.

• Derivation:

  1. For a free SV, ξ_i = 0 and y_i (Σ_j α_j y_j K(x_j, x_i) + b) = 1.

  2. Because y_i ∈ {−1, +1}, multiplying by y_i gives b = y_i − Σ_j α_j y_j K(x_j, x_i).

• Average for stability:

  • Solvers stop at a tolerance, so each free SV gives a slightly different b.

  • Averaging over all free SVs reduces this numerical noise.

• Why not use bounded SVs (α_i = C)?: Their slack ξ_i is unknown, so the equality doesn't hold. They only give inequalities on b.

• If no free SVs exist:

  • The KKT inequalities from the α = 0 and α = C points bound b to an interval.

  • libsvm takes the midpoint of that interval.

• In a linear SVM: The same formula becomes b = y_i − w·x_i, with w = Σ α_j y_j x_j.

The Gram matrix K, with K_ij = K(x_i, x_j) = φ(x_i)·φ(x_j), holds all pairwise similarities between training points in feature space. It is the only form in which the data enter the SVM dual.

• Defines the dual QP:

  • The dual maximizes Σ α_i − ½ αᵀQα, where Q_ij = y_i y_j K_ij.

  • Q is the Hessian of the dual objective.

• Geometry: The Gram matrix encodes all lengths, angles, and distances in feature space. For example, ‖φ(x_i) − φ(x_j)‖² = K_ii + K_jj − 2K_ij.

• Must be positive semi-definite:

  • PSD keeps the dual concave (a convex QP) with a global optimum.

  • For a valid kernel, the Gram matrix is PSD for any dataset.

• Diagnostic for hyperparameters:

  • A very large RBF γ makes K close to the identity: every point is similar only to itself, which leads to memorization.

  • A very small γ makes K close to all-ones: all points look alike, which leads to underfitting.

• Computational cost:

  • It needs O(n²) memory, which is why solvers like libsvm compute rows on demand and cache them (cache_size in scikit-learn).

  • You can supply it directly with SVC(kernel='precomputed').

Choose a linear kernel when the data is already high-dimensional or likely close to linearly separable, when the dataset is large, or when you need interpretability. In these cases RBF adds cost and tuning with little or no accuracy gain.

• High-dimensional, sparse features:

  • Text (bag-of-words, TF-IDF) and genomics data are typical examples.

  • When the number of features is large relative to the number of samples, classes are often already linearly separable. Mapping to an even higher space mainly adds overfitting risk.

• Many samples:

  • Kernel SVMs scale roughly O(n²) to O(n³) in time and need O(n²) kernel memory.

  • Linear solvers (LinearSVC / liblinear, SGDClassifier) scale about linearly and handle millions of rows.

• Interpretability and speed:

  • You get an explicit weight vector w, which gives feature importances.

  • Prediction is a single dot product instead of a sum over the support vectors.

  • There is only C to tune, rather than C and γ.

• When RBF is the better choice:

  • Use RBF for low-dimensional, dense features with clearly nonlinear structure and a moderate n (up to tens of thousands).

  • Note: a well-tuned RBF can approximate linear behavior (small γ), so it is rarely worse in accuracy. The trade-off is compute and tuning effort.

• Practical workflow: start with linear as a baseline, then try RBF if the linear model underfits.

Prefer a polynomial kernel when a linear model underfits because the target depends on interactions between features (products and conjunctions) of bounded order, and you want that structure without the fully local flexibility of RBF.

• Signals that favor polynomial:

  • Linear SVM training and validation accuracy are both low, a sign of high bias.

  • Domain knowledge suggests pairwise or low-order interactions, for example "word A AND word B" in NLP or pixel co-occurrences in images.

  • Degree-2 kernels were historically strong for parsing and digit recognition.

• Advantages over doing it by hand:

  • The kernel trick models all C(n + d, d) monomials implicitly, avoiding an explicit blow-up in the number of features.

  • The degree acts as an interpretable complexity knob: d = 2 means pairwise interactions.

• Costs and caveats:

  • There are more hyperparameters (degree, gamma, coef0, C), and high degrees are numerically unstable. Scale or normalize the features first.

  • It carries kernel-SVM scaling costs. For large n, explicit PolynomialFeatures (degree 2) plus a linear SVM is often faster.

  • If the nonlinearity is not interaction-shaped, RBF usually generalizes better.

A polynomial kernel K(x, z) = (γ·x·z + r)^d equals an inner product between explicit feature vectors that contain every product of input features up to degree d. Expanding the power generates those interaction terms automatically, but the kernel only ever computes one dot product and raises it to a power, so the expanded vectors are never built.

• Worked example (d = 2, r = 1, γ = 1):

  • (x·z + 1)^2 = Σ_i Σ_j x_i x_j z_i z_j + 2 Σ_i x_i z_i + 1

  • This is exactly φ(x)·φ(z) where φ(x) = [x_i x_j for all i, j; √2·x_i; 1].

  • So the model contains pairwise interactions (x_1 x_2), squares, linear terms and a bias.

• Why this is cheaper than manual expansion:

  • The explicit space has C(p + d, d) dimensions. With p = 1000 and d = 3 that is about 167 million features.

  • The kernel costs O(p) per pair of points no matter what d is.

  • The dual SVM touches the data only through dot products, so swapping in K is enough (the kernel trick).

• What the hyperparameters control:

  • d sets the highest interaction order.

  • r (coef0 in scikit-learn) weights lower-order terms relative to higher ones. With r = 0 you get only degree-d monomials.

  • The binomial coefficients fix the relative weights of the terms. You cannot pick which interactions to include.

• Caveats:

  • Scale the features first: large values blow up (x·z)^d numerically.

  • High d overfits quickly, so keep it in the 2 to 3 range.

Exact kernel SVM training scales between roughly O(n²) and O(n³) in the number of samples n, and needs O(n²) kernel evaluations and memory if the full Gram matrix is used. This is why it becomes impractical beyond a few hundred thousand samples.

• Where the cost comes from:

  • The dual is a quadratic program over n variables α_i that is coupled through the n × n kernel matrix.

  • Computing that matrix costs O(n² · p), and storing it costs O(n²) (about 80 GB of float64 at n = 100k).

  • A generic interior-point QP solver costs O(n³).

• In practice (SMO / libsvm):

  • Decomposition solvers update two α values at a time and cache kernel rows instead of storing the whole matrix.

  • Empirical scaling is about O(n²) to O(n^2.3). It gets worse with large C, many support vectors, noisy data, or a cache that is too small.

  • The scikit-learn docs state SVC fit time as between O(p · n²) and O(p · n³).

• For contrast:

  • A linear SVM (liblinear, SGD) trains in about O(n · p) per pass.

  • Kernel approximations such as Nyström or random Fourier features turn the problem into a linear one in O(n · m), where m is the number of components.

Kernel SVMs break down mainly on scale: training is quadratic to cubic in n, and prediction is linear in the number of support vectors. They also struggle with noisy or high-dimensional data, poorly scaled or heterogeneous features, and requirements they do not natively meet, such as probabilities, interpretability, or online learning.

• Computational limits:

  • Large n (beyond about 10^5) makes O(n²) kernel memory and O(n²–n³) training infeasible.

  • Prediction costs O(n_SV · p), which is too slow for low-latency serving when the support-vector count is large.

  • Multiclass SVC uses one-vs-one, training k(k−1)/2 models.

• Statistical failure modes:

  • Heavily overlapping or noisy classes push many points to become support vectors, giving a slow model with little gain.

  • With p ≫ n, or many irrelevant features, RBF distances lose meaning and the model overfits.

  • Results are sensitive to C, γ, and the kernel choice, and the search is expensive at scale.

  • Under class imbalance, the boundary is biased toward the majority class unless you use class_weight.

• Data and requirement mismatches:

  • Unscaled, categorical, or missing-value-heavy tabular data generally suits tree ensembles better.

  • There are no native probabilities. Platt scaling (probability=True) adds an internal cross-validation step and can be poorly calibrated.

  • The model is hard to interpret, and it is not suited to streaming or incremental updates.

• Alternatives: Linear SVM or logistic regression, kernel approximation (Nyström, RBFSampler) combined with SGDClassifier, gradient boosting, or neural networks.

A kernel SVM predicts with f(x) = Σ α_i y_i K(x_i, x) + b over its support vectors. Per-sample latency is therefore O(n_SV · p), and n_SV tends to grow with training-set size. Logistic regression computes w·x + b in O(p), independent of n.

• Why n_SV grows with n:

  • Every margin violator and misclassified point becomes a support vector.

  • On noisy data, the support-vector count grows roughly linearly with n, at least in proportion to the Bayes error rate.

  • So doubling the training data can roughly double inference cost.

• Logistic regression is a fixed-size model: After training, only w (p numbers) is kept. Latency and memory do not depend on how much data was used.

• Nuances:

  • A linear-kernel SVM collapses to w = Σ α_i y_i x_i, so it is also O(p).

  • The kernel SVM must also store all support vectors, so model size scales with n_SV · p.

• Mitigations: Lower C or γ carefully, use reduced-set or budgeted SVMs, or use explicit kernel approximations (Nyström, random Fourier features) with a linear model, which gives O(m · p) with a fixed m.

Platt scaling fits a sigmoid P(y=1 | f) = 1 / (1 + exp(A·f + B)) to the SVM's decision values f(x). It is a one-feature logistic regression whose parameters A and B are learned by maximum likelihood on held-out scores.

• Why it's needed:

  • An SVM outputs a signed margin distance, not a probability. Its scale depends on C, the kernel, and the data.

  • The sigmoid assumes the class likelihoods are roughly monotonic in f, which empirically holds well for SVMs.

• Procedure:

  1. Get decision values on data not used to train the SVM (a hold-out set or cross-validation folds). Training-set scores are biased because many points sit exactly at the margin.

  2. Minimize cross-entropy over A and B, using regularized targets t₊ = (N₊+1)/(N₊+2) and t₋ = 1/(N₋+2) instead of hard 0/1 labels to avoid overfitting.

  3. A is typically negative, so larger margins map to higher probability.

• Multi-class: Pairwise OvO probabilities are combined by pairwise coupling (Wu, Lin & Weng). This is what LIBSVM does.

• Practical notes:

  • SVC(probability=True) runs an internal 5-fold CV, which makes training slower.

  • Its predict_proba argmax can disagree with predict.

  • CalibratedClassifierCV(method='sigmoid') does the same job explicitly. Use 'isotonic' when you have lots of data and the distortion is not sigmoid-shaped.

The epsilon-insensitive loss ignores any error smaller than ε and penalizes larger errors linearly: L = max(0, |y − f(x)| − ε). Predictions inside a tube of half-width ε around f(x) are treated as correct.

• Primal optimization problem:

  • Minimize ½‖w‖² + C Σ(ξᵢ + ξᵢ*).

  • Constraints: yᵢ − f(xᵢ) ≤ ε + ξᵢ, f(xᵢ) − yᵢ ≤ ε + ξᵢ*, and ξ, ξ* ≥ 0.

  • The slacks ξ and ξ* measure how far a point lies above or below the tube.

• Dead zone gives sparsity:

  • Points strictly inside the tube have zero loss and zero dual coefficients (αᵢ = αᵢ* = 0).

  • Only points on or outside the tube become support vectors, so f(x) = Σ(αᵢ − αᵢ*) K(xᵢ, x) + b.

• Linear penalty outside the tube adds robustness: The loss grows like absolute error, not squared error, so outliers have bounded influence: each dual coefficient is capped at C.

• Interpretation:

  • ε encodes the noise level you are willing to tolerate, in the units of y.

  • ε = 0 reduces the loss to L1 (absolute error) regression with L2 regularization.

Increasing ε widens the tube, so more points fall inside it at zero loss. That leaves fewer support vectors, a sparser and flatter model, and a higher risk of underfitting.

• Support vector count drops:

  • Only points on or outside the tube have nonzero dual coefficients.

  • A wider tube contains more points, so fewer qualify as support vectors and prediction gets faster.

• The fit becomes flatter and coarser: Fewer points constrain the function, so minimizing ‖w‖ pushes toward a flatter fit that ignores detail smaller than ε.

• Extremes:

  1. ε = 0: nearly every point is a support vector, and the model behaves like L1 regression with regularization.

  2. ε larger than the spread of y: the whole dataset fits in the tube. You get no support vectors and an essentially constant prediction.

• Practical tuning:

  • ε is in target units, so set it relative to the noise level of y, or scale y first.

  • Too small an ε fits noise. Too large an ε misses real structure.

Both use the same machinery: a regularized convex problem, kernels, and sparse support vectors. They invert the geometric goal, though. Classification seeks a boundary that keeps points out of a margin. Regression seeks a function whose ε-tube keeps points inside it.

• Objective:

  • Support vector classification (SVC) maximizes the margin between classes.

  • Support vector regression (SVR) minimizes ‖w‖ to get the flattest function that keeps most points within ±ε of the targets.

• Loss:

  • SVC uses hinge loss max(0, 1 − y·f(x)), which is one-sided and depends on the class sign.

  • SVR uses ε-insensitive loss max(0, |y − f(x)| − ε), which is two-sided and gives two slack variables per point.

• What counts as a support vector:

  • In SVC, points on or inside the margin, or misclassified, are support vectors. Well-classified points far from the boundary are ignored.

  • In SVR, points on or outside the tube are support vectors. Well-fit points inside the tube are ignored.

• Hyperparameters:

  • Both use C and the kernel parameters.

  • SVR adds ε, which sets error tolerance in target units. SVC has no equivalent, because its margin width is fixed at 1 by scaling.

• Output:

  • SVC thresholds sign(f(x)) to produce a label.

  • SVR uses f(x) directly as a real-valued prediction.

Ordinary least squares (OLS) minimizes squared error over every point, with no regularization. Support vector regression (SVR) minimizes an ε-insensitive linear loss plus an L2 penalty. That makes SVR more robust, sparse in support vectors, and kernelizable, at the cost of hyperparameters and no closed form.

• Loss function:

  • OLS uses Σ(y − ŷ)², so every residual counts and large residuals dominate quadratically.

  • SVR ignores residuals smaller than ε and penalizes larger ones linearly, which bounds each outlier's influence (dual coefficients are capped at C).

• Regularization:

  • OLS has none.

  • SVR includes ½‖w‖², so it is closer to ridge regression with a robust loss. C sets the trade-off.

• Solution:

  • OLS has the closed form (XᵀX)⁻¹Xᵀy, and every point affects it.

  • SVR is solved as a quadratic program. Only points on or outside the tube (the support vectors) define the fit.

• Nonlinearity:

  • SVR's dual form takes kernels naturally.

  • OLS needs explicit feature expansion to model nonlinear relationships.

• Statistics versus practice:

  • OLS is BLUE (best linear unbiased estimator) under the Gauss-Markov assumptions and supports inference: standard errors, p-values, confidence intervals.

  • SVR is a predictive tool. It needs C, ε, and kernel tuning plus feature scaling, and it scales poorly to very large n when kernelized.

ε decides which errors count at all, and C decides how much those counted errors cost relative to model flatness. Their effects are coupled, so they must be tuned together.

• Roles in the objective ½‖w‖² + C Σ(ξ + ξ*):

  • ε sets the tube width, so only deviations beyond ε create slack.

  • C scales the penalty on that slack against the flatness term ‖w‖².

• ε gates C's influence:

  • With a large ε, few points leave the tube, so the slack sum is small and C barely matters. The model stays flat whatever C is.

  • With a small ε, most points generate slack, so C dominates the fit.

• Corner cases:

  1. High C, small ε: the model chases every point. Expect many support vectors and overfitting.

  2. Low C, large ε: the model is very flat and sparse. Expect underfitting.

  3. High C, large ε: the fit is tight to points outside a coarse tube, which suits low noise with a known tolerance.

  4. Low C, small ε: many points become support vectors, but each is lightly weighted, giving a smooth, L1-like fit.

• Tuning advice:

  • Scale y first, or set ε relative to the noise standard deviation of y.

  • Grid-search C on a log scale, jointly with ε and the kernel's gamma.

  • Watch the support vector count as a diagnostic.

A one-class SVM (Schölkopf's ν-formulation) maps data into kernel feature space. There it finds the maximum-margin hyperplane separating the training points from the origin. Mapped back to input space with an RBF kernel, that hyperplane becomes a closed boundary around the dense region of normal data. ν controls the share of training points allowed outside it.

• Optimization problem:

  • Minimize ½‖w‖² + (1/(νn)) Σξᵢ − ρ, subject to w·φ(xᵢ) ≥ ρ − ξᵢ and ξᵢ ≥ 0.

  • Pushing ρ up moves the hyperplane away from the origin, which tightens the region around the data.

  • The slacks let some points fall on the origin side; these are flagged as training outliers.

  • Decision rule: f(x) = sign(Σ αᵢ K(xᵢ, x) − ρ), where +1 means normal and −1 means anomaly.

• Why the boundary wraps the data:

  • With RBF, K(x, x) = 1, so all points lie on a unit sphere in feature space.

  • Cutting a cap off that sphere corresponds to enclosing the data. This makes the method equivalent to SVDD (smallest enclosing ball).

• What ν controls (0 < ν ≤ 1):

  • It is an upper bound on the fraction of training points treated as outliers.

  • It is a lower bound on the fraction of support vectors.

  • Both fractions converge to ν as n grows, so ν is effectively your expected contamination rate.

• Interplay with gamma:

  • gamma sets boundary shape: a large value gives a tight, wiggly boundary that can overfit; a small value gives a smooth, loose one.

  • ν sets how much data is allowed outside the boundary.

  • Without labeled anomalies, tuning is hard, so it is usually guided by domain knowledge of the anomaly rate.

In high dimensions, data is usually close to linearly separable already, so a kernel adds little; a linear SVM gives a max-margin, well-regularized boundary whose training and prediction cost scale with the number of non-zero entries, not the vocabulary size.

• Linear separability comes for free:

  • With tens of thousands of TF-IDF or n-gram features and fewer samples, a hyperplane can usually separate classes (Cover's theorem intuition).

  • Text classes are often signalled by individual words, i.e. roughly additive evidence that a linear model captures well.

• The margin controls overfitting: When features vastly outnumber samples, many hyperplanes fit perfectly; maximizing the margin (L2 penalty on w) picks a robust one, and generalization bounds depend on the margin rather than the dimension.

• Computational efficiency on sparse input:

  • Dot products w·x only touch non-zero features, so cost is O(nnz).

  • Primal solvers (LinearSVC/liblinear, SGDClassifier(loss="hinge")) scale roughly linearly in samples, unlike kernel SVMs that need an O(n²) kernel matrix.

• Interpretability: each weight maps to a word or n-gram.

• Caveat: scale with MaxAbsScaler or normalized TF-IDF, not centering, which would destroy sparsity.

Prefer tree ensembles (random forests, gradient boosting) for large, heterogeneous tabular data: mixed types, missing values, unscaled features, and interactions that align with individual features. Kernel SVMs shine on smaller, homogeneous, continuous data with smooth boundaries.

• Heterogeneous tabular features: Categorical, ordinal and numeric columns on different scales: trees split on thresholds and need no scaling; an RBF kernel needs a meaningful global distance, which mixed types lack.

• Missing values and outliers: Libraries like XGBoost/LightGBM handle NaNs natively; threshold splits are robust to extreme values that distort distances.

• Large sample sizes: Kernel SVM training is roughly O(n²) to O(n³) with O(n²) memory; boosting scales near-linearly to millions of rows.

• Axis-aligned or step-like structure: Rules like "income > X and age < Y" or sharp thresholds are natural for trees but awkward for smooth kernels.

• Many irrelevant features: Trees implicitly select features; an RBF kernel weights all dimensions equally, so noise features corrupt distances.

• Operational needs: little tuning (vs. sensitive C and gamma), feature importances, native multiclass and calibrated-ish probabilities.

• SVM still wins on: small/medium dense continuous data, smooth boundaries, very high-dimensional data (often with a linear kernel).

Both score a query by similarity to stored training points, but a kernel SVM uses a learned, sparse, weighted sum over its support vectors, while KNN takes an unweighted vote among the k nearest training points with no training-time optimization.

• Kernel SVM: f(x) = sign(Σ αᵢ yᵢ K(xᵢ, x) + b):

  • The αᵢ are learned by solving a convex margin-maximization problem; only support vectors have αᵢ > 0.

  • Every support vector contributes, weighted by kernel similarity: a global, smooth decision function.

  • Model stores only the support vectors, so prediction cost is O(#SV).

• KNN: majority vote of the k closest points:

  • Lazy learner: no training, no learned weights; all points are kept.

  • Purely local: only k neighbors matter, others have zero influence; prediction is O(n) per query without index structures.

• Key differences:

  1. Which points matter: SVM chooses them during training (boundary points); KNN chooses them per query (nearest points).

  2. Weighting: SVM weights by learned α times kernel value; KNN uses equal (or distance) weights.

  3. Regularization: SVM controls complexity via C and the kernel bandwidth; KNN via k.

  4. Output: SVM gives a signed distance to a hyperplane in feature space; KNN gives a vote proportion.

• Connection: an RBF SVM with very large gamma behaves locally, like a nearest-neighbor classifier over support vectors.

An SVM picks the boundary that maximizes the margin, determined only by points near it (hinge loss); logistic regression picks the boundary that maximizes the likelihood of all points (log loss), so every point influences it.

• SVM: hinge loss max(0, 1 − y·f(x)) + ||w||²:

  • Zero loss once a point is correctly classified beyond the margin, so well-classified points have no influence.

  • The solution depends only on support vectors: a geometric, sparse criterion.

• Logistic regression: log loss log(1 + e^(−y·f(x))):

  • Never exactly zero, so all points pull on the boundary, distant ones weakly.

  • Optimizes calibrated probabilities, not margin geometry.

• Consequences:

  • Separable data: SVM gives a unique max-margin boundary; unregularized LR's weights diverge (regularized LR approaches the max-margin direction).

  • Outputs: SVM gives a score/class (probabilities need Platt scaling); LR gives probabilities directly.

  • Robustness: SVM ignores far, easy points; LR is affected by the whole distribution, including class imbalance and outliers in the bulk.

  • In practice the two losses are similar, so on many problems the boundaries are close.

Choose an SVM when the true boundary is smooth or oblique (not aligned with feature axes) and features are continuous and comparably scaled; a decision tree can only build axis-parallel, staircase boundaries and approximates diagonal or curved ones poorly.

• Oblique linear boundaries: A boundary like x₁ + x₂ = 1 is one hyperplane for a linear SVM but needs many stair-step splits for a tree, which overfits and generalizes jaggedly.

• Smooth curved boundaries: Circles, spirals or blobs suit an RBF kernel, which produces smooth contours; trees produce rectangles.

• Clear gap between classes: Max-margin puts the boundary in the middle of the gap; a tree's split threshold is placed somewhat arbitrarily within it.

• High-dimensional, dense continuous data with small n: Margin regularization generalizes well; a single tree has high variance there.

• Prefer a tree when: Structure is axis-aligned or rule-like (thresholds on single features), features are mixed or categorical, scales differ, or interpretability as rules is required.

Choose a linear SVM when you care about the classification boundary rather than probabilities, especially with high-dimensional data, a clear margin, or noisy easy points you want ignored; choose logistic regression when you need calibrated probabilities.

• Favor linear SVM when:

  • Only the label or a ranking score is needed (no probability outputs).

  • Data is (nearly) separable: SVM returns a stable max-margin solution where unregularized LR weights blow up.

  • High-dimensional sparse features (text): hinge loss often gives slightly better accuracy, and LinearSVC is fast.

  • You want the boundary driven only by hard, near-boundary examples (support vectors), not the bulk of easy points.

• Favor logistic regression when:

  • You need probabilities for thresholds, risk scoring or expected-cost decisions (SVM needs Platt scaling via CalibratedClassifierCV).

  • Interpretability as log-odds or odds ratios matters.

  • Classes overlap heavily: LR models the uncertainty explicitly.

• Reality check: performance is often very similar; pick by output needs and validate both.

The weight vector w is the normal to the separating hyperplane: the sign of each wⱼ says which class feature j pushes toward, and its magnitude (on standardized features) says how strongly it moves the decision score f(x) = w·x + b.

• Sign and magnitude:

  • wⱼ > 0: increasing feature j pushes toward the positive class; wⱼ < 0: toward the negative class.

  • A unit increase in xⱼ changes the score by wⱼ (holding others fixed).

• Score meaning: f(x)/||w|| is the signed geometric distance to the hyperplane: a margin/confidence measure, not a probability or log-odds (unlike logistic regression).

• Caveats:

  1. Compare magnitudes only after standardizing features; otherwise scale drives size.

  2. Correlated features split or swap weight, so individual weights can be unstable or misleading.

  3. With L2 regularization weights shrink but rarely hit zero; use L1 (LinearSVC(penalty="l1", dual=False)) for sparse, selection-style interpretation.

  4. Only valid for linear kernels; with nonlinear kernels w lives in an implicit feature space and is not directly accessible.

Tune them jointly with a cross-validated grid (or random/Bayesian) search over log-spaced values, because they interact: gamma sets how local each support vector's influence is, C sets how hard the model fights to classify every point, and both push toward overfitting when large.

• What each controls:

  • gamma (kernel width, roughly 1/(2σ²)): large means narrow bumps and wiggly boundaries (overfit); small means smooth, near-linear boundaries (underfit).

  • C (margin vs. training error): large penalizes misclassification heavily (narrow margin, overfit); small allows violations (wider margin, more regularization).

• Why tune jointly: Good models lie along a diagonal ridge in log(C)-log(gamma) space: a higher gamma often pairs with a lower C and vice versa, so tuning one at a time misses the optimum.

• Practical procedure:

  1. Scale features first (e.g. StandardScaler inside a Pipeline); RBF distances are meaningless otherwise.

  2. Coarse log grid: e.g. C in 1e-2…1e3, gamma in 1e-4…1e1 (centred near gamma='scale' = 1/(n_features·Var(X))).

  3. Stratified k-fold CV with a metric that fits the problem (F1/AUC if imbalanced).

  4. Refine with a finer grid around the best region; if the best value is on a grid edge, extend the grid.

  5. Confirm on a held-out test set.

• Diagnostics:

  • Plot a CV heatmap to see the ridge and how flat the optimum is; prefer the simpler (smaller C, smaller gamma) model among near-ties.

  • Nearly all points becoming support vectors with high gamma signals memorization.

• Cost: each fit is roughly O(n²)–O(n³), so on large data search on a subsample or use HalvingGridSearchCV / Bayesian optimization.

The minimum is two support vectors, one from each class, whatever the dimension n. The max-margin hyperplane is then the perpendicular bisector of the segment joining them.

• Why at least one per class:

  • The KKT condition Σ αᵢ yᵢ = 0 (with αᵢ ≥ 0) means the nonzero multipliers must include both labels. So one support vector alone cannot satisfy it.

  • Geometrically, the margin is set by the closest opposing points, so each side needs a point that touches it.

• Why two are enough:

  • If x₊ and x₋ are the closest pair, set w ∝ x₊ − x₋ and put the boundary through their midpoint.

  • Margin maximisation fixes the orientation, so you do not need n points to pin down the hyperplane.

• Common confusion:

  • A general hyperplane in ℝⁿ has n+1 parameters and needs n points to pass through. For a max-margin SVM in general position, n+1 is an upper bound on the number of margin support vectors, not a lower bound.

  • With a soft margin, many more points become support vectors: all points on or inside the margin.

For separable data, the hard-margin SVM is equivalent to finding the two closest points between the convex hulls of the two classes. The optimal hyperplane is the perpendicular bisector of the segment joining those points, and the margin equals the distance between the hulls.

• The equivalence:

  • The dual can be rewritten as: minimise ‖Σᵢ∈₊ λᵢxᵢ − Σⱼ∈₋ λⱼxⱼ‖ with λ ≥ 0 and the weights of each class summing to 1. That is exactly the nearest-points problem between two convex hulls.

  • w points along the difference of the two nearest points, and the margin width equals the hull-to-hull distance.

• What it explains:

  • Support vectors are the hull vertices or faces that contain the nearest points. Interior points have zero weight and cannot affect the boundary.

  • The data are separable exactly when the hulls do not intersect. Overlapping hulls are why a hard-margin SVM becomes infeasible.

• Soft-margin extension: reduced convex hulls:

  • Cap each weight, λᵢ ≤ D. This shrinks each hull toward its centroid until the hulls separate.

  • Smaller D means more shrinkage, which corresponds to a smaller C. A single outlier can no longer stretch its hull far.

A large margin restricts the effective capacity of the hypothesis class independently of the input dimension. Generalization bounds that depend on margin rather than on the number of features are therefore tighter, which is why SVMs can generalize in very high-dimensional or kernel feature spaces.

• Margin-based VC dimension:

  • Hyperplanes with margin γ on data inside a ball of radius R have VC dimension ≤ min(⌈R²/γ²⌉, n) + 1.

  • A larger γ means lower capacity, even when n is huge or infinite (as with the RBF kernel).

• Rademacher and fat-shattering bounds:

  • Test error ≤ (fraction of training points with margin < γ) + roughly O(R / (γ√m)).

  • This is a trade-off: a bigger margin shrinks the complexity term but may cost some margin violations. That trade-off is what C controls.

• Sparsity (leave-one-out) bound:

  • Expected LOO error ≤ E[#support vectors] / m.

  • Removing a non-support vector does not change the solution, so fewer support vectors means a tighter bound.

• Intuition:

  • A wide margin tolerates small perturbations of the inputs, so noise or slight distribution shift is less likely to flip predictions.

  • There are fewer "wiggly" separators to choose from, so there is less room to fit noise.