57 Classification Interview Questions and Answers (2026)

In multi-class classification each sample belongs to exactly one of K mutually exclusive classes. In multi-label classification each sample can belong to any subset of the labels: zero, one, or several at once.

• Multi-class: one label per sample:

  • Example: classify an image as cat, dog, or bird. It is exactly one of them.

  • Predicted probabilities form a single distribution that sums to 1 (softmax), and the prediction is the argmax.

• Multi-label: a set of labels per sample:

  • Example: tag a movie as action, comedy, and romance at the same time.

  • Each label is an independent yes/no decision (one sigmoid per label), and each is thresholded separately.

• Consequences of the difference:

  • Targets: a one-hot vector or class index for multi-class, versus a multi-hot binary vector for multi-label.

  • Loss: categorical cross-entropy for multi-class, versus binary cross-entropy summed over labels for multi-label.

  • Metrics: accuracy and macro-F1 for multi-class, versus Hamming loss, subset accuracy, and micro/macro-F1 per label for multi-label.

• Key test: ask whether the classes are mutually exclusive. If they are, the problem is multi-class. If not, it is multi-label.

Multi-class targets encode one active class per sample, as an integer index or a one-hot vector. Multi-label targets encode a set of active classes, as a multi-hot binary vector (an N×K indicator matrix).

• Multi-class encoding:

  • Integer index (e.g. 2), used by sparse_categorical_crossentropy or PyTorch CrossEntropyLoss.

  • Or one-hot ([0,0,1]): exactly one 1 per row, and each row sums to 1.

  • The LabelEncoder in scikit-learn produces indices.

• Multi-label encoding:

  • Multi-hot ([1,0,1]): any number of 1s, including all zeros.

  • The MultiLabelBinarizer in scikit-learn turns label sets into this matrix.

  • Pairs with per-label sigmoid plus BCEWithLogitsLoss and float targets.

• Common pitfall: feeding multi-hot targets to a softmax/categorical loss forces probability mass to be split across true labels, so the model cannot be confident about all of them.

A majority-class baseline is a trivial classifier that always predicts the most frequent class in the training data. Any real model must beat it to show it has learned something.

• What it gives you:

  • Its accuracy equals the majority-class proportion. For example, 95% negatives gives 95% accuracy.

  • This exposes misleading accuracy on imbalanced data: a 95%-accurate model may be no better than the baseline.

• Its limits:

  • Recall for minority classes is 0, so macro-F1 and balanced accuracy expose it immediately.

  • ROC-AUC is 0.5 because it produces constant scores.

• How to implement and compare:

  • Use DummyClassifier(strategy="most_frequent") in scikit-learn.

  • Compare against other simple baselines too: stratified random guessing, and a simple model such as logistic regression.

  • For multi-label tasks, the analogue is predicting each label's majority value. This is often "all zeros", which can still score a low Hamming loss.

Binary classification predicts one of two mutually exclusive classes (spam/not spam). It is the building block of the other types: multi-class is the generalisation to K exclusive classes, and multi-label is a set of K binary problems.

• Binary setup:

  • A single output passes through a sigmoid to give P(positive), then a threshold (0.5 by default, tunable) decides the class.

  • The loss is binary cross-entropy. Metrics include precision, recall, F1, and ROC/PR-AUC.

• Relation to multi-class:

  • Binary is multi-class with K=2. A 2-way softmax is equivalent to a sigmoid.

  • Binary-only learners (e.g. SVMs) are extended to multi-class through decomposition. One-vs-Rest uses K classifiers. One-vs-One uses K(K-1)/2 classifiers.

• Relation to multi-label:

  • Binary relevance trains one independent binary classifier per label.

  • Classifier chains feed earlier label predictions into later ones to capture label correlations.

• Relation to multi-output: Several target variables are predicted at once, and each can be binary or multi-class.

The margin is how far data points sit from the decision boundary, measured on the correct side. A large margin means predictions are confident and robust to small perturbations, which is why SVMs explicitly maximize it.

• Functional margin:

  • Defined as y·f(x) with labels y ∈ {-1, +1}.

  • A positive value means the point is correctly classified; a larger value means more confident.

  • It is scale-dependent: doubling w doubles it.

• Geometric margin:

  • Defined as y(w·x + b)/||w||: the actual distance to the hyperplane, so it is scale-invariant.

  • The margin of the classifier is the minimum of this value over the training points.

• SVM view:

  • Hard-margin SVM fixes the functional margin at 1 and minimizes ||w||, so the margin width is 2/||w||.

  • The points that lie on the margin are the support vectors. They alone define the boundary.

  • Soft margin adds slack variables, and C trades margin width against training violations.

• Why it matters:

  • Generalization bounds depend on margin, not dimension, which explains why SVMs work in very high-dimensional kernel spaces.

  • Margin also appears in loss functions: hinge loss penalizes y·f(x) < 1, and boosting increases margins as it adds rounds.

The boundary is linear when the decision rule is a threshold on a linear function of the input features: w·x + b = 0. It becomes non-linear when the model's decision function is non-linear in the original inputs, through engineered features, kernels, hidden layers, or local/partition-based rules.

• Linear in the inputs gives a linear boundary:

  • Logistic regression, linear SVM, the perceptron, and LDA all produce a hyperplane.

  • A monotonic link such as the sigmoid doesn't change this: p = 0.5 still occurs exactly where w·x + b = 0.

• Sources of non-linearity:

  • Feature maps: polynomial or interaction terms give a boundary that is linear in φ(x) but curved in x.

  • Kernels: RBF or polynomial SVMs apply the same idea implicitly.

  • Neural networks: non-linear activations between layers. Without activations, a deep network collapses to a single linear model.

  • Trees and ensembles: axis-aligned splits produce piecewise, staircase-shaped boundaries.

  • k-NN: the boundary follows the local data, giving pieces of a Voronoi diagram.

• Generative models depend on their assumptions:

  • LDA uses a shared covariance, so the boundary is linear.

  • QDA uses per-class covariances, so the boundary is quadratic.

  • Gaussian Naive Bayes is generally quadratic, but linear when the variances are shared.

• Key insight: "linear" is always relative to the feature space. The same model can be linear in transformed features and non-linear in the raw inputs.

A decision boundary is the surface in feature space where the classifier's predicted class changes. Points on one side get class A, points on the other get class B, and on the boundary itself the model is exactly indifferent (for example, P(y=1|x) = 0.5 or f(x) = 0).

• Formal definition:

  • It is the set of x where the top two class scores tie.

  • In d dimensions it is usually a (d−1)-dimensional surface: a line in 2D, a plane in 3D, a hyperplane beyond that.

• Shape reflects the model:

  • Linear models give a hyperplane.

  • Kernel SVMs and neural networks give curved boundaries.

  • Trees give axis-aligned boxes.

  • k-NN gives jagged boundaries that track the data.

• It depends on the threshold:

  • Changing the decision threshold (e.g. 0.5 to 0.8) moves the boundary while keeping the same model.

  • This is how you trade precision against recall.

• Multi-class case: There are several pairwise boundaries, which partition the space into class regions.

• Diagnostic value:

  • An overly wiggly boundary that wraps individual training points signals overfitting (high variance).

  • A boundary too rigid to fit the true class shape signals underfitting (high bias).

One-vs-one (OvO) splits a K-class problem into K(K-1)/2 binary problems, one for every pair of classes. At prediction time each pairwise classifier votes for one of its two classes, and the class with the most votes wins.

• Training:

  • For each pair (i, j), train a classifier using only the samples from classes i and j. All other classes are ignored.

  • Example: 4 classes gives 6 classifiers (A-B, A-C, A-D, B-C, B-D, C-D).

• Prediction:

  1. Run the sample through all pairwise classifiers.

  2. Each classifier adds one vote to the class it prefers. Some variants sum confidences instead of counting hard votes.

  3. Predict the class with the most votes, breaking ties if needed.

• Advantages:

  • Each subproblem is small and roughly balanced, since it holds two classes of similar size.

  • Pairwise boundaries are often simpler than one-vs-all boundaries.

  • It suits kernel SVMs, whose training cost grows superlinearly with n. This is why libsvm and sklearn.svm.SVC use OvO internally.

• Drawbacks:

  • The number of models grows quadratically, which raises inference cost and memory.

  • Votes can tie.

  • "Incompetent" classifiers still vote. An A-vs-B model must vote on a class-C sample it never saw during training.

  • Votes are not probabilities. Getting probabilities needs pairwise coupling (as in SVC(probability=True)).

Ordinal classification (ordinal regression) predicts a discrete label from classes that have a natural order but no meaningful numeric spacing. Examples are ratings of 1–5 stars, disease severity graded mild, moderate or severe, and education level.

• It sits between classification and regression:

  • It is unlike nominal classes, because order matters: confusing 1 with 5 is worse than confusing 1 with 2.

  • It is unlike regression, because the gaps between levels are not assumed equal or even measurable.

• Common approaches:

  • Cumulative link models (ordered logit/probit): a single latent score with learned thresholds.

  • Cumulative binary decomposition: K−1 tasks of the form "is y > k?" (Frank & Hall, CORAL, CORN in deep learning).

  • Regression plus rounding or tuned cut-points: a simple baseline that assumes equal spacing.

• Evaluation uses order-aware metrics:

  • Mean absolute error measured on class ranks.

  • Quadratic weighted kappa.

  • Plain accuracy is not enough, because it ignores how far off a prediction is.

Hierarchical classification is classification where the labels are organised in a taxonomy, either a tree or a DAG. A prediction must respect that structure: if an item is labelled "Labrador", it is also implicitly "Dog" and "Animal". The hierarchy can be used both to structure the model and to grade its mistakes.

• Problem variants:

  • Tree versus DAG: in a DAG a node can have multiple parents.

  • Single-path versus multi-path: multi-path is hierarchical multi-label classification.

  • Mandatory leaf versus non-mandatory leaf: in the latter, prediction may stop at an internal node when unsure.

• Modelling strategies:

  • Flat: ignore the hierarchy, predict leaves directly, and infer ancestors from the chosen leaf.

  • Local classifiers, which are usually applied top-down:

  • one classifier per node;

  • one classifier per parent node;

  • or one classifier per level.

  • Global (big-bang): a single model that predicts the whole structure, often with a hierarchy-aware loss or constraints.

• Evaluation:

  • Hierarchical precision, recall and F1 compare the ancestor-augmented label sets, giving partial credit to near-misses.

  • Mistaking two sibling classes is penalised less than a cross-branch error.

Use independent sigmoids when the classes are not mutually exclusive: a sample can have several labels, or none. Softmax forces the outputs to compete and sum to 1, which assumes exactly one correct class.

• Multi-label tasks:

  • Examples: tagging, multi-disease detection from an X-ray, detecting several objects present in an image.

  • Each sigmoid gives P(label present) independently, so raising one label's score doesn't lower another's.

• "None of the above" or open-set cases: With sigmoids, all outputs can be low. Softmax always assigns most of the mass to some class, even for out-of-distribution inputs.

• Per-label thresholds: Thresholds can be tuned per label to trade precision against recall, which is useful with imbalanced labels.

• Training details: Use BCEWithLogitsLoss with multi-hot targets, optionally with pos_weight for rare labels.

• Keep softmax when exactly one class is true: Its coupling then helps: it gives calibrated competition between classes and a single argmax decision.

Multi-label is a special case of multi-output where every output is binary (label present or absent). Multi-output (multi-output multi-class) predicts several target variables at once, and each can have more than two classes.

• Multi-label:

  • There are K binary targets drawn from one label vocabulary. Example: genres {action, comedy, drama}.

  • Targets form a multi-hot vector.

• Multi-output:

  • There are several distinct target variables, each multi-class. Example: predict a fruit's type {apple, pear} and its color {red, green, yellow}.

  • Each output has its own class set. In neural networks this means one softmax head per output, with the losses summed.

• Hierarchy of problem types:

  • Binary sits inside multi-class (one output, K classes).

  • Multi-label is multi-output with every output binary.

  • Multi-output generalises both.

• In scikit-learn:

  • Wrap any classifier with MultiOutputClassifier to fit one estimator per target.

  • Use ClassifierChain to model dependencies between targets.

Choose multi-label when a single input can legitimately have several labels at once, or none. In other words, the categories describe overlapping attributes rather than mutually exclusive alternatives.

• Signals that point to multi-label:

  • Annotators naturally assign several tags. Examples: article topics, product attributes, symptoms.

  • Forcing one label would lose information or require arbitrary tie-breaking.

  • A "none of these" state is valid.

• Why not fake it with multi-class:

  • Label powerset (each label combination becomes one class) explodes combinatorially (2^K) and gives sparse, unseen combinations.

  • Picking only the "primary" label creates noisy targets and penalises correct secondary predictions.

• When multi-class is still right:

  • The business needs exactly one decision (e.g. route to one queue).

  • The classes are genuinely exclusive by definition (e.g. digit 0–9).

• Practical implications:

  • Use sigmoid outputs, BCE loss, and per-label thresholds.

  • Evaluate with micro/macro-F1 and Hamming loss.

  • Consider label correlations (classifier chains, shared encoders).

Treat it as a multi-output problem: two separate targets predicted from the same ticket. Category is a multi-class head, and severity is an ordinal head. Usually they share one encoder and train with a combined loss.

• Category head:

  • If each ticket goes to exactly one queue, use softmax with cross-entropy.

  • If a ticket can span teams (billing and technical), switch to a multi-label head with sigmoids.

• Severity head:

  • Severity levels are ordered (low < medium < high < critical). Use ordinal regression or cumulative-link / CORAL-style outputs, or at least a cost matrix that penalises distant mistakes more.

  • Evaluate with MAE across levels, or weighted kappa, and recall on critical tickets.

• Architecture choices:

  1. Shared encoder with two heads and loss = L_cat + λ·L_sev. This is data-efficient and exploits correlations between the targets.

  2. Two independent models. This is simpler to debug, retrain, and own separately.

  3. Chain the models: feed the predicted category into the severity model if severity depends on category.

• Avoid flattening into category×severity combo classes:

  • It multiplies the number of classes and makes rare combinations sparse.

  • It also discards the ordering of severity.

• Operational concerns:

  • Handle class imbalance, since critical tickets are rare.

  • Tune the thresholds so critical tickets are biased toward recall.

  • Send low-confidence predictions to human triage.

Map the inputs into a new feature space where the classes can be separated by a hyperplane. A linear boundary in the transformed space corresponds to a non-linear boundary in the original space.

• Classic examples:

  • XOR: adding the product x1·x2 makes the four points linearly separable.

  • Concentric circles: adding r² = x1² + x2² means a single threshold on radius separates the classes.

• Ways to transform:

  1. Explicit feature engineering: polynomial or interaction terms, logs, binning, domain ratios.

  2. The kernel trick: SVMs compute inner products K(x, x') = φ(x)·φ(x') directly, never building φ. The RBF kernel corresponds to an infinite-dimensional space.

  3. Learned representations: hidden layers of a neural network learn φ, and the final layer is a linear classifier on top of it.

  4. Random features: random Fourier features approximate kernels cheaply.

• Why higher dimensions help: Cover's theorem: data projected non-linearly into a higher-dimensional space is more likely to be linearly separable.

• Costs:

  • More features raise variance, so you need regularization (for SVMs, C and the RBF gamma).

  • Explicit polynomial expansions grow combinatorially in size.

Prefer a generative classifier, which models P(x|y)P(y), when data is scarce, partially labeled, or incomplete, or when its distributional assumptions roughly hold. It converges faster and can use structure that a discriminative P(y|x) model ignores.

• Small training sets:

  • Ng & Jordan showed Naive Bayes reaches its (higher) asymptotic error in O(log d) samples, while logistic regression needs O(d).

  • So the generative model often wins early, and the discriminative model overtakes it as data grows.

• Missing features: A model of the joint distribution can marginalize out missing inputs in a principled way, with no ad hoc imputation.

• Unlabeled data (semi-supervised learning): Unlabeled x informs P(x), for example through EM with a mixture model.

• Shifting class priors: If the deployment base rates differ from training, you can swap in a new P(y) without retraining.

• Outlier and novelty detection:

  • A low P(x) flags inputs unlike anything seen in training.

  • A discriminative model will confidently classify such inputs anyway.

• Adding classes incrementally: A new class only needs its own P(x|y) fitted.

• Caveat:

  • With plenty of labeled data and misspecified assumptions (e.g. correlated features under Naive Bayes), a discriminative model usually achieves lower error.

  • Generative probabilities are also often poorly calibrated.

One-vs-rest (OvR) trains K models, and each sees all n samples. One-vs-one (OvO) trains K(K-1)/2 models, and each sees only about 2n/K samples. OvO therefore has more models but smaller training sets. Which strategy is cheaper to train depends on how the base learner scales with n. OvO always costs more at inference.

• OvR training: about K · T(n), where T(n) is the cost to train one model on n samples.: Cost grows linearly in K, but every model sees the full dataset.

• OvO training: about K²/2 · T(2n/K):

  • Linear learner (T ∝ n): K²/2 · 2n/K ≈ K·n, which is roughly the same as OvR.

  • Superlinear learner such as a kernel SVM (T ∝ n² or worse): K²/2 · (2n/K)² ≈ 2n². That is independent of K and cheaper than OvR's K·n².

• Inference and memory:

  • OvR evaluates and stores K models.

  • OvO evaluates and stores K(K-1)/2 models. At K=100 that is 4,950 versus 100, so OvO becomes impractical for many classes.

• Rule of thumb:

  • Use OvO for kernel methods with moderate K.

  • Use OvR, or a native softmax model, for large K or when latency matters.

When two or more classes receive the same number of pairwise votes, implementations break the tie using the classifiers' confidence scores. If no confidence scores are used, they fall back to a deterministic rule such as picking the lowest class index.

• Why ties happen: Hard votes are small integers, so ties are common. Cyclic preferences are one cause (A beats B, B beats C, C beats A).

• Confidence-based tie-breaking:

  • Sum each class's decision values (margins or probabilities) across its pairwise classifiers.

  • Among the tied classes, pick the one with the highest total.

  • sklearn's OneVsOneClassifier adds a small, bounded transform of the summed confidences to the vote counts. This can reorder tied classes but never overturns a real vote difference.

• Index-based fallback:

  • libsvm and SVC by default return the first tied class in label order. This is arbitrary and biased toward low-index classes.

  • SVC(break_ties=True, decision_function_shape='ovr') instead breaks ties using decision-function confidence, at extra compute cost.

• Alternatives: Pairwise coupling estimates calibrated class probabilities from the pairwise probabilities, then takes the argmax. This avoids discrete ties entirely.

Some models represent all K classes in a single model and output a distribution or vote over all of them directly. These include tree-based models, instance-based models, probabilistic generative models, and softmax-based models. Classic SVMs and binary logistic regression are the main families that need decomposition.

• Decision trees and their ensembles (random forest, extra trees): Splits use multi-class impurity (Gini or entropy), and leaves store class distributions.

• k-nearest neighbors: The prediction is a majority vote among neighbors over any number of labels.

• Generative models (Naive Bayes, LDA, QDA): They model P(x|y)P(y) for each class and take the argmax of the posterior.

• Softmax-based models:

  • Multinomial logistic regression (LogisticRegression with the lbfgs solver) is one example.

  • Neural networks with a K-way softmax output and cross-entropy loss are another.

  • Classes are learned jointly, and the probabilities sum to 1.

• Gradient boosting (XGBoost multi:softprob, LightGBM multiclass): It grows one tree per class per round, but optimizes a joint softmax loss rather than independent binary tasks.

• Need decomposition: Standard SVMs and the perceptron need OvR or OvO. Multi-class SVM formulations such as Crammer-Singer exist but are less common.

One-vs-rest pits one class against the union of all other K-1 classes. With K balanced classes, each binary problem therefore has a positive:negative ratio of about 1:(K-1), regardless of how balanced the original data was.

• The arithmetic:

  • 10 balanced classes give each classifier 10% positives and 90% negatives.

  • 100 classes give 1% positives. Imbalance worsens as K grows.

• Consequences:

  • Each model is biased toward predicting "rest". It can reach high accuracy while having poor recall on its own class.

  • The "rest" class is heterogeneous (many sub-clusters), which makes its boundary harder to learn.

  • Score scales differ across classifiers, which complicates the final argmax.

• Mitigations:

  • Use class weights (class_weight='balanced') or resampling within each binary task.

  • Calibrate the scores before comparing them.

  • Use OvO, whose pairs stay balanced, or a native softmax model.

The Bayes-optimal classifier is the decision rule that minimizes expected loss under the true data distribution. Under 0-1 loss it predicts argmax_y P(y|x). No classifier can achieve lower risk, so it serves as a theoretical benchmark rather than something you can build exactly.

• General form:

  • Choose the action a minimizing Σ_y L(a, y) P(y|x).

  • With 0-1 loss this reduces to picking the most probable class.

• Its error is the Bayes error rate: E_x[1 - max_y P(y|x)]: the floor any model can reach on that feature set.

• Why you can't build it directly:

  • It requires the true P(y|x), which is unknown.

  • Real models approximate it, either generatively (P(x|y)P(y)) or discriminatively (P(y|x)).

  • Consistent learners approach it as data grows.

• Alternative meaning (Mitchell):

  • Averaging predictions over all hypotheses, weighted by their posterior given the data: P(y|D) = Σ_h P(y|h) P(h|D).

  • No single hypothesis does better on average given the same hypothesis space and prior.

A decision score only needs to rank examples correctly: higher means more likely positive. A calibrated posterior must also match observed frequencies: among cases scored 0.8, about 80% should be positive. Many models output good rankings but poorly calibrated values.

• Uncalibrated scores:

  • Examples: SVM margins, raw logits, boosted-tree scores.

  • Even predict_proba outputs can be off: naive Bayes pushes to extremes, random forests avoid 0 and 1, and modern deep nets tend to be overconfident.

  • They are fine for ranking metrics (AUC) and for picking a threshold empirically.

• Why true probabilities matter:

  • Cost-sensitive thresholds such as p > C_FP/(C_FP+C_FN) assume real probabilities.

  • So do combining models, expected-value decisions, and correcting for prior shift.

• Calibration methods (fit on held-out data):

  • Platt scaling: fits a sigmoid on the score.

  • Isotonic regression: a monotone, nonparametric fit that needs more data.

  • Temperature scaling: a single parameter for neural-net logits.

  • Because these maps are monotone, ranking and AUC are unchanged.

• How to check calibration: Reliability diagram, expected calibration error (ECE), Brier score, log loss.

Label powerset (LP) turns every distinct label combination into its own class. With L labels there are up to 2^L possible classes, so the multi-class problem grows exponentially while the data available per class shrinks.

• Exponential class space: 10 labels can give 1,024 classes; 30 labels can give over a billion. In practice the count is capped at min(2^L, N) observed combinations, but that is still very large.

• Data sparsity and imbalance:

  • Many combinations appear only once or a few times, so their classes are badly underfit.

  • A few common combinations dominate, which gives a severe long-tail imbalance.

• Cannot generalize to unseen combinations: LP can only predict label sets it saw during training, even when each individual label is well represented.

• Computational cost: More classes means larger output layers, more one-vs-rest models, and slower training and inference.

• Mitigations:

  • RAkEL builds LP models on small random label subsets (k-labelsets) and combines them by voting.

  • Pruned problem transformation removes or remaps rare combinations.

Binary relevance (BR) trains one independent classifier per label, so it cannot use correlations between labels. It throws away co-occurrence information and can output label sets that are inconsistent or implausible.

• Ignores useful co-occurrence signal: Knowing an image is tagged "beach" makes "sea" more likely. BR cannot exploit this, which particularly hurts rare labels that could borrow strength from correlated common ones.

• Inconsistent outputs:

  • It may predict mutually exclusive labels together (e.g. "desert" and "snow").

  • It may violate hierarchies, such as predicting "poodle" without "dog".

• Optimizes marginals, not the joint:

  • BR is reasonable for per-label metrics like Hamming loss.

  • It is suboptimal for subset accuracy (exact match), which depends on the joint distribution P(y1..yL | x).

• Remedies: Classifier chains, stacking (2BR), label powerset or RAkEL, and neural nets with a shared representation across label heads.

• Upside of independence: it is simple, parallelizable, scales linearly in L, and can predict any label combination.

A single 0.5 cutoff assumes every label's scores are calibrated, balanced, and equally costly to get wrong, which is rarely true. Tuning a threshold per label adapts to each label's base rate, calibration, and error costs, which usually improves F1 or recall on rare labels.

• Different base rates: For rare labels the predicted probabilities rarely exceed 0.5, so recall collapses. A lower threshold recovers the positives.

• Different calibration: Each label's model may be over- or under-confident in its own way (class weighting and resampling both distort scores).

• The metric's optimum is not 0.5: For F1 with calibrated probabilities, the optimal threshold is roughly half the best achievable F1, which is often well below 0.5.

• Different costs: A missed "toxic" tag may cost far more than a false alarm, while a spurious "sports" tag is cheap.

• How to tune:

  1. Use a held-out validation set or out-of-fold predictions.

  2. For each label, sweep thresholds and pick the one that maximizes the target metric (per-label F1 or expected cost).

• Pitfall: overfitting on rare labels:

  • A label with only a few positives gives noisy thresholds.

  • Use cross-validation, shrink toward a global threshold, or fall back to the global value when positives are scarce.

Pick the decision rule that matches your evaluation metric and product constraints:

• Per-label thresholds: when the number of labels per instance varies naturally and you care about precision, recall, or F1.

• Top-k: when the output size is fixed by design and you care about ranking quality.

• Hybrids or a cardinality predictor: when neither alone fits.

• Per-label (or global) thresholds:

  • The output size adapts to each instance: zero, one, or many labels.

  • Best for metrics like F1, Hamming loss, and subset accuracy.

  • It depends on calibration and can return empty sets. A common fix is to always emit the argmax label.

• Top-k:

  • Always returns k labels, ranked by score.

  • Fits UIs ("show 3 tags"), recommendation, and metrics like precision@k and nDCG.

  • Only relative scores matter, so it is robust to poor calibration.

  • It forces false positives when an instance has fewer than k true labels, and misses labels when it has more.

• Hybrids:

  • Top-k plus a minimum score, i.e. at most k labels above threshold t.

  • Rank-cut at the largest gap between consecutive sorted scores.

• Predict the cardinality: Train a meta-model that predicts how many labels an instance has, then take that many top-ranked labels.

• Deciding in practice:

  1. Look at the label-cardinality distribution. If it is tight, top-k works; if it is wide, prefer thresholds.

  2. Align the rule with the business metric and UI constraints.

  3. Compare the candidate rules on validation data using that metric.

In a top-down hierarchical classifier, each level only chooses among the children of the node selected above it. A wrong decision at an upper level therefore sends the example down the wrong branch, and no later classifier can recover it. Errors compound with depth.

• Why it compounds:

  • Leaf accuracy is roughly the product of the routing accuracies along the path.

  • For example, 0.95 at each of 4 levels gives about 0.81 end to end.

• Train/test mismatch makes it worse:

  • Child classifiers are usually trained only on examples that truly belong to their parent.

  • At test time they also receive misrouted examples, so they confidently pick one of their own children for an input that belongs elsewhere.

• Mitigations:

  • Soft routing: multiply probabilities along each path and pick the best overall path, or keep the top-k branches (beam search) instead of committing greedily.

  • Allow stopping at an internal node when confidence is low (non-mandatory leaf prediction).

  • Train child classifiers on the parent's actual predictions, or add an "other/reject" class so misrouted inputs can be flagged.

  • Use a flat or global model, possibly with hierarchy-aware losses, which avoids hard sequential decisions altogether.

Standard multi-class classification treats labels as unordered and interchangeable. Ordinal classification uses the fact that labels have a rank, which changes how the model is built, how errors are costed and how results are evaluated.

• Error cost:

  • Multi-class: every mistake costs the same under 0-1 or cross-entropy loss.

  • Ordinal: predicting 4 for a true 5 should cost less than predicting 1.

• Model structure:

  • Multi-class: a softmax over K independent class scores. This discards the order and can produce bimodal predictions, such as high probability on both 1 and 5.

  • Ordinal: one latent score plus ordered thresholds, or cumulative "y > k" tasks. Both build in monotonic structure and share information across neighbouring classes.

• Data efficiency:

  • Ordinal models need fewer parameters per class and borrow strength from adjacent levels.

  • This helps rare middle or extreme classes.

• Contrast with regression: Regressing on 1..K assumes equal spacing between levels. Ordinal methods learn the thresholds instead.

• Metrics: Use MAE on ranks or quadratic weighted kappa rather than plain accuracy or macro-F1 alone.

You need to enforce the "true-path rule": a node may only be predicted if all its ancestors are predicted. Equivalently, a child's score must never exceed its parent's. This can be guaranteed by the inference procedure, by post-processing scores, or by building the constraint into the model.

• Consistent by construction:

  • Top-down inference only evaluates children of nodes that were already accepted.

  • Leaf-only flat prediction followed by adding all ancestors of the chosen leaf.

• Post-hoc score correction (for multi-label or DAG outputs):

  • Bottom-up: set each parent's score to max(own, children), then threshold.

  • Top-down: set each child's score to min(own, parent), or multiply conditional probabilities along the path.

  • With a single shared threshold, monotone scores give consistent label sets automatically.

• Constraint-aware models:

  • Model P(child | parent) so that the marginal P(child) = P(child | parent)·P(parent) ≤ P(parent).

  • Use a max-constraint layer (e.g. C-HMCNN) or a hierarchy-violation penalty in the loss.

  • Use structured decoding that searches only over valid label sets.

• Training hygiene: Propagate ground-truth labels up so that every positive leaf also marks its ancestors positive. Otherwise the model learns inconsistency from the data.

The reject option lets a classifier abstain ("I don't know") instead of predicting when its confidence is too low. Rejected cases go to a human or a fallback system. The model trades coverage (the fraction of inputs it predicts on) for lower error on the inputs it does accept.

• Chow's rule (1970):

  • Reject if max_k p(k|x) < t.

  • With a misclassification cost of 1 and an abstention cost of c, the optimal threshold is t = 1 - c.

• Error-reject trade-off:

  • Raising the threshold lowers the error on accepted samples but rejects more inputs.

  • Performance is summarized by risk-coverage curves; this framing is also called selective classification.

• Two kinds of rejection:

  • Ambiguity rejection: the input lies near a decision boundary, where classes overlap.

  • Distance or novelty rejection: the input lies far from all the training data (out-of-distribution).

  • Max-probability thresholding handles the first well but the second poorly.

• Requirement: the probabilities must be calibrated (e.g., via temperature scaling). Otherwise the threshold does not mean what the cost analysis assumes.

Predictive entropy measures how spread out the predicted class distribution is. The idea is that out-of-distribution (OOD) inputs tend to give flatter, higher-entropy predictions. You flag an input as OOD when its entropy exceeds a threshold chosen on in-distribution validation data.

• Score:

  • H(x) = -Σ_k p(k|x) log p(k|x).

  • It is 0 for a one-hot prediction and log K for a uniform one.

• Thresholding:

  • Pick the threshold so that, for example, 95% of in-distribution validation inputs are accepted.

  • Evaluate with AUROC and FPR@95%TPR on held-out OOD sets.

• Limitations:

  • A single softmax network is often overconfident on OOD inputs, so their entropy can stay low.

  • High entropy also arises from in-distribution ambiguity (aleatoric noise), so entropy alone confuses "hard" inputs with "novel" ones.

• Making it work better:

  • Compute entropy from an averaged predictive distribution: deep ensembles or MC dropout.

  • Decompose total entropy = expected entropy (aleatoric) + mutual information (epistemic).

  • Mutual information, i.e. disagreement between ensemble members, is the better OOD signal.

Usually start binary: the core business decision (block, review, or allow) is binary, and fraud labels are scarce and noisy. Move to multi-class or a two-stage design only when different fraud types need different actions, and you have enough labeled examples per type.

• Why binary is the default:

  • Positives are already rare (often under 1%). Splitting them into types leaves some classes with almost no examples.

  • Type labels are often unreliable: chargeback reason codes are inconsistent, and many cases are never investigated deeply.

  • One fraud score maps cleanly onto a single threshold tuned to cost (fraud loss vs. friction for good customers).

• When multi-class earns its place:

  • Different responses per type: account takeover triggers a step-up login, while card testing triggers velocity blocks.

  • Different features or patterns per type: a shared binary model can underfit a minority type that the more common types drown out.

  • Investigators need routing to specialized queues.

• Common hybrid designs:

  1. Hierarchical: a binary fraud gate, then a type classifier that runs only on flagged cases.

  2. Multi-label: one transaction can show several fraud signals at once, so use per-type sigmoid outputs rather than a softmax.

  3. Multi-task: a shared representation with a binary head plus auxiliary type heads. The type heads can help the binary task.

• Interview framing: let the decision and the label quality drive the choice, not modeling elegance. Novel fraud types also argue for a binary model plus anomaly detection, since a fixed type taxonomy can't name attacks it hasn't seen.

Walk it as a pipeline where each step constrains the next: define the tags and the decisions they drive, get reliable labels, choose the problem shape and a baseline, evaluate on the right metric, then convert scores into actions with explicit thresholds.

1. Define labels and use:

   • Build a tag taxonomy with written definitions, edge cases, and examples, then ask what each tag drives (search, ads, moderation).

   • Decide whether tags are mutually exclusive (multi-class) or can co-occur (multi-label). Content tagging is usually multi-label.

2. Collect labels:

   • Use annotation guidelines and measure inter-annotator agreement (e.g. Cohen's kappa). Low agreement means the definition is broken, not the model.

   • Handle the long tail of rare tags with targeted sampling or active learning, and treat weak labels (user tags) as noisy.

3. Data splits:

   • Split by time or by creator to avoid leakage from near-duplicates.

   • Keep a clean, expert-labeled test set.

4. Model:

   • Start with a baseline such as TF-IDF plus logistic regression with one-vs-rest heads.

   • Then try a fine-tuned encoder with sigmoid outputs and BCEWithLogitsLoss, plus class weighting for rare tags.

5. Evaluate:

   • Report per-tag precision/recall and PR-AUC.

   • Use macro-F1 so rare tags count, and micro-F1 for overall volume.

   • Run error analysis by slice (language, content type).

6. Calibrate and set the decision rule:

   • Calibrate scores first (Platt or isotonic) so probabilities mean something.

   • Set per-tag thresholds from costs: high precision for user-visible or moderation tags, high recall for retrieval tags.

   • Optionally use three zones: auto-tag above a high threshold, send to human review in the middle band, drop below it.

7. Ship and monitor:

   • Track drift in tag rates and audit samples.

   • Retrain as the taxonomy evolves.

With many features (p) and few samples (n), the variance term dominates. A flexible model can fit noise and spurious correlations perfectly, while a constrained model's bias costs far less than the variance it avoids.

• Separability is cheap when p ≫ n:

  • With p ≥ n, almost any labeling of the training points is linearly separable, even random labels.

  • A high-capacity model therefore reaches zero training error easily, and that number tells you nothing about generalization.

• Spurious correlations:

  • With thousands of features, some will correlate with the label by chance.

  • Flexible models exploit these, and the correlations vanish on new data.

• Curse of dimensionality:

  • Distances concentrate, so local methods (k-NN, RBF kernels) lose meaning.

  • Data becomes too sparse to estimate complex interactions.

• Why simple models win:

  • Fewer effective parameters mean lower variance.

  • Strong regularization (L1/lasso for sparsity, L2/ridge) or Naive Bayes-style independence assumptions act as useful priors.

  • The true signal is often approximately linear or sparse anyway (e.g. genomics, text).

• Practical implications:

  • Use nested cross-validation and keep feature selection inside each fold to avoid selection leakage.

  • Prefer linear SVM or regularized logistic regression as baselines.

  • Add capacity only if validation scores justify it.

With tens of thousands to millions of classes (extreme classification), a flat K-way model becomes too expensive to train, store, and serve. Most classes also have very few examples. Solutions avoid touching all K classes for every sample.

• Compute and memory:

  • The softmax normalization and its gradient cost O(K) per example. The output layer alone can hold billions of parameters.

  • OvR means K models, and OvO means about K²/2, which is infeasible at this scale.

  • Inference latency grows if every class must be scored.

• Data issues:

  • Label frequencies follow a long-tailed distribution. Most classes have only a handful of examples, and some new ones have none.

  • Labels are often missing, noisy, or semantically overlapping, which is frequently a multi-label setting.

• Evaluation: Plain accuracy is uninformative. Use precision@k and recall@k, plus propensity-scored metrics to credit tail labels.

• Common solutions:

  • Approximate the softmax: hierarchical softmax (O(log K)), sampled softmax, or negative sampling.

  • Use tree or label-partitioning methods (e.g. Parabel), which route a sample to a small subset of labels.

  • Use embedding-based retrieval: embed inputs and labels, shortlist candidates with approximate nearest neighbor search, then re-rank. This also handles new labels zero-shot.

Each OvR classifier is trained independently on a different binary problem. Its raw output (an SVM margin, logit, or tree score) is therefore on its own arbitrary scale, so a score of 2.0 from one model does not mean the same confidence as 2.0 from another. Taking the argmax over such scores can systematically favor some classes.

• Sources of scale mismatch:

  • Each subproblem has a different class prior and imbalance (1 vs K-1), which shifts each model's bias term.

  • Separability differs per class. An easy class gets large margins, and a hard class gets small ones.

  • Regularization affects weight norms differently per model, and SVM margins are not probabilities at all.

• Result:

  • The argmax reflects scale artifacts, not true likelihood.

  • The "probabilities" do not sum to 1, and the model can assign a high score to several classes or to none.

• Fixes:

  • Calibrate each binary model on held-out data with Platt scaling or isotonic regression (CalibratedClassifierCV), then normalize across classes.

  • Or train jointly with a multinomial softmax, which puts all classes on a shared scale.

Error-correcting output codes (ECOC) assign each class a binary codeword and train one classifier per bit. A sample is then predicted as the class whose codeword is nearest to the vector of bit predictions. With well-separated codewords, a few individual classifiers can be wrong and the correct class is still recovered. OvR has no such redundancy.

• How it works:

  1. Build a K × L code matrix. Each column defines a binary split, a group of classes versus the rest.

  2. Train L binary classifiers, one per column.

  3. Decode by choosing the class with the minimum Hamming distance, or a loss-based distance using soft scores.

• Error correction:

  • If the minimum Hamming distance between codewords is d, up to ⌊(d-1)/2⌋ bit errors can be corrected.

  • OvR is the identity code, with d = 2. It corrects zero errors: a single wrong classifier can flip the prediction.

• Flexible code length:

  • L > K adds redundancy for robustness, which acts like an ensemble and reduces variance.

  • L ≈ log₂K compresses the problem into far fewer classifiers than OvR (OutputCodeClassifier(code_size=...)).

• Caveats:

  • Random class groupings can create unnatural, hard binary problems.

  • Correction only works if the bit errors are fairly independent.

  • The output is less interpretable than per-class scores.

In ECOC each class gets a binary codeword (a row of the code matrix), and each column trains one binary classifier. The minimum Hamming distance between codewords sets how many classifier mistakes the decoder can absorb and still recover the right class, so it is the main quantity you maximize when designing the matrix.

• Row separation gives error correction:

  • If the minimum pairwise row distance is d, nearest-codeword decoding corrects up to floor((d-1)/2) wrong bits.

  • One-vs-rest is an ECOC with d = 2, so it corrects nothing: one wrong binary classifier can flip the prediction.

• Column separation keeps the classifiers diverse:

  • Columns should be far from each other and from each other's complements. Identical or complementary columns define the same dichotomy, so they make the same mistakes.

  • Correlated errors break the independence assumption behind the correction guarantee.

• Decoding:

  • Hard decoding: pick the class whose codeword is closest in Hamming distance to the predicted bit vector.

  • Loss-based or soft decoding: use classifier margins or probabilities instead of hard bits. It usually works better.

• Design tradeoffs:

  • Longer codes raise d but cost more classifiers to train and run.

  • Some column splits group dissimilar classes together and are hard to learn, so their high error can cancel the distance gain.

  • In practice codes are random or optimized. For example, OutputCodeClassifier(code_size=...) in scikit-learn uses random codes.

Native multi-class models (softmax regression, trees, naive Bayes, kNN, neural nets) learn one coherent model over all classes. Reductions (one-vs-rest, one-vs-one, ECOC) break the problem into binary tasks so you can use binary-only learners such as SVMs. The tradeoffs are cost, score consistency and calibration, imbalance, and flexibility.

• Training and inference cost:

  • OvR: K models, each trained on all data.

  • OvO: K(K-1)/2 models, each on a small subset. This can be cheaper for learners with super-linear cost (kernel SVMs), but prediction is slow for large K.

  • Native: one model with shared parameters and features, which is usually cheapest for large K.

• Score consistency:

  • OvR scores come from separately trained models, so they are not on a common scale. Argmax can be unreliable, and the scores don't form a probability distribution.

  • OvO voting produces ties and ambiguous regions.

  • Native softmax gives a normalized posterior directly.

• Class imbalance: Each OvR subproblem is 1 vs K-1, which is heavily imbalanced even when the classes themselves are balanced.

• Flexibility (where reductions win):

  • They let you use any strong binary learner.

  • Subproblems are embarrassingly parallel.

  • You can tune or threshold each class separately.

  • With OvR, adding a class means training only one new model.

• Rule of thumb:

  • Prefer native models when they exist and you need probabilities or have many classes.

  • Use reductions when your best learner is binary-only or you need per-class control.

Switch to retrieval when the label space is large, long-tailed, or changing. In that setting, learning a separate output weight per class fails because many classes have few examples, and retraining for every new class is impractical. Encoding inputs and classes into a shared space and doing nearest-neighbor search avoids both problems. The number of classes alone isn't the trigger: a 50k-way softmax is perfectly feasible on modern hardware.

• Signals that favor retrieval:

  • Classes are added or removed often (products, entities, documents). New classes only need an embedding, with no retraining.

  • Long tail: many classes have few or zero examples. Embedding classes from their side information (text, images, metadata) enables few-shot and zero-shot prediction.

  • Classes have structure or similarity worth sharing across classes.

  • Inference: ANN search (FAISS, ScaNN) is sublinear in K, whereas a full softmax scores every class.

• How it's trained:

  • Use a two-tower or dual encoder with contrastive loss, sampled softmax, or in-batch negatives.

  • Hard-negative mining matters a lot for quality.

• Costs:

  • ANN search is approximate, so recall can be lost.

  • Scores are not calibrated probabilities.

  • You often need a retrieve-then-rerank stage, which adds system complexity.

• When to keep the softmax:

  • The label set is fixed and closed.

  • There is plenty of data per class.

  • You need calibrated probabilities and the full output layer fits in compute.

For each possible decision, compute its expected loss under the posterior, then pick the decision with the minimum: a* = argmin_a Σ_y L(a, y) P(y|x). Under 0-1 loss this is argmax posterior. With asymmetric costs the decision boundary moves away from 0.5 toward the costlier mistake.

• Conditional risk:

  • R(a|x) = Σ_y L(a, y) P(y|x), where L(a, y) is the cost of predicting a when the truth is y.

  • In matrix form: risks = L @ p, then take the argmin.

• Binary closed form:

  • Predict positive if p > (C_FP - C_TN) / ((C_FP - C_TN) + (C_FN - C_TP)).

  • With zero cost for correct predictions, the threshold is C_FP / (C_FP + C_FN).

  • Example: a missed fraud costing 9x a false alarm gives threshold 0.1.

• Prerequisites and extensions:

  • Posteriors must be calibrated and reflect deployment priors. Otherwise the rule is optimal for the wrong distribution.

  • A reject or abstain option can be added as an extra row with a fixed cost.

  • The loss matrix can have more actions than classes.