74 Unsupervised Learning Interview Questions and Answers (2026)

Structure is any regularity that makes the data non-random: the points are not spread uniformly and independently across feature space, so they can be described more compactly than by listing them one by one. Unsupervised learning tries to find and model that regularity.

• Groupings (clusters): Points concentrate in dense regions separated by sparse ones, for example customer segments.

• Low-dimensional geometry (manifolds, subspaces): High-dimensional data lies near a lower-dimensional surface, so a few latent factors explain most of the variance. Examples: PCA components, image manifolds.

• Dependencies and correlations: Features co-vary or co-occur. Examples: items bought together (association rules), conditional independencies in a graphical model.

• Density shape: The probability mass is non-uniform, with modes, tails and low-density regions. Low-density regions define what counts as an anomaly.

• Latent or hierarchical organization: Hidden causes generate the observations, such as topics generating words, or nested groups (hierarchies).

• Key caveat: structure is defined relative to an assumption:

  • The choice of distance metric, feature scaling and model family decides what counts as "similar" or "simple".

  • So you should confirm that discovered structure is real by checking stability, comparing against a null or shuffled baseline, and confirming it is useful downstream.

The core categories are clustering, dimensionality reduction, density estimation, anomaly detection and association/dependency discovery. Representation learning (self-supervised) and generative modeling are modern extensions of the same idea.

• Clustering: group similar points: Examples: k-means, hierarchical clustering, DBSCAN, Gaussian mixtures.

• Dimensionality reduction: compress features while keeping structure: Examples: PCA, autoencoders, t-SNE and UMAP (the last two mainly for visualization).

• Density estimation: model P(X): Examples: kernel density estimation, GMMs, normalizing flows.

• Anomaly / novelty detection: flag points that don't fit learned normal behavior: Examples: Isolation Forest, one-class SVM, reconstruction error.

• Association rule mining: find co-occurrence rules: Examples: Apriori, FP-Growth for market-basket analysis.

• Representation and generative learning: Learn embeddings with contrastive or masked objectives, or sample new data with VAEs, GANs and diffusion models.

• These categories overlap: a GMM clusters data and estimates density at the same time, and density estimates drive anomaly detection.

Histograms are the simplest density estimator, but they are discontinuous, highly sensitive to arbitrary binning choices, and scale terribly with dimension. This is why smoother methods like kernel density estimation (KDE) usually replace them.

• Sensitivity to bin width:

  • Bins that are too narrow give a noisy, spiky estimate (high variance). Bins that are too wide hide structure such as multiple modes (high bias).

  • There is no universally correct width. Rules like Sturges, Scott, or Freedman–Diaconis are only heuristics.

• Sensitivity to bin origin (anchor): Shifting the bin edges while keeping the width fixed can change the apparent shape, for example making a bimodal distribution look unimodal.

• Discontinuous, blocky estimate:

  • The density jumps at bin edges and is flat inside each bin, so it is a poor fit for smooth true densities.

  • It has no useful derivatives, which matters for mode-finding or gradient-based methods.

• Curse of dimensionality:

  • With M bins per axis, d dimensions need M^d bins. Most bins end up empty, and memory explodes.

  • In practice histograms are only usable in 1 to 2 (maybe 3) dimensions.

• Zero density in empty bins: Unseen regions get probability exactly 0. This gives log-likelihood of -∞ and fragile anomaly scores.

• Fix: KDE replaces hard bins with a smooth kernel centred on each point. This removes the origin dependence and the discontinuities, though a bandwidth still has to be chosen.

Density estimation means learning the probability density p(x) that generated a set of unlabeled samples. With that density you can ask how likely any point is, and often draw new samples from it.

• Why it is unsupervised: There are no targets. The goal is to model the structure of the data distribution itself.

• Main families:

  • Parametric: assume a form (such as a Gaussian) and fit its parameters, usually by maximum likelihood.

  • Non-parametric: let the data define the shape (histograms, KDE, k-NN density).

  • Semi-parametric or mixtures: Gaussian Mixture Models.

  • Deep density models: normalizing flows, autoregressive models, and VAEs (the latter give a likelihood bound).

• Uses:

  • Anomaly detection: flag points with low p(x).

  • Generative sampling and data synthesis.

  • Clustering: modes of the density, as in mean-shift.

  • Generative classifiers via Bayes' rule with p(x|y).

  • Visualizing distributions.

• Evaluation: Use the average log-likelihood on held-out data. Training likelihood rewards overfitting, for example a KDE with a tiny bandwidth.

Support is how often an itemset appears in the data: the fraction of all transactions that contain every item in the set. It measures popularity or frequency, and it is the main filter used to decide which itemsets are worth mining.

• Definition:

  • support(X) = (number of transactions containing X) / (total transactions).

  • For a rule X → Y, support = support(X ∪ Y): how often X and Y occur together.

  • Example: {bread, milk} in 200 of 1,000 baskets gives support = 0.2.

• Role in mining:

  • A minimum support threshold defines the frequent itemsets.

  • Anti-monotone (downward closure): a superset can't have higher support than its subsets. Apriori uses this to prune any candidate that has an infrequent subset.

• Trade-offs:

  • Threshold too high: you miss rare but valuable patterns, such as expensive items.

  • Threshold too low: the number of itemsets explodes and many spurious rules appear.

  • Support alone says nothing about association strength, so pair it with confidence and lift.

Confidence measures how reliable a rule X → Y is: of the transactions containing X, the fraction that also contain Y. It estimates the conditional probability P(Y | X).

• Formula:

  • confidence(X → Y) = support(X ∪ Y) / support(X).

  • Example: {diapers} in 100 baskets and {diapers, beer} in 60 gives confidence = 0.6.

• Directional: confidence(X → Y) ≠ confidence(Y → X) in general, because the denominators differ.

• Key weakness: it ignores how common Y already is:

  • If 90% of baskets contain milk, almost any rule X → milk has high confidence even when X and milk are unrelated or negatively correlated.

  • Fix this by checking lift = confidence / support(Y), or by using measures such as conviction.

• In practice, rules are generated from frequent itemsets and filtered by a minimum confidence threshold.

Lift compares how often X and Y occur together with how often they would co-occur if they were independent. It tells you whether X genuinely raises the likelihood of Y (lift > 1), has no effect (= 1), or lowers it (< 1).

• Formula:

  • lift(X → Y) = support(X ∪ Y) / (support(X) · support(Y)) = confidence(X → Y) / support(Y).

  • It is the ratio of P(Y | X) to the baseline P(Y).

• Interpretation:

  • > 1: positive association (complements). Lift = 2 means Y is twice as likely given X.

  • = 1: statistically independent.

  • < 1: negative association (substitutes).

• Properties:

  • Symmetric: lift(X → Y) = lift(Y → X), so it measures association, not direction or causation.

  • It corrects confidence's bias toward popular consequents.

• Caveat: Lift is unstable for rare itemsets: small counts can produce huge values. Combine it with a minimum support threshold or a significance test.

An itemset is just a set of items that appear together, with no direction. An association rule X → Y is a directional, if-then statement built by splitting a frequent itemset into an antecedent and a consequent.

• Itemset:

  • For example {bread, butter, milk}. A k-itemset has k items.

  • Measured by support (frequency). It is "frequent" if it meets the minimum support threshold.

• Association rule:

  • For example {bread, butter} → {milk}, where X and Y are disjoint and X ∪ Y is the itemset.

  • Measured by support, confidence and lift, so it carries implication strength, not just frequency.

• How they relate in mining:

  1. Find frequent itemsets (Apriori, FP-Growth). This is the expensive step.

  2. From each frequent itemset, generate rules from its non-empty splits and keep those above minimum confidence or lift.

• One itemset of size k can produce up to 2^k − 2 rules, and all of them share the same support.

With no labels, you evaluate from several independent angles: internal metrics on the data itself, stability under perturbation, usefulness for a downstream task, and human or domain validation. No single number is enough, so the goal is converging evidence.

• Intrinsic (internal) metrics:

  • Clustering: silhouette, Davies-Bouldin, Calinski-Harabasz (compactness vs separation).

  • Density models: held-out log-likelihood; topic models: perplexity or coherence; dimensionality reduction: reconstruction error, explained variance, trustworthiness.

  • Useful for comparing settings (e.g. choosing k), but they encode the algorithm's own assumptions.

• Stability: Re-run across seeds, bootstrap samples, or subsets and compare results with ARI or NMI; real structure reproduces.

• Extrinsic / downstream evaluation:

  • Use clusters or embeddings as features for a task that does have labels or a business metric (classifier accuracy, A/B test lift).

  • If a small labeled sample exists, compute external metrics (ARI, NMI, purity) against it.

• Baselines and sanity checks: Compare against the same metric on shuffled or uniform-random data; structure should clearly beat the null.

• Human and domain review: Inspect cluster exemplars, profile features per group, and check whether experts find the groups meaningful and actionable.

A positive pair is two inputs the model should map close together because they share the semantic content you care about. A negative pair is two inputs it should push apart. What counts as "the same" is a design choice, and that choice determines which invariances the representation learns.

• Positive pairs (common definitions):

  • Two augmented views of the same instance (SimCLR, MoCo): crops, color jitter, blur of one image.

  • Naturally co-occurring signals: an image and its caption (CLIP), nearby frames in video, adjacent audio segments, context and target in CPC.

  • Same-label samples when labels exist (Supervised Contrastive learning).

• Negative pairs:

  • Usually any other instance in the batch or in a memory bank or queue, treated as a different identity.

  • They prevent collapse: without them, mapping everything to one point would trivially satisfy "pull positives together".

• The objective ties them together: InfoNCE treats the positive as the correct class among 1 + K candidates (a softmax over similarities divided by temperature τ).

• Pitfalls:

  • False negatives: two different images of the same class get pushed apart. This matters more with small datasets or few classes.

  • Poorly chosen positives teach the wrong invariance. For example, color augmentation is harmful if color is the label.

  • Some methods drop explicit negatives entirely (BYOL, SimSiam, VICReg). They avoid collapse in other ways: stop-gradient, a predictor head, or variance and covariance regularisation.

Representation learning means automatically learning a transformation of raw data into features (embeddings) that make downstream tasks easier, instead of hand-engineering those features.

• What makes a representation "good":

  • It keeps task-relevant information and discards nuisance variation, such as lighting or pose for object identity.

  • It makes structure simple: classes become close to linearly separable, and similar items sit nearby.

  • It is compact and ideally disentangled, with separate factors of variation captured in separate directions.

  • It is transferable, so one representation serves many tasks.

• Main families of approaches:

  • Classical unsupervised methods: PCA, ICA, manifold learning.

  • Reconstruction: autoencoders, VAEs, masked modeling (BERT, MAE).

  • Contrastive and joint-embedding methods: SimCLR, CLIP, BYOL.

  • Predictive methods: word2vec, next-token prediction.

  • Supervised learning as a by-product: the penultimate layer of an ImageNet classifier.

• How representations are evaluated:

  • Linear probe: a frozen encoder with a linear classifier on top.

  • k-NN accuracy in the embedding space.

  • Fine-tuning performance, and transfer to other tasks or domains.

• Why it matters: labels are scarce. A good representation learned from unlabeled data lets a small labeled set go much further.

Because the bottleneck has fewer dimensions than the input, the network can't copy the input through it. It is forced to learn a compressed code that keeps the factors of variation most needed to reconstruct the data. In effect, it learns coordinates on the low-dimensional manifold where the data lives.

• Linear case equals PCA:

  • With linear layers and MSE loss, the optimal code spans the same subspace as the top-k principal components.

  • The learned basis need not be orthogonal or ordered by variance.

• Nonlinear case is a nonlinear generalization of PCA: It can capture curved manifolds, giving more compact codes than PCA for the same dimension.

• What it does NOT guarantee:

  • Semantic or disentangled features: it keeps whatever lowers reconstruction error. That may be high-variance details like background or lighting rather than the content you care about.

  • A regular latent space: points between codes may decode to garbage, so it is not a generative model (unlike a VAE).

  • Protection from memorization: a very powerful encoder and decoder can still memorize training samples despite the bottleneck.

• Uses: dimensionality reduction, feature extraction, compression, and anomaly detection (poorly reconstructed inputs fall off the learned manifold).

Reconstruction loss measures how much information about the input is lost after passing it through the encoder and decoder. It is the discrepancy between x and x̂ = decoder(encoder(x)). Minimizing it forces the code to retain whatever is needed to rebuild the input.

• Probabilistic view:

  • The loss is the negative log-likelihood of the input under the decoder's output distribution, p(x|z).

  • Two common pairings:

  • MSE corresponds to a Gaussian decoder (continuous data).

  • Binary cross-entropy corresponds to a Bernoulli decoder (binary or [0,1] data).

• The loss defines "what counts as similar":

  • Pixel-wise MSE treats every pixel equally and averages over uncertainty, which produces blurry reconstructions.

  • Perceptual or feature-space losses emphasize structure humans care about.

• How to interpret it:

  • Per-sample error works as an anomaly score: inputs unlike the training data reconstruct poorly.

  • Low loss alone does not mean useful features: an unconstrained model achieves it by learning the identity. The loss is only meaningful together with a bottleneck or regularizer.

Clustering is a technique: a family of algorithms that group similar points. Customer segmentation is a business goal: dividing customers into groups you can act on. Clustering is one way to build segments, but segmentation adds requirements that clustering does not care about.

• Clustering:

  • Examples include K-means, GMM, DBSCAN and hierarchical clustering.

  • Success is judged by geometry: silhouette score, within-cluster variance, stability.

  • It applies to any data: pixels, documents, sensors.

• Customer segmentation:

  • Segments can be built from rules (RFM tiers, demographics), from clustering, or from a mix of both.

  • Success is judged by business value. Segments should be:

  • Interpretable, because marketers need a persona they can name.

  • Actionable, meaning each segment gets a different strategy.

  • Large enough to target, and stable over time.

  • Distinct on the outcomes the business cares about (churn, LTV).

• Practical implication:

  • The clustering solution with the best silhouette score may be the wrong segmentation. Features, k and post-processing are usually chosen for business usefulness.

  • Validation is profiling the segments plus A/B testing the strategies, not just internal cluster metrics.

Supervised learning has a target and a loss tied to it, so "correct" is defined by the labels. Unsupervised learning has no ground truth, so you must decide yourself what a good answer looks like, and many different answers can be equally valid.

• No objective comes for free:

  • You must pick a proxy, such as within-cluster variance, likelihood or reconstruction error.

  • Optimizing that proxy may not match what the business actually wants.

• The answer is ill-posed:

  • The same data can be grouped by color, by shape or by size; each grouping is "right" for a different purpose.

  • The number of clusters or latent dimensions is often undefined.

• Results depend heavily on modeling choices: The distance metric, feature scaling, preprocessing and algorithm all encode assumptions, and changing them changes the result.

• Evaluation is indirect:

  • Internal metrics (for example silhouette score) reward a particular geometry, not usefulness.

  • Real validation requires stability tests, human judgment or downstream task performance.

• Practical consequence: specify the intended use up front: For example, "segments that differ in churn behavior" is a usable spec; "find clusters" is not.

Frame a problem as unsupervised when the goal is to discover or describe structure rather than predict a known target, or when the relevant labels don't exist and can't be defined or obtained reliably.

• Ask: is there a well-defined target?:

  • If the outcome is already known and recorded (churned, fraud confirmed), supervised learning usually wins.

  • If the question is "what kinds of X exist?" or "what is unusual?", the problem is exploratory, which points to unsupervised learning.

• Typical signals for unsupervised:

  • Segmentation with no predefined categories.

  • Anomalies that are rare, keep changing, or have never been seen before (new fraud patterns).

  • Compression, deduplication or recommendation by similarity.

  • Preprocessing or feature learning for a later supervised model.

• Check feasibility:

  • Can you define what "good" means (for example actionable segments or alert precision)?

  • Can a domain expert interpret and act on the output?

• Consider hybrids:

  • If a few labels exist, semi-supervised or weakly supervised learning may beat a purely unsupervised approach.

  • Unsupervised output often bootstraps labeling later.

Choose an unsupervised method when labels are slow, expensive, ambiguous or structurally unavailable, and when an approximate, structure-based answer delivers value now. Ideally, use it to make later labeling cheaper.

• Labels are fundamentally scarce or undefined:

  • Examples: novel attacks, new failure modes, cold-start products.

  • Waiting won't produce enough positives to train on.

• Labeling cost or latency is too high: Examples: expert annotation (medical, legal) or outcomes that take months to observe.

• The task is exploratory: You don't yet know which categories matter, so labeling now would lock in the wrong taxonomy.

• The data distribution drifts fast: Labels go stale quickly, and unsupervised baselines adapt without relabeling.

• Use it as a bridge, not a dead end:

  • Pretrain representations on unlabeled data to reduce how many labels you need later.

  • Cluster the data and label representative points (active learning).

• When to wait instead: The decision needs a precise target, errors are costly, and labels are reasonably attainable. In that case a supervised model will usually outperform.

Overfitting means fitting noise specific to the sample instead of the underlying distribution. An unsupervised model fits the data itself, so it can memorize sampling noise and quirks just as a supervised model memorizes label noise.

• The data is the target: Density estimation, reconstruction and clustering objectives all score how well the model explains the training points.

• Concrete examples:

  • KDE with a tiny bandwidth puts spikes on each training point, giving high training likelihood and poor held-out likelihood.

  • k-means with very large k keeps lowering inertia (k = n gives zero) while the clusters stop meaning anything.

  • An over-capacity autoencoder learns a near-identity map that reconstructs training data well but generalizes poorly.

  • A GMM with many components can collapse a component onto a single point, sending the likelihood to infinity.

• How to detect it:

  • Compare held-out log-likelihood or reconstruction error against training values.

  • Check cluster stability across bootstrap samples.

  • Use information criteria (BIC, AIC) to compare model sizes.

• How to control it: Limit capacity (bottleneck size, number of components), add regularization or priors, and tune smoothing such as bandwidth.

Learning P(X) uses only the inputs, with no target Y, which makes it unsupervised. It is generative because a model of P(X) describes how the data is produced, so you can sample new X and score the likelihood of any point.

• Unsupervised: the signal comes from X alone:

  • The objective is typically maximum likelihood, maximizing Σ log P(xᵢ) over training points.

  • No human annotation is needed.

• Generative versus discriminative:

  • Discriminative models learn P(Y|X), only the decision boundary.

  • Generative models capture the full joint structure of X, which lets you sample new data, impute missing features and evaluate density.

• It subsumes other unsupervised tasks:

  • Low P(x) means an anomaly.

  • Mixture components act as clusters.

  • Latent variables provide representations or dimensionality reduction.

• Model families:

  • Explicit density models: GMMs, autoregressive models (language models factor P(x) = Π P(xₜ|x<ₜ)), normalizing flows.

  • Approximate density: VAEs, which optimize a lower bound on the likelihood (the ELBO).

  • Implicit models: GANs, which can sample but give no density.

• Nuance: next-token prediction is often called self-supervised: Its "labels" are carved out of X itself, so it is still unsupervised modeling of P(X).

Bandwidth controls the bias–variance trade-off in KDE. Too small undersmooths into a spiky, overfit estimate. Too large oversmooths and erases real structure. Bandwidth matters far more than the choice of kernel.

• Too small: high variance, low bias:

  • Each sample becomes its own narrow bump, so the estimate is jagged and full of spurious modes.

  • It overfits: training likelihood is very high, but density between samples drops toward 0, so held-out likelihood is poor.

  • For anomaly detection, normal points that fall in gaps look anomalous, causing false positives.

• Too large: high bias, low variance:

  • The estimate approaches a single broad kernel shape. Distinct modes merge, peaks flatten, and tails get fattened.

  • It underfits: the result is stable but wrong, and outliers may receive deceptively high density.

• Theory:

  • The bias of the estimate grows roughly with h². The variance shrinks roughly with 1/(n·h^d).

  • So the optimal h shrinks with n (about n^(-1/5) in 1D) but shrinks slowly in high dimensions.

• Practical note: with features on different scales, a single scalar bandwidth over- and under-smooths at the same time. Standardize the features or use a bandwidth matrix.

Parametric estimation assumes the density belongs to a fixed-form family with a finite number of parameters, then fits them. Non-parametric estimation makes minimal shape assumptions, so its effective complexity grows with the amount of data.

• Parametric (e.g. a Gaussian, or exponential-family models):

  • Parameters are fit by maximum likelihood or Bayesian inference, for example μ and Σ for a Gaussian.

  • Pros: data-efficient, compact, fast to evaluate, interpretable, and works in higher dimensions.

  • Cons: if the assumed form is wrong (model misspecification, e.g. fitting a single Gaussian to bimodal data), the bias never goes away, no matter how much data you add.

• Non-parametric (histograms, KDE, k-NN density):

  • The estimate is built directly from the samples. Only smoothing hyperparameters (bin width, bandwidth, k) need to be chosen.

  • Pros: flexible. It can capture arbitrary shapes, multimodality, and skew, and it is consistent as n → ∞.

  • Cons: must store all the data, has O(n) evaluation cost per query, is sensitive to the smoothing hyperparameter, and suffers badly from the curse of dimensionality.

• Middle ground:

  • Mixture models such as GMMs are parametric per component but very flexible as K grows.

  • Normalizing flows are parametric but highly expressive.

• Rule of thumb: use parametric methods when you have a good prior on the shape or limited data and many dimensions. Use non-parametric methods for low-dimensional data with unknown shape and enough samples.

Fit a density model to (mostly) normal data, then score each point by how unlikely it is under that model, typically with the negative log-density. Points in low-density regions are flagged as anomalies.

• Procedure:

  1. Preprocess: scale features, and possibly reduce dimension.

  2. Fit a density estimator: Gaussian, GMM, KDE, or a normalizing flow.

  3. Score each point with s(x) = -log p̂(x). Log-space avoids numerical underflow.

  4. Threshold the score, for example at the 99th percentile of training scores, at an expected contamination rate, or tuned on a few labeled anomalies.

• Choice of model:

  • A single Gaussian is equivalent to Mahalanobis-distance scoring.

  • A GMM handles multimodal normal behaviour.

  • KDE is flexible in low dimensions.

• Pitfalls:

  • Contaminated training data inflates density around the anomalies themselves. Use robust fitting or clean the data first.

  • Smoothing choices (bandwidth, K) directly shift which points look rare.

  • In high dimensions density estimates become unreliable. Deep generative models can even assign high likelihood to out-of-distribution data.

  • A single global threshold misses local anomalies in clusters of varying density. Local Outlier Factor (LOF) addresses this by comparing a point's density to that of its neighbours.

A GMM models the density as a weighted sum of K Gaussians, p(x) = Σ_k π_k N(x | μ_k, Σ_k). After fitting with EM, it gives an explicit, smooth, normalized density that you can evaluate and sample from.

• Why it works:

  • With enough components, Gaussian mixtures can approximate essentially any smooth density.

  • The mixing weights π_k are non-negative and sum to 1, so p(x) integrates to 1.

• Fitting with Expectation-Maximization (EM):

  1. E-step: compute responsibilities, i.e. the posterior probability that each point came from each component.

  2. M-step: re-estimate π, μ, and Σ from those weighted points.

  3. Repeat until the log-likelihood converges. EM only finds a local optimum, so use multiple initializations (n_init).

• Using it as a density estimator:

  • score_samples(X) returns log p(x), used for likelihood, anomaly scoring, or comparing models.

  • sample(n) draws new data: pick a component by π, then sample from its Gaussian.

• Key choices:

  • Number of components K: choose by BIC/AIC or held-out log-likelihood. A BayesianGaussianMixture can prune unused components automatically.

  • covariance_type (full, tied, diag, spherical): trades flexibility against the number of parameters, which matters in high dimensions.

• Pitfall: a component can collapse onto a single point, driving the likelihood to infinity (a singular covariance). reg_covar adds a small diagonal term to the covariance to prevent this.

Start with a rule of thumb (Scott or Silverman) as a baseline. For anything that matters, choose the bandwidth by cross-validated held-out log-likelihood, or by a plug-in method like Sheather–Jones when the data is skewed or multimodal.

• Rules of thumb (fast, assume roughly Gaussian data):

  • Silverman (1D): h = 0.9 · min(σ, IQR/1.34) · n^(-1/5).

  • Scott: h ∝ σ · n^(-1/(d+4)).

  • Both tend to oversmooth multimodal data, merging peaks together.

• Plug-in methods:

  • Sheather–Jones estimates the unknown curvature term in the formula for the optimal (AMISE) bandwidth.

  • It is a strong default for 1D data with non-Gaussian shape.

• Cross-validation (most general):

  • Likelihood CV: pick the h that maximizes the log-likelihood on held-out or leave-one-out data.

  • Least-squares CV minimizes the estimated integrated squared error instead.

  • CV is data-driven and works in any dimension, but it is slower and can be noisy for small n.

• Practical tips:

  • Standardize the features first, or use a full bandwidth matrix, so a single h makes sense across dimensions.

  • Search h on a log scale.

  • Use adaptive (variable) bandwidths when density varies widely: narrow kernels in dense regions, wide kernels in the tails.

  • Sanity-check visually in 1D/2D, and remember that the downstream task (e.g. anomaly recall) may prefer a different h.

Choose a GMM when you need a compact, fast, scalable model that also works in moderate dimensions or gives interpretable components. Choose KDE when the data is low-dimensional, the sample size is manageable, and the shape is complex enough that Gaussian components would fit it poorly.

• Favour a GMM when:

  • The data is large: a GMM evaluates in O(K) per query, while KDE is O(n) and must store every training point.

  • There are more dimensions: shared covariance structure generalizes better than a fixed kernel, whose estimates degrade quickly as d grows.

  • You want interpretability or clustering: components and responsibilities give soft cluster assignments.

  • Correlated or elongated clusters exist: full covariances capture them, whereas KDE needs a bandwidth matrix to do so.

  • You need cheap sampling or a model that is easy to deploy and serialize.

• Favour KDE when:

  • The data is low-dimensional (roughly 1 to 3 dimensions) with irregular shapes, sharp features, or non-Gaussian clusters that would need many Gaussian components.

  • You want few assumptions and only one hyperparameter (the bandwidth), with no local optima or EM initialization issues.

  • The data is small to medium, so storing it and evaluating against all points is cheap.

• Trade-offs to mention:

  • A GMM is biased when the true components are not Gaussian, and choosing K adds model-selection work.

  • KDE is unbiased in the limit but high-variance in high dimensions.

  • Both can be compared fairly by held-out log-likelihood.

The cluster assumption says that points in the same cluster (a high-density region) are likely to share a label. Equivalently, the decision boundary should pass through low-density regions, which is what lets unlabelled data inform where the boundary goes.

• Why it matters: Unlabelled data reveals p(x). That only helps predict y if p(x) and p(y|x) are linked, and this assumption supplies the link.

• Low-density separation form: Methods that implement it include transductive SVMs (maximize margin over unlabelled points), entropy minimization, and pseudo-labelling.

• Related assumptions:

  • Smoothness: nearby points in high-density regions have similar outputs.

  • Manifold: data lies on a low-dimensional manifold, and labels vary smoothly along it. This underpins graph methods.

• When it fails: If classes overlap heavily, or one cluster contains several classes, unlabelled data can push the boundary to the wrong place and hurt accuracy.

Pseudo-labelling trains a model on labelled data, uses it to predict labels for unlabelled data, keeps the confident predictions as "pseudo-labels", and retrains on the combined set. This effectively pushes decision boundaries away from dense unlabelled regions.

• Basic loop:

  1. Train a model on the labelled set.

  2. Predict on the unlabelled data, then take the argmax class as a hard label (or keep a sharpened soft label).

  3. Keep only predictions above a confidence threshold, such as max probability ≥ 0.95.

  4. Retrain on labelled plus pseudo-labelled data, often down-weighting the pseudo-label loss.

  5. Repeat, or do it online per mini-batch in deep learning.

• Why it works: It is equivalent to entropy minimization: it encourages confident predictions on unlabelled data, which enforces the cluster assumption.

• Risks:

  • Confirmation bias: wrong pseudo-labels get reinforced.

  • Class imbalance drift: easy or majority classes dominate the accepted pseudo-labels.

• Mitigations:

  • Use high or class-adaptive thresholds (FlexMatch) and ramp up the unlabelled loss weight gradually.

  • Generate labels with a separate teacher (noisy student, mean teacher), or combine with augmentation as in FixMatch.

  • Validate on a clean labelled holdout, never on pseudo-labelled data.

Consistency regularization adds an unsupervised loss that forces the model to give the same prediction for an unlabelled input under different perturbations (augmentations, dropout, noise). This makes the function smooth in dense regions, so boundaries settle in low-density gaps.

• Objective:

  • Total loss = supervised cross-entropy on labelled data + λ · distance(f(aug1(x)), f(aug2(x))) on unlabelled x.

  • The distance is typically MSE or KL. λ is usually ramped up from 0 during training.

• Key variants:

  • Π-model: two stochastic forward passes of the same network.

  • Temporal ensembling and Mean Teacher: the target comes from an EMA of past predictions or of the weights, so targets are more stable.

  • VAT: the perturbation is the adversarial direction that changes the prediction most.

  • UDA and FixMatch: a weak-augmentation prediction (thresholded into a pseudo-label for FixMatch) is the target for a strongly augmented view.

• Why it works: It encodes the smoothness and cluster assumptions: invariance to label-preserving changes, with unlabelled data mapping out where those invariances must hold.

• Pitfalls:

  • Augmentations must preserve the label, or the model learns the wrong invariances.

  • Collapse to constant predictions is a risk; the supervised loss and target stop-gradient prevent it.

Label propagation builds a similarity graph over labelled and unlabelled points, then iteratively lets each node take a weighted average of its neighbours' label distributions while labelled nodes stay fixed (or nearly fixed). Labels diffuse along dense paths until convergence.

• Algorithm:

  1. Build the graph using kNN or an RBF kernel: W_ij = exp(−||x_i − x_j||² / 2σ²).

  2. Row-normalize to a transition matrix T = D⁻¹W.

  3. Initialize a label matrix Y: one-hot rows for labelled points, zeros (or uniform) for unlabelled ones.

  4. Iterate Y ← TY, then reset the labelled rows to their true labels (clamping).

  5. At convergence, predict the argmax of each unlabelled row.

• Interpretation:

  • Equivalent to a random walk from each unlabelled node: the label is the probability of first hitting each labelled class.

  • It minimizes the graph energy Σ W_ij ||f_i − f_j||² (harmonic function), so labels are smooth across strong edges.

• Label spreading variant: Uses symmetric normalization with soft clamping, Y ← αSY + (1−α)Y₀, so noisy labels can be corrected.

• Practical notes:

  • Quality depends on the graph: σ or k, and the distance metric (embeddings help).

  • A dense graph costs O(n²), so use sparse kNN graphs. It is transductive: new points need a graph rebuild or an inductive extension.

  • In scikit-learn: LabelPropagation and LabelSpreading, with unlabelled points marked -1.

Self-training feeds the model's own predictions back as training labels, so any wrong label gets reinforced. A confidence threshold admits only pseudo-labels whose predicted probability is high, where accuracy is much higher. This trades fewer pseudo-labels for cleaner ones and slows the feedback loop of compounding errors.

• The problem: confirmation bias: A wrong pseudo-label pulls the boundary toward it, which makes nearby wrong predictions more confident in the next round, and so on. Errors snowball.

• How the threshold helps:

  • If the model is reasonably calibrated, high-confidence predictions are mostly correct, so the noise rate of added labels drops.

  • Confident points tend to lie deep inside clusters, far from the boundary. Ambiguous boundary points, where errors concentrate, are deferred until the model improves.

  • Iterating creates a curriculum: easy points first, then harder ones as the model gains confidence.

• Trade-offs:

  • Too high: few points are added, little gain, and hard classes are never pseudo-labelled.

  • Too low: noise floods in and causes degradation.

  • Neural nets are often overconfident, so a high softmax score is not a guarantee. Calibrate first (temperature scaling) or use ensemble agreement.

• Refinements:

  • Class-wise or adaptive thresholds (FlexMatch) avoid starving minority or hard classes.

  • Weight samples by confidence instead of using a hard cutoff, and re-evaluate pseudo-labels each round so earlier mistakes can be dropped.

  • Combine with augmentation consistency (FixMatch) or a separate teacher model to decorrelate errors.

FP-Growth mines frequent itemsets without generating candidates: it compresses the database into a prefix tree (the FP-tree) and grows patterns recursively from it, whereas Apriori repeatedly generates and tests candidate itemsets with a full database scan per level.

• Apriori's bottleneck:

  • Level-wise generate-and-test: build candidate k-itemsets from frequent (k-1)-itemsets, then scan the data to count them.

  • Many candidates (e.g. 10^4 frequent items give about 5×10^7 candidate pairs) and one scan per itemset length.

• FP-Growth's approach:

  1. Two scans: one to count single items, one to insert each transaction (items sorted by frequency) into the FP-tree, so shared prefixes are stored once.

  2. For each item, extract its conditional pattern base (the prefix paths ending in it), build a conditional FP-tree, and recurse (divide and conquer).

• Result: far less I/O and no candidate explosion; usually much faster on dense data or at low support.

• Trade-offs:

  • The tree must fit in memory and may compress poorly on very sparse data.

  • Recursion is harder to parallelize (Spark's PFP partitions the work to address this).

  • Both algorithms output the same frequent itemsets; only the efficiency differs.

Confidence P(B|A) ignores how common B already is. If B is popular, the rule A→B gets high confidence even when A has no effect on B, or even makes B less likely.

• Classic example:

  • 80% of all baskets contain milk, and 75% of bread baskets contain milk. Confidence(bread→milk) = 0.75 looks strong.

  • Yet buying bread slightly lowers the chance of milk (0.75 < 0.80): a negative association.

• Fix: compare against the baseline:

  • Lift = conf(A→B) / P(B): above 1 is positive, 1 means independent, below 1 is negative.

  • Leverage = P(A,B) − P(A)P(B); conviction penalizes rules whose consequent is very common.

  • Null-invariant measures (Kulczynski, cosine) are less distorted by the many transactions that contain neither item.

• Even lift isn't proof of a real effect:

  • With low support it is noisy, so test significance (chi-square, Fisher's exact test) and correct for multiple comparisons.

  • It is still correlation: confounders like seasonality or promotions can drive it. Causal claims need an experiment.

Spurious rules come from three main sources: popular items co-occurring by chance, hidden common causes, and testing a huge number of rules on finite data. They look strong on support or confidence but don't reflect a real dependency.

• Base-rate effects: Very frequent items (bags, milk) appear in high-confidence rules with almost anything.

• Confounding and aggregation:

  • A third factor drives both items: season, promotions, store layout, or a customer segment.

  • Pooling heterogeneous segments can create or reverse associations (Simpson's paradox).

• Multiple testing: Millions of candidate rules guarantee that some pass thresholds by pure chance, especially low-support rules with high lift.

• Data artifacts: Bundles and kits, default add-ons, duplicate transactions, bot traffic, and B2B bulk orders.

• Mitigations:

  • Use lift or leverage plus significance tests with FDR or Bonferroni correction.

  • Stratify by time or segment, and validate on a holdout period.

  • Clean out bundles and outliers, and A/B test any rule before acting on it.

A rule is redundant when a simpler or more general rule already conveys the same information with equal or better support and confidence, so it adds nothing new.

• More specific antecedent, no gain:

  • If {A}→C has confidence 0.8, then {A,B}→C with confidence about 0.8 is redundant: adding B didn't change anything.

  • A specialization is only worth keeping if it improves confidence or lift meaningfully (minimum improvement criterion).

• Derivable consequents: If A→{B,C} holds, then A→B and A→C, with at least the same support and confidence, are implied by it.

• How redundancy is removed:

  • Mine closed itemsets (no superset has the same support) or maximal itemsets.

  • Generate non-redundant bases: minimal antecedent, maximal consequent.

  • Prune by improvement or by productive rules (significantly better than every sub-rule).

• Why it matters: far fewer rules, which are easier to review and less prone to multiple-testing noise.

The Apriori property (downward closure) says every subset of a frequent itemset is also frequent; equivalently, any superset of an infrequent itemset is infrequent. So once an itemset fails minimum support, its entire superset lattice can be pruned without counting.

• Why it holds: Support is anti-monotone: adding items can only keep the set of matching transactions the same or shrink it.

• How Apriori uses it, level by level:

  1. Join: build candidate k-itemsets only from frequent (k-1)-itemsets.

  2. Prune: drop any candidate that has an infrequent (k-1)-subset, before scanning the data.

  3. Count the surviving candidates in one database pass and keep those meeting minimum support.

• Effect:

  • Instead of 2^n possible itemsets, only extensions of frequent sets are explored.

  • Example: if {beer} is infrequent, no itemset containing beer is ever generated.

• Limit: at low support or on dense data, the frequent (k-1)-sets can still yield huge candidate sets, which motivates FP-Growth and Eclat.

There is no universal value. Set thresholds from business constraints and the data's frequency distribution, then iterate: support controls which patterns are worth acting on and the compute cost, while confidence (paired with lift) controls how reliable a rule must be.

• Start from the business use:

  • Set minimum support from actionability: how many transactions per period would make a promotion or shelf placement worthwhile (e.g. at least 100 baskets per month).

  • Set confidence from the cost of a wrong action: a cheap recommendation slot tolerates lower confidence than a costly bundle discount.

• Inspect the data:

  • Plot the item frequency distribution, which is usually long-tailed. Too high a support finds only the obvious items; too low causes a rule explosion.

  • Consider per-category or multiple minimum supports so rare but valuable items aren't lost.

• Tune iteratively:

  1. Sweep thresholds and track the rule count and runtime; aim for a reviewable number of rules.

  2. Filter by lift > 1 (plus a significance test) rather than confidence alone.

  3. Check that rules are stable on a holdout time period.

  4. Have domain experts review the rules, then A/B test them on business KPIs (attach rate, revenue).

Downstream task evaluation judges an unsupervised model by how much its output (clusters, embeddings, reduced features, anomaly scores) improves a separate task that has a measurable objective. It answers "is this representation useful?" rather than "does it look good by its own criterion?"

• How it works:

  1. Train the unsupervised model on (often large, unlabeled) data.

  2. Feed its output into a supervised task with labels: e.g. a classifier on embeddings, cluster IDs as features, anomaly scores checked against confirmed incidents.

  3. Compare the task metric (accuracy, F1, AUC, revenue) against a baseline without the unsupervised component.

• Common protocols:

  • Linear probing: freeze the representation and train only a simple linear model, so the score reflects representation quality, not classifier capacity.

  • k-NN evaluation on embeddings; fine-tuning for a looser, more practical measure.

  • Business-side: A/B test of a segmentation-driven campaign or recommendation.

• Why it matters: Intrinsic metrics (silhouette, likelihood) can look great while being useless for the actual goal; downstream metrics tie evaluation to value.

• Caveats:

  • Results are task-specific: good for one task does not mean good in general.

  • Needs some labels or a measurable outcome, and must avoid leakage (fit the unsupervised step only on training data).

You re-fit the model under controlled perturbations (different seeds, bootstrap samples, subsamples, small noise) and measure how much the outputs agree. Real structure should reproduce; results that shift with each run are likely artifacts of initialization or sampling.

• Sources of variation to test:

  • Random seeds / initializations (k-means, GMM, t-SNE, UMAP, neural embeddings).

  • Resampling: bootstrap or 80% subsamples of the rows.

  • Small perturbations: added noise, feature dropout, hyperparameter nudges.

• Agreement metrics:

  • Clusterings: pairwise Adjusted Rand Index or NMI between runs (label-permutation invariant); on subsamples, compare only on the shared points.

  • Per-cluster stability: Jaccard overlap of each cluster with its best match across bootstraps (e.g. stable if mean Jaccard > 0.75).

  • Embeddings: overlap of k-nearest-neighbor sets, or Procrustes / CCA alignment, since coordinates themselves are arbitrary.

  • Components (PCA, NMF): cosine similarity of matched components.

• Using it for model selection:

  • Plot stability vs k; choices of k with high, low-variance agreement are preferred.

  • Caveat: stability is necessary, not sufficient. k=2 or a degenerate solution can be very stable yet uninformative, so pair it with quality metrics.

Supervised models have a ground truth to score against on held-out data; unsupervised models do not, so there is no single objective notion of "correct" and every metric you choose smuggles in assumptions about what good structure means.

• No ground truth: Supervised: compare predictions to labels (accuracy, RMSE) on a test set. Unsupervised: nothing to compare to.

• Multiple valid answers: The same customers can be grouped by spend, behavior, or geography; all may be "right" depending on purpose.

• Internal metrics are biased toward particular shapes:

  • Silhouette and Davies-Bouldin favor compact, convex clusters, so they penalize valid density-based or elongated clusters (DBSCAN).

  • Metrics often improve trivially with complexity (inertia always drops as k grows).

• Sensitivity to choices: Scaling, distance metric, feature selection, and seeds can change results substantially, so you must also test stability.

• Usefulness is external: Real value is judged downstream or by domain experts, which is slower, costlier, and more subjective than a test-set score.

A pretext task is a surrogate task whose labels are generated automatically from the data itself. Solving it forces the network to learn useful features. The task's own output is then discarded, and the learned encoder is reused for downstream tasks.

• How it works:

  1. Transform or hide part of the input in a known way (rotate it, crop it, mask it, shuffle it).

  2. Train the model to predict what was done or what was hidden. The supervision is free.

  3. Keep the encoder, then fine-tune or linear-probe it on the real task.

• Classic examples:

  • Vision: rotation prediction, jigsaw puzzles, relative patch position, colorization, inpainting, masked patches (MAE).

  • NLP: next-token prediction (GPT), masked tokens (BERT).

  • Contrastive learning is also a pretext task: instance discrimination, meaning "which sample is the other view of me?"

• What makes a good pretext task:

  • Solving it should require semantic understanding. To predict rotation, you must know which way is "up" for objects.

  • It must not allow shortcuts. Jigsaw models learned to exploit chromatic aberration and patch-edge continuity until those cues were suppressed (with color dropping, gaps and jitter between patches).

  • Pretext performance is not the goal. Features from late layers can over-specialise to the pretext task, so intermediate layers sometimes transfer better.

Masked prediction hides part of the input and trains the model to reconstruct the missing part from the visible context. Doing this well requires modelling the relationships and structure in the data, which produces strong general-purpose representations.

• Core recipe:

  • Corrupt the input by replacing a subset of tokens or patches with a [MASK] token, or by dropping them.

  • Predict the original content. The loss is usually computed only on the masked positions.

• Key instances:

  • BERT masks 15% of tokens: 80% become [MASK], 10% become a random token, 10% stay unchanged. This reduces the mismatch between pre-training and fine-tuning, where no [MASK] appears.

  • MAE masks about 75% of image patches. The encoder sees only the visible patches, and a lightweight decoder reconstructs pixels. This is cheap and effective.

  • BEiT predicts discrete visual tokens. data2vec and I-JEPA predict latent features of the masked regions rather than raw pixels.

  • Speech: wav2vec 2.0 and HuBERT mask audio frames.

• Masking ratio reflects redundancy:

  • Text is information-dense, so a low ratio is enough.

  • Images are highly redundant: at low ratios, neighbouring pixels can simply be interpolated. A high ratio forces semantic reasoning.

• Compared with contrastive learning:

  • Masked prediction needs no negatives and relies less on hand-designed augmentations.

  • Its features often need fine-tuning: linear-probe results for MAE are weaker than its fine-tuned results.

In instance-based contrastive learning, augmentations are what create positive pairs. They therefore define the task and decide which invariances the representation learns. Without them the task is trivial, and with poor ones the model learns shortcuts.

• They make the task non-trivial: If both views were identical, the model could match them on low-level pixel statistics and learn nothing semantic.

• They encode the desired invariances: Pulling two views together teaches the model to ignore whatever differs between them (crop, color, blur), so only content shared across views survives.

• They block shortcuts:

  • SimCLR found that random crop alone lets the model match views using color histograms. Adding color distortion removes that cue.

  • The composition of crop and color jitter mattered more than any single augmentation.

• Strength matters:

  • Contrastive methods benefit from stronger augmentation than supervised training does.

  • Augmentations that are too strong or mismatched destroy task-relevant information. Color invariance hurts flower or bird classification, and rotation invariance confuses 6 with 9.

• Implications: Augmentation design is domain-specific (images, audio, graphs, tabular data). This is a key weakness, and part of why masked modelling and multimodal pairing (CLIP) are attractive alternatives.

A standard autoencoder maps each input to a single deterministic point in latent space and only learns to reconstruct. A VAE is a probabilistic generative model: the encoder outputs a distribution q(z|x), and the latent space is regularized toward a prior p(z). That makes it possible to sample new data by decoding z ~ p(z).

• Objective:

  • A standard AE minimizes reconstruction loss only.

  • A VAE maximizes the ELBO, which has two terms:

  • A reconstruction term, E[log p(x|z)].

  • A penalty, KL(q(z|x) || p(z)), usually toward N(0, I).

• Latent space structure:

  • AE latents can be irregular, with "holes": random points decode to nonsense.

  • In a VAE, the KL term and sampling noise make the space continuous and dense. Nearby codes decode to similar outputs, and interpolation is meaningful.

• Training mechanics: The VAE samples z using the reparameterization trick (z = μ + σ·ε), so gradients can flow through the sampling step.

• Trade-off:

  • VAE reconstructions are often blurrier, because the KL term competes with reconstruction (tuned in β-VAE).

  • If the decoder is too powerful, the posterior can collapse: the decoder ignores z.

Nothing in the reconstruction loss itself prevents it. An autoencoder avoids learning the identity function only because some constraint makes perfect copying impossible or costly, forcing it to capture structure in the data instead.

• Architectural constraint: Undercomplete bottleneck: a code smaller than the input can't hold everything, so the model keeps the most informative factors.

• Regularized (possibly overcomplete) autoencoders:

  • Sparse AE: an L1 or KL penalty on activations means only a few units fire per input.

  • Denoising AE: the model reconstructs a clean x from a corrupted x̃, so copying its input is wrong by design.

  • Contractive AE: it penalizes the encoder Jacobian, making the code insensitive to small input changes.

  • VAE: the KL term and sampling noise limit how much information the code carries.

• Capacity still matters:

  • An overcomplete AE with no regularization can trivially learn the identity.

  • Even with a bottleneck, an extremely powerful encoder and decoder could memorize training samples. In principle even a 1-D code that indexes each example would work, so you also limit depth, width, or training.

• Diagnostic: if training reconstruction is near perfect but the features are useless downstream, or anomaly scores are flat, the model is likely copying rather than learning structure.

A denoising autoencoder corrupts the input (with noise or masking) but is trained to reconstruct the clean original. Because input and target differ, it can't succeed by copying. It must learn the statistical dependencies of the data to infer what was corrupted, which yields robust features instead of a memorized identity mapping.

• Training setup:

  1. Corrupt the input: x̃ = corrupt(x), for example with Gaussian noise, masking or dropout of inputs, or salt-and-pepper noise.

  2. Encode and decode it: x̂ = decoder(encoder(x̃)).

  3. Minimize the loss against the clean x, not x̃.

• Why it regularizes:

  • The identity function is no longer optimal, so DAEs work even with overcomplete codes.

  • Fresh corruption every epoch acts like data augmentation, making exact memorization of training points much harder.

  • Geometrically, the model learns to map off-manifold points back onto the data manifold. Its reconstruction direction approximates the score, ∇log p(x) (Alain & Bengio).

• Noise level is a key hyperparameter:

  • Too little noise: behavior drifts back toward copying the input.

  • Too much noise: the input is unrecoverable, so the model outputs blurry averages.

• Legacy: the same idea underlies masked modeling (BERT, MAE) and diffusion models, which denoise across many noise levels.

The two terms of the VAE loss pull in opposite directions. Reconstruction loss makes the latent code carry enough information to rebuild the input. The KL divergence makes the encoder's distribution over codes stay close to a simple prior, so the latent space is smooth and can be sampled from. Together they form the negative ELBO.

• Reconstruction loss: fidelity:

  • It is the expected log-likelihood E_q[log p(x|z)], in practice MSE (Gaussian decoder) or BCE (Bernoulli decoder).

  • On its own it turns the VAE into a plain autoencoder: codes become precise but scattered, and the gaps between them decode to garbage.

• KL divergence: regularization toward the prior:

  • KL(q(z|x) || p(z)) with p(z) = N(0, I) penalizes posteriors that are too narrow or sit far from the origin.

  • The result is overlapping, continuous regions in latent space. Interpolation works, and samples drawn from the prior decode to plausible data.

  • It also acts as an information bottleneck, limiting how many nats the code can carry about x.

• The trade-off:

  • Too much KL weight gives blurry outputs, or posterior collapse where the code is ignored.

  • Too little gives sharp reconstructions but poor generation.

  • β-VAE scales the KL term by β to tune this balance. A larger β encourages disentanglement at some cost to fidelity.

Unsupervised models expose the structure of unlabeled data. That lets annotators label groups instead of single points, focus on the most informative examples, and catch errors. The result is much less labelling effort for the same model quality.

• Batch labelling through clustering:

  • Cluster embeddings, review a few points per cluster, then assign the label to the whole cluster.

  • Only impure or borderline clusters need individual review.

• Smarter sample selection:

  • Pick cluster centroids or core-set points so the labelled seed set covers the data's diversity.

  • This avoids labelling thousands of near-duplicates and pairs well with active learning.

• Label propagation and similarity search:

  • Spread a few labels to nearest neighbours in embedding space, using kNN or graph-based semi-supervised methods.

  • "Find similar" tools let an annotator retrieve and tag many examples of a rare class at once.

• Defining the taxonomy:

  • Topic models or clusters reveal which categories actually exist before a label schema is fixed.

  • They also surface emerging classes that the current schema misses.

• Quality control:

  • Anomaly detection flags outliers and corrupted inputs.

  • A label that disagrees with its cluster or neighbours is a likely mislabel and can be sent for re-review.

• Caveat: clusters follow feature similarity, not necessarily your label semantics, so always spot-check before bulk assignment.

Fit the unsupervised model on the training data only. Use it to transform each row into new features: cluster memberships, distances, embeddings or anomaly scores. Append those to, or substitute them for, the raw features in the supervised model. Do all of this inside a pipeline so the features are refit per fold and nothing leaks.

• Common feature types:

  • Cluster ID (one-hot encoded): coarse, but it captures nonlinear groupings.

  • Distances to centroids (KMeans.transform): richer than a single hard ID.

  • Soft memberships: GMM predict_proba, or LDA topic proportions for text.

  • Low-dimensional embeddings from PCA, autoencoders or self-supervised encoders: these compress and denoise the input.

  • Anomaly scores from Isolation Forest, or autoencoder reconstruction error, useful for fraud and quality tasks.

• Avoiding leakage:

  • Fitting the clusterer or PCA on all the data, test set included, leaks distribution information.

  • Put the transform in a Pipeline so cross-validation refits it on each training fold.

  • Semi-supervised exception: fitting on extra unlabeled data is fine, but held-out evaluation rows still must not influence the fit.

• Production concerns:

  • Version the unsupervised model together with the supervised one.

  • Retraining the clusterer reshuffles cluster IDs and changes embeddings, so the downstream model must be retrained too.

  • Confirm the features add value with an ablation against the raw-feature baseline.

Freeze when labelled data is scarce and the downstream task is close to the pre-training domain. Fine-tune when you have enough labels and the task or domain differs meaningfully. Decide empirically by comparing a linear probe against fine-tuning on a validation set.

• Favour freezing when:

  • You have few labels: fine-tuning millions of parameters on hundreds of examples overfits.

  • The domain is similar, so the features already separate the classes (a strong linear probe is the signal).

  • Compute or latency is limited, or one backbone must serve many tasks (embeddings can be cached).

  • You need to keep general knowledge and avoid catastrophic forgetting.

• Favour fine-tuning when:

  • You have plenty of labelled data.

  • There is real domain shift: medical images versus ImageNet, or legal text versus web text.

  • The task needs features the pre-training objective did not emphasise, such as fine-grained or spatial detail.

  • There is a large gap between linear-probe and fine-tuned validation scores.

• Middle-ground options:

  • Unfreeze only the top layers. Lower layers hold generic features, higher layers task-specific ones.

  • Gradual unfreezing, plus discriminative learning rates that are smaller for lower layers.

  • Parameter-efficient tuning (adapters, LoRA, prompt tuning): good adaptation with little forgetting and storage.

  • Linear probe first, then fine-tune (LP-FT). This avoids a random head distorting the pretrained features.

• Decision process: start frozen as a baseline, fine-tune with a low learning rate and early stopping, and compare on held-out data. Also weigh the cost of each option.

Use unsupervised learning when you have no trustworthy signal about the target concept, or when you want to discover structure rather than predict a known outcome. Use weak supervision when cheap, noisy sources of the target exist, such as heuristics, rules, distant labels or proxies.

• Weak supervision needs a defined target:

  • Noisy labeling functions, keyword rules, knowledge-base matches or crowd labels approximate Y.

  • Tools like Snorkel denoise and combine these sources into training labels.

• Choose unsupervised when:

  • The categories aren't known yet (exploratory segmentation, topic discovery).

  • You cannot write even rough heuristics for the target, as with novel anomalies.

  • The available proxies are biased in ways that would mislead the model more than help it.

  • The goal is general-purpose representations or compression, not a specific task.

• Choose weak supervision when:

  • Domain experts can express their knowledge as rules or patterns.

  • The task needs a specific decision boundary that unsupervised structure won't line up with.

• They combine well:

  • Unsupervised clustering can reveal categories, and you then write labeling functions for them.

  • Alternatively, pretrain unsupervised representations and fine-tune them on the weak labels.

In high dimensions, data becomes exponentially sparse relative to volume, so nonparametric density estimators (histograms, KDE, k-NN) need exponentially more samples to keep the same accuracy; with realistic data sizes, estimates become noisy, oversmoothed, or degenerate.

• Volume explodes, local neighbourhoods empty out:

  • A histogram with k bins per axis has k^d cells; most are empty even with millions of points.

  • To capture a fixed fraction of data in a local window, its side length must approach the full range of each axis, so "local" estimates stop being local.

• Convergence rate collapses: KDE's optimal MSE rate is about n^(-4/(4+d)): at d=1 it is n^(-0.8), at d=20 it is about n^(-0.17), so halving error needs vastly more data.

• Bandwidth dilemma worsens:

  • A small bandwidth gives spiky estimates that are near zero away from training points (high variance).

  • A large bandwidth blurs all structure into a blob (high bias).

• Distances concentrate:

  • Nearest and farthest neighbour distances become similar, so distance-based kernels and k-NN densities lose contrast.

  • Most Gaussian mass sits in a thin shell, not near the mode, which breaks intuition about where density is high.

• Mitigations:

  • Exploit low intrinsic dimension: reduce dimensions first (PCA, autoencoders), then estimate density.

  • Impose structure: parametric models, GMMs with constrained covariance, or factorized and graphical models.

  • Use deep density models such as normalizing flows and autoregressive models, which share parameters across dimensions.

Score each estimator by how much probability it assigns to held-out data: the average held-out log-likelihood. Higher is better, because it estimates (up to a constant) the negative KL divergence from the unknown true distribution, and that constant is the same for every model.

• Why held-out log-likelihood works:

  • KL(p||q) = E_p[log p] − E_p[log q]. The first term is fixed, so maximizing E_p[log q] (estimated by the sample mean on test data) minimizes KL without knowing p.

  • It must be held out: on training data, a tiny-bandwidth KDE achieves near-infinite likelihood by memorizing.

• Practical protocol:

  1. Use k-fold cross-validation (or a fixed test split) and compute mean log-density per point.

  2. Compare models on the same splits and report variance or paired differences across folds.

  3. For continuous models, ensure densities are normalized. For discrete versus continuous data, compare in bits per dimension, with dequantization.

• Complementary checks:

  • Information criteria (AIC, BIC) for parametric models of differing complexity.

  • Sample-based tests: draw samples and compare them with real data via two-sample tests (MMD, KS on marginals) or a classifier two-sample test.

  • Calibration: probability integral transform (PIT) histograms or quantile coverage for 1-D or conditional densities.

  • Downstream task performance, such as anomaly detection AUC on labelled incidents.

• Caveat: High likelihood does not guarantee good samples, and good samples do not guarantee high likelihood. Pick the metric that matches the use case.

Frame it as density estimation when the business needs "how likely or typical is this point?" (a continuous score). Frame it as clustering when it needs "which group does this belong to?" (a discrete, interpretable partition to act on).

• Signals for density estimation:

  • Anomaly, fraud, or novelty detection: you need to flag low-probability events.

  • Risk and capacity planning: you need likelihoods, quantiles, or tail probabilities.

  • Generating synthetic data, or using a prior inside a larger probabilistic model.

  • The data may have no natural groups (a single smooth or skewed distribution).

• Signals for clustering:

  • The decision is categorical: customer segments, a different campaign per group, routing tickets.

  • Stakeholders need a few named, describable groups.

  • You are exploring structure or compressing data (prototypes, deduplication).

• Ask these questions:

  1. What action follows: a threshold on a score, or a per-group treatment?

  2. Is the output consumed as a probability (calibration matters) or as a label?

  3. How will success be measured: held-out likelihood or detection precision, versus segment stability, separation, and business lift?

• They overlap:

  • A Gaussian mixture model does both: components give soft clusters, and the mixture gives a density.

  • Density-based clustering (DBSCAN, HDBSCAN, mean-shift) defines clusters as high-density regions. If both outputs are needed, choose a model that provides both.

Confirmation bias is when a self-training model learns from its own wrong pseudo-labels: errors get added to the training set as if they were ground truth, the model fits them more confidently, and the next round of pseudo-labels repeats and amplifies the same mistakes.

• Why it happens:

  • Pseudo-labels come from the model's own predictions, so they reflect its current biases (wrong boundaries, class imbalance).

  • Training on them lowers loss on those errors, so the model becomes more confident in them. This is a self-reinforcing feedback loop.

  • Neural nets are often overconfident, so a high softmax score is no guarantee of correctness.

• Symptoms:

  • Errors concentrate on hard or minority classes; predictions drift toward majority classes.

  • Training confidence rises while held-out labeled accuracy stalls or drops.

• Mitigations:

  • Confidence thresholds (e.g. FixMatch's 0.95), possibly per class or adaptive (FlexMatch).

  • Soft labels or label smoothing instead of hard argmax, and mixup.

  • A separate teacher: EMA teacher (Mean Teacher), co-training or two networks labeling for each other.

  • Strong augmentation on the student plus consistency regularization, and re-generating labels each round instead of freezing them.

  • Class-balanced selection of pseudo-labels.