65 Random Forests Interview Questions and Answers (2026)

A Random Forest regressor predicts the arithmetic mean of the individual trees' predictions. Each tree outputs the average target value of the leaf the sample lands in, and the forest averages those outputs.

• Per-tree prediction: The sample is routed down the tree to a leaf. The leaf's value is the mean of the training targets in it (with the default squared-error criterion).

• Forest prediction:

  • The forest computes ŷ = (1/B) Σ ŷ_b over all B trees, with equal weight per tree.

  • In scikit-learn, RandomForestRegressor.predict is exactly this mean.

• Why averaging helps: Averaging many high-variance, low-bias trees reduces variance. The reduction is largest when trees are weakly correlated.

• Consequences:

  • Predictions are always within the range of the training targets, so the forest cannot extrapolate.

  • The spread of the per-tree predictions gives a rough uncertainty signal. Quantile regression forests go further and keep the full leaf distributions to estimate quantiles instead of just the mean.

Both train trees on bootstrap samples and aggregate their outputs. The difference is that a Random Forest also considers only a random subset of features at each split. This decorrelates the trees, so averaging them reduces variance further.

• Bagged trees: every split searches all p features:

  • If a few features are very strong, almost every tree splits on them near the root.

  • The trees therefore end up looking similar and are highly correlated.

• Random Forest: every split searches m < p random features:

  • The subset size is controlled by max_features. Common choices are √p for classification and about p/3 for regression.

  • Weaker features get a chance to be chosen, so the trees become more diverse.

• Why decorrelation matters:

  • The variance of an average of B trees with pairwise correlation ρ is ρσ² + (1−ρ)σ²/B.

  • Adding more trees only shrinks the second term. Lowering ρ is the only way to reduce the first term, and feature subsampling does exactly that.

• Bagging is a special case of Random Forest with max_features equal to all features.

• Trade-off: each individual tree gets slightly worse (a small bias increase), but the ensemble's variance usually drops by much more.

A Random Forest is an ensemble of decision trees. Each tree is trained on a bootstrap sample and considers random feature subsets at each split. It predicts by combining the trees: majority vote (or averaged probabilities) for classification, and the mean for regression.

• Training: two sources of randomness:

  1. Row sampling: each tree is fit on a bootstrap sample (n rows drawn with replacement).

  2. Feature sampling: each split only searches a random subset of max_features features.

  3. Trees are usually grown deep (low bias, high variance) and are not pruned.

• Prediction:

  • Each tree routes the sample to a leaf and outputs that leaf's class distribution or mean value.

  • Classification uses a majority vote. scikit-learn instead averages predict_proba across trees and takes the argmax.

  • Regression averages the trees' numeric predictions.

• Why it works:

  • Averaging many diverse, weakly correlated trees cancels out their individual errors.

  • The result has much lower variance than a single tree, at roughly the same bias.

• Practical perks:

  • Little tuning is needed.

  • Out-of-bag error gives a free validation estimate.

  • Feature importances come built in.

  • Trees are independent, so training parallelizes easily.

Bootstrap sampling draws n observations with replacement from a training set of size n. A Random Forest gives each tree its own bootstrap sample so the trees see different data, which makes them diverse enough that averaging reduces variance.

• What a bootstrap sample looks like:

  • It has the same size as the original data.

  • Some rows appear several times and others not at all. About 63.2% of rows are unique and about 36.8% are left out.

• Why the forest uses it:

  1. Diversity: each tree fits a slightly different dataset, so the trees' errors are less correlated.

  2. Variance reduction: averaging decorrelated, high-variance trees stabilizes predictions.

  3. Out-of-bag (OOB) evaluation: the rows a tree never saw act as a built-in validation set. Each row is scored only by the trees that left it out (oob_score=True).

• Cost: Each tree sees fewer distinct points, which adds a little bias per tree. This is usually outweighed by the variance gain.

• Controlled in scikit-learn by bootstrap (default True) and max_samples.

Bagging trains many copies of the same model, each on a different bootstrap sample of the data. It then aggregates their predictions (averaging for regression, voting for classification) to get a lower-variance ensemble.

• Procedure:

  1. Draw B bootstrap samples, each of size n with replacement.

  2. Fit one base model on each sample, independently and in parallel.

  3. Aggregate the outputs: mean for regression, majority vote or averaged probabilities for classification.

• What it fixes:

  • It mainly reduces variance and leaves bias roughly unchanged.

  • It works best with unstable, low-bias learners such as deep decision trees.

  • It helps little for stable models like linear regression or k-NN with large k.

• Limitation:

  • Bootstrap samples overlap heavily, so the models stay correlated and that correlation caps the variance reduction.

  • Random Forest adds per-split feature subsampling to break this correlation.

• Contrast with boosting:

  • Bagging builds models in parallel and independently, targeting variance.

  • Boosting builds models sequentially, each correcting the previous ones' errors, targeting mainly bias.

• Bonus: the left-out rows give out-of-bag error estimates without a separate validation set.

Each deep tree is low-bias but high-variance: its errors depend heavily on the particular training sample. Averaging lets these idiosyncratic errors partly cancel, while the expected prediction (and so the bias) stays the same.

• Independent case: The mean of B i.i.d. predictions with variance σ² has variance σ²/B.

• Realistic correlated case:

  • Trees share data and features, so with pairwise correlation ρ: Var = ρσ² + (1 − ρ)σ²/B.

  • More trees shrink only the second term; the ρσ² floor remains.

• Bias is unchanged: E[average] = E[single tree], so averaging fixes variance, not bias; hence you grow deep, low-bias trees.

• Why Random Forests add randomness: Bootstrapping and feature subsampling lower ρ, so averaging removes more variance.

Random Forest hyperparameters fall into three groups: ensemble size, the randomness that decorrelates trees, and the complexity limits on each tree. Names below follow scikit-learn.

• Ensemble size:

  • n_estimators: number of trees.

  • More trees lower variance until the error plateaus. Past that point they only cost compute and do not overfit.

• Randomness and decorrelation:

  • max_features: number of features considered at each split. This is the most important knob, because it trades tree strength against correlation.

  • bootstrap / max_samples: whether each tree trains on a resample, and how large that resample is.

• Tree complexity (regularization):

  • max_depth, max_leaf_nodes: cap the size of each tree.

  • min_samples_split, min_samples_leaf: the minimum number of samples needed to split a node or to form a leaf. These smooth predictions.

  • min_impurity_decrease: a split happens only if it improves impurity by at least this amount.

• Split quality and class handling:

  • criterion: the impurity measure. Classification uses gini or entropy; regression uses squared_error and similar.

  • class_weight: reweights classes when the data is imbalanced.

• Utility settings (do not change the model's capacity):

  • oob_score: computes a free validation estimate from out-of-bag samples.

  • n_jobs: parallelism. random_state: reproducibility.

Averaging reduces variance but not bias. The forest therefore wants low-bias, high-variance trees, which fully grown trees are, and lets averaging over decorrelated trees remove the variance. Pruning would add bias that the ensemble cannot undo.

• Bias–variance logic:

  • A deep tree fits fine structure: low bias, but high variance.

  • The bias of the average is roughly the bias of a single tree, so any bias added by pruning carries straight through to the forest.

• Variance is handled elsewhere:

  • Bootstrap samples and random feature subsets decorrelate the trees.

  • Their individual overfitting errors partly cancel when averaged, shrinking the (1 − ρ)σ²/B term.

• Practical bonus: No pruning step means no cost-complexity tuning and faster, simpler training.

• Caveats:

  • On very noisy or small data, mild limits often help: min_samples_leaf or max_depth. Common defaults include leaf size 5 for regression.

  • Fully grown trees can also be very large in memory and slow at prediction time.

Tree splits are thresholds on one feature at a time, so they depend only on the ordering of values, not their scale. Any monotonic rescaling produces the same partitions and the same predictions.

• Splits are order-based:

  • A split like x_j ≤ t divides samples the same way whether x_j is in metres or kilometres. The threshold just moves.

  • Impurity reduction is computed from the target values on each side, which scaling doesn't change.

• Features are never combined: There are no distances (unlike kNN or SVM), no dot products, and no gradient descent on weights (unlike linear models or neural nets), so one feature's range can't dominate another's.

• Same reason outliers in features matter little: a split only cares which side of the threshold a value falls on.

• When you'd still scale:

  • The pipeline includes scale-sensitive steps (PCA, kNN imputation) or other models you're comparing against.

  • Non-monotonic transforms (binning, interactions) are a different matter: they do change what the trees can learn.

OOB error is a built-in validation estimate. Each training sample is predicted using only the trees that did not see it in their bootstrap sample, and the error of those predictions is measured against the true labels.

• Where OOB samples come from:

  • Each tree trains on a bootstrap sample (drawn with replacement) containing about 63.2% unique rows.

  • The remaining ~36.8% (≈ 1/e) are that tree's out-of-bag rows.

• How it is computed:

  1. For each sample, collect predictions only from the trees where it was OOB.

  2. Aggregate them: majority vote or averaged probabilities for classification, the mean for regression.

  3. Compare the aggregated predictions with the true values across all samples to get the misclassification rate, MSE, or another metric.

• Why it is useful:

  • It is close to an unbiased estimate of generalization error, comparable to cross-validation, without a holdout set or refitting.

  • In scikit-learn, set oob_score=True (requires bootstrap=True), then read oob_score_ and oob_decision_function_ or oob_prediction_.

• Caveats:

  • It is slightly pessimistic, because each sample is scored by only about a third of the trees.

  • It is invalid when rows are not independent (time series, grouped or duplicated data), and it ignores leakage from preprocessing done on the full dataset.

MDI (Gini importance) adds up the weighted impurity reduction from every split that uses a feature, across all trees. It then averages over the forest and normalizes the scores to sum to 1.

• Per split at node t on feature j:

  • Decrease = (N_t/N) × [I(t) − (N_L/N_t)·I(L) − (N_R/N_t)·I(R)].

  • I is Gini or entropy for classification, and variance (MSE) for regression.

  • Weighting by N_t/N means splits near the root, which affect many samples, count more.

• Aggregation:

  1. Sum the decreases over all nodes in a tree that split on j.

  2. Normalize within each tree, then average over all trees.

  3. Normalize the final scores to sum to 1 (as in feature_importances_ in scikit-learn).

• Properties:

  • Essentially free: it is computed from the training-time tree structure.

  • It uses training data only, so it reflects how much a feature was used to fit, not how much it helps generalization.

  • It is biased toward high-cardinality features and splits credit among correlated features.

Random Forests give strong, robust accuracy with little tuning or preprocessing, because averaging many decorrelated trees cuts variance. The costs are interpretability, model size and latency, and typically a small accuracy gap to well-tuned boosting.

• Advantages:

  • Low variance: bagging plus feature subsampling (max_features) decorrelates the trees, so the average overfits far less than a single deep tree.

  • Little preprocessing: no scaling is needed, it is fairly robust to outliers, and it captures nonlinearities and interactions automatically.

  • Few sensitive hyperparameters: adding more trees does not cause overfitting; it only gives diminishing returns.

  • Free validation: the out-of-bag (OOB) error estimates generalization without a holdout set.

  • Embarrassingly parallel: trees are trained independently (n_jobs=-1).

  • Useful extras: feature importances, proximities, and handling of high-dimensional data.

• Disadvantages:

  • Hard to interpret: hundreds of deep trees cannot be read as rules.

  • Large and slow at inference: memory grows with the number of trees and their depth, which matters on latency-sensitive or edge deployments.

  • No extrapolation: regression predictions are bounded by the target range seen in training.

  • Biased impurity importance: it favors high-cardinality and continuous features and splits credit among correlated features. Permutation importance or SHAP is safer.

  • Usually beaten by tuned GBDTs on tabular data. It is also weaker on very sparse, high-dimensional text, where linear models often win.

  • Imbalanced classes need care: use class_weight, balanced subsampling, or threshold tuning.

Choose a single decision tree when transparency, simplicity, or deployment constraints matter more than the last few points of accuracy. A shallow tree is a model a human can read, audit, and implement as plain rules.

• Interpretability is required:

  • Regulated decisions (credit, medical triage) may demand an explanation for each decision that maps to explicit rules.

  • Stakeholders can validate the logic directly, which builds trust and makes domain errors easy to spot.

• Deployment constraints: Microsecond latency, tiny memory footprints, embedded devices, or the need to express the model as if/else code or a SQL CASE statement.

• The problem is genuinely simple: When a few axis-aligned rules capture the signal, a forest adds cost without adding accuracy.

• Exploration and communication: As a quick baseline, for understanding segments and interactions, or as a surrogate model that explains a complex model.

• Caveats to mention:

  • Single trees have high variance: small changes in the data can change the structure.

  • Constrain them (max_depth, min_samples_leaf, cost-complexity pruning via ccp_alpha) and check their stability with cross-validation.

Each tree is grown on its own bootstrap sample with its own random feature choices, and no tree depends on any other. All B trees can therefore be built simultaneously on separate cores or machines, and the only communication needed is collecting them at the end.

• No sequential dependency: Contrast with boosting: tree t fits the residuals of trees 1..t-1, so trees must be built in order. Boosting can only parallelize work within a tree (e.g., split search).

• Minimal coordination:

  • Each worker only needs the data and an independent random seed.

  • Aggregation (vote or average) is a trivial reduce step.

• In practice:

  • In scikit-learn, n_jobs=-1 distributes trees across cores via joblib.

  • Spark MLlib distributes the work across a cluster.

  • Speedup is near-linear until cores exceed B.

• Prediction is parallel for the same reason: every tree scores independently.

• Caveats:

  • Each worker may need a copy of the data, which drives memory pressure with process-based parallelism.

  • Seeds must be set per tree for reproducibility.

Hard voting gives each tree one vote for its predicted class and picks the majority. Soft voting averages the trees' predicted class probabilities and picks the class with the highest mean probability. Breiman's original Random Forest used hard voting; scikit-learn uses soft voting.

• Hard voting:

  • Prediction = mode of the trees' class labels.

  • A tree that is 51% confident counts the same as one that is 100% confident.

• Soft voting:

  • A tree's probability is the class fraction of training samples in its leaf.

  • The forest averages these probabilities and takes the argmax. This is what RandomForestClassifier.predict does via predict_proba.

  • Because it uses each tree's confidence, it usually gives slightly better accuracy and smoother, more useful probability scores.

• When they coincide:

  • With fully grown trees and pure leaves, every tree outputs probability 0 or 1, so soft and hard voting give the same answer.

  • They diverge when leaves are impure, for example with min_samples_leaf > 1 or shallow trees.

• Caveat:

  • Even averaged forest probabilities are often poorly calibrated, typically pushed away from 0 and 1.

  • Calibrate them (e.g. CalibratedClassifierCV) if you need true probabilities.

Each of the n draws misses a given observation with probability 1 − 1/n. Across all n draws it is missed with probability (1 − 1/n)ⁿ, which approaches 1/e ≈ 0.368 as n grows. So about 1 − 0.368 = 63.2% of the original observations appear in the sample at least once.

• Derivation:

  1. One draw picks a specific row with probability 1/n.

  2. Draws are independent (sampling with replacement), so P(never picked in n draws) = (1 − 1/n)ⁿ.

  3. Using the limit (1 − 1/n)ⁿ → e⁻¹ ≈ 0.3679, P(picked at least once) ≈ 0.632.

• Converges quickly:

  • For n = 10 the value is about 0.651, and for n = 100 about 0.634.

  • So 63.2% is a good approximation for any realistic dataset.

• It is an expectation:

  • The expected number of unique rows is n(1 − (1 − 1/n)ⁿ). Any single sample varies around this.

  • The sample still has n rows in total; the remaining ~36.8% of slots are filled by duplicates.

• Why it matters:

  • The ~36.8% of rows left out of each tree are its out-of-bag (OOB) rows.

  • They provide a free validation estimate, and each row is OOB for about a third of the trees.

With bootstrap=False, every tree trains on the full dataset, so the only diversity comes from per-split feature subsampling. Each tree's bias stays about the same or drops slightly, but the trees become more correlated, so the ensemble's variance rises.

• Bias: roughly unchanged, possibly slightly lower:

  • Each tree now sees all n distinct points instead of about 63.2% of them.

  • Individual trees therefore fit the data a bit better.

• Variance: higher:

  • Ensemble variance is ρσ² + (1−ρ)σ²/B, where ρ is the correlation between trees.

  • Removing row resampling raises ρ, so averaging cancels less of the error.

• Extreme case: max_features set to all features:

  • With no randomness left, all trees are essentially identical (apart from random tie-breaking between equally good splits).

  • The forest collapses to a single deep tree with that tree's full variance.

• Side effects:

  • There are no out-of-bag rows, so oob_score is unavailable and you need cross-validation instead.

  • How much this hurts depends on how strong the remaining randomness is.

  • Extra Trees (ExtraTreesClassifier) turns bootstrapping off by default and still stays diverse, because it also randomizes split thresholds.

max_samples sets how many rows each tree's bootstrap sample contains. Smaller samples mean less overlap between trees' training sets, so trees are more diverse (less correlated), but each tree is individually weaker.

• What it controls (scikit-learn):

  • Only active when bootstrap=True; an int is an absolute count, a float is a fraction of n_samples.

  • Default None draws n rows with replacement (about 63.2% unique rows per tree).

• Why smaller samples increase diversity:

  • Drawing a fraction f of n with replacement gives roughly 1 - e^(-f) unique rows; two trees then share fewer rows, so they fit different noise.

  • Lower overlap lowers tree correlation ρ, which lowers the ensemble's variance floor ρσ².

• The cost: Each tree sees less data: higher bias and higher individual variance (σ²), so too small hurts accuracy.

• Side effects:

  • Faster training per tree; more out-of-bag rows per tree for OOB estimates.

  • Useful on very large datasets where full-size bootstraps are redundant.

• Tune it alongside max_features with OOB score or CV; both trade tree strength for decorrelation.

Pasting trains each ensemble member on a random subset of rows drawn without replacement; bagging draws with replacement (bootstrap). Both aggregate by averaging or voting; they differ only in how the training subsets are sampled.

• Sampling:

  • Bagging: duplicates possible; a full-size bootstrap contains about 63.2% unique rows.

  • Pasting: no duplicates; the subset must be smaller than the dataset, otherwise every model sees identical data and there is no diversity.

• Diversity and bias/variance:

  • Bootstrap adds extra randomness (duplicate weighting), usually giving slightly more diverse models: slightly higher bias, lower ensemble variance.

  • Pasting's diversity is controlled entirely by the subset size.

• Out-of-bag evaluation: Bagging gets OOB estimates for free; in scikit-learn oob_score=True requires bootstrap=True.

• Origin: Breiman's "Pasting small votes" (1999), aimed at datasets too large to fit in memory.

Without it, every tree would pick the same strong features at the top splits and end up structurally similar, so their errors would line up. Restricting each split to a random subset of features forces trees to take different paths, making their errors less aligned.

• The problem with plain bagging:

  • Bootstrap samples overlap heavily, so greedy split search keeps choosing the same dominant features near the root.

  • Similar top splits produce similar partitions and correlated predictions.

• What subsampling changes:

  • At each node only m of p features compete, so the top feature is often absent and other informative features get used.

  • Different splits early cascade into very different trees.

• Why it matters: Ensemble variance is ρσ² + (1 − ρ)σ²/B; lower ρ lowers the floor that more trees can't remove.

• Trade-off: Smaller m means lower correlation but weaker individual trees; common defaults are √p (classification) and p/3 (Breiman, regression). Tune max_features.

Per-split sampling injects randomness at every node while still letting each tree use every feature somewhere. Sampling once per tree would permanently blind a tree to excluded features, making many trees weak, and adds less diversity overall.

• Problem with once-per-tree (Random Subspace style):

  • If a key feature is left out, the whole tree can never use it: high bias.

  • Interactions involving excluded features are impossible for that tree.

• Benefits of per-split sampling:

  • Strength: important features are likely available at some node, so trees stay accurate.

  • Diversity: randomness compounds down the tree, and varied root splits create different structures, lowering correlation.

  • Net result: a better strength/correlation trade-off, which Breiman's generalization bound shows is what drives forest error.

• Cost: negligible, since split search already scans features at each node; sampling just evaluates fewer.

Adding trees only averages more independent draws from the same distribution. It lowers Monte Carlo variance and makes the generalization error converge to a fixed limit, without making the model more flexible. Overfitting depends on how complex each tree is and how correlated the trees are, not on how many there are.

• Convergence:

  • Breiman showed via the strong law of large numbers that forest error converges almost surely as the number of trees B grows to infinity.

  • Test error decreases and then plateaus. It does not turn back upward.

• Why capacity does not grow:

  • Trees are trained independently. No tree is fit to the errors of the others.

  • Each extra tree just refines the estimate of the same expected prediction, E[tree(x)].

• Contrast with boosting:

  • Boosting fits each new tree to the current residuals, so more rounds keep adding capacity and can overfit.

  • That is why boosting needs early stopping and Random Forests do not.

• Caveats:

  • A forest can still overfit for other reasons: deep trees on noisy data, or leakage. Adding trees just will not fix that.

  • Beyond the plateau, extra trees only cost training time, memory and prediction latency.

min_samples_leaf forces every leaf to contain at least k training samples. This stops trees from isolating individual (possibly noisy) points, so each prediction averages over a neighbourhood. Trees become shallower and smoother: variance goes down and bias goes up.

• How it acts:

  • A split is considered only if both children keep at least k samples. This implicitly limits depth.

  • It accepts either an integer count or a float fraction of the samples.

• Why it regularizes:

  • With k = 1, a leaf can memorize a single label. A larger k averages at least k targets, damping noise.

  • Regression: leaf means become more stable. Classification: leaf class proportions become less extreme, giving better-calibrated probabilities.

• Trade-off:

  • If k is too large, trees underfit and miss small but real patterns. Averaging cannot recover that lost bias.

  • Defaults: scikit-learn uses 1. R randomForest uses nodesize 1 for classification and 5 for regression.

• Compared with max_depth: min_samples_leaf adapts to the data: dense regions can still be split finely, while sparse regions stop early.

• Tuning: Tune k with OOB score or cross-validation. It helps most on noisy targets.

The classic defaults from Breiman and the Elements of Statistical Learning are √p features per split for classification and p/3 for regression, where p is the number of features. Libraries differ, though: scikit-learn's regressor considers all features by default.

• Classic recommendation (R randomForest, H2O):

  • Classification: floor(√p).

  • Regression: floor(p/3), with a minimum of 1.

• scikit-learn:

  • RandomForestClassifier: max_features="sqrt".

  • RandomForestRegressor: max_features=1.0 (all features) since v1.1, which was previously "auto". By default this makes it bagged trees rather than a decorrelated forest.

• Why regression uses more features: Regression trees lose strength faster when the best predictors are hidden from a split, so a larger subset keeps individual trees accurate.

• Treat defaults as starting points:

  • max_features is the key strength/correlation knob and is worth tuning with OOB score.

  • It matters most with many noisy features (where √p may miss the signal) or with highly correlated features.

Random Forests resist overfitting from adding more trees, not overfitting in general. They still overfit when deep trees memorize noise that averaging cannot cancel, or when the data or validation setup lets them learn patterns that will not generalize.

• Noisy targets with fully grown trees:

  • With leaf size 1, every tree fits label noise. Because trees share many of the same samples, that noise is correlated and survives averaging.

  • Fix: raise min_samples_leaf or limit max_depth.

• Small n, high p, few informative features:

  • Random splits on many irrelevant features find spurious structure.

  • A small max_features often excludes the true signal from a split, making this worse.

• High-cardinality or ID-like features: Impurity-based splitting favours features with many unique values, such as user IDs or timestamps, which can isolate individual rows.

• Leakage and dependent samples:

  • Duplicate or near-duplicate rows, grouped data (e.g. several rows per patient) and time series all inflate OOB and random-CV scores.

  • The cause is that correlated copies land in both training and evaluation. Use group or time-based splits.

• Class imbalance and tiny minority classes: Leaves memorize the few minority examples, so recall looks good in training and collapses on new data.

• How to diagnose: Near-100% training accuracy is normal for a forest. Judge overfitting by the gap between training score and OOB or properly grouped cross-validation score.

Lowering max_features makes trees less correlated, which lowers ensemble variance, but makes each tree weaker, which raises bias. Near 1, splits are almost random, so bias usually dominates and accuracy drops. The best value sits in between.

• Why variance falls:

  • Ensemble variance ≈ ρσ² + (1−ρ)σ²/B (ρ = correlation between trees, σ² = single-tree variance, B = number of trees).

  • Fewer candidate features per split means trees stop picking the same strong features, so ρ drops and the irreducible ρσ² floor shrinks.

• Why bias rises:

  • Each split picks the best of only a few random features, often not a good one. Trees become shallower in useful signal and fit the true function worse.

  • At max_features=1 the split feature is purely random (only the threshold is optimized), so you get nearly totally randomized trees.

  • Individual trees also get noisier (higher σ²), which partly offsets the gain from lower ρ.

• It depends on the data:

  • With many irrelevant features, small values hurt badly because most splits land on noise.

  • With many correlated, informative features, small values can work well.

• Practice: start from defaults (sqrt(p) for classification, p/3 is the classic regression rule) and tune using OOB or CV error.

Error drops quickly and then plateaus, usually somewhere in the low hundreds of trees. Adding more trees doesn't cause overfitting. Past the plateau it only adds compute, memory and latency for negligible gain.

• Why it plateaus:

  • In ρσ² + (1−ρ)σ²/B, more trees (B) only shrink the second term. The correlation floor ρσ² remains.

  • Breiman showed the generalization error converges as B grows, so more trees can't overfit. They just stop helping.

• When you need more trees:

  • Many features or a small max_features: each tree is weaker and noisier, so averaging takes longer to converge.

  • Stable feature importances, well-resolved predicted probabilities and proximities converge more slowly than accuracy does.

• How to find the point:

  • Plot OOB error against the number of trees using warm_start=True, and stop where the curve flattens.

  • Treat it as a cost/accuracy trade-off rather than a hyperparameter to tune for accuracy.

A Balanced Random Forest (Chen, Liaw & Breiman, 2004) undersamples the majority class separately for every tree. Each tree trains on a class-balanced bootstrap, while the ensemble as a whole still sees most of the majority data.

• How each tree is built:

  1. Draw a bootstrap sample of the minority class.

  2. Randomly draw the same number of majority-class samples, with replacement.

  3. Grow a normal randomized tree on this balanced set, then aggregate trees by voting as usual.

• Why it works:

  • Splits are no longer dominated by the majority class, so trees actually learn minority-class regions.

  • Plain undersampling throws data away. Here, different trees see different majority subsets, so little information is lost, and the extra diversity also reduces variance.

  • Each tree trains on a smaller set, which also makes training cheaper.

• Alternative: a weighted RF (class_weight) keeps all data and instead penalizes minority errors more. Which works better is dataset-dependent.

• Caveats:

  • Predicted probabilities are biased toward the minority class. Recalibrate them or tune the decision threshold.

  • With very few minority samples, the same points repeat in every tree, which risks overfitting them.

  • Evaluate with PR-AUC, recall/precision or balanced accuracy, not plain accuracy.

• Implementation: imblearn.ensemble.BalancedRandomForestClassifier (its sampling_strategy parameter controls the target ratio).

Both reweight classes inversely to their frequency. 'balanced' computes the weights once from the full training labels. 'balanced_subsample' recomputes them for each tree from that tree's bootstrap sample.

• Formula (same for both):

  • n_samples / (n_classes * np.bincount(y))

  • The weights multiply sample weights, so they affect impurity calculations and leaf predictions.

• 'balanced': Uses global class counts. A bootstrap that happens to contain fewer minority samples isn't fully balanced.

• 'balanced_subsample':

  • Uses the counts in each tree's bootstrap, so every tree is exactly balanced in total weight.

  • Only differs from 'balanced' when bootstrap=True.

• When it matters:

  • With severe imbalance or a tiny minority class, the minority count varies a lot between bootstraps, so per-tree reweighting is more consistent.

  • Otherwise the results are usually very similar. Pick one via CV.

• Neither option resamples data. That is what distinguishes them from a Balanced Random Forest, which undersamples.

A Random Forest averages independent trees, so a bad point's influence gets diluted. Gradient boosting builds trees sequentially to correct remaining errors, so mislabeled points and target outliers keep producing large residuals that later trees chase.

• Random Forest:

  • Each noisy point appears in only about 63% of bootstraps, and trees don't react to each other's mistakes.

  • Individual deep trees do memorize noise, but in different ways, so averaging cancels much of it.

• Gradient boosting:

  • Each tree fits the gradients (residuals) of the current ensemble. Points it can't fit, such as wrong labels, keep large gradients.

  • Later trees spend capacity fitting those points, which bakes the noise into the model.

  • It's worst with AdaBoost's exponential loss and with squared loss on target outliers. Dietterich (2000) showed AdaBoost degrades more than bagging under label noise.

• Outliers in features affect both models little, since splits are rank-based. The difference is about noise in the target.

• Making boosting more robust: Robust losses (Huber, absolute error, quantile), a small learning rate, row subsampling, and early stopping on a validation set.

• Caveat: RF regression leaves output the mean of their samples, so extreme targets still pull leaf values. Larger min_samples_leaf helps.

Support depends on the implementation. Breiman's original method and R's randomForest have built-in imputation and native factors. scikit-learn's RF accepts NaN natively since version 1.4, but categoricals must be encoded. In practice, most pipelines impute and encode before training.

• Missing values:

  • Native NaN (scikit-learn ≥ 1.4)

  • At each split, the tree learns which child the missing values go to, choosing the side that gives the best impurity reduction.

  • If a feature had no missing values during training, missing values at prediction go to the child with the most samples.

  • Breiman's approach: start with a rough fill (median or mode, e.g. na.roughfix), then refine iteratively using proximity-weighted values (rfImpute).

  • Common practical approach: a SimpleImputer plus a missing-indicator column, so that missingness itself can be a signal.

• Categorical features:

  • Native support (R randomForest, H2O): splits on subsets of categories. R limits this to about 53 levels.

  • Ordinal encoding: works fine for trees despite the arbitrary order, because repeated splits can isolate individual categories. It's cheap and dense.

  • One-hot encoding: fine for low cardinality. With many levels, each dummy is rarely chosen, which spreads importance thinly and weakens trees.

  • Target encoding: good for high cardinality, but it must be done out-of-fold to avoid leakage.

• Pitfalls:

  • Fit imputers and encoders inside a Pipeline so CV doesn't leak information.

  • Handle categories that appear only at prediction time (handle_unknown).

  • MDI importance is inflated for high-cardinality features.

Every Random Forest regression prediction is an average of training target values, so it can never be larger than the largest training target or smaller than the smallest. Outside the training feature range, the forest just returns a flat value from the edge leaves.

• Leaves store means of training targets:

  • A regression tree predicts the mean y of the training samples that land in a leaf.

  • The forest averages these leaf means across trees, so the output is a weighted average (a convex combination) of observed targets.

  • A convex combination is always bounded by [min(y), max(y)].

• Splits are piecewise constant:

  • A split like x > 10 sends every value beyond the training range into the same outermost leaf.

  • The model learns no slope, so a trend cannot continue. Predictions stay constant past the edge of the data.

• Contrast with linear models: A fitted coefficient keeps applying to new x values, so a linear model extrapolates (correctly or not).

• Practical consequences:

  • Forecasting a trending series (sales, prices) will flatten out at historical levels.

  • Gradient-boosted trees have the same limitation, because they also sum constant leaf values learned within the training range.

• Common fixes:

  1. Detrend or difference the target, model the residual with the forest, then add the trend back.

  2. Hybrid approach: fit a linear model first, then fit the forest on its residuals.

  3. Use models with linear leaves (model trees, LightGBM linear_tree), or use a different model family when extrapolation matters.

More trees make the OOB estimate more reliable. Each sample gets more OOB votes, fewer samples are left with no OOB prediction, and the upward bias shrinks. Once the OOB error curve plateaus, adding more trees changes little.

• Votes per sample:

  • Each sample is OOB in about 36.8% of trees, so with B trees it gets roughly 0.37·B votes.

  • With few trees, each OOB prediction comes from a tiny ensemble, so it is noisy and depends heavily on the random seed.

• Samples with no OOB prediction:

  • The probability that a sample is in-bag for every tree is about 0.632^B, which is roughly 1% at B=10 and negligible past about 50 trees.

  • scikit-learn warns when this happens. Those samples are effectively excluded, which biases the estimate.

• Pessimistic bias:

  • OOB error measures a sub-forest about a third the size of the full model, so it overstates the full model's error.

  • The gap is large with few trees and vanishes as both error curves flatten.

• In practice:

  • Plot OOB error against n_estimators (cheaply, using warm_start=True) and pick a value on the plateau.

  • After the plateau, the remaining uncertainty comes from the finite dataset, not the number of trees.

For training instance i, collect predictions only from the trees whose bootstrap sample did not include i, then aggregate them the same way the forest normally does.

• Steps:

  1. Identify the set O_i of trees where i was out-of-bag (about 37% of all trees).

  2. Pass x_i through each tree in O_i.

  3. Aggregate the results. For classification, take a majority vote; scikit-learn instead averages class probabilities and takes the argmax. For regression, take the mean of the tree outputs.

• Why exclude the in-bag trees: Those trees have already fit i, often perfectly in deep trees, so including them would leak the label and make the error look optimistic.

• Edge case: If O_i is empty (possible with very few trees), i has no OOB prediction and is left out of the OOB score.

Use the OOB score when you want a fast, "free" generalization estimate for a bagged model on i.i.d. data. Use k-fold CV when you need a fair comparison across model types, leakage-safe pipelines, non-i.i.d. splits, or a variance estimate of the score.

• Prefer OOB when:

  • Training is expensive or data is large. OOB needs one fit instead of k.

  • You are doing quick tuning of n_estimators, max_features, or min_samples_leaf.

  • Rows are i.i.d., preprocessing is label-free or done inside the trees, and data is too scarce to sacrifice a holdout.

• Prefer k-fold CV when:

  • Comparing against non-bagged models (linear models, boosting), which should be evaluated with the same protocol.

  • The pipeline has learned steps such as target encoding, feature selection, or imputation. Fit on the full data, these leak into OOB, whereas CV refits them per fold.

  • Data is grouped or temporal and needs GroupKFold or TimeSeriesSplit.

  • You need a spread of scores across folds, a custom metric, or nested CV for unbiased tuning.

  • bootstrap=False is set, in which case OOB is unavailable.

• Nuance:

  • OOB slightly overstates error, since each sample is scored by about a third of the trees, but it is usually close to CV given enough trees.

  • If you tune heavily on OOB, keep a final holdout set, because OOB becomes optimistic once it is used for selection.

For each tree, measure its error on its own OOB rows. Then shuffle one feature's values within those rows and measure the error again. The average increase across trees is that feature's importance (Breiman's original RF importance).

• Algorithm for feature j:

  1. For tree b, compute the error e_b on its OOB samples.

  2. Randomly permute column j among those OOB samples, which breaks the link between X_j and y.

  3. Recompute the error to get e_b(j), and record the difference d_b = e_b(j) − e_b.

  4. Importance = mean of d_b over all trees. The scaled version divides by the standard deviation of d_b, giving a z-score.

• Why OOB:

  • The score reflects predictive value on unseen data, so it is less inflated by overfitting than MDI.

  • No separate validation set is needed.

• Implementations:

  • R randomForest(importance=TRUE) reports this as type=1 (Mean Decrease Accuracy).

  • scikit-learn's permutation_importance permutes on a whole held-out set you pass in, not on per-tree OOB rows.

• Caveats:

  • With correlated features, shuffling creates unrealistic rows, and a correlated partner can mask the loss.

  • Results are noisy, so use several repeats.

A feature with many distinct values offers many candidate split points. The greedy search therefore has more chances to find a split that lowers training impurity purely by chance. MDI credits these lucky splits because it is measured on training data.

• Multiple-comparisons effect:

  • A binary feature has 1 possible split, while a continuous or ID-like feature has up to n−1.

  • The best of many noisy splits shows a larger impurity drop than the best of one, even if neither carries any signal.

• Deep trees amplify it:

  • Near the leaves, nodes contain few samples, where spurious splits easily produce "pure" children.

  • High-cardinality features win these splits and accumulate importance.

• Training-set evaluation: MDI never checks whether these splits generalize, so a random ID column can rank highly (Strobl et al., 2007).

• Mitigations:

  • Use permutation importance on held-out or OOB data.

  • Use conditional inference forests (party::cforest), which select split variables with unbiased statistical tests.

  • Add a random noise column as a sanity baseline, or limit depth with min_samples_leaf.

Choose a Random Forest when you want a robust, low-maintenance model that is hard to overfit and fast to train in parallel. Tuned GBDTs usually win on raw accuracy, but they demand more tuning and are more sensitive to noise.

• Different error targets:

  • RF averages deep, low-bias trees to reduce variance.

  • GBDT adds shallow trees sequentially to reduce bias, so it can keep fitting until it overfits.

• Pick RF when:

  • There is little time or compute for tuning. RF works well near its defaults, and more trees never hurts. GBDT needs learning rate, number of trees, depth, and early stopping tuned jointly.

  • Labels are noisy or the dataset is small. Boosting chases hard or mislabeled points; averaging dilutes them.

  • You want parallel training across trees, or OOB error in place of a validation split.

  • You need a strong, stable baseline, or you need RF-specific tools such as proximities and quantile regression forests.

• Pick GBDT when:

  • You need maximum accuracy on tabular data and can afford careful tuning (XGBoost, LightGBM, CatBoost).

  • You need custom losses, monotonic constraints, or smaller and faster models (shallow trees).

A standard random forest searches for the optimal threshold of each candidate feature. Extremely Randomized Trees (ExtraTrees) instead draw one random threshold per candidate feature and pick the best of those random splits. This adds randomness to the split itself, not only to the data and feature choice.

• Random forest split:

  1. Sample max_features candidate features at the node.

  2. For each feature, evaluate all possible cut points (via sorting) and keep the one with the best impurity reduction.

  3. Choose the best feature/threshold pair.

• ExtraTrees split:

  1. Sample max_features candidate features, same as RF.

  2. For each feature, draw a threshold uniformly between that feature's min and max within the node.

  3. Score only those random splits and choose the best one.

• Sampling difference: ExtraTrees grows each tree on the full training set by default (bootstrap=False in scikit-learn), while RF bootstraps.

• Consequences:

  • Faster splits, more decorrelated trees, and smoother decision boundaries.

  • Variance drops further at the cost of slightly more bias.

  • With max_features=1, the trees become fully random, independent of the labels.

An Isolation Forest is an unsupervised anomaly detector, not a predictor. It builds completely random trees and scores points by how quickly they get isolated: anomalies sit in sparse regions, so they need few splits. A Random Forest is supervised and chooses splits that best predict a label.

• Objective:

  • RF learns a mapping from X to y for classification or regression.

  • Isolation Forest needs no labels and estimates how separable each point is from the rest.

• Split rule:

  • RF picks the split that maximizes impurity reduction (Gini, entropy, MSE).

  • Isolation Forest picks a random feature and a random threshold between that feature's min and max. There is no optimization criterion.

• Output:

  • RF outputs a majority vote or an average of leaf values.

  • Isolation Forest outputs an anomaly score from the average path length E[h(x)], normalized by the expected path length c(n): s = 2^(-E[h(x)]/c(n)).

  • A score near 1 indicates an anomaly; a score well below 0.5 indicates a normal point.

• Tree growth and sampling:

  • Isolation Forest uses small subsamples drawn without replacement (default 256). Trees are capped at about log2(256) = 8 levels, because only short paths matter.

  • RF bootstraps the full dataset and grows deep trees.

• Practical note:

  • A decision threshold must be chosen, e.g. via contamination in IsolationForest.

  • Axis-parallel random splits can produce artifacts. Extended Isolation Forest addresses this with oblique cuts.

For roughly balanced trees, training costs about O(B · m · N log² N), where m is the number of features tried per split (mtry, e.g. √M or M/3). With presorting or histogram tricks this drops to about O(B · m · N log N).

• Cost per node: For each of the m candidate features, sort the n samples in the node (O(n log n)) and scan the split points (O(n)).

• Cost per level: The nodes at one depth partition the N samples, so a level costs O(m · N log N).

• Cost per tree:

  • A balanced tree has depth about log N, giving O(m · N log² N).

  • Worst case is a degenerate, chain-like tree with depth about N: O(m · N² log N).

• Multiply by B trees. The cost is linear in B, and wall-clock time divides by the number of cores.

• Why random feature selection helps: Using m ≪ M makes an RF tree cheaper than a full CART tree, which costs O(M · N log² N).

• Practical modifiers:

  • A bootstrap sample contains about 0.632N unique points.

  • max_depth or min_samples_leaf cap the depth.

  • Histogram binning replaces sorting with O(n) bucketing.

Inference latency per sample grows as O(B · d): each of the B trees does one root-to-leaf walk of about d comparisons. Memory grows as O(B · nodes per tree), and nodes per tree can reach 2^d. So both costs are linear in the number of trees, but depth costs linear time and up to exponential memory.

• Latency:

  • Doubling B doubles the work. Doubling the depth roughly doubles the comparisons per tree.

  • Depth matters more than the comparison count suggests: deeper trees are bigger, so the pointer-chasing traversal suffers more cache misses and branch mispredictions.

  • Trees can be scored in parallel, which reduces wall-clock latency but not total compute.

• Memory:

  • Each node stores a feature index, a threshold, child pointers and a leaf value. In scikit-learn, every node holds a value array with one entry per class.

  • The node count is bounded by data, not just by depth. A fully grown tree has up to about N leaves (2N-1 nodes), so the model is roughly O(B · N), which can reach gigabytes on large datasets.

  • min_samples_leaf and max_leaf_nodes shrink the model roughly proportionally.

• Accuracy trade-off: test accuracy usually plateaus well before B gets large, so extra trees mostly add cost.

Both train each estimator on a random subset of features. Random Subspace keeps all training rows; Random Patches also subsamples rows, so each estimator sees a random "patch" of the data matrix (rows × columns).

• Random Subspace (Ho, 1998):

  • Feature subset per estimator, all samples used.

  • Best when features are many and samples are relatively few.

• Random Patches (Louppe & Geurts, 2012):

  • Random subset of samples and of features per estimator.

  • More diversity and lower memory/compute; suits large datasets in both dimensions.

• In scikit-learn's BaggingClassifier:

  • Subspace: bootstrap=False, max_samples=1.0, max_features<1.0

  • Patches: max_samples<1.0 and max_features<1.0

• Contrast with Random Forest: Both methods fix the feature subset once per estimator; Random Forest resamples candidate features at every split (plus bootstraps rows).

The variance is ρσ² + ((1 − ρ)/B)σ². As B grows, the second term vanishes but ρσ² remains, so correlation sets a floor that adding trees cannot break.

• Derivation:

  1. Var((1/B) Σ Tᵦ) = (1/B²)[Σ Var(Tᵦ) + Σ_{i≠j} Cov(Tᵢ, Tⱼ)]

  2. There are B variance terms (σ² each) and B(B − 1) covariance terms (ρσ² each).

  3. = σ²/B + ((B − 1)/B)ρσ² = ρσ² + (1 − ρ)σ²/B

• Limiting cases:

  • ρ = 0: σ²/B (full variance reduction).

  • ρ = 1: σ² (averaging identical trees does nothing).

• Practical implications:

  • More trees have diminishing returns but never overfit through this term.

  • To go lower, reduce ρ (smaller max_features, bootstrapping) without inflating σ² too much.

Tree correlation rises sharply. Whenever the dominant feature is among a node's candidates it wins the split, so most trees share the same top splits and make similar predictions. This raises the ρσ² variance floor.

• Mechanism:

  • The feature is a candidate at a node with probability m/p; with max_features near p (bagging), it is almost always chosen at the root.

  • Bootstrap samples barely change that choice, so tree structures converge.

• Lowering max_features helps, with a cost:

  • More nodes lack the feature, so trees use other features and decorrelate.

  • Those trees are weaker (higher σ² and bias); tune via OOB or CV.

• Side effects:

  • Impurity-based importance concentrates on that feature and can mask others.

  • If the signal truly lives in one feature, ensembling gains are inherently limited; a simpler model or boosting may do as well.

Removing one tree raises ensemble variance only slightly. It removes only part of the variance term that averaging can shrink, and with hundreds of trees the effect is negligible. Expected bias does not change, because every tree is an identically distributed random draw and none is special.

• Variance of an average of B correlated trees:

  • Var = ρσ² + (1 − ρ)σ²/B (σ²: single-tree variance, ρ: pairwise correlation between trees).

  • The ρσ² floor is fixed by tree correlation. Only the second term depends on B.

• Going from B to B − 1 trees:

  • The increase is (1 − ρ)σ²/(B(B − 1)), which shrinks roughly as 1/B².

  • Example: with B = 500 the increase is about 4×10⁻⁶·(1 − ρ)σ², so it is effectively zero.

• Bias is unchanged in expectation: Every tree has the same expected prediction, so averaging fewer of them does not shift the mean.

• Practical effects:

  • Individual predictions move a little, and OOB estimates get slightly noisier, since each sample has fewer OOB trees.

  • The effect only matters when B is small (e.g. 10 trees), where the 1/B term is still large.

Breiman (2001) bounded the forest's generalization error by PE* ≤ ρ̄(1 − s²)/s². Error falls when individual trees are strong and when they are weakly correlated, so a good forest balances the two.

• Strength s:

  • The expected margin: how much the vote for the correct class exceeds the vote for the best wrong class, averaged over the data.

  • The bound requires s > 0. Larger s gives a smaller bound.

• Correlation ρ̄:

  • The mean correlation between trees' raw margin functions, i.e. how often trees make the same mistakes.

  • Error scales linearly with ρ̄, so decorrelated trees are what make averaging pay off.

• The tension, controlled mainly by max_features (mtry):

  • Smaller mtry lowers correlation but also lowers each tree's strength.

  • Larger mtry raises strength but also raises correlation.

  • Bootstrap sampling and random feature selection exist to cut ρ̄ without destroying s.

• Caveats:

  • The bound is usually loose. Its value is conceptual, not as a numeric predictor of error.

  • The same paper shows error converges as trees are added (strong law of large numbers), which is why more trees do not overfit.