Machine Learning Full Course · Episode 34. Every code sample below was run with Python 3.13, NumPy 2.5.3 and scikit-learn 1.9.1; the output shown is the real output. Sunita's tiffin customers and kitchen days are made up for teaching; the mini project uses the breast cancer dataset bundled with scikit-learn.
The big idea: bagging (Episode 33) averages many deep trees, but when one feature is much stronger than the rest, every tree asks that feature first, so the trees look alike and make the same mistakes. Averaging correlated mistakes can't cancel them: the variance of the average is ρσ² + (1 − ρ)σ²/B, and more trees only shrink the second part. A random forest fixes this with feature subsampling: at every split, a tree may only choose among a few randomly picked features (max_features). Each tree gets a bit worse, but the trees become decorrelated, so the forest gets better. The number of trees (n_estimators) only needs to be "enough". Forests also predict numbers (random forest regression), and they come with feature importances, but the built-in impurity importance can rank useless, many-valued columns highly; permutation importance on held-out data is the honest check. Extra trees go one step further and pick split thresholds at random too.
Words you need first
- Decision tree, split, root, leaf, Gini impurity,
max_depth: Episodes 29-30. The root is the first question a tree asks. - Bootstrap sample, bagging, out-of-bag (OOB) score: Episode 33.
- Bias and variance: error from a model that is too simple; error from a model that changes a lot with the training data (Episode 20).
- Correlation: how strongly two quantities move together, from −1 to 1; 0 means unrelated (Episode 11).
- RMSE (root mean squared error): the typical size of a regression miss, in the target's units (Episode 19).
- Test set, validation, cross-validation: Episodes 15 and 26.
Bagged trees look alike
Sunita runs a tiffin (home-cooked lunch box) service with 3,000 made-up customers and 10 facts about each: missed meals last month, rating (1-5), distance in km, months as a customer, price paid, spice level, complaint calls, age, family size and veg days per week. The label: does the customer renew next month (1) or leave (0)? The hidden rule mixes most features, with missed meals the strongest; age is not in the rule at all.
We bag 100 full trees exactly as in Episode 33, and for each tree we record its root feature and whether it is wrong on each test customer. The error correlation is the average correlation between two trees' "wrong or right" patterns.
import numpy as np
rng = np.random.default_rng(34); n = 3000 # Sunita's tiffin service: 3,000 made-up customers
names = "missed_meals rating distance_km months price_rs spice calls age family veg_days".split()
P, U, I = rng.poisson, rng.uniform, rng.integers # random counts, decimals, whole numbers
X = np.c_[P(3,n), U(1,5,n), U(0,12,n), I(1,36,n), U(60,160,n), I(1,6,n), P(2,n), I(18,65,n), I(1,7,n), I(0,8,n)]
s = X @ [-1.2,.8,-.12,.04,-.01,0,-.25,0,0,.15] + .5*np.sin(X[:,5]) + .3*(X[:,8]>3) + .8*(X[:,1]>3.5)*(X[:,2]<4)
y = (s + rng.normal(0, 0.8, n) > np.median(s)).astype(int) # hidden made-up rule · 1 = renews next month
X_tr, y_tr, X_te, y_te = X[:2000], y[:2000], X[2000:], y[2000:]
from sklearn.tree import DecisionTreeClassifier as DT; from collections import Counter
def bag(k, mf=None): # ep33's bagging · mf = features tried at each split
boot, P, roots = np.random.default_rng(0), [], Counter(); acc = lambda p: (p == y_te).mean()
for _ in range(k):
i = boot.integers(0, 2000, 2000) # a bootstrap sample
t = DT(max_features=mf, random_state=int(boot.integers(10**6))).fit(X_tr[i], y_tr[i])
roots[names[t.tree_.feature[0]]] += 1; P.append(t.predict(X_te)) # which feature is at the root?
P = np.array(P); corr = np.corrcoef(P != y_te)[np.triu_indices(k, 1)].mean() # same mistakes?
print(f"mf={mf!s:<5} 1 tree {acc(P[0]):.3f} forest {acc(P.mean(0) > .5):.3f} corr {corr:.2f}")
print(" roots:", dict(roots.most_common(3)))
bag(100)
Output:
mf=None 1 tree 0.811 forest 0.868 corr 0.38
roots: {'missed_meals': 100}
- One tree 0.811 → 100 bagged trees 0.868: bagging works.
- All 100 roots are
missed_meals, and the trees' errors are correlated (0.38): when one tree is wrong about a customer, many others are wrong too.
Why the trees agree, and why that limits bagging
- A tree chooses each split greedily: it tries every feature and keeps the best. A bootstrap sample changes the rows a little, not the winner, so every tree starts with the same question and grows similar branches below it.
- For B models, each with variance σ² and pairwise correlation ρ, the variance of their average is
Var(average) = ρσ² + (1 − ρ)σ²/B
- Adding trees (bigger B) only shrinks the second term. The first term, ρσ², is a floor you can't average through. To go lower you need a smaller ρ: trees that disagree more.
Feature subsampling: the random forest
The fix: at every split (not once per tree), pick a random subset of the features and choose the best split among only those. In scikit-learn this is max_features:
max_features |
Features tried at each split (10 features) |
|---|---|
None or 1.0 |
all 10 (plain bagging) |
"sqrt" |
⌊√10⌋ = 3 |
"log2" |
⌊log₂10⌋ = 3 |
0.5 |
50% = 5 |
1 (an integer) |
exactly 1 |
Real-life picture: a quiz team where everybody studies the same easy chapter misses the same questions; if the teacher hands each student three random chapters at a time, each student is weaker but the team covers more.
import numpy as np
rng = np.random.default_rng(34); n = 3000 # Sunita's tiffin service: 3,000 made-up customers
names = "missed_meals rating distance_km months price_rs spice calls age family veg_days".split()
P, U, I = rng.poisson, rng.uniform, rng.integers # random counts, decimals, whole numbers
X = np.c_[P(3,n), U(1,5,n), U(0,12,n), I(1,36,n), U(60,160,n), I(1,6,n), P(2,n), I(18,65,n), I(1,7,n), I(0,8,n)]
s = X @ [-1.2,.8,-.12,.04,-.01,0,-.25,0,0,.15] + .5*np.sin(X[:,5]) + .3*(X[:,8]>3) + .8*(X[:,1]>3.5)*(X[:,2]<4)
y = (s + rng.normal(0, 0.8, n) > np.median(s)).astype(int) # hidden made-up rule · 1 = renews next month
X_tr, y_tr, X_te, y_te = X[:2000], y[:2000], X[2000:], y[2000:]
from sklearn.tree import DecisionTreeClassifier as DT; from collections import Counter
def bag(k, mf=None): # ep33's bagging · mf = features tried at each split
boot, P, roots = np.random.default_rng(0), [], Counter(); acc = lambda p: (p == y_te).mean()
for _ in range(k):
i = boot.integers(0, 2000, 2000) # a bootstrap sample
t = DT(max_features=mf, random_state=int(boot.integers(10**6))).fit(X_tr[i], y_tr[i])
roots[names[t.tree_.feature[0]]] += 1; P.append(t.predict(X_te)) # which feature is at the root?
P = np.array(P); corr = np.corrcoef(P != y_te)[np.triu_indices(k, 1)].mean() # same mistakes?
print(f"mf={mf!s:<5} 1 tree {acc(P[0]):.3f} forest {acc(P.mean(0) > .5):.3f} corr {corr:.2f}")
print(" roots:", dict(roots.most_common(3)))
for mf in [None, "sqrt", 1]: bag(100, mf)
Output:
mf=None 1 tree 0.811 forest 0.868 corr 0.38
roots: {'missed_meals': 100}
mf=sqrt 1 tree 0.781 forest 0.881 corr 0.25
roots: {'missed_meals': 34, 'rating': 20, 'distance_km': 15}
mf=1 1 tree 0.652 forest 0.877 corr 0.11
roots: {'calls': 16, 'missed_meals': 14, 'distance_km': 13}
- With
"sqrt", roots spread across many features. A single tree gets worse (0.811 → 0.781) because it often can't use the best feature, but the error correlation drops (0.38 → 0.25) and the forest improves (0.868 → 0.881). - With
max_features=1, trees are too weak (0.652); correlation is tiny (0.11) but the forest (0.877) is no better than"sqrt". There is a sweet spot between "all features" and "one feature".

Random forest = bagging + feature subsampling:
- Draw a bootstrap sample of rows for each tree.
- At every split, pick
max_featuresfeatures at random and use the best split among them. - Grow deep trees; combine by majority vote (classification) or average (regression).
Feature subsampling produces decorrelated trees (lower ρ), so averaging cancels more of their mistakes.
RandomForestClassifier, n_estimators and OOB
import numpy as np
rng = np.random.default_rng(34); n = 3000 # Sunita's tiffin service: 3,000 made-up customers
names = "missed_meals rating distance_km months price_rs spice calls age family veg_days".split()
P, U, I = rng.poisson, rng.uniform, rng.integers # random counts, decimals, whole numbers
X = np.c_[P(3,n), U(1,5,n), U(0,12,n), I(1,36,n), U(60,160,n), I(1,6,n), P(2,n), I(18,65,n), I(1,7,n), I(0,8,n)]
s = X @ [-1.2,.8,-.12,.04,-.01,0,-.25,0,0,.15] + .5*np.sin(X[:,5]) + .3*(X[:,8]>3) + .8*(X[:,1]>3.5)*(X[:,2]<4)
y = (s + rng.normal(0, 0.8, n) > np.median(s)).astype(int) # hidden made-up rule · 1 = renews next month
X_tr, y_tr, X_te, y_te = X[:2000], y[:2000], X[2000:], y[2000:]
from sklearn.ensemble import RandomForestClassifier
for k in [1, 10, 50, 100, 300]:
f = RandomForestClassifier(n_estimators=k, oob_score=k >= 50, random_state=0).fit(X_tr, y_tr)
oob = f" OOB {f.oob_score_:.3f}" if k >= 50 else ""
print(f"{k:>4} trees: test {f.score(X_te, y_te):.3f}{oob}")
print("default max_features:", f.max_features, " features per split:", int(np.sqrt(10)))
Output:
1 trees: test 0.752
10 trees: test 0.870
50 trees: test 0.886 OOB 0.875
100 trees: test 0.888 OOB 0.880
300 trees: test 0.886 OOB 0.886
default max_features: sqrt features per split: 3
n_estimatorsis the number of trees. The score climbs quickly (1 tree 0.752, 10 trees 0.870) and then stays flat; the small ups and downs (0.886, 0.888, 0.886) are noise from a 1,000-customer test set.- The OOB score tracks the test score without using the test set.
- The default
max_featuresfor the classifier is"sqrt".

- With 1-2 trees, bagging is better (each tree can use the best feature); from about 10 trees on, the forest pulls ahead.
- More trees never make a forest overfit more; they just stop helping and cost time. Choose "enough" (a few hundred is common). The settings that actually change the result are
max_featuresand how deep trees grow (max_depth,min_samples_leaf).
Main RandomForestClassifier parameters
| Parameter | Meaning | Default (scikit-learn 1.9.1) |
|---|---|---|
n_estimators |
number of trees | 100 |
max_features |
features tried per split | "sqrt" (regressor: 1.0) |
max_depth, min_samples_leaf |
limit tree growth | None, 1 (grow fully) |
bootstrap |
bootstrap samples of rows | True |
oob_score |
compute the OOB score after fitting | False |
max_samples |
rows per bootstrap sample | None (= n) |
class_weight |
weights for imbalanced classes (Episode 41) | None |
n_jobs |
CPU cores (−1 = all) | None |
random_state |
seed for repeatable results | None |
The defaults above were printed with:
from sklearn.ensemble import RandomForestClassifier, RandomForestRegressor
for M in [RandomForestClassifier, RandomForestRegressor]:
p = M().get_params()
print(M.__name__, {k: p[k] for k in ["n_estimators", "max_features", "max_depth", "min_samples_leaf", "bootstrap", "max_samples"]})
Output:
RandomForestClassifier {'n_estimators': 100, 'max_features': 'sqrt', 'max_depth': None, 'min_samples_leaf': 1, 'bootstrap': True, 'max_samples': None}
RandomForestRegressor {'n_estimators': 100, 'max_features': 1.0, 'max_depth': None, 'min_samples_leaf': 1, 'bootstrap': True, 'max_samples': None}
Random forest regression
Sunita's kitchen: 1,200 made-up days with temperature (°C), day of week (0-6) and rain (mm); target = tiffins ordered. Orders rise with heat then fall on very hot days, jump at weekends, rise with rain, plus noise with standard deviation 15.
import numpy as np
from sklearn.tree import DecisionTreeRegressor
from sklearn.ensemble import RandomForestRegressor
rng = np.random.default_rng(5); n = 1200 # Sunita's kitchen: 1,200 made-up days
X = np.c_[rng.uniform(15, 45, n), rng.integers(0, 7, n), rng.uniform(0, 60, n)] # temp, day, rain
y = 200 + 3*X[:, 0] - 0.24*(X[:, 0] - 30)**2 + 25*(X[:, 1] >= 5) + 0.8*X[:, 2] + rng.normal(0, 15, n)
X_tr, y_tr, X_te, y_te = X[:900], y[:900], X[900:], y[900:] # y = tiffins ordered that day
rmse = lambda m: np.sqrt(np.mean((m.fit(X_tr, y_tr).predict(X_te) - y_te) ** 2))
print(f"one full tree RMSE {rmse(DecisionTreeRegressor(random_state=0)):.1f}")
print(f"tree, depth 5 RMSE {rmse(DecisionTreeRegressor(max_depth=5, random_state=0)):.1f}")
f = RandomForestRegressor(n_estimators=300, n_jobs=1, random_state=0)
print(f"forest, 300 trees RMSE {rmse(f):.1f} default max_features {f.max_features}")
print("38 C, Saturday, 20 mm rain ->", f.predict([[38, 5, 20]]).round(), "tiffins")
Output:
one full tree RMSE 21.2
tree, depth 5 RMSE 16.9
forest, 300 trees RMSE 15.8 default max_features 1.0
38 C, Saturday, 20 mm rain -> [328.] tiffins
- One full tree memorises noise (RMSE 21.2); a depth-5 tree is better (16.9); the forest is best (15.8), within about one tiffin of the noise floor (≈15).
- Each regression tree's leaf predicts the average of its training rows; the forest averages the trees' numbers.
- For
RandomForestRegressorthe defaultmax_featuresis1.0(all features), so by default it is plain bagging of regression trees. With many features, try smaller values such as0.33or"sqrt".
Which feature matters? Impurity vs permutation importance
Impurity-based importance (feature_importances_): every time a tree splits on a feature, impurity (Gini for classification, variance for regression) drops; add up those drops per feature over all trees and normalise so they sum to 1. Fast and free, but:
- it is measured on the training data, and
- it favours features with many unique values (high cardinality): a deep tree can split on them again and again to separate a few training rows, even when they are pure noise.
Permutation importance: score the fitted model on held-out data; shuffle one column (breaking its link with the label, keeping its values); score again; the drop is that feature's importance. Repeat the shuffle several times (n_repeats) and report mean ± standard deviation. Real-life picture: swap one spice in the dal for a random jar and taste how much worse it got.
We add a test column that cannot matter: a unique random customer ID.
import numpy as np
rng = np.random.default_rng(34); n = 3000 # Sunita's tiffin service: 3,000 made-up customers
names = "missed_meals rating distance_km months price_rs spice calls age family veg_days".split()
P, U, I = rng.poisson, rng.uniform, rng.integers # random counts, decimals, whole numbers
X = np.c_[P(3,n), U(1,5,n), U(0,12,n), I(1,36,n), U(60,160,n), I(1,6,n), P(2,n), I(18,65,n), I(1,7,n), I(0,8,n)]
s = X @ [-1.2,.8,-.12,.04,-.01,0,-.25,0,0,.15] + .5*np.sin(X[:,5]) + .3*(X[:,8]>3) + .8*(X[:,1]>3.5)*(X[:,2]<4)
y = (s + rng.normal(0, 0.8, n) > np.median(s)).astype(int) # hidden made-up rule · 1 = renews next month
X_tr, y_tr, X_te, y_te = X[:2000], y[:2000], X[2000:], y[2000:]
from sklearn.ensemble import RandomForestClassifier
from sklearn.inspection import permutation_importance
X2 = np.c_[X, np.random.default_rng(7).permutation(n)]; cols = names + ["customer_id"] # unique, useless ID
f = RandomForestClassifier(100, n_jobs=1, random_state=0).fit(X2[:2000], y_tr)
imp = f.feature_importances_ # impurity-based importance
p = permutation_importance(f, X2[2000:], y_te, n_repeats=5, random_state=0) # shuffle each column 5 times
print(f"{'feature':<13} impurity permutation (test)")
for k in np.argsort(-imp):
print(f"{cols[k]:<13} {imp[k]:.3f} {p.importances_mean[k]:+.3f} ± {p.importances_std[k]:.3f}")
Output:
feature impurity permutation (test)
missed_meals 0.388 +0.295 ± 0.011
rating 0.175 +0.109 ± 0.005
distance_km 0.089 +0.035 ± 0.006
price_rs 0.065 +0.025 ± 0.006
months 0.059 +0.020 ± 0.005
customer_id 0.053 -0.001 ± 0.003
age 0.046 +0.000 ± 0.002
calls 0.039 +0.009 ± 0.001
veg_days 0.035 +0.008 ± 0.002
family 0.026 +0.002 ± 0.003
spice 0.026 +0.003 ± 0.004
- Impurity importance ranks the useless
customer_id6th (0.053), abovecalls,veg_days,spiceandfamily, which are really in the rule; uselessagegets 0.046. - Permutation importance on the test set gives
customer_id−0.001 ± 0.003 andage+0.000: zero within the noise.callsandveg_daysnow rank above them.

| Impurity importance | Permutation importance | |
|---|---|---|
| Data used | training data | any held-out data (validation/test) |
| Cost | free after fitting | one re-score per feature per repeat |
| Models | tree-based only | any fitted model |
| Bias | favours many-valued features | none from cardinality |
| Weakness | can rank noise highly | correlated twins share credit; both look unimportant |
Rules of thumb: use impurity importance for a quick look; use permutation importance on held-out data when a decision depends on it; check correlated features first. Importance means the model uses a feature, not that the feature causes the outcome.
Extra trees (extremely randomised trees)
- Random forest: random features at each split, then the best threshold for each.
- Extra trees: random features and a random threshold for each candidate feature (one random cut between its minimum and maximum), then the best of those random cuts.
- No threshold search → usually faster training; trees differ even more. In scikit-learn,
ExtraTreesClassifier/ExtraTreesRegressorusebootstrap=Falseby default: every tree sees all rows.
import numpy as np
rng = np.random.default_rng(34); n = 3000 # Sunita's tiffin service: 3,000 made-up customers
names = "missed_meals rating distance_km months price_rs spice calls age family veg_days".split()
P, U, I = rng.poisson, rng.uniform, rng.integers # random counts, decimals, whole numbers
X = np.c_[P(3,n), U(1,5,n), U(0,12,n), I(1,36,n), U(60,160,n), I(1,6,n), P(2,n), I(18,65,n), I(1,7,n), I(0,8,n)]
s = X @ [-1.2,.8,-.12,.04,-.01,0,-.25,0,0,.15] + .5*np.sin(X[:,5]) + .3*(X[:,8]>3) + .8*(X[:,1]>3.5)*(X[:,2]<4)
y = (s + rng.normal(0, 0.8, n) > np.median(s)).astype(int) # hidden made-up rule · 1 = renews next month
X_tr, y_tr, X_te, y_te = X[:2000], y[:2000], X[2000:], y[2000:]
import time
from sklearn.ensemble import RandomForestClassifier, ExtraTreesClassifier
for name, M in [("random forest", RandomForestClassifier), ("extra trees", ExtraTreesClassifier)]:
t = time.perf_counter(); m = M(n_estimators=200, n_jobs=1, random_state=0).fit(X_tr, y_tr)
secs, depth = time.perf_counter() - t, np.mean([e.get_depth() for e in m.estimators_])
print(f"{name:<13} test {m.score(X_te, y_te):.3f} fit {secs:.2f} s depth {depth:.1f}")
Output:
random forest test 0.886 fit 0.47 s depth 14.8
extra trees test 0.883 fit 0.37 s depth 21.0
- Accuracy is about the same. Fit times change from machine to machine and run to run (they will differ from the video); skipping the threshold search usually saves time, more so on bigger data. Extra trees' trees are deeper (21.0 vs 14.8 levels) because random cuts separate customers less efficiently. Try both; sometimes one wins, sometimes the other.
Common mistakes
import numpy as np
rng = np.random.default_rng(34); n = 3000 # Sunita's tiffin service: 3,000 made-up customers
names = "missed_meals rating distance_km months price_rs spice calls age family veg_days".split()
P, U, I = rng.poisson, rng.uniform, rng.integers # random counts, decimals, whole numbers
X = np.c_[P(3,n), U(1,5,n), U(0,12,n), I(1,36,n), U(60,160,n), I(1,6,n), P(2,n), I(18,65,n), I(1,7,n), I(0,8,n)]
s = X @ [-1.2,.8,-.12,.04,-.01,0,-.25,0,0,.15] + .5*np.sin(X[:,5]) + .3*(X[:,8]>3) + .8*(X[:,1]>3.5)*(X[:,2]<4)
y = (s + rng.normal(0, 0.8, n) > np.median(s)).astype(int) # hidden made-up rule · 1 = renews next month
X_tr, y_tr, X_te, y_te = X[:2000], y[:2000], X[2000:], y[2000:]
from sklearn.ensemble import RandomForestClassifier
f = RandomForestClassifier(n_estimators=300)
print(f.feature_importances_) # forgot to call fit
Output:
sklearn.exceptions.NotFittedError: This RandomForestClassifier instance is not fitted yet. Call 'fit' with appropriate arguments before using this estimator.
| Mistake | What you see | Why | Fix |
|---|---|---|---|
| Bagging when one feature dominates | 100 of 100 roots the same, error correlation 0.38 | Trees are correlated | max_features="sqrt" (a random forest) |
max_features=1 |
single trees 0.652 | Trees too weak | Tune max_features (try sqrt, 0.3-0.5) |
Tuning n_estimators like a delicate knob |
0.886 → 0.888 → 0.886 | Flat after enough trees; differences are noise | Enough trees, then tune other settings |
| Trusting impurity importance | customer_id ranked 6th |
Training data; many unique values | Permutation importance on held-out data |
| Reading importance as cause | "missed meals cause leaving" | Importance = what the model uses | Experiments or causal methods |
Importances before fit |
NotFittedError: This RandomForestClassifier instance is not fitted yet. ... |
No trees exist yet | Call fit first |
Expecting RandomForestRegressor to subsample features |
default max_features=1.0 |
Regressor default = bagging | Set max_features yourself |
Mini project: a random forest on real data
The breast cancer dataset bundled with scikit-learn (569 tumour samples, 30 numeric measurements, label = malignant or benign):
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.inspection import permutation_importance
data = load_breast_cancer()
X_tr, X_te, y_tr, y_te = train_test_split(data.data, data.target, test_size=0.25, random_state=0, stratify=data.target)
f = RandomForestClassifier(n_estimators=500, oob_score=True, n_jobs=-1, random_state=0).fit(X_tr, y_tr)
print(f"OOB {f.oob_score_:.3f} test {f.score(X_te, y_te):.3f}")
imp = np.argsort(-f.feature_importances_)[:5]
print("top 5 by impurity: ", [str(data.feature_names[i]) for i in imp])
p = permutation_importance(f, X_te, y_te, n_repeats=20, random_state=0)
top = np.argsort(-p.importances_mean)[:5]
print("top 5 by permutation:", [str(data.feature_names[i]) for i in top])
print("largest permutation drop:", p.importances_mean.max().round(3))
Output:
OOB 0.965 test 0.944
top 5 by impurity: ['worst perimeter', 'worst concave points', 'worst radius', 'worst area', 'mean concave points']
top 5 by permutation: ['worst area', 'worst perimeter', 'worst concave points', 'worst radius', 'worst texture']
largest permutation drop: 0.007
- OOB (0.965) and test (0.944) are close; with only 143 test samples, a single test number wobbles by a couple of points.
- Both rankings put the "worst" (largest) size and shape measurements on top, but the largest permutation drop is under one point: shuffling any single column barely hurts. Many of the 30 measurements are near-duplicates (radius, perimeter and area of the same tumour are strongly correlated), so when one is shuffled the forest reads its twins. That is the "correlated twins" weakness of permutation importance; group correlated features (or shuffle them together) before interpreting either ranking.
Practice
1. Plot (or print) test accuracy for max_features = 1 … 10 on Sunita's data.
Solution
import numpy as np
rng = np.random.default_rng(34); n = 3000 # Sunita's tiffin service: 3,000 made-up customers
names = "missed_meals rating distance_km months price_rs spice calls age family veg_days".split()
P, U, I = rng.poisson, rng.uniform, rng.integers # random counts, decimals, whole numbers
X = np.c_[P(3,n), U(1,5,n), U(0,12,n), I(1,36,n), U(60,160,n), I(1,6,n), P(2,n), I(18,65,n), I(1,7,n), I(0,8,n)]
s = X @ [-1.2,.8,-.12,.04,-.01,0,-.25,0,0,.15] + .5*np.sin(X[:,5]) + .3*(X[:,8]>3) + .8*(X[:,1]>3.5)*(X[:,2]<4)
y = (s + rng.normal(0, 0.8, n) > np.median(s)).astype(int) # hidden made-up rule · 1 = renews next month
X_tr, y_tr, X_te, y_te = X[:2000], y[:2000], X[2000:], y[2000:]
from sklearn.ensemble import RandomForestClassifier
for mf in range(1, 11):
f = RandomForestClassifier(n_estimators=300, max_features=mf, n_jobs=-1, random_state=0).fit(X_tr, y_tr)
print(f"max_features={mf:>2}: test {f.score(X_te, y_te):.3f}")
Output:
max_features= 1: test 0.878
max_features= 2: test 0.886
max_features= 3: test 0.886
max_features= 4: test 0.878
max_features= 5: test 0.872
max_features= 6: test 0.877
max_features= 7: test 0.872
max_features= 8: test 0.869
max_features= 9: test 0.867
max_features=10: test 0.860
The best values are in the low-to-middle range; max_features=10 is plain bagging. Differences of a few thousandths are noise on 1,000 test customers; use cross-validation (Episode 26) to choose.
2. Add a copy of rating (rating out of 100) and compare impurity and permutation importance.
Solution
import numpy as np
rng = np.random.default_rng(34); n = 3000 # Sunita's tiffin service: 3,000 made-up customers
names = "missed_meals rating distance_km months price_rs spice calls age family veg_days".split()
P, U, I = rng.poisson, rng.uniform, rng.integers # random counts, decimals, whole numbers
X = np.c_[P(3,n), U(1,5,n), U(0,12,n), I(1,36,n), U(60,160,n), I(1,6,n), P(2,n), I(18,65,n), I(1,7,n), I(0,8,n)]
s = X @ [-1.2,.8,-.12,.04,-.01,0,-.25,0,0,.15] + .5*np.sin(X[:,5]) + .3*(X[:,8]>3) + .8*(X[:,1]>3.5)*(X[:,2]<4)
y = (s + rng.normal(0, 0.8, n) > np.median(s)).astype(int) # hidden made-up rule · 1 = renews next month
X_tr, y_tr, X_te, y_te = X[:2000], y[:2000], X[2000:], y[2000:]
from sklearn.ensemble import RandomForestClassifier
from sklearn.inspection import permutation_importance
X2 = np.c_[X, X[:, 1] * 20]; cols = names + ["rating_x20"] # a perfect twin of rating
f = RandomForestClassifier(300, n_jobs=-1, random_state=0).fit(X2[:2000], y_tr)
p = permutation_importance(f, X2[2000:], y_te, n_repeats=10, random_state=0)
for k in [1, 10]:
print(f"{cols[k]:<11} impurity {f.feature_importances_[k]:.3f} permutation {p.importances_mean[k]:+.3f}")
Output:
rating impurity 0.110 permutation +0.034
rating_x20 impurity 0.114 permutation +0.030
Without the twin, rating had impurity 0.174 and permutation +0.113. Now the two copies split the impurity credit, and shuffling either one hurts much less, because the forest can still read the other copy. Neither method shows that "rating" as a fact matters a lot; check for correlated columns before you interpret importances.
3. RandomForestRegressor with max_features=0.33 on the kitchen data. Better than the default?
Solution
import numpy as np
from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import cross_val_score
rng = np.random.default_rng(5); n = 1200
X = np.c_[rng.uniform(15, 45, n), rng.integers(0, 7, n), rng.uniform(0, 60, n)]
y = 200 + 3*X[:, 0] - 0.24*(X[:, 0] - 30)**2 + 25*(X[:, 1] >= 5) + 0.8*X[:, 2] + rng.normal(0, 15, n)
for mf in [1.0, 0.33]:
m = RandomForestRegressor(n_estimators=300, max_features=mf, n_jobs=-1, random_state=0)
s = -cross_val_score(m, X, y, cv=5, scoring="neg_root_mean_squared_error")
print(f"max_features={mf}: CV RMSE {s.mean():.1f}")
Output:
max_features=1.0: CV RMSE 16.2
max_features=0.33: CV RMSE 16.4
With only three features there is little to decorrelate, so the default (all features) is about as good or better. Feature subsampling pays off when there are many features and a few dominate.
4. Random forest vs extra trees with 5-fold cross-validation on Sunita's data.
Solution
import numpy as np
rng = np.random.default_rng(34); n = 3000 # Sunita's tiffin service: 3,000 made-up customers
names = "missed_meals rating distance_km months price_rs spice calls age family veg_days".split()
P, U, I = rng.poisson, rng.uniform, rng.integers # random counts, decimals, whole numbers
X = np.c_[P(3,n), U(1,5,n), U(0,12,n), I(1,36,n), U(60,160,n), I(1,6,n), P(2,n), I(18,65,n), I(1,7,n), I(0,8,n)]
s = X @ [-1.2,.8,-.12,.04,-.01,0,-.25,0,0,.15] + .5*np.sin(X[:,5]) + .3*(X[:,8]>3) + .8*(X[:,1]>3.5)*(X[:,2]<4)
y = (s + rng.normal(0, 0.8, n) > np.median(s)).astype(int) # hidden made-up rule · 1 = renews next month
X_tr, y_tr, X_te, y_te = X[:2000], y[:2000], X[2000:], y[2000:]
from sklearn.ensemble import RandomForestClassifier, ExtraTreesClassifier
from sklearn.model_selection import cross_val_score
for M in [RandomForestClassifier, ExtraTreesClassifier]:
s = cross_val_score(M(n_estimators=300, n_jobs=-1, random_state=0), X, y, cv=5)
print(f"{M.__name__:<23} CV accuracy {s.mean():.3f} ± {s.std():.3f}")
Output:
RandomForestClassifier CV accuracy 0.886 ± 0.013
ExtraTreesClassifier CV accuracy 0.892 ± 0.013
The two are within noise of each other here. Extra trees are often a good, fast alternative; the only way to know on your data is to compare with cross-validation.
5. By hand: 100 trees each with variance σ² = 1. What is the variance of their average if ρ = 0? If ρ = 0.4? What is the limit as B → ∞ when ρ = 0.4?
Solution
ρ = 0: 0 + 1/100 = 0.01. ρ = 0.4: 0.4 + 0.6/100 = 0.406. Limit: 0.4: correlation, not the number of trees, sets the floor.
Quick quiz
- 100 bagged trees all have the same root. Which change lowers their correlation itself: more trees,
max_features="sqrt", deeper trees, orbootstrap=False?max_features="sqrt". - In a random forest, are random features chosen once per tree or at every split? At every split.
- Does adding more trees make a random forest overfit? No; the score levels off.
- Default
max_featuresforRandomForestClassifier? ForRandomForestRegressor?"sqrt";1.0(all features). - Why can impurity importance rank a random ID column highly? It is computed on training data and favours many-valued columns that trees can split on repeatedly.
- What do extra trees randomise besides the features? The split threshold.
Interview questions
- How does a random forest differ from bagged decision trees? It also picks a random subset of features at every split (
max_features), which decorrelates the trees. - Why does decorrelation help? Var(average) = ρσ² + (1 − ρ)σ²/B; lowering ρ lowers the floor that more trees can't remove.
- Which hyperparameters matter most in a random forest?
max_featuresand tree size (max_depth,min_samples_leaf);n_estimatorsjust needs to be large enough. - Can a random forest overfit? Individual deep trees overfit, but adding trees doesn't increase overfitting; very deep trees on noisy data can still give an optimistic model, so tune leaf size with validation.
- Impurity vs permutation importance? Impurity: from training splits, fast, biased towards high-cardinality features. Permutation: drop in held-out score when a column is shuffled, model-agnostic, but correlated features share credit.
- What is the OOB score in a forest? Accuracy (or R² for regressors) of each training row predicted only by trees that didn't see it.
- Random forest vs extra trees? Extra trees also randomise thresholds and by default don't bootstrap; usually faster, similar accuracy, slightly more bias and less variance.
- Do random forests need feature scaling? No; tree splits depend only on the order of values.
- Random forest regression: how is the prediction made? The average of the trees' predictions (each leaf predicts the mean of its training rows), so it can't predict outside the range of training targets.
- When would you not use a random forest? When you need a very small or fast model, smooth extrapolation, or the best accuracy on large tabular data, where gradient boosting (Episodes 35-36) usually wins.
Glossary
| Term | Meaning (plain words) | Example from this episode |
|---|---|---|
| Random forest | Bagged decision trees that also pick random features at every split | RandomForestClassifier 0.886 |
| Root (of a tree) | The first question a tree asks | missed_meals in all 100 bagged trees |
| Greedy split | Choosing the best split right now, without looking ahead | always the strongest feature first |
| Error correlation | How often trees are wrong on the same rows | 0.38 (bagging) → 0.25 (forest) |
| ρ (rho) | Correlation between two models' predictions or errors | in ρσ² + (1 − ρ)σ²/B |
| σ² (sigma squared) | Variance of one model's prediction: how much it wobbles | one tree's wobble |
| Feature subsampling | Considering only a random subset of features at a split | 3 of 10 features |
max_features |
scikit-learn's setting for feature subsampling | "sqrt", 1.0, 0.33, 1 |
| Decorrelated trees | Trees whose errors are less alike | roots spread over many features |
n_estimators |
The number of trees in the ensemble | 1 → 300 |
| OOB score | Score from trees that never saw each row (Episode 33) | 0.886 at 300 trees |
| Random forest regression | A forest of regression trees whose numbers are averaged | RandomForestRegressor, RMSE 15.8 |
| Noise floor | The error no model can beat because of random noise | ≈ 15 tiffins |
| Feature importance | A score for how much a model relies on a feature | feature_importances_ |
| Impurity-based importance | Total impurity drop from splits on a feature, from training data | customer_id 0.053 |
| High cardinality | A column with many distinct values | a unique customer ID |
| Permutation importance | Drop in held-out score when one column is shuffled | missed_meals +0.295 |
n_repeats |
How many times each column is shuffled | 5 |
| Correlated twins | Two features carrying the same information | rating and rating × 20 |
| Extra trees (extremely randomised trees) | Trees with random features and random thresholds | ExtraTreesClassifier 0.883 |
NotFittedError |
Error when using a model before fit |
feature_importances_ before fit |
Summary
- Bagged trees all start with the strongest feature, so their errors are correlated, and correlated errors set a floor: ρσ² + (1 − ρ)σ²/B.
- Random forest = bootstrap rows + random features at every split (
max_features). Each tree is a bit worse; the decorrelated forest is better. n_estimators: enough, then flat; forests don't overfit by adding trees. Tunemax_featuresand tree size instead. The OOB score checks it for free.- Random forest regression averages regression trees; the regressor's default uses all features.
- Impurity importance is fast but biased towards many-valued columns; permutation importance on held-out data is the honest check.
- Extra trees randomise thresholds too: similar accuracy, often faster.
- Next episode: boosting from scratch, where each new tree learns from the previous trees' mistakes.