SyntaxPilot
Courses/Machine Learning/Lesson 30
Video coming soonThis lesson’s video is being made

Start with the notes below; the video appears here as soon as it is on YouTube.

Machine Learning · Lesson 30 of 90

Decision trees in practice

DecisionTreeClassifier and Regressormax_depthmin_samples_splitmin_samples_leafpre-pruning vs post-pruningcost-complexity pruning (ccp_alpha)regression trees (variance reduction)impurity-based feature importancetree visualisationinstability of trees

Machine Learning Full Course · Episode 30. Every code sample below was run with Python 3.13, NumPy 2.5.3 and scikit-learn 1.9.1; the output shown is the real output. The kirana customers and chai sales are made up for teaching; the mini project uses the diabetes dataset bundled with scikit-learn.

The big idea: an unlimited tree memorises noise. Stop it early with pre-pruning (max_depth, min_samples_split, min_samples_leaf) or grow it fully and cut it back with post-pruning (cost-complexity pruning, ccp_alpha), choosing the strength by cross-validation. A pruned tree can be drawn and read. Its feature importances are easy to get but biased towards features with many unique values. Trees are unstable (high variance), which ensembles fix later. Regression trees split by variance reduction and predict the mean of a leaf, so their predictions are steps that cannot extrapolate.

Words you need first

  • Decision tree, root, node, leaf, split, depth, Gini impurity, CART: Episode 29.
  • Overfitting / underfitting; variance: memorising noise / missing the pattern; how much a model changes with different training data (Episode 20).
  • Cross-validation (CV): scoring on several train/validation splits and averaging (Episode 26).
  • Hyperparameter: a setting we choose, not learned (Episode 8).
  • Regression: predicting a number (Episode 1). Variance of numbers: average squared distance from their mean (Episode 10). MSE / MAE: mean squared / mean absolute error (Episodes 18-19).

The problem: the tree that never stops asking

Same 2,000 made-up customers as Episode 29 (income in thousand rupees, years Raju has known them, repaid or not). Unlimited, the tree grows 284 leaves, some holding a single customer: a memory, not a pattern.

Pre-pruning

  • Pruning: reducing a tree's size to stop overfitting.
  • Pre-pruning (early stopping): stop growing before the tree is complete.
  • max_depth: maximum number of questions on any root-to-leaf path.
  • min_samples_split: a node may split only if it holds at least this many training rows.
  • min_samples_leaf: every leaf must keep at least this many rows (a split that would create a smaller leaf is not allowed).
  • Others: max_leaf_nodes (cap on leaves), min_impurity_decrease (a split must reduce weighted impurity by at least this much).
import numpy as np
from sklearn.tree import DecisionTreeClassifier as DT
rng = np.random.default_rng(29); n = 2000                  # the same made-up kirana customers
X = np.c_[rng.integers(10, 81, n), rng.integers(0, 11, n)]  # income (thousand Rs), years known
ok = ((X[:, 0] >= 30) & (X[:, 1] >= 2)) | (X[:, 0] >= 60)
y = np.where(rng.random(n) < 0.1, ~ok, ok).astype(int)      # 1 = repaid (10% noise)
X_tr, y_tr, X_te, y_te = X[:1500], y[:1500], X[1500:], y[1500:]
def report(name, t):                                       # pre-pruning: stop the tree growing early
    tr, te = t.score(X_tr, y_tr), t.score(X_te, y_te)
    print(f"{name:<22} leaves {t.get_n_leaves():>3}  train {tr:.3f}  test {te:.3f}")
report("no limits", DT(random_state=0).fit(X_tr, y_tr))
report("max_depth=3", DT(max_depth=3, random_state=0).fit(X_tr, y_tr))
report("min_samples_split=100", DT(min_samples_split=100, random_state=0).fit(X_tr, y_tr))
report("min_samples_leaf=50", DT(min_samples_leaf=50, random_state=0).fit(X_tr, y_tr))

Output:

no limits              leaves 284  train 0.917  test 0.818
max_depth=3            leaves   8  train 0.903  test 0.880
min_samples_split=100  leaves  30  train 0.903  test 0.880
min_samples_leaf=50    leaves  23  train 0.903  test 0.880

All three brakes lift test accuracy from 0.818 to 0.880 with far fewer leaves. min_samples_leaf is often the most useful: a leaf's vote is only trustworthy with enough rows. Pre-pruning's risk: a split that looks useless may enable a great split below it (e.g. XOR-like patterns), and early stopping can't know that.

Post-pruning: cost-complexity pruning

Real-life example: a mali (gardener) lets the hedge grow, then trims weak branches. To decide which, think of Raju's shelves: every product pays shelf rent and stays only if it earns more than the rent.

  • Post-pruning: grow the full tree, then remove branches that don't pay for themselves.
  • Cost-complexity pruning (CCP): minimise R_α(T) = R(T) + α × |leaves(T)| where R(T) is the tree's total training impurity (weighted by samples) and α is the "rent" per leaf.
  • α = 0 keeps the full tree; growing α removes the weakest link first (the branch whose impurity reduction per extra leaf is smallest); a huge α leaves a single leaf (the majority class).
  • cost_complexity_pruning_path(X, y) returns every α at which a branch is cut (ccp_alphas) and the matching impurities.
  • ccp_alpha is a hyperparameter: pick it with cross-validation.
import numpy as np
from sklearn.tree import DecisionTreeClassifier as DT
rng = np.random.default_rng(29); n = 2000                  # the same made-up kirana customers
X = np.c_[rng.integers(10, 81, n), rng.integers(0, 11, n)]  # income (thousand Rs), years known
ok = ((X[:, 0] >= 30) & (X[:, 1] >= 2)) | (X[:, 0] >= 60)
y = np.where(rng.random(n) < 0.1, ~ok, ok).astype(int)      # 1 = repaid (10% noise)
X_tr, y_tr, X_te, y_te = X[:1500], y[:1500], X[1500:], y[1500:]
from sklearn.model_selection import cross_val_score
path = DT(random_state=0).cost_complexity_pruning_path(X_tr, y_tr)   # grow fully, then prune
alphas = path.ccp_alphas
print(len(alphas), "pruning levels, alpha from", alphas[0], "to", alphas[-1].round(3))
for a in [0.0, 0.0005, 0.001, 0.002, 0.005, 0.05, 0.15]:
    t = DT(ccp_alpha=a, random_state=0)
    cv = cross_val_score(t, X_tr, y_tr, cv=5).mean(); t.fit(X_tr, y_tr)
    print(f"ccp_alpha {a:<6}  leaves {t.get_n_leaves():>3}  CV {cv:.3f}  test {t.score(X_te, y_te):.3f}")

Output:

119 pruning levels, alpha from 0.0 to 0.206
ccp_alpha 0.0     leaves 284  CV 0.827  test 0.818
ccp_alpha 0.0005  leaves  25  CV 0.878  test 0.876
ccp_alpha 0.001   leaves   8  CV 0.894  test 0.882
ccp_alpha 0.002   leaves   4  CV 0.901  test 0.880
ccp_alpha 0.005   leaves   4  CV 0.902  test 0.880
ccp_alpha 0.05    leaves   3  CV 0.855  test 0.856
ccp_alpha 0.15    leaves   2  CV 0.837  test 0.822
  • 119 pruning levels between α = 0 and 0.206.
  • α = 0.0005 already cuts 284 leaves to 25; α ≈ 0.002 gives 4 leaves, CV 0.901, test 0.880.
  • α = 0.15 leaves just 2 leaves (one question): underfitting.

Pre- vs post-pruning

Pre-pruning Post-pruning (CCP)
When while growing after growing fully
Knobs max_depth, min_samples_split, min_samples_leaf, max_leaf_nodes ccp_alpha
Speed fast (never grows the big tree) slower (grows it, then prunes)
Risk may stop before a useful deeper split none of that; needs CV for α

Reading a tree: visualisation

from sklearn.tree import plot_tree, export_text
t = DT(ccp_alpha=0.002, random_state=0).fit(X_tr, y_tr)
plot_tree(t, feature_names=["income", "years"], class_names=["not repaid", "repaid"], filled=True, rounded=True)
print(export_text(t, feature_names=["income", "years"]))

The pruned tree

Each box shows: the question; gini (impurity of the node); samples (training rows reaching it); value = [not repaid, repaid] counts; class = the majority. Left = True (yes), right = False. Colour: orange = mostly not repaid, blue = mostly repaid, darker = purer.

Read as rules: income ≤ 29.5 → no udhaar; above that, known ≥ 2 years → udhaar; known < 2 years → only if income > 59.5. This is exactly the hidden rule used to make the data, and the useless split from Episode 29 is gone. A small tree is one of the few models you can explain line by line to a customer or a regulator.

Feature importance

  • Impurity-based feature importance (feature_importances_): for each feature, the total weighted impurity decrease from all splits on it, normalised to sum to 1. Also called mean decrease in impurity (MDI).
import numpy as np
from sklearn.tree import DecisionTreeClassifier as DT
rng = np.random.default_rng(29); n = 2000                  # the same made-up kirana customers
X = np.c_[rng.integers(10, 81, n), rng.integers(0, 11, n)]  # income (thousand Rs), years known
ok = ((X[:, 0] >= 30) & (X[:, 1] >= 2)) | (X[:, 0] >= 60)
y = np.where(rng.random(n) < 0.1, ~ok, ok).astype(int)      # 1 = repaid (10% noise)
X_tr, y_tr, X_te, y_te = X[:1500], y[:1500], X[1500:], y[1500:]
phone = rng.integers(0, 10**6, n)                          # last 6 digits of phone: pure noise
X3 = np.c_[X, phone]; names = ["income", "years", "phone"]
models = [("no limits", DT(random_state=0)), ("ccp_alpha=0.002", DT(ccp_alpha=0.002, random_state=0))]
for name, t in models:
    t.fit(X3[:1500], y_tr)
    imp = ", ".join(f"{f} {v:.2f}" for f, v in zip(names, t.feature_importances_))
    print(f"{name:<16} test {t.score(X3[1500:], y_te):.3f}")
    print(f"  importance: {imp}")

Output:

no limits        test 0.790
  importance: income 0.60, years 0.17, phone 0.23
ccp_alpha=0.002  test 0.880
  importance: income 0.82, years 0.18, phone 0.00
  • The phone digits are pure noise, yet the full tree gives them 0.23, more than years known (0.17), and test accuracy drops to 0.790.
  • Why: a feature with many unique values offers many thresholds; deep in the tree, with a handful of rows left, some threshold always happens to separate them. Pure luck, counted as importance. And the importance is computed on training data, so memorised noise gets credit.
  • The pruned tree gives phone 0.00. Prune first; for a reliable check use permutation importance on held-out data (Episode 34).

Why trees are unstable

import numpy as np
from sklearn.tree import DecisionTreeClassifier as DT
rng = np.random.default_rng(29); n = 2000                  # the same made-up kirana customers
X = np.c_[rng.integers(10, 81, n), rng.integers(0, 11, n)]  # income (thousand Rs), years known
ok = ((X[:, 0] >= 30) & (X[:, 1] >= 2)) | (X[:, 0] >= 60)
y = np.where(rng.random(n) < 0.1, ~ok, ok).astype(int)      # 1 = repaid (10% noise)
X_tr, y_tr, X_te, y_te = X[:1500], y[:1500], X[1500:], y[1500:]
from sklearn.tree import export_text
for name, kw in [("no limits", {}), ("max_depth=3", {"max_depth": 3})]:
    preds = []
    for seed in [1, 2]:                                    # two shops, slightly different 80% of data
        i = np.random.default_rng(seed).choice(1500, 1200, replace=False)
        t = DT(random_state=0, **kw).fit(X_tr[i], y_tr[i]); preds.append(t.predict(X_te))
        print(f"{name:<11} sample {seed}: leaves {t.get_n_leaves():>3}  root:",
              export_text(t, feature_names=["income", "years"]).splitlines()[0])
    print(f"{name:<11} the two trees disagree on {np.mean(preds[0] != preds[1]):.1%} of test customers")

Output:

no limits   sample 1: leaves 231  root: |--- income <= 29.50
no limits   sample 2: leaves 247  root: |--- income <= 29.50
no limits   the two trees disagree on 6.6% of test customers
max_depth=3 sample 1: leaves   8  root: |--- income <= 29.50
max_depth=3 sample 2: leaves   8  root: |--- income <= 29.50
max_depth=3 the two trees disagree on 0.0% of test customers
  • Two trees trained on two random 80% samples of the same data: full trees (231 and 247 leaves) disagree on 6.6% of new customers; depth-3 trees agree on all of them.
  • Instability (high variance): a small change in the training data changes the tree, because each split decides what every node below it sees.
  • Fixes: prune or limit depth (less variance, maybe more bias), or average many trees trained on different samples: bagging and random forests (Episodes 33-34).

Regression trees

Real-life example: Raju's chai stall: 300 made-up days, temperature vs cups sold. Hot days sell fewer cups. Gini needs classes; for numbers, a "messy" group is one whose values are spread out.

  • Regression tree: a tree whose leaves predict numbers (DecisionTreeRegressor).
  • Variance reduction: split quality = variance of the parent − size-weighted variance of the children. scikit-learn calls this criterion "squared_error" (variance is the MSE around the mean). "absolute_error" (MAE around the median) and "poisson" also exist.
  • Leaf prediction: the mean of the leaf's training targets.
import numpy as np
from sklearn.tree import DecisionTreeRegressor, export_text
rng = np.random.default_rng(30)                            # 300 made-up days at Raju's chai stall
temp = rng.uniform(10, 45, 300).round(1)                   # temperature, degrees C
cups = (260 - 4 * temp + rng.normal(0, 15, 300)).round()   # cups of chai sold
def var_drop(t):                                           # variance reduction of "temp <= t"
    m = temp <= t
    return cups.var() - (m.mean() * cups[m].var() + (~m).mean() * cups[~m].var())
for t in [15, 27.5, 35]:
    print(f"temp <= {t:<4}: variance {cups.var():.0f} -> drop {var_drop(t):.0f}")
tree = DecisionTreeRegressor(max_depth=2).fit(temp.reshape(-1, 1), cups)
print(export_text(tree, feature_names=["temp"], decimals=1).strip())
temps = [[5], [12], [22], [32], [42], [50]]                # 5 and 50 C: outside the data
print("5, 12, 22, 32, 42, 50 C ->", tree.predict(temps).round(1), "cups")

Output:

temp <= 15  : variance 2003 -> drop 604
temp <= 27.5: variance 2003 -> drop 1310
temp <= 35  : variance 2003 -> drop 1147
|--- temp <= 27.2
|   |--- temp <= 18.9
|   |   |--- value: [204.4]
|   |--- temp >  18.9
|   |   |--- value: [167.4]
|--- temp >  27.2
|   |--- temp <= 35.8
|   |   |--- value: [136.6]
|   |--- temp >  35.8
|   |   |--- value: [93.5]
5, 12, 22, 32, 42, 50 C -> [204.4 204.4 167.4 136.6  93.5  93.5] cups
  • The variance of cups is 2,003. Splitting at 27.5 °C removes 1,310 of it, more than at 15 or 35; the tree picks 27.2 °C.
  • Leaves predict 204.4, 167.4, 136.6 and 93.5 cups: a staircase.
  • 5 °C predicts the same as 12 °C, and 50 °C the same as 42 °C: a tree cannot extrapolate beyond the range of its training data. For a trend that continues, a linear model extrapolates; a tree just repeats the edge leaf.

Regression trees predict in steps

Depth 2: 4 flat steps. No limit: 196 leaves, a jagged line through every point (overfitting). Prune regression trees with the same knobs.

Decision trees at a glance

Strengths readable rules, no scaling, handles mixed features and interactions, fast prediction, captures non-linear patterns
Weaknesses overfits without limits, unstable (high variance), axis-parallel boundaries, no extrapolation, importance biased to many-valued features
Key hyperparameters max_depth, min_samples_leaf, min_samples_split, ccp_alpha, criterion
Classifier / regressor DecisionTreeClassifier (Gini/entropy, majority leaf) / DecisionTreeRegressor (squared error, mean leaf)

Common mistakes

Mistake What you see Why Fix
No limits on the tree 284 leaves, test 0.818 Memorises noise Pre- or post-prune
Guessing ccp_alpha α = 0.15: CV 0.837 Too much rent: underfits Pruning path + CV
Trusting a full tree's importances phone 0.23 > years 0.17 Many values = lucky splits; training data Prune; permutation importance
Betting on one deep tree Two trees disagree on 6.6% High variance Prune, or ensembles
Extrapolating with a tree 50 °C → 93.5, same as 42 °C Leaf = mean of seen rows Linear model for trends
Passing a 1-D array to fit ValueError: Expected 2D array, got 1D array instead: scikit-learn wants (rows, features) temp.reshape(-1, 1)
Negative ccp_alpha InvalidParameterError: The 'ccp_alpha' parameter of DecisionTreeClassifier must be a float in the range [0.0, inf). Got -1 instead. α is a rent ≥ 0 Use values from the pruning path

The 1-D mistake, reproduced:

import numpy as np
from sklearn.tree import DecisionTreeRegressor
temp = np.array([12.0, 22.0, 32.0, 42.0])            # 1-D: shape (4,)
cups = np.array([205, 167, 137, 94])
DecisionTreeRegressor().fit(temp, cups)

Output:

Traceback (most recent call last):
  ... (library frames) ...
ValueError: Expected 2D array, got 1D array instead:
array=[12. 22. 32. 42.].
Reshape your data either using array.reshape(-1, 1) if your data has a single feature or array.reshape(1, -1) if it contains a single sample.

Mini project: a regression tree on real data

The diabetes dataset (bundled with scikit-learn): 10 numeric measurements (age, sex, body mass index, blood pressure and six blood serum measurements) for 442 patients; the target is a quantitative measure of disease progression one year after baseline.

import numpy as np
from sklearn.datasets import load_diabetes
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.tree import DecisionTreeRegressor
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error as mae
data = load_diabetes()                                       # bundled with scikit-learn
X_tr, X_te, y_tr, y_te = train_test_split(data.data, data.target, test_size=0.25, random_state=0)
print(X_tr.shape, "target: disease progression one year after baseline")
full = DecisionTreeRegressor(random_state=0).fit(X_tr, y_tr)
grid = GridSearchCV(DecisionTreeRegressor(random_state=0), {"max_depth": [2, 3, 4, 5, 6],
                    "min_samples_leaf": [5, 10, 20, 40]}, cv=5, scoring="neg_mean_absolute_error").fit(X_tr, y_tr)
lin = LinearRegression().fit(X_tr, y_tr)
print(f"full tree   leaves {full.get_n_leaves():>3}  test MAE {mae(y_te, full.predict(X_te)):.1f}")
print(f"tuned tree  leaves {grid.best_estimator_.get_n_leaves():>3}  test MAE {mae(y_te, grid.predict(X_te)):.1f}  {grid.best_params_}")
print(f"linear regression       test MAE {mae(y_te, lin.predict(X_te)):.1f}")
top = np.argsort(grid.best_estimator_.feature_importances_)[::-1][:3]
print("top features of the tuned tree:", [data.feature_names[i] for i in top])

Output:

(331, 10) target: disease progression one year after baseline
full tree   leaves 318  test MAE 60.4
tuned tree  leaves   8  test MAE 53.5  {'max_depth': 3, 'min_samples_leaf': 10}
linear regression       test MAE 45.1
top features of the tuned tree: ['s5', 'bmi', 'bp']
  • The full tree (318 leaves) has the worst error; tuning max_depth and min_samples_leaf by CV helps (8 leaves).
  • Plain linear regression still wins here: this target changes smoothly with the features, and a single tree can only make steps. Trees shine as parts of ensembles (Episodes 33-36).
  • The tuned tree's top features are s5 (a serum measurement), bmi (body mass index) and bp (average blood pressure).

Practice

1. Grid-search max_depth × min_samples_leaf with 5-fold CV.

Solution
import numpy as np
from sklearn.tree import DecisionTreeClassifier as DT
rng = np.random.default_rng(29); n = 2000                  # the same made-up kirana customers
X = np.c_[rng.integers(10, 81, n), rng.integers(0, 11, n)]  # income (thousand Rs), years known
ok = ((X[:, 0] >= 30) & (X[:, 1] >= 2)) | (X[:, 0] >= 60)
y = np.where(rng.random(n) < 0.1, ~ok, ok).astype(int)      # 1 = repaid (10% noise)
X_tr, y_tr, X_te, y_te = X[:1500], y[:1500], X[1500:], y[1500:]
from sklearn.model_selection import GridSearchCV
grid = {"max_depth": [2, 3, 4, 6, 10, None], "min_samples_leaf": [1, 5, 20, 50, 100]}
search = GridSearchCV(DT(random_state=0), grid, cv=5).fit(X_tr, y_tr)
print("best settings:", search.best_params_, f" CV {search.best_score_:.3f}")
print(f"test accuracy {search.score(X_te, y_te):.3f}  leaves {search.best_estimator_.get_n_leaves()}")

Output:

best settings: {'max_depth': 3, 'min_samples_leaf': 1}  CV 0.902
test accuracy 0.880  leaves 8

2. Plot the CV score against ccp_alpha for every level of the pruning path.

Solution
import numpy as np
from sklearn.tree import DecisionTreeClassifier as DT
rng = np.random.default_rng(29); n = 2000                  # the same made-up kirana customers
X = np.c_[rng.integers(10, 81, n), rng.integers(0, 11, n)]  # income (thousand Rs), years known
ok = ((X[:, 0] >= 30) & (X[:, 1] >= 2)) | (X[:, 0] >= 60)
y = np.where(rng.random(n) < 0.1, ~ok, ok).astype(int)      # 1 = repaid (10% noise)
X_tr, y_tr, X_te, y_te = X[:1500], y[:1500], X[1500:], y[1500:]
from sklearn.model_selection import cross_val_score
import matplotlib; matplotlib.use("Agg"); import matplotlib.pyplot as plt
alphas = DT(random_state=0).cost_complexity_pruning_path(X_tr, y_tr).ccp_alphas
cv = [cross_val_score(DT(ccp_alpha=a, random_state=0), X_tr, y_tr, cv=5).mean() for a in alphas]
best = alphas[int(np.argmax(cv))]
print(f"{len(alphas)} alphas tried; best ccp_alpha {best:.5f}, CV {max(cv):.3f}")
plt.semilogx(alphas[1:], cv[1:], marker="o"); plt.xlabel("ccp_alpha"); plt.ylabel("5-fold CV accuracy")
plt.savefig("ccp_curve.png", dpi=90)

Output:

119 alphas tried; best ccp_alpha 0.00126, CV 0.899

The curve rises steeply from α = 0, stays flat around 0.0005-0.005, then drops as α removes useful splits. Any α on the plateau is fine (in the episode, α = 0.005 gave CV 0.902; different α between two path levels give the same tree, so CV values cluster).

3. Add a unique customer-ID column. What importance does it get?

Solution
import numpy as np
from sklearn.tree import DecisionTreeClassifier as DT
rng = np.random.default_rng(29); n = 2000                  # the same made-up kirana customers
X = np.c_[rng.integers(10, 81, n), rng.integers(0, 11, n)]  # income (thousand Rs), years known
ok = ((X[:, 0] >= 30) & (X[:, 1] >= 2)) | (X[:, 0] >= 60)
y = np.where(rng.random(n) < 0.1, ~ok, ok).astype(int)      # 1 = repaid (10% noise)
X_tr, y_tr, X_te, y_te = X[:1500], y[:1500], X[1500:], y[1500:]
cid = np.arange(n)                                           # customer ID: 0, 1, 2, ... unique for everyone
X3 = np.c_[X, cid]
for name, t in [("no limits", DT(random_state=0)), ("min_samples_leaf=50", DT(min_samples_leaf=50, random_state=0))]:
    t.fit(X3[:1500], y_tr)
    print(f"{name:<20} importance income {t.feature_importances_[0]:.2f}  years {t.feature_importances_[1]:.2f}"
          f"  customer_id {t.feature_importances_[2]:.2f}  test {t.score(X3[1500:], y_te):.3f}")

Output:

no limits            importance income 0.60  years 0.17  customer_id 0.22  test 0.824
min_samples_leaf=50  importance income 0.81  years 0.18  customer_id 0.01  test 0.880

An ID is unique for every customer, so the full tree uses it to separate the last few rows: importance 0.22 for a meaningless number. With min_samples_leaf=50 it almost disappears.

4. Prune the chai regression tree with ccp_alpha and compare cross-validated MAE.

Solution
import numpy as np
from sklearn.tree import DecisionTreeRegressor
from sklearn.model_selection import cross_val_score
rng = np.random.default_rng(30)                              # the same 300 made-up chai days
temp = rng.uniform(10, 45, 300).round(1).reshape(-1, 1)
cups = (260 - 4 * temp[:, 0] + rng.normal(0, 15, 300)).round()
for a in [0, 10, 50, 200, 1000]:                             # alpha is in squared cups here
    r = DecisionTreeRegressor(ccp_alpha=a, random_state=0)
    mae = -cross_val_score(r, temp, cups, cv=5, scoring="neg_mean_absolute_error").mean()
    print(f"ccp_alpha {a:<5} leaves {r.fit(temp, cups).get_n_leaves():>3}  CV MAE {mae:.1f} cups")

Output:

ccp_alpha 0     leaves 196  CV MAE 16.0 cups
ccp_alpha 10    leaves   6  CV MAE 13.1 cups
ccp_alpha 50    leaves   4  CV MAE 14.5 cups
ccp_alpha 200   leaves   3  CV MAE 17.8 cups
ccp_alpha 1000  leaves   2  CV MAE 22.9 cups

For regression trees α is measured in squared target units (cups²), so useful values are much larger than for classification. About 6 leaves gives the lowest error; too much pruning underfits.

5. By hand: a regression leaf holds 120, 130, 140, 150 cups. Prediction? Its variance?

Solution

Mean = 540 / 4 = 135. Variance = ((−15)² + (−5)² + 5² + 15²) / 4 = (225 + 25 + 25 + 225) / 4 = 125.

Quick quiz

  1. Which knob guarantees every leaf has at least 50 rows? min_samples_leaf=50.
  2. Bigger ccp_alpha means a bigger or smaller tree? Smaller.
  3. A full tree gives a random ID column high importance. Why? Many unique values give lucky splits, measured on training data.
  4. What does a regression tree's leaf predict? The mean of its training targets.
  5. Prediction at 50 °C if training data stops at 45 °C? The same as the edge leaf (no extrapolation).

Interview questions

  1. How do you prevent a decision tree from overfitting? Pre-pruning (max_depth, min_samples_leaf, min_samples_split, max_leaf_nodes) or post-pruning (cost-complexity, ccp_alpha), tuned by cross-validation; or use ensembles.
  2. Explain cost-complexity pruning. Minimise training impurity + α × number of leaves; increasing α removes the weakest-link branches first; choose α by CV.
  3. Pre-pruning vs post-pruning? Pre stops early (fast, may miss deeper useful splits); post grows fully then prunes (slower, usually safer).
  4. How is feature importance computed in a tree, and what's its bias? Total weighted impurity decrease per feature, normalised; biased towards high-cardinality features and computed on training data. Use permutation importance on held-out data to check.
  5. Why are decision trees unstable? Small data changes alter early splits, which change the whole subtree: high variance. Bagging/random forests average it out.
  6. How does a regression tree choose splits and make predictions? Maximum variance (squared error) reduction; leaf predicts the mean.
  7. Can a tree extrapolate? No: predictions are always one of the leaf means seen in training.
  8. How do you read value in plot_tree? The per-class counts (or the mean, for regression) of the training rows reaching that node.
  9. When would you prefer a single tree over a more accurate model? When the decision must be explained as rules (credit, medical triage, compliance) and a small accuracy loss is acceptable.
  10. Do trees need scaling? No; splits compare one feature at a time against a threshold.

Glossary

Term Meaning (plain words) Example from this episode
Pruning Shrinking a tree to stop overfitting 284 → 4 leaves
Pre-pruning Stopping growth early max_depth=3
max_depth Most questions on any path 3
min_samples_split Rows a node needs before it may split 100
min_samples_leaf Rows every leaf must keep 50
Post-pruning Grow fully, then cut weak branches cost-complexity pruning
Cost-complexity pruning Minimise impurity + α × leaves ccp_alpha=0.002
ccp_alpha (α) "Rent" per leaf; bigger = smaller tree 0.15 → 2 leaves
Pruning path Every α where a branch is cut 119 levels
Weakest link The branch that gains least per extra leaf cut first as α grows
plot_tree / export_text Draw / print a fitted tree the 4-leaf tree figure
value Class counts (or mean) at a node [578, 922] at the root
Feature importance (MDI) Share of total impurity decrease from a feature income 0.82
High cardinality Many unique values phone digits, customer ID
Permutation importance Score drop when a feature is shuffled (Episode 34) the reliable check
Instability / high variance Small data change → different model 6.6% disagreement
Regression tree Tree predicting numbers chai cups vs temperature
Variance reduction Parent variance − weighted children variance 1,310 at 27.5 °C
squared_error scikit-learn's variance-reduction criterion default for DecisionTreeRegressor
Extrapolation Predicting outside the training range 50 °C → 93.5 (flat)
Step function Prediction that jumps between flat levels 4 steps at depth 2

Summary

  • Stop overgrowth with pre-pruning (max_depth, min_samples_split, min_samples_leaf) or post-pruning (ccp_alpha), tuned by CV.
  • A pruned tree is readable: question, gini, samples, value, class.
  • Feature importances of overgrown trees credit noise; prune first, check with permutation importance.
  • Trees are unstable; pruning and ensembles fix it.
  • Regression trees split by variance reduction, predict leaf means in steps, and can't extrapolate.
  • Next episode: support vector machines.