Machine Learning Full Course · Episode 35. Every code sample below was run with Python 3.13, NumPy 2.5.3 and scikit-learn 1.9.1; the output shown is the real output. Sunita's tiffin customers, kitchen days and wedding orders are made up for teaching; the mini project uses the diabetes dataset bundled with scikit-learn.
The big idea: bagging and random forests train many models independently and average them, which reduces variance. Boosting trains models one after another, each focusing on what the earlier ones got wrong, which reduces bias. The building block is a weak learner, a model only a little better than guessing, classically a decision stump (a tree with one split). AdaBoost keeps a sample weight per row and makes the wrongly classified rows heavier for the next stump. Gradient boosting starts from the average and adds small trees fitted to the residuals (what is still wrong); in general it fits pseudo-residuals, the negative gradient of any loss, so it is gradient descent where each step is a tree. A learning rate (shrinkage) below 1 makes each tree fix only part of the error: more trees are needed, but the model generalises better. Unlike a forest, boosting can overfit if you add too many trees.
Words you need first
- Decision tree, depth, leaf, Gini impurity: Episodes 29-30.
- Bagging, bootstrap, random forest: Episodes 33-34.
- Bias and variance: too simple vs too wobbly (Episode 20).
- Residual: true value minus prediction (Episode 18).
- Derivative, gradient, gradient descent, learning rate: Episodes 6-8.
- Loss function, squared loss, absolute loss, RMSE: Episodes 8, 18-19.
- Natural log (ln) and e ≈ 2.718: ln is the inverse of eˣ; ln(1) = 0, ln of a number above 1 is positive.
Weak learners and decision stumps
- A weak learner is a model that is only slightly better than random guessing.
- A decision stump is a decision tree with
max_depth=1: one question, two leaves. Real-life picture: a gate guard who checks only one thing ("ID card?").
Sunita's 3,000 made-up tiffin customers from Episode 34 (10 features; label: renews next month):
import numpy as np
rng = np.random.default_rng(34); n = 3000 # Sunita's tiffin service: 3,000 made-up customers
names = "missed_meals rating distance_km months price_rs spice calls age family veg_days".split()
P, U, I = rng.poisson, rng.uniform, rng.integers # random counts, decimals, whole numbers
X = np.c_[P(3,n), U(1,5,n), U(0,12,n), I(1,36,n), U(60,160,n), I(1,6,n), P(2,n), I(18,65,n), I(1,7,n), I(0,8,n)]
s = X @ [-1.2,.8,-.12,.04,-.01,0,-.25,0,0,.15] + .5*np.sin(X[:,5]) + .3*(X[:,8]>3) + .8*(X[:,1]>3.5)*(X[:,2]<4)
y = (s + rng.normal(0, 0.8, n) > np.median(s)).astype(int) # hidden made-up rule · 1 = renews next month
X_tr, y_tr, X_te, y_te = X[:2000], y[:2000], X[2000:], y[2000:]
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import BaggingClassifier, RandomForestClassifier
stump = DecisionTreeClassifier(max_depth=1, random_state=0).fit(X_tr, y_tr) # ONE question
f, t = stump.tree_.feature[0], stump.tree_.threshold[0]
print(f"the stump asks: {names[f]} <= {t}? test {stump.score(X_te, y_te):.3f}")
bag = BaggingClassifier(DecisionTreeClassifier(max_depth=1), n_estimators=100, random_state=0)
print(f"100 bagged stumps: test {bag.fit(X_tr, y_tr).score(X_te, y_te):.3f}")
rf = RandomForestClassifier(n_estimators=300, random_state=0).fit(X_tr, y_tr)
print(f"random forest (ep34): test {rf.score(X_te, y_te):.3f}")
Output:
the stump asks: missed_meals <= 2.5? test 0.776
100 bagged stumps: test 0.776
random forest (ep34): test 0.886
- One question ("missed meals ≤ 2.5?") already gets 0.776.
- 100 bagged stumps: 0.776, no gain. Every bootstrap picks the same best question with almost the same threshold, so the vote is one stump repeated. A stump's problem is bias (one question is too simple), and bagging only reduces variance.
The boosting idea
Real-life picture: a student's mistakes notebook. After each mock test, she spends more time on the questions she got wrong; each round fixes the gaps left by the earlier rounds.
Boosting: train models in sequence; each new model pays more attention to the examples the ensemble so far gets wrong; combine all models with weights.
| Bagging / random forest | Boosting | |
|---|---|---|
| Training | independent, in parallel | one after another |
| Each new model | sees a random sample | focuses on current mistakes |
| Base model | deep, unstable trees | shallow trees (stumps to depth ~6) |
| Mainly reduces | variance | bias |
| More models | never worse, just slower | can overfit: choose the count on validation data |
AdaBoost from scratch
AdaBoost (adaptive boosting), for labels y ∈ {−1, +1}:
- Start with equal sample weights wᵢ = 1/n. A sample weight says how much a row counts during training.
- Fit a stump h using the weights (
fit(X, y, sample_weight=w)). Its weighted error err = the sum of weights of the rows it gets wrong. - Give the stump a say: α = ½ · ln((1 − err) / err). err near 0 → large α; err near 0.5 (a coin toss) → α near 0.
- Update: wᵢ ← wᵢ · exp(−α · yᵢ · h(xᵢ)), then divide by the sum so the weights add up to 1. Wrong rows (y·h = −1) get heavier by e^α; right rows (y·h = +1) get lighter.
- Repeat. Final prediction: sign(α₁h₁(x) + α₂h₂(x) + …).
Example: err = 0.2 → α = ½ ln(0.8/0.2) = ½ ln 4 ≈ 0.693; wrong rows are multiplied by e^0.693 = 2, right rows by e^−0.693 = 0.5 (before rescaling).
import numpy as np
rng = np.random.default_rng(34); n = 3000 # Sunita's tiffin service: 3,000 made-up customers
names = "missed_meals rating distance_km months price_rs spice calls age family veg_days".split()
P, U, I = rng.poisson, rng.uniform, rng.integers # random counts, decimals, whole numbers
X = np.c_[P(3,n), U(1,5,n), U(0,12,n), I(1,36,n), U(60,160,n), I(1,6,n), P(2,n), I(18,65,n), I(1,7,n), I(0,8,n)]
s = X @ [-1.2,.8,-.12,.04,-.01,0,-.25,0,0,.15] + .5*np.sin(X[:,5]) + .3*(X[:,8]>3) + .8*(X[:,1]>3.5)*(X[:,2]<4)
y = (s + rng.normal(0, 0.8, n) > np.median(s)).astype(int) # hidden made-up rule · 1 = renews next month
X_tr, y_tr, X_te, y_te = X[:2000], y[:2000], X[2000:], y[2000:]
from sklearn.tree import DecisionTreeClassifier as DT; from sklearn.ensemble import AdaBoostClassifier
ys, w, F = 2 * y_tr - 1, np.full(2000, 1 / 2000), np.zeros(1000) # labels -1/+1 · equal sample weights · test score
for m in range(1, 201):
stump = DT(max_depth=1, random_state=0).fit(X_tr, ys, sample_weight=w); h = stump.predict(X_tr)
err = w[h != ys].sum(); alpha = 0.5 * np.log((1 - err) / err) # weighted error · its say in the vote
w = w * np.exp(-alpha * ys * h); w = w / w.sum() # wrong ones get heavier
F += alpha * stump.predict(X_te); acc = np.mean((F > 0) == y_te) # running vote for the test customers
if m in [1, 3, 10, 50, 200]:
print(f"round {m:>3}: err {err:.3f} alpha {alpha:.3f} max w x{w.max() * 2000:.0f} test {acc:.3f}")
ada = AdaBoostClassifier(DT(max_depth=1), n_estimators=200, random_state=0).fit(X_tr, y_tr)
print(f"scikit-learn AdaBoostClassifier: test {ada.score(X_te, y_te):.3f}")
Output:
round 1: err 0.211 alpha 0.661 max w x2 test 0.776
round 3: err 0.294 alpha 0.437 max w x7 test 0.822
round 10: err 0.366 alpha 0.275 max w x9 test 0.857
round 50: err 0.489 alpha 0.023 max w x11 test 0.905
round 200: err 0.493 alpha 0.014 max w x17 test 0.907
scikit-learn AdaBoostClassifier: test 0.907
- Test accuracy: 0.776 (round 1) → 0.822 (3) → 0.857 (10) → 0.905 (50) → 0.907 (200), better than the random forest's 0.886.
- The weighted error of new stumps climbs towards 0.5 and α shrinks: the weights pile onto the hardest customers (the heaviest weighs 17 times its starting weight), so later stumps work on difficult cases and get small votes.
- scikit-learn's
AdaBoostClassifier(it uses the SAMME formula; for two classes its α is a constant multiple of ours, so the votes and predictions are the same) also gives 0.907.

Why can stumps reach 0.907? Each stump uses one feature, so a sum of stumps is a sum of one-feature effects. Sunita's hidden rule is mostly such a sum, so boosted stumps fit it well. To model interactions (like "rating > 3.5 and distance < 4"), use depth-2 or deeper trees.
Gradient boosting: fitting residuals
Real-life picture: a tailor cuts a kurta in a standard size, measures what is still off at the fitting (the residual), fixes that, measures again, and repeats.
Gradient boosting for squared loss:
- Start with a constant: F₀ = the average of y.
- Compute residuals rᵢ = yᵢ − F(xᵢ).
- Fit a small regression tree to the residuals.
- Update F ← F + η · tree(x), where η is the learning rate.
- Repeat.

Sunita's kitchen: 1,200 made-up days (temperature, day of week, rain) → tiffins ordered, noise standard deviation 15.
import numpy as np
from sklearn.tree import DecisionTreeRegressor
rng = np.random.default_rng(5); n = 1200 # Sunita's kitchen: 1,200 made-up days
X = np.c_[rng.uniform(15, 45, n), rng.integers(0, 7, n), rng.uniform(0, 60, n)] # temp, day, rain
y = 200 + 3*X[:, 0] - 0.24*(X[:, 0] - 30)**2 + 25*(X[:, 1] >= 5) + 0.8*X[:, 2] + rng.normal(0, 15, n)
X_tr, y_tr, X_te, y_te = X[:900], y[:900], X[900:], y[900:] # y = tiffins ordered that day
rmse = lambda pred, true: np.sqrt(np.mean((pred - true) ** 2))
from sklearn.ensemble import GradientBoostingRegressor
for lr in [1.0, 0.1]: # learning rate = shrinkage
F_tr, F_te = np.full(900, y_tr.mean()), np.full(300, y_tr.mean()) # start: average day
print(f"lr={lr} start: test RMSE {rmse(F_te, y_te):.1f}")
for m in range(1, 501):
r = y_tr - F_tr # residuals: what is still wrong
tree = DecisionTreeRegressor(max_depth=3, random_state=0).fit(X_tr, r)
F_tr += lr * tree.predict(X_tr); F_te += lr * tree.predict(X_te)
if m in [1, 5, 50, 500]:
print(f" {m:>3} trees: train {rmse(F_tr, y_tr):5.1f} test {rmse(F_te, y_te):5.1f}")
sk = GradientBoostingRegressor(n_estimators=50, learning_rate=0.1, max_depth=3, random_state=0)
print(f"scikit-learn, 50 trees, lr 0.1: test {rmse(sk.fit(X_tr, y_tr).predict(X_te), y_te):.1f}")
Output:
lr=1.0 start: test RMSE 37.2
1 trees: train 19.8 test 21.6
5 trees: train 14.5 test 17.4
50 trees: train 8.1 test 21.1
500 trees: train 0.1 test 22.5
lr=0.1 start: test RMSE 37.2
1 trees: train 35.4 test 34.7
5 trees: train 27.1 test 27.0
50 trees: train 13.2 test 15.6
500 trees: train 7.2 test 17.2
scikit-learn, 50 trees, lr 0.1: test 15.6
- Learning rate 1.0: fast (37.2 → 17.4 test RMSE in 5 trees), then the test error climbs to 22.5 while training error falls to 0.1: each full-strength tree chases leftover noise. Overfitting.
- Learning rate 0.1: slower, but 15.6 at 50 trees, at the noise floor (≈15). scikit-learn's
GradientBoostingRegressorwith the same settings gives the same 15.6: we built gradient boosting.
Shrinkage (learning rate)
- Shrinkage = adding only a fraction η of each tree: F ← F + η · tree. Same role as the learning rate in gradient descent (Episode 8).
- Small η: each tree fixes a little; many trees share the work; noise picked up by one tree is mostly undone by later ones.
- Smaller η needs more trees: learning rate and number of trees trade off.
- Boosting can overfit: with enough trees it fits the training noise. Choose the number of trees on validation data (Episode 36 automates this with early stopping).

Pseudo-residuals: gradient boosting for any loss
For squared loss L = ½(y − F)², the derivative with respect to the prediction F is −(y − F), so the negative gradient is y − F, exactly the residual. Fitting residuals moves each prediction downhill on the loss: gradient descent, with one tree per step.
For any differentiable loss, compute the pseudo-residual for every row:
rᵢ = −∂L(yᵢ, F)/∂F
and fit a tree to it. Examples:
| Loss | L(y, F) | Pseudo-residual −∂L/∂F | Leaf value |
|---|---|---|---|
| Squared | ½(y − F)² | y − F | mean of residuals in the leaf |
| Absolute | |y − F| | sign(y − F) | median of residuals in the leaf |
| Log loss (binary, F = log-odds) | −[y ln p + (1 − y) ln(1 − p)], p = sigmoid(F) | y − p | Newton step: Σ(y − p) / Σ p(1 − p) |
Absolute loss grows slowly for big misses, so one huge outlier can't dominate. Test: add 400 tiffins to 30 random training days (made-up wedding orders):
import numpy as np
from sklearn.tree import DecisionTreeRegressor
rng = np.random.default_rng(5); n = 1200 # Sunita's kitchen: 1,200 made-up days
X = np.c_[rng.uniform(15, 45, n), rng.integers(0, 7, n), rng.uniform(0, 60, n)] # temp, day, rain
y = 200 + 3*X[:, 0] - 0.24*(X[:, 0] - 30)**2 + 25*(X[:, 1] >= 5) + 0.8*X[:, 2] + rng.normal(0, 15, n)
X_tr, y_tr, X_te, y_te = X[:900], y[:900], X[900:], y[900:] # y = tiffins ordered that day
rmse = lambda pred, true: np.sqrt(np.mean((pred - true) ** 2))
y_bad = y_tr.copy(); y_bad[np.random.default_rng(1).choice(900, 30, replace=False)] += 400 # 30 wedding orders
def boost(y, loss, k=100, lr=0.1):
F = np.full(900, y.mean() if loss == "squared" else np.median(y)); P = np.full(300, F[0])
for _ in range(k):
r = y - F if loss == "squared" else np.sign(y - F) # pseudo-residuals
tree = DecisionTreeRegressor(max_depth=3, random_state=0).fit(X_tr, r)
lt, le = tree.apply(X_tr), tree.apply(X_te) # the leaf of every day
step = {L: (np.mean if loss == "squared" else np.median)((y - F)[lt == L]) for L in np.unique(lt)}
F += lr * np.array([step[L] for L in lt]); P += lr * np.array([step[L] for L in le])
return rmse(P, y_te)
for loss in ["squared", "absolute"]:
print(f"{loss:<8} loss: clean data RMSE {boost(y_tr, loss):.1f} with 30 outliers {boost(y_bad, loss):.1f}")
Output:
squared loss: clean data RMSE 15.9 with 30 outliers 37.9
absolute loss: clean data RMSE 15.6 with 30 outliers 16.4
- Clean data: both ≈ 15.6-15.9.
- With 30 wedding days: squared loss 37.9 (the model learned to expect weddings), absolute loss 16.4 (barely hurt). Same algorithm; only the loss changed. In scikit-learn:
GradientBoostingRegressor(loss="absolute_error"), or"huber"(squared for small misses, absolute for big ones).
Common mistakes
import numpy as np
rng = np.random.default_rng(34); n = 3000 # Sunita's tiffin service: 3,000 made-up customers
names = "missed_meals rating distance_km months price_rs spice calls age family veg_days".split()
P, U, I = rng.poisson, rng.uniform, rng.integers # random counts, decimals, whole numbers
X = np.c_[P(3,n), U(1,5,n), U(0,12,n), I(1,36,n), U(60,160,n), I(1,6,n), P(2,n), I(18,65,n), I(1,7,n), I(0,8,n)]
s = X @ [-1.2,.8,-.12,.04,-.01,0,-.25,0,0,.15] + .5*np.sin(X[:,5]) + .3*(X[:,8]>3) + .8*(X[:,1]>3.5)*(X[:,2]<4)
y = (s + rng.normal(0, 0.8, n) > np.median(s)).astype(int) # hidden made-up rule · 1 = renews next month
X_tr, y_tr, X_te, y_te = X[:2000], y[:2000], X[2000:], y[2000:]
from sklearn.tree import DecisionTreeClassifier as DT
ys, w, F = y_tr, np.full(2000, 1 / 2000), np.zeros(1000) # forgot: labels must be -1 / +1
for m in range(1, 201):
stump = DT(max_depth=1, random_state=0).fit(X_tr, ys, sample_weight=w); h = stump.predict(X_tr)
err = w[h != ys].sum(); alpha = 0.5 * np.log((1 - err) / err)
w = w * np.exp(-alpha * ys * h); w = w / w.sum()
F += alpha * stump.predict(X_te)
if m in [1, 10, 200]: print(f"round {m:>3}: err {err:.3f} test {np.mean((F > 0) == y_te):.3f}")
Output:
round 1: err 0.211 test 0.776
round 10: err 0.324 test 0.776
round 200: err 0.324 test 0.776
With 0/1 labels, y·h = 0 whenever y = 0 or h = 0, so exp(−α·0) = 1: a customer whose true label is 0 but who was predicted 1 gets no extra weight. The mistakes notebook never records those mistakes and boosting freezes at the first stump's accuracy. No error message, just a wrong result.
A real error message, for a negative learning rate:
from sklearn.ensemble import GradientBoostingRegressor
GradientBoostingRegressor(learning_rate=-0.1).fit([[1], [2]], [1, 2])
Output:
sklearn.utils._param_validation.InvalidParameterError: The 'learning_rate' parameter of GradientBoostingRegressor must be a float in the range [0.0, inf). Got -0.1 instead.
| Mistake | What you see | Why | Fix |
|---|---|---|---|
| Bagging stumps | 0.776 → 0.776 | Stumps have bias, not variance | Boost them |
| 0/1 labels in the AdaBoost formula | stuck at 0.776, no error | y·h = 0: weights never change | Labels −1/+1 (ys = 2*y - 1) |
| Learning rate 1.0 with many trees | train 0.1, test 22.5 | Each tree chases noise | η ≈ 0.05-0.1 and more trees |
| Too many rounds even with small η | 15.6 → 17.2 | Boosting can overfit | Choose the count on validation data |
| Squared loss with big outliers | RMSE 37.9 | Huge residuals dominate the trees | loss="absolute_error" or "huber" |
| Negative learning rate | InvalidParameterError: The 'learning_rate' parameter ... must be a float in the range [0.0, inf) |
Not a valid step size | A small positive value |
Mini project: gradient boosting on real data, choosing the tree count
The diabetes dataset bundled with scikit-learn (442 patients, 10 numeric features, target = disease progression one year after baseline). staged_predict returns the prediction after 1, 2, 3, … trees, so we can pick the tree count on a validation split:
import numpy as np
from sklearn.datasets import load_diabetes
from sklearn.model_selection import train_test_split
from sklearn.ensemble import GradientBoostingRegressor, RandomForestRegressor
from sklearn.metrics import mean_absolute_error as mae
X, y = load_diabetes(return_X_y=True)
X_rest, X_te, y_rest, y_te = train_test_split(X, y, test_size=0.25, random_state=0)
X_tr, X_va, y_tr, y_va = train_test_split(X_rest, y_rest, test_size=0.25, random_state=0)
gb = GradientBoostingRegressor(n_estimators=1000, learning_rate=0.05, max_depth=2, random_state=0).fit(X_tr, y_tr)
va = [mae(y_va, p) for p in gb.staged_predict(X_va)]
best = int(np.argmin(va)) + 1
print(f"best tree count on validation: {best} validation MAE {min(va):.1f} with all 1000 trees: {va[-1]:.1f}")
final = GradientBoostingRegressor(n_estimators=best, learning_rate=0.05, max_depth=2, random_state=0).fit(X_rest, y_rest)
rf = RandomForestRegressor(500, min_samples_leaf=5, random_state=0).fit(X_rest, y_rest)
print(f"test MAE: gradient boosting {mae(y_te, final.predict(X_te)):.1f} random forest {mae(y_te, rf.predict(X_te)):.1f}")
Output:
best tree count on validation: 274 validation MAE 44.5 with all 1000 trees: 46.7
test MAE: gradient boosting 47.3 random forest 47.2
- The validation error is lowest at a modest number of trees and worse with all 1,000: boosting overfits if left running.
- After choosing the count, we refit on train + validation and score once on the untouched test set. On this small, smooth dataset the two ensembles are close; boosting usually shines on larger tabular data (Episode 36).
Practice
1. AdaBoost with depth-2 trees instead of stumps on Sunita's data: better or worse?
Solution
import numpy as np
rng = np.random.default_rng(34); n = 3000 # Sunita's tiffin service: 3,000 made-up customers
names = "missed_meals rating distance_km months price_rs spice calls age family veg_days".split()
P, U, I = rng.poisson, rng.uniform, rng.integers # random counts, decimals, whole numbers
X = np.c_[P(3,n), U(1,5,n), U(0,12,n), I(1,36,n), U(60,160,n), I(1,6,n), P(2,n), I(18,65,n), I(1,7,n), I(0,8,n)]
s = X @ [-1.2,.8,-.12,.04,-.01,0,-.25,0,0,.15] + .5*np.sin(X[:,5]) + .3*(X[:,8]>3) + .8*(X[:,1]>3.5)*(X[:,2]<4)
y = (s + rng.normal(0, 0.8, n) > np.median(s)).astype(int) # hidden made-up rule · 1 = renews next month
X_tr, y_tr, X_te, y_te = X[:2000], y[:2000], X[2000:], y[2000:]
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import AdaBoostClassifier
for d in [1, 2, 3]:
a = AdaBoostClassifier(DecisionTreeClassifier(max_depth=d), n_estimators=200, random_state=0).fit(X_tr, y_tr)
print(f"depth {d}: test {a.score(X_te, y_te):.3f}")
Output:
depth 1: test 0.907
depth 2: test 0.893
depth 3: test 0.887
Here stumps win (0.907) and deeper trees are slightly worse (0.893, 0.887). Sunita's rule is mostly a sum of one-feature effects, which stumps fit exactly; AdaBoost with deeper trees has stronger learners that start fitting the 3% of noisy-looking customers sooner (AdaBoost keeps making hard rows heavier). On data with strong interactions, depth 2-4 often wins. Tune the depth with validation.
2. Gradient boosting with learning rates 0.05, 0.1 and 0.3: best tree count and test RMSE.
Solution
import numpy as np
from sklearn.tree import DecisionTreeRegressor
rng = np.random.default_rng(5); n = 1200 # Sunita's kitchen: 1,200 made-up days
X = np.c_[rng.uniform(15, 45, n), rng.integers(0, 7, n), rng.uniform(0, 60, n)] # temp, day, rain
y = 200 + 3*X[:, 0] - 0.24*(X[:, 0] - 30)**2 + 25*(X[:, 1] >= 5) + 0.8*X[:, 2] + rng.normal(0, 15, n)
X_tr, y_tr, X_te, y_te = X[:900], y[:900], X[900:], y[900:] # y = tiffins ordered that day
rmse = lambda pred, true: np.sqrt(np.mean((pred - true) ** 2))
for lr in [0.05, 0.1, 0.3]:
F_tr, F_te, best = np.full(900, y_tr.mean()), np.full(300, y_tr.mean()), (1e9, 0)
for m in range(1, 601):
tree = DecisionTreeRegressor(max_depth=3, random_state=0).fit(X_tr, y_tr - F_tr)
F_tr += lr * tree.predict(X_tr); F_te += lr * tree.predict(X_te)
best = min(best, (rmse(F_te, y_te), m))
print(f"lr {lr}: best test RMSE {best[0]:.1f} at {best[1]} trees")
Output:
lr 0.05: best test RMSE 15.5 at 100 trees
lr 0.1: best test RMSE 15.5 at 44 trees
lr 0.3: best test RMSE 15.8 at 17 trees
Smaller learning rates reach their best later and the best is a little lower or similar. (Here we peeked at the test set to show the curve; in real work, pick the count on a validation set.)
3. Gradient boosting for classification (log loss) from scratch.
Solution
For binary labels y ∈ {0, 1}, keep F as the log-odds; p = sigmoid(F). The pseudo-residual of log loss is y − p, and each leaf moves by a Newton step Σ(y − p) / Σ p(1 − p).
import numpy as np
rng = np.random.default_rng(34); n = 3000 # Sunita's tiffin service: 3,000 made-up customers
names = "missed_meals rating distance_km months price_rs spice calls age family veg_days".split()
P, U, I = rng.poisson, rng.uniform, rng.integers # random counts, decimals, whole numbers
X = np.c_[P(3,n), U(1,5,n), U(0,12,n), I(1,36,n), U(60,160,n), I(1,6,n), P(2,n), I(18,65,n), I(1,7,n), I(0,8,n)]
s = X @ [-1.2,.8,-.12,.04,-.01,0,-.25,0,0,.15] + .5*np.sin(X[:,5]) + .3*(X[:,8]>3) + .8*(X[:,1]>3.5)*(X[:,2]<4)
y = (s + rng.normal(0, 0.8, n) > np.median(s)).astype(int) # hidden made-up rule · 1 = renews next month
X_tr, y_tr, X_te, y_te = X[:2000], y[:2000], X[2000:], y[2000:]
from sklearn.tree import DecisionTreeRegressor
sig = lambda z: 1 / (1 + np.exp(-z))
p0 = y_tr.mean(); F = np.full(2000, np.log(p0 / (1 - p0))); P = np.full(1000, F[0]) # start: log-odds of the base rate
for m in range(1, 201):
p = sig(F); r = y_tr - p # pseudo-residuals of log loss
tree = DecisionTreeRegressor(max_depth=2, random_state=0).fit(X_tr, r)
lt, le = tree.apply(X_tr), tree.apply(X_te)
step = {L: r[lt == L].sum() / (p * (1 - p))[lt == L].sum() for L in np.unique(lt)}
F += 0.1 * np.array([step[L] for L in lt]); P += 0.1 * np.array([step[L] for L in le])
if m in [1, 10, 50, 200]: print(f"{m:>3} trees: test accuracy {np.mean((P > 0) == y_te):.3f}")
Output:
1 trees: test accuracy 0.776
10 trees: test accuracy 0.825
50 trees: test accuracy 0.880
200 trees: test accuracy 0.898
This is what GradientBoostingClassifier does for two classes. Predictions are log-odds; sigmoid turns them into probabilities.
4. GradientBoostingRegressor(loss="huber") on the wedding data.
Solution
import numpy as np
from sklearn.tree import DecisionTreeRegressor
rng = np.random.default_rng(5); n = 1200 # Sunita's kitchen: 1,200 made-up days
X = np.c_[rng.uniform(15, 45, n), rng.integers(0, 7, n), rng.uniform(0, 60, n)] # temp, day, rain
y = 200 + 3*X[:, 0] - 0.24*(X[:, 0] - 30)**2 + 25*(X[:, 1] >= 5) + 0.8*X[:, 2] + rng.normal(0, 15, n)
X_tr, y_tr, X_te, y_te = X[:900], y[:900], X[900:], y[900:] # y = tiffins ordered that day
rmse = lambda pred, true: np.sqrt(np.mean((pred - true) ** 2))
from sklearn.ensemble import GradientBoostingRegressor
y_bad = y_tr.copy(); y_bad[np.random.default_rng(1).choice(900, 30, replace=False)] += 400
for loss in ["squared_error", "absolute_error", "huber"]:
g = GradientBoostingRegressor(loss=loss, n_estimators=100, learning_rate=0.1, max_depth=3, random_state=0)
print(f"{loss:<15} test RMSE {rmse(g.fit(X_tr, y_bad).predict(X_te), y_te):.1f}")
Output:
squared_error test RMSE 37.7
absolute_error test RMSE 16.1
huber test RMSE 22.6
Huber loss is squared for small misses and absolute for large ones; the switch point is the alpha quantile of the current absolute residuals (default 0.9). It lands in between here (22.6): much better than squared loss, worse than absolute loss, because with 30 huge outliers the 90% point still sits fairly high early on. Lowering alpha makes it more robust (alpha=0.8 gives 17.0 on this data); absolute loss is the most robust but ignores the size of normal misses.
5. By hand: a stump has weighted error 0.25. What is α? By what factor are a wrong row's and a right row's weights multiplied (before rescaling)?
Solution
α = ½ ln(0.75/0.25) = ½ ln 3 ≈ 0.549. Wrong: × e^0.549 ≈ 1.73. Right: × e^−0.549 ≈ 0.577.
Quick quiz
- Gradient boosting, squared loss, η = 1: start 270, tree 1 adds +20, true value 300. What does tree 2 learn for this row? 10.
- Does bagging mainly reduce bias or variance? Boosting? Variance; bias.
- In AdaBoost, what happens to the weight of a misclassified row? It goes up (× e^α before rescaling).
- What is a pseudo-residual? The negative gradient of the loss with respect to the current prediction, per row.
- Can adding more trees make gradient boosting overfit? Yes.
- Why use a learning rate below 1? Smaller steps generalise better; many trees share the work.
Interview questions
- What is boosting? Training weak learners sequentially, each focusing on the current ensemble's errors, and combining them with weights; it mainly reduces bias.
- Bagging vs boosting? Independent models averaged (variance) vs sequential models that correct each other (bias); bagging parallelises easily, boosting is sequential and can overfit with too many rounds.
- How does AdaBoost reweight samples? wᵢ ← wᵢ·exp(−α yᵢ h(xᵢ)), normalised; α = ½ ln((1 − err)/err).
- Why is it called gradient boosting? Each tree fits the negative gradient of the loss (pseudo-residuals): gradient descent in function space.
- What does the learning rate (shrinkage) do? Scales each tree's contribution; smaller values need more trees but usually generalise better.
- Which loss for outliers? Absolute or Huber loss instead of squared.
- Typical tree depth in gradient boosting? Shallow: 2-8 (or limited leaves); stumps capture only additive effects.
- How do you choose the number of trees? Validation curve or early stopping (Episode 36), never the test set.
- Is AdaBoost sensitive to label noise? Yes: mislabelled rows keep getting heavier, so later learners chase them.
- What is a weak learner? A model only slightly better than chance, e.g. a decision stump.
Glossary
| Term | Meaning (plain words) | Example from this episode |
|---|---|---|
| Boosting | Training models one after another, each fixing earlier mistakes | AdaBoost, gradient boosting |
| Weak learner | A model only a little better than guessing | a stump at 0.776 |
| Decision stump | A decision tree with one split (depth 1) | "missed_meals ≤ 2.5?" |
| AdaBoost (adaptive boosting) | Boosting by reweighting misclassified rows | 0.776 → 0.907 |
| Sample weight | How much one row counts during training | starts at 1/2000 |
| Weighted error | Total weight of the rows a model gets wrong | 0.211 in round 1 |
| α (alpha) | A learner's say in the final vote | ½ ln((1 − err)/err) |
| Labels −1/+1 | The label convention AdaBoost's formula needs | ys = 2*y - 1 |
| Gradient boosting | Adding trees fitted to the (pseudo-)residuals | 37.2 → 15.6 RMSE |
| Residual | True value minus current prediction | what is still wrong |
| Fitting residuals | Training the next tree to predict the residuals | the tailor's fittings |
| Pseudo-residual | Negative gradient of the loss for each row | y − F (squared), sign(y − F) (absolute) |
| Shrinkage | Adding only a fraction of each tree | η = 0.1 |
| Learning rate (η) | The fraction used for shrinkage | 1.0 vs 0.1 |
| Outlier | A value far from the rest | 30 wedding days (+400) |
| Absolute loss | |y − F|; grows slowly for big misses | robust to weddings |
| Huber loss | Squared for small misses, absolute for big ones | loss="huber" |
staged_predict |
Predictions after each added tree | choose the tree count |
AdaBoostClassifier, GradientBoostingRegressor |
scikit-learn's boosting models | 0.907, 15.6 |
Summary
- A decision stump is a weak learner; bagging can't fix it because its problem is bias.
- Boosting trains learners in sequence on the current mistakes and reduces bias.
- AdaBoost: sample weights, α = ½ ln((1 − err)/err), wrong rows get heavier; labels must be −1/+1.
- Gradient boosting: start at the average, fit small trees to residuals; in general to pseudo-residuals (negative gradient of any loss).
- Shrinkage: a small learning rate with more trees generalises better; boosting can overfit, so pick the tree count on validation data.
- Next episode: XGBoost, LightGBM and CatBoost, with early stopping.