Machine Learning Full Course · Episode 36. Every code sample below was run with Python 3.13, NumPy 2.5.3, pandas, scikit-learn 1.9.1, XGBoost 3.4.1, LightGBM 4.7.0 and CatBoost 1.2.10; the output shown is the real output (fit times will differ on your machine). Sunita's 20,000 app customers are made up for teaching; the mini project uses the breast cancer dataset bundled with scikit-learn.
The big idea: the gradient boosting we built in Episode 35 is a GBDT (gradient boosted decision trees). On real business tables it hits three problems: it is slow (every split tries every unique value), it can't read text columns, and we must guess the number of trees. The three libraries people use fix these. All of them use histogram-based splitting (bin each feature once, try only bin edges). XGBoost adds a regularised objective (a penalty per leaf and on leaf values). LightGBM grows trees leaf-wise (always split the leaf that helps most; size knob num_leaves). CatBoost handles text categories natively with ordered target statistics. With early stopping, a validation set decides when to stop adding trees, so the learning rate and n_estimators stop being a guessing game. subsample and colsample bring the bagging and random-forest ideas into boosting.
Sources for the product facts in this episode: the XGBoost paper (Chen and Guestrin, KDD 2016, arXiv 1603.02754), the LightGBM features documentation, the CatBoost paper (Prokhorenkova et al., NeurIPS 2018) and the CatBoost documentation on categorical features.
Words you need first
- Gradient boosting, pseudo-residuals, learning rate (shrinkage): Episode 35.
- Categorical feature: a column whose values are names from a fixed list (area, plan) (Episode 13).
- One-hot encoding: one 0/1 column per category (Episode 16).
- Train / validation / test split, target leakage: Episode 15.
- Regularisation, ridge (L2) penalty: Episode 21.
- Log loss: the loss for predicted probabilities; lower is better (Episodes 12, 22).
- Bagging, random forest,
max_features: Episodes 33-34.
Real data breaks Episode 35
Sunita's tiffin service moved to an app: 20,000 made-up customers with six number columns and two text columns, area (30 neighbourhoods) and plan (veg, nonveg, jain). Some areas like Sunita much more than others. Split: 12,000 train, 4,000 validation, 4,000 test.
import numpy as np, pandas as pd
rng = np.random.default_rng(36); n = 20000; areas = [f"area_{i:02d}" for i in range(30)] # Sunita's app (made up)
P, U, I, C = rng.poisson, rng.uniform, rng.integers, rng.choice
df = pd.DataFrame({"missed_meals": P(3,n), "rating": U(1,5,n), "distance_km": U(0,12,n), "months": I(1,36,n),
"price_rs": U(60,160,n), "calls": P(2,n), "area": C(areas,n), "plan": C(["veg","nonveg","jain"],n)})
s = (df.iloc[:, :6] @ [-1.2,.8,-.12,.04,-.01,-.25] + df.area.map(dict(zip(areas, rng.normal(0,1,30))))
+ .4*(df.plan == "jain") + .8*(df.rating > 3.5)*(df.distance_km < 4)) # hidden made-up rule
y = (s + rng.normal(0, 0.8, n) > s.median()).astype(int) # 1 = renews next month
X = df.astype({"area": "category", "plan": "category"}) # text columns → pandas categories
X_tr, X_va, X_te, y_tr, y_va, y_te = X[:12000], X[12000:16000], X[16000:], y[:12000], y[12000:16000], y[16000:]
from sklearn.ensemble import GradientBoostingClassifier
print(X[["missed_meals", "rating", "area", "plan"]].head(3))
try: GradientBoostingClassifier(n_estimators=300).fit(X_tr, y_tr) # ep35's way
except ValueError as e: print("ValueError:", e) # the real error, without the long traceback
Output:
missed_meals rating area plan
0 3 1.609491 area_21 nonveg
1 4 2.436860 area_00 nonveg
2 1 3.630504 area_15 veg
ValueError: could not convert string to float: 'area_21'
A tree split asks "is x ≤ t?"; "area_21" is not a number, so scikit-learn's classic GradientBoostingClassifier refuses it. (The programs catch the error with try/except and print its type and message, so the real message is visible without the long library traceback.)
One-hot, and the speed problem
import numpy as np, pandas as pd
rng = np.random.default_rng(36); n = 20000; areas = [f"area_{i:02d}" for i in range(30)] # Sunita's app (made up)
P, U, I, C = rng.poisson, rng.uniform, rng.integers, rng.choice
df = pd.DataFrame({"missed_meals": P(3,n), "rating": U(1,5,n), "distance_km": U(0,12,n), "months": I(1,36,n),
"price_rs": U(60,160,n), "calls": P(2,n), "area": C(areas,n), "plan": C(["veg","nonveg","jain"],n)})
s = (df.iloc[:, :6] @ [-1.2,.8,-.12,.04,-.01,-.25] + df.area.map(dict(zip(areas, rng.normal(0,1,30))))
+ .4*(df.plan == "jain") + .8*(df.rating > 3.5)*(df.distance_km < 4)) # hidden made-up rule
y = (s + rng.normal(0, 0.8, n) > s.median()).astype(int) # 1 = renews next month
X = df.astype({"area": "category", "plan": "category"}) # text columns → pandas categories
X_tr, X_va, X_te, y_tr, y_va, y_te = X[:12000], X[12000:16000], X[16000:], y[:12000], y[12000:16000], y[16000:]
import time, xgboost, lightgbm
from sklearn.ensemble import GradientBoostingClassifier
Xd = pd.get_dummies(X, dtype=float) # one-hot: area_00 ... area_29, plan_jain ...
print("columns after one-hot:", Xd.shape[1], " unique ratings:", X.rating.nunique())
for name, m in [("sklearn GradientBoosting", GradientBoostingClassifier(n_estimators=100, max_depth=3)),
("XGBoost (hist)", xgboost.XGBClassifier(n_estimators=100, max_depth=3, n_jobs=1)),
("LightGBM", lightgbm.LGBMClassifier(n_estimators=100, num_leaves=8, n_jobs=1, verbose=-1))]:
t = time.perf_counter(); m.fit(Xd[:12000], y_tr); secs = time.perf_counter() - t
print(f"{name:<25} fit {secs:5.2f} s test {m.score(Xd[16000:], y_te):.3f}")
Output:
columns after one-hot: 39 unique ratings: 20000
sklearn GradientBoosting fit 1.89 s test 0.882
XGBoost (hist) fit 0.11 s test 0.900
LightGBM fit 0.08 s test 0.892
- Each model runs on one CPU core (
n_jobs=1) so the times are comparable. One-hot turns 8 columns into 39;ratingalone has 20,000 different values. - The classic implementation is more than ten times slower than XGBoost and LightGBM (100 trees each), and their accuracy is similar or better (XGBoost's default learning rate is 0.3, LightGBM's and scikit-learn's 0.1, so the scores differ). Exact times change from run to run. On millions of rows, that is minutes versus hours.
Histogram-based splitting
Real-life picture: to split a class by height, the exact way measures everyone to the millimetre and tries a line between every pair of students; the quick way puts students into height groups (140-145 cm, 145-150 cm, …), counts each group, and tries lines only between groups.
- Exact split search: for every feature, sort the values and try every unique value as a threshold (20,000 candidates for
rating). - Histogram-based: bin each feature once into a few hundred bins; at each split, add up the gradient statistics per bin (a histogram) and try only the bin edges. Much less work, almost the same splits. The LightGBM documentation describes exactly this ("bucket continuous feature values into discrete bins"); the XGBoost paper proposes candidate split points from feature percentiles and maps values into the buckets between them.
- GBDT (gradient boosted decision trees) is the general name for gradient boosting with trees; XGBoost, LightGBM and CatBoost are all GBDT libraries.
XGBoost: a regularised objective
From the XGBoost paper (Chen and Guestrin, KDD 2016), training minimises
objective = Σ loss(yᵢ, ŷᵢ) + Σₖ Ω(treeₖ), Ω(tree) = γ·T + ½·λ·Σⱼ wⱼ²
- T = number of leaves in the tree; wⱼ = the value leaf j adds to the prediction.
- γ (gamma,
gamma/min_split_loss): a fee per leaf; a split is kept only if it lowers the loss by more than γ. - λ (lambda,
reg_lambda): an L2 (ridge) penalty on leaf values, pulling them towards 0 (Episode 21).reg_alphaadds an L1 penalty. - The paper also uses shrinkage (the learning rate
eta/learning_rate), column subsampling borrowed from random forests, and a default direction learned for missing values at every node, so rows with gaps still reach a leaf.
A small worked example of how λ shrinks a leaf: for squared loss, the best value of a leaf holding rows with residuals r is w = Σr / (n + λ). Residuals 10, 12 and 8 (n = 3): λ = 0 → w = 30/3 = 10; λ = 3 → w = 30/6 = 5. A leaf with few rows is pulled towards 0 the most.
Early stopping
Real-life picture: tasting dal while it cooks; it improves, then gets worse if you wait; stop when it hasn't improved for a while and keep the best moment.
Early stopping: after each tree, score the model on a validation set; if the score hasn't improved for early_stopping_rounds trees in a row, stop and keep the best number of trees (best_iteration). Set n_estimators high and let early stopping choose.
import numpy as np, pandas as pd
rng = np.random.default_rng(36); n = 20000; areas = [f"area_{i:02d}" for i in range(30)] # Sunita's app (made up)
P, U, I, C = rng.poisson, rng.uniform, rng.integers, rng.choice
df = pd.DataFrame({"missed_meals": P(3,n), "rating": U(1,5,n), "distance_km": U(0,12,n), "months": I(1,36,n),
"price_rs": U(60,160,n), "calls": P(2,n), "area": C(areas,n), "plan": C(["veg","nonveg","jain"],n)})
s = (df.iloc[:, :6] @ [-1.2,.8,-.12,.04,-.01,-.25] + df.area.map(dict(zip(areas, rng.normal(0,1,30))))
+ .4*(df.plan == "jain") + .8*(df.rating > 3.5)*(df.distance_km < 4)) # hidden made-up rule
y = (s + rng.normal(0, 0.8, n) > s.median()).astype(int) # 1 = renews next month
X = df.astype({"area": "category", "plan": "category"}) # text columns → pandas categories
X_tr, X_va, X_te, y_tr, y_va, y_te = X[:12000], X[12000:16000], X[16000:], y[:12000], y[12000:16000], y[16000:]
import xgboost
for lr in [0.3, 0.1, 0.05]:
m = xgboost.XGBClassifier(n_estimators=3000, learning_rate=lr, max_depth=4, enable_categorical=True,
early_stopping_rounds=20, n_jobs=1)
m.fit(X_tr, y_tr, eval_set=[(X_va, y_va)], verbose=False) # watch the validation loss
print(f"lr {lr:<5} early stop: best {m.best_iteration + 1:>4} trees test {m.score(X_te, y_te):.3f}")
m = xgboost.XGBClassifier(n_estimators=500, learning_rate=0.1, max_depth=4, enable_categorical=True, n_jobs=1)
print(f"lr 0.1 no early stop: all 500 trees test {m.fit(X_tr, y_tr).score(X_te, y_te):.3f}")
Output:
lr 0.3 early stop: best 55 trees test 0.896
lr 0.1 early stop: best 168 trees test 0.898
lr 0.05 early stop: best 332 trees test 0.901
lr 0.1 no early stop: all 500 trees test 0.892
- Learning rate 0.3 → best at 55 trees; 0.1 → 168; 0.05 → 332. Smaller learning rate, more trees, slightly better test accuracy (0.896 → 0.898 → 0.901).
- 500 trees without stopping: 0.892, worse than every early-stopped model: overfit.
- In XGBoost 3,
predictandscoreautomatically use the best iteration after early stopping.

Recipe: pick the smallest learning rate you can afford in time, set n_estimators very high, use early stopping on a validation set (never the test set), then report the test score once.
LightGBM: leaf-wise growth

- Level-wise (depth-wise) growth: split every node of one level before going deeper; most decision tree learners grow this way.
- Leaf-wise (best-first) growth: always split the leaf with the largest loss reduction, wherever it is. LightGBM does this; with the same number of leaves it usually reaches a lower loss.
- Size knob:
num_leaves(maximum leaves per tree, default 31). The LightGBM documentation warns that leaf-wise growth may over-fit when the data is small and offersmax_depthto limit it;min_child_samples(minimum rows per leaf) also helps. - LightGBM is an open-source library maintained under Microsoft's GitHub organisation (its documentation is © Microsoft Corporation).
import numpy as np, pandas as pd
rng = np.random.default_rng(36); n = 20000; areas = [f"area_{i:02d}" for i in range(30)] # Sunita's app (made up)
P, U, I, C = rng.poisson, rng.uniform, rng.integers, rng.choice
df = pd.DataFrame({"missed_meals": P(3,n), "rating": U(1,5,n), "distance_km": U(0,12,n), "months": I(1,36,n),
"price_rs": U(60,160,n), "calls": P(2,n), "area": C(areas,n), "plan": C(["veg","nonveg","jain"],n)})
s = (df.iloc[:, :6] @ [-1.2,.8,-.12,.04,-.01,-.25] + df.area.map(dict(zip(areas, rng.normal(0,1,30))))
+ .4*(df.plan == "jain") + .8*(df.rating > 3.5)*(df.distance_km < 4)) # hidden made-up rule
y = (s + rng.normal(0, 0.8, n) > s.median()).astype(int) # 1 = renews next month
X = df.astype({"area": "category", "plan": "category"}) # text columns → pandas categories
X_tr, X_va, X_te, y_tr, y_va, y_te = X[:12000], X[12000:16000], X[16000:], y[:12000], y[12000:16000], y[16000:]
import lightgbm
for rows in [12000, 1000]:
for leaves in [8, 31, 255]:
m = lightgbm.LGBMClassifier(n_estimators=200, num_leaves=leaves, n_jobs=1, verbose=-1)
m.fit(X_tr[:rows], y_tr[:rows]) # area and plan: native categories
print(f"{rows:>5} rows, num_leaves {leaves:>3}: test {m.score(X_te, y_te):.3f}")
Output:
12000 rows, num_leaves 8: test 0.904
12000 rows, num_leaves 31: test 0.896
12000 rows, num_leaves 255: test 0.886
1000 rows, num_leaves 8: test 0.873
1000 rows, num_leaves 31: test 0.861
1000 rows, num_leaves 255: test 0.863
- With 12,000 rows, more leaves is slightly worse here (0.904 → 0.886); with 1,000 rows the drop is bigger (0.873 → 0.861). Small data, small
num_leaves.
Categorical features
Giving XGBoost the text columns directly:
import numpy as np, pandas as pd
rng = np.random.default_rng(36); n = 20000; areas = [f"area_{i:02d}" for i in range(30)] # Sunita's app (made up)
P, U, I, C = rng.poisson, rng.uniform, rng.integers, rng.choice
df = pd.DataFrame({"missed_meals": P(3,n), "rating": U(1,5,n), "distance_km": U(0,12,n), "months": I(1,36,n),
"price_rs": U(60,160,n), "calls": P(2,n), "area": C(areas,n), "plan": C(["veg","nonveg","jain"],n)})
s = (df.iloc[:, :6] @ [-1.2,.8,-.12,.04,-.01,-.25] + df.area.map(dict(zip(areas, rng.normal(0,1,30))))
+ .4*(df.plan == "jain") + .8*(df.rating > 3.5)*(df.distance_km < 4)) # hidden made-up rule
y = (s + rng.normal(0, 0.8, n) > s.median()).astype(int) # 1 = renews next month
X = df.astype({"area": "category", "plan": "category"}) # text columns → pandas categories
X_tr, X_va, X_te, y_tr, y_va, y_te = X[:12000], X[12000:16000], X[16000:], y[:12000], y[12000:16000], y[16000:]
import xgboost
m = xgboost.XGBClassifier(n_estimators=300, n_jobs=1)
try: m.fit(df[:12000], y_tr) # df: area and plan are still text
except ValueError as e: print("ValueError:", e)
Output:
ValueError: DataFrame.dtypes for data must be int, float, bool or category. When categorical type is supplied, the experimental DMatrix parameter`enable_categorical` must be set to `True`. Invalid columns:area: str, plan: str
The fix is in the message: convert the text to the pandas category dtype (each name stored once with a small code) and pass enable_categorical=True. That is what every other program here does (df.astype({"area": "category", "plan": "category"})).
Native categorical handling vs one-hot:
- One-hot: 30 areas → 30 columns; a split can separate only one area from all the others.
- Native: a split can send any group of areas left and the rest right. The LightGBM documentation describes sorting the categories by their accumulated gradient statistics and then finding the best cut in that order.
- XGBoost:
enable_categorical=Truewith category dtype. LightGBM: category dtype is enough. CatBoost: name the columns incat_features; plain text is fine.
CatBoost and ordered target statistics
CatBoost turns each category into a number: roughly the average label of earlier rows with that category (target statistics, a form of target encoding; Episode 39 covers target encoding in depth). If a row's own label went into its category's average, the model would see part of the answer during training: target leakage. CatBoost's documentation describes the fix:
- Shuffle the rows into a random order (several random permutations are used).
- Encode each row using only the rows before it: value = (countInClass + prior) / (totalCount + 1), where countInClass counts earlier rows of that category with label 1, totalCount counts earlier rows of that category, and the prior is a starting constant.
Example from the documentation with prior 0.05: the first "rock" row gets 0.05; a later "rock" row whose only earlier "rock" had label 0 gets (0 + 0.05)/2 = 0.025. The CatBoost paper (NeurIPS 2018) presents this categorical processing together with ordered boosting, both designed to fight a prediction shift caused by target leakage.
import numpy as np, pandas as pd
rng = np.random.default_rng(36); n = 20000; areas = [f"area_{i:02d}" for i in range(30)] # Sunita's app (made up)
P, U, I, C = rng.poisson, rng.uniform, rng.integers, rng.choice
df = pd.DataFrame({"missed_meals": P(3,n), "rating": U(1,5,n), "distance_km": U(0,12,n), "months": I(1,36,n),
"price_rs": U(60,160,n), "calls": P(2,n), "area": C(areas,n), "plan": C(["veg","nonveg","jain"],n)})
s = (df.iloc[:, :6] @ [-1.2,.8,-.12,.04,-.01,-.25] + df.area.map(dict(zip(areas, rng.normal(0,1,30))))
+ .4*(df.plan == "jain") + .8*(df.rating > 3.5)*(df.distance_km < 4)) # hidden made-up rule
y = (s + rng.normal(0, 0.8, n) > s.median()).astype(int) # 1 = renews next month
X = df.astype({"area": "category", "plan": "category"}) # text columns → pandas categories
X_tr, X_va, X_te, y_tr, y_va, y_te = X[:12000], X[12000:16000], X[16000:], y[:12000], y[12000:16000], y[16000:]
import catboost
cb = catboost.CatBoostClassifier(iterations=3000, learning_rate=0.2, depth=4, cat_features=["area", "plan"],
early_stopping_rounds=20, thread_count=1, verbose=0)
cb.fit(df[:12000], y_tr, eval_set=(df[12000:16000], y_va)) # df: area and plan as plain text
print(f"CatBoost on raw text: best {cb.best_iteration_ + 1} trees test {cb.score(df[16000:], y_te):.3f}")
Output:
CatBoost on raw text: best 219 trees test 0.902
- Raw text in, no encoding by hand: best at 219 trees (learning rate 0.2), test 0.902, close to the best XGBoost below (0.905). CatBoost calls the number of trees
iterationsand the tree depthdepth.
subsample and colsample
subsample: each tree trains on a random fraction of the rows (without replacement). LightGBM:bagging_fraction(withbagging_freq).colsample_bytree: each tree sees a random fraction of the columns. LightGBM:feature_fraction. (XGBoost also hascolsample_bylevelandcolsample_bynode.)- These bring the bagging (Episode 33) and random forest (Episode 34) ideas into boosting: trees differ more, chase less noise, and train faster.
import numpy as np, pandas as pd
rng = np.random.default_rng(36); n = 20000; areas = [f"area_{i:02d}" for i in range(30)] # Sunita's app (made up)
P, U, I, C = rng.poisson, rng.uniform, rng.integers, rng.choice
df = pd.DataFrame({"missed_meals": P(3,n), "rating": U(1,5,n), "distance_km": U(0,12,n), "months": I(1,36,n),
"price_rs": U(60,160,n), "calls": P(2,n), "area": C(areas,n), "plan": C(["veg","nonveg","jain"],n)})
s = (df.iloc[:, :6] @ [-1.2,.8,-.12,.04,-.01,-.25] + df.area.map(dict(zip(areas, rng.normal(0,1,30))))
+ .4*(df.plan == "jain") + .8*(df.rating > 3.5)*(df.distance_km < 4)) # hidden made-up rule
y = (s + rng.normal(0, 0.8, n) > s.median()).astype(int) # 1 = renews next month
X = df.astype({"area": "category", "plan": "category"}) # text columns → pandas categories
X_tr, X_va, X_te, y_tr, y_va, y_te = X[:12000], X[12000:16000], X[16000:], y[:12000], y[12000:16000], y[16000:]
import xgboost
for ss in [1.0, 0.8, 0.5]: # row and column sampling per tree
m = xgboost.XGBClassifier(n_estimators=3000, learning_rate=0.1, max_depth=4, subsample=ss, colsample_bytree=ss,
enable_categorical=True, early_stopping_rounds=20, random_state=0, n_jobs=1)
m.fit(X_tr, y_tr, eval_set=[(X_va, y_va)], verbose=False)
print(f"XGBoost subsample = colsample_bytree = {ss}: test {m.score(X_te, y_te):.3f}")
Output:
XGBoost subsample = colsample_bytree = 1.0: test 0.898
XGBoost subsample = colsample_bytree = 0.8: test 0.905
XGBoost subsample = colsample_bytree = 0.5: test 0.905
- 0.898 → 0.905 with 80% (and also 50%) of rows and columns. Differences under about a point on 4,000 test rows are close to noise; confirm with cross-validation (Episode 26).
Which library when?
| XGBoost | LightGBM | CatBoost | |
|---|---|---|---|
| Tree growth (default) | level-wise (max_depth) |
leaf-wise (num_leaves) |
depth-limited (depth) |
| Categories | category dtype + enable_categorical=True |
category dtype | raw text in cat_features |
| Main size knob | max_depth |
num_leaves (+ max_depth, min_child_samples) |
depth |
| Number of trees | n_estimators |
n_estimators |
iterations |
| Row / column sampling | subsample, colsample_bytree |
bagging_fraction, feature_fraction |
subsample, rsm |
| Good first choice when | general tabular data | large data, speed matters | many categorical columns |
All three are strong; on a given dataset the differences are usually small compared with good features (Episodes 38-40) and honest validation. Try two, tune lightly with early stopping, and compare with cross-validation.
Common mistakes
import numpy as np, pandas as pd
rng = np.random.default_rng(36); n = 20000; areas = [f"area_{i:02d}" for i in range(30)] # Sunita's app (made up)
P, U, I, C = rng.poisson, rng.uniform, rng.integers, rng.choice
df = pd.DataFrame({"missed_meals": P(3,n), "rating": U(1,5,n), "distance_km": U(0,12,n), "months": I(1,36,n),
"price_rs": U(60,160,n), "calls": P(2,n), "area": C(areas,n), "plan": C(["veg","nonveg","jain"],n)})
s = (df.iloc[:, :6] @ [-1.2,.8,-.12,.04,-.01,-.25] + df.area.map(dict(zip(areas, rng.normal(0,1,30))))
+ .4*(df.plan == "jain") + .8*(df.rating > 3.5)*(df.distance_km < 4)) # hidden made-up rule
y = (s + rng.normal(0, 0.8, n) > s.median()).astype(int) # 1 = renews next month
X = df.astype({"area": "category", "plan": "category"}) # text columns → pandas categories
X_tr, X_va, X_te, y_tr, y_va, y_te = X[:12000], X[12000:16000], X[16000:], y[:12000], y[12000:16000], y[16000:]
import xgboost
m = xgboost.XGBClassifier(n_estimators=3000, early_stopping_rounds=20, enable_categorical=True, n_jobs=1)
try: m.fit(X_tr, y_tr) # forgot eval_set
except ValueError as e: print("ValueError:", e)
Output:
ValueError: Must have at least 1 validation dataset for early stopping.
| Mistake | What you see | Why | Fix |
|---|---|---|---|
Text columns into classic GradientBoostingClassifier |
ValueError: could not convert string to float: 'area_21' |
Splits need numbers | Encode, or use a GBDT library with native categories |
| Raw text into XGBoost | ValueError: DataFrame.dtypes for data must be int, float, bool or category. ... Invalid columns:area: str, plan: str |
Text, not category dtype | astype("category") + enable_categorical=True |
| A fixed, large number of trees | 500 trees: test 0.892 | Overfitting | Early stopping on a validation set |
| Early stopping on the test set | an optimistic test score | The test set chose the model | A separate validation set |
Large num_leaves on small data |
0.873 → 0.861 | Deep leaf-wise branches fit a few rows | Small num_leaves, max_depth, min_child_samples |
early_stopping_rounds without eval_set |
ValueError: Must have at least 1 validation dataset for early stopping. |
Nothing to watch | Pass eval_set=[(X_va, y_va)] |
Lowering the learning rate but not raising n_estimators |
stops at the cap, underfit | ~10× smaller rate needs ~10× more trees | Raise n_estimators; let early stopping decide |
Mini project: LightGBM with early stopping on real data
import lightgbm as lgb
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
X, y = load_breast_cancer(return_X_y=True, as_frame=True)
X_rest, X_te, y_rest, y_te = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)
X_tr, X_va, y_tr, y_va = train_test_split(X_rest, y_rest, test_size=0.25, random_state=0, stratify=y_rest)
m = lgb.LGBMClassifier(n_estimators=5000, learning_rate=0.02, num_leaves=8, min_child_samples=10,
subsample=0.8, subsample_freq=1, colsample_bytree=0.8, verbose=-1, random_state=0)
m.fit(X_tr, y_tr, eval_set=[(X_va, y_va)], callbacks=[lgb.early_stopping(100, verbose=False)])
print(f"best trees: {m.best_iteration_} validation log loss {m.best_score_['valid_0']['binary_logloss']:.3f}")
print(f"test accuracy {m.score(X_te, y_te):.3f}")
imp = sorted(zip(m.booster_.feature_importance("gain"), X.columns), reverse=True)[:3]
print("top 3 by gain:", [name for _, name in imp])
Output:
best trees: 176 validation log loss 0.124
test accuracy 0.937
top 3 by gain: ['worst concave points', 'worst area', 'mean concave points']
- LightGBM's early stopping is a callback (
lgb.early_stopping(rounds));subsampleneedssubsample_freq ≥ 1to take effect. feature_importance("gain")is LightGBM's impurity-style importance (total loss reduction); remember Episode 34's warning and confirm with permutation importance before trusting it.
Practice
1. XGBoost with gamma = 0, 1 and 5 (early stopping). What happens to the tree size?
Solution
import numpy as np, pandas as pd
rng = np.random.default_rng(36); n = 20000; areas = [f"area_{i:02d}" for i in range(30)] # Sunita's app (made up)
P, U, I, C = rng.poisson, rng.uniform, rng.integers, rng.choice
df = pd.DataFrame({"missed_meals": P(3,n), "rating": U(1,5,n), "distance_km": U(0,12,n), "months": I(1,36,n),
"price_rs": U(60,160,n), "calls": P(2,n), "area": C(areas,n), "plan": C(["veg","nonveg","jain"],n)})
s = (df.iloc[:, :6] @ [-1.2,.8,-.12,.04,-.01,-.25] + df.area.map(dict(zip(areas, rng.normal(0,1,30))))
+ .4*(df.plan == "jain") + .8*(df.rating > 3.5)*(df.distance_km < 4)) # hidden made-up rule
y = (s + rng.normal(0, 0.8, n) > s.median()).astype(int) # 1 = renews next month
X = df.astype({"area": "category", "plan": "category"}) # text columns → pandas categories
X_tr, X_va, X_te, y_tr, y_va, y_te = X[:12000], X[12000:16000], X[16000:], y[:12000], y[12000:16000], y[16000:]
import xgboost, json
for g in [0, 1, 5]:
m = xgboost.XGBClassifier(n_estimators=3000, learning_rate=0.1, max_depth=6, gamma=g,
enable_categorical=True, early_stopping_rounds=50)
m.fit(X_tr, y_tr, eval_set=[(X_va, y_va)], verbose=False)
trees = m.get_booster().get_dump()[: m.best_iteration + 1]
leaves = sum(t.count("leaf=") for t in trees) / len(trees)
print(f"gamma {g}: {m.best_iteration + 1:>3} trees, {leaves:5.1f} leaves per tree, test {m.score(X_te, y_te):.3f}")
Output:
gamma 0: 90 trees, 53.7 leaves per tree, test 0.894
gamma 1: 96 trees, 47.2 leaves per tree, test 0.897
gamma 5: 94 trees, 24.2 leaves per tree, test 0.901
A larger γ charges more per leaf, so splits that only help a little are dropped and trees get smaller: a regularisation knob for deep trees.
2. LightGBM with num_leaves 8 vs 31, with and without max_depth=4, on 1,000 rows.
Solution
import numpy as np, pandas as pd
rng = np.random.default_rng(36); n = 20000; areas = [f"area_{i:02d}" for i in range(30)] # Sunita's app (made up)
P, U, I, C = rng.poisson, rng.uniform, rng.integers, rng.choice
df = pd.DataFrame({"missed_meals": P(3,n), "rating": U(1,5,n), "distance_km": U(0,12,n), "months": I(1,36,n),
"price_rs": U(60,160,n), "calls": P(2,n), "area": C(areas,n), "plan": C(["veg","nonveg","jain"],n)})
s = (df.iloc[:, :6] @ [-1.2,.8,-.12,.04,-.01,-.25] + df.area.map(dict(zip(areas, rng.normal(0,1,30))))
+ .4*(df.plan == "jain") + .8*(df.rating > 3.5)*(df.distance_km < 4)) # hidden made-up rule
y = (s + rng.normal(0, 0.8, n) > s.median()).astype(int) # 1 = renews next month
X = df.astype({"area": "category", "plan": "category"}) # text columns → pandas categories
X_tr, X_va, X_te, y_tr, y_va, y_te = X[:12000], X[12000:16000], X[16000:], y[:12000], y[12000:16000], y[16000:]
import lightgbm
for leaves in [8, 31]:
for depth in [-1, 4]:
m = lightgbm.LGBMClassifier(n_estimators=300, num_leaves=leaves, max_depth=depth, verbose=-1)
m.fit(X_tr[:1000], y_tr[:1000])
print(f"num_leaves {leaves:>2}, max_depth {depth:>2}: test {m.score(X_te, y_te):.3f}")
Output:
num_leaves 8, max_depth -1: test 0.872
num_leaves 8, max_depth 4: test 0.871
num_leaves 31, max_depth -1: test 0.855
num_leaves 31, max_depth 4: test 0.870
max_depth=-1 means no limit. A depth limit of 4 caps a tree at 16 leaves, so it tames num_leaves=31 on small data.
3. CatBoost vs one-hot XGBoost on only 1,000 training rows.
Solution
import numpy as np, pandas as pd
rng = np.random.default_rng(36); n = 20000; areas = [f"area_{i:02d}" for i in range(30)] # Sunita's app (made up)
P, U, I, C = rng.poisson, rng.uniform, rng.integers, rng.choice
df = pd.DataFrame({"missed_meals": P(3,n), "rating": U(1,5,n), "distance_km": U(0,12,n), "months": I(1,36,n),
"price_rs": U(60,160,n), "calls": P(2,n), "area": C(areas,n), "plan": C(["veg","nonveg","jain"],n)})
s = (df.iloc[:, :6] @ [-1.2,.8,-.12,.04,-.01,-.25] + df.area.map(dict(zip(areas, rng.normal(0,1,30))))
+ .4*(df.plan == "jain") + .8*(df.rating > 3.5)*(df.distance_km < 4)) # hidden made-up rule
y = (s + rng.normal(0, 0.8, n) > s.median()).astype(int) # 1 = renews next month
X = df.astype({"area": "category", "plan": "category"}) # text columns → pandas categories
X_tr, X_va, X_te, y_tr, y_va, y_te = X[:12000], X[12000:16000], X[16000:], y[:12000], y[12000:16000], y[16000:]
import catboost, xgboost
cb = catboost.CatBoostClassifier(iterations=500, learning_rate=0.05, depth=4, verbose=0,
cat_features=["area", "plan"], allow_writing_files=False)
cb.fit(df[:1000], y_tr[:1000])
Xd = pd.get_dummies(X, dtype=float)
xg = xgboost.XGBClassifier(n_estimators=500, learning_rate=0.05, max_depth=4).fit(Xd[:1000], y_tr[:1000])
print(f"CatBoost (raw text): {cb.score(df[16000:], y_te):.3f} XGBoost (one-hot): {xg.score(Xd[16000:], y_te):.3f}")
Output:
CatBoost (raw text): 0.873 XGBoost (one-hot): 0.874
A tie (0.873 vs 0.874). With only 30 areas and about 33 customers per area, one-hot columns work fine. CatBoost's advantage is convenience (raw text in, no encoding code) and robustness when a column has hundreds or thousands of categories, where one-hot becomes huge and sparse; on a small, low-cardinality table like this, don't expect a gain. Always compare on your own data.
4. Choose the number of trees with 5-fold cross-validation (xgboost.cv).
Solution
import numpy as np, pandas as pd
rng = np.random.default_rng(36); n = 20000; areas = [f"area_{i:02d}" for i in range(30)] # Sunita's app (made up)
P, U, I, C = rng.poisson, rng.uniform, rng.integers, rng.choice
df = pd.DataFrame({"missed_meals": P(3,n), "rating": U(1,5,n), "distance_km": U(0,12,n), "months": I(1,36,n),
"price_rs": U(60,160,n), "calls": P(2,n), "area": C(areas,n), "plan": C(["veg","nonveg","jain"],n)})
s = (df.iloc[:, :6] @ [-1.2,.8,-.12,.04,-.01,-.25] + df.area.map(dict(zip(areas, rng.normal(0,1,30))))
+ .4*(df.plan == "jain") + .8*(df.rating > 3.5)*(df.distance_km < 4)) # hidden made-up rule
y = (s + rng.normal(0, 0.8, n) > s.median()).astype(int) # 1 = renews next month
X = df.astype({"area": "category", "plan": "category"}) # text columns → pandas categories
X_tr, X_va, X_te, y_tr, y_va, y_te = X[:12000], X[12000:16000], X[16000:], y[:12000], y[12000:16000], y[16000:]
import xgboost
d = xgboost.DMatrix(X_tr, label=y_tr, enable_categorical=True)
params = {"objective": "binary:logistic", "eta": 0.1, "max_depth": 4, "eval_metric": "logloss"}
cv = xgboost.cv(params, d, num_boost_round=3000, nfold=5, early_stopping_rounds=50, seed=0)
print("trees chosen by 5-fold CV:", len(cv))
print("CV log loss at that point:", round(cv["test-logloss-mean"].iloc[-1], 4))
Output:
trees chosen by 5-fold CV: 156
CV log loss at that point: 0.2335
xgboost.cv stops when the mean validation loss across the 5 folds stops improving; the returned table has one row per kept tree. Close to the 168 trees picked by the single validation split, but steadier because it averages 5 splits.
5. By hand: with learning rate 0.1 early stopping kept 168 trees. Roughly how many trees would you expect at learning rate 0.02?
Solution
About 5 times as many: ≈ 800-850 trees (each tree adds a fifth as much). Set n_estimators well above that, e.g. 5,000, so early stopping can find the point.
Quick quiz
- Learning rate 0.1 → best at 168 trees. At 0.01? Roughly 10× more trees.
- What does histogram-based splitting try as thresholds? Bin edges, not every unique value.
- In Ω = γT + ½λΣw², what is T? The number of leaves.
- What does LightGBM grow at each step? The leaf with the largest loss reduction (leaf-wise).
- Why does CatBoost encode each row using only earlier rows? To avoid target leakage from the row's own label.
- What does early stopping need besides
early_stopping_rounds? A validation set (eval_set).
Interview questions
- XGBoost vs classic gradient boosting? Regularised objective (γT + ½λΣw²), second-order (Newton) leaf values, histogram/approximate splits, missing-value default directions, column subsampling, fast parallel implementation.
- What is histogram-based split finding? Bin features once; accumulate gradient statistics per bin; evaluate only bin edges. Faster and less memory, almost no accuracy loss.
- Leaf-wise vs level-wise growth? Leaf-wise splits the best leaf anywhere (lower loss for a given leaf count, risk of overfitting on small data); level-wise splits all nodes of a level (more balanced trees).
- How do you set learning rate and number of trees? Small learning rate, large
n_estimators, early stopping on validation data. - How does CatBoost handle categorical features? Ordered target statistics: encode each row with the average label of earlier rows (random permutation) plus a prior, to avoid target leakage.
- What do
subsampleandcolsample_bytreedo? Random row/column sampling per tree; reduces overfitting and speeds up training. - Name regularisation knobs in XGBoost.
gamma,reg_lambda,reg_alpha,max_depth,min_child_weight,subsample,colsample_*,learning_rate. - Why not early-stop on the test set? The test set would then help choose the model, so its score becomes optimistic.
- Do GBDT libraries need feature scaling? No; splits depend only on value order (bins are per feature).
- When might a random forest be preferred over GBDT? Very little tuning time, noisy small data, or when you want something robust with few knobs; GBDT usually wins with tuning.
Glossary
| Term | Meaning (plain words) | Example from this episode |
|---|---|---|
| GBDT (gradient boosted decision trees) | Gradient boosting where each learner is a tree | XGBoost, LightGBM, CatBoost |
| XGBoost | A GBDT library with a regularised objective (KDD 2016 paper) | XGBClassifier |
| LightGBM | A GBDT library with histogram bins and leaf-wise growth | LGBMClassifier |
| CatBoost | A GBDT library with native text-category handling (NeurIPS 2018 paper) | CatBoostClassifier, 0.902 |
| Exact split search | Trying every unique value as a threshold | 20,000 candidates for rating |
| Bin | A range of values grouped together | rating 3.10-3.12 |
| Histogram-based splitting | Binning features and trying only bin edges | a few hundred candidates |
| Objective | What training minimises | loss + penalties |
| Regularised objective | Loss plus a complexity penalty | Ω = γT + ½λΣw² |
γ (gamma, min_split_loss) |
Fee per leaf; minimum gain for a split | gamma=5 → smaller trees |
λ (reg_lambda) |
L2 penalty on leaf values | w = Σr/(n + λ) |
| Leaf value (w) | What a leaf adds to the prediction | 10 → 5 with λ = 3 |
| Level-wise (depth-wise) growth | Split every node of a level first | most tree learners |
| Leaf-wise (best-first) growth | Split the leaf with the biggest loss drop | LightGBM |
num_leaves |
LightGBM's maximum leaves per tree | 8, 31, 255 |
| Early stopping | Stop adding trees when validation stops improving | best at 168 trees |
early_stopping_rounds |
How many non-improving trees to allow | 20 |
eval_set |
The validation data early stopping watches | [(X_va, y_va)] |
best_iteration |
Index of the best tree count (0-based in XGBoost) | 167 → 168 trees |
| Validation log loss | Log loss on the validation set | the curves' minimum |
| Native categorical handling | Splitting on groups of categories directly | enable_categorical=True |
| pandas category dtype | Text stored once with integer codes | astype("category") |
| Target statistics (target encoding) | Replacing a category by the average label | area_21 → ≈ 0.8 |
| Ordered target statistics | Using only earlier rows in a random order, plus a prior | CatBoost |
| Prior | Starting value in the encoding formula | 0.05 in the docs' example |
| Target leakage | Training on information that contains the answer | a row's own label in its encoding |
| Prediction shift | The bias that leakage causes, named in the CatBoost paper | fought by ordered boosting |
subsample |
Fraction of rows per tree | 0.8 |
colsample_bytree |
Fraction of columns per tree | 0.8 |
iterations |
CatBoost's name for the number of trees | 3000 cap, 219 kept |
Summary
- Episode 35's gradient boosting is a GBDT; on real tables it is slow, can't read text and needs a tree count.
- Histogram-based splitting bins features once and tries only bin edges: same accuracy, much faster.
- XGBoost minimises loss + γT + ½λΣw²; LightGBM grows leaf-wise (
num_leaves); CatBoost reads text categories with ordered target statistics. - Early stopping on a validation set picks the number of trees; smaller learning rates need more trees.
- Native categories beat one-hot for many-valued columns;
subsample/colsample_bytreeadd healthy randomness. - Next episode: stacking and blending, a model that learns how to combine other models.