Machine Learning Full Course · Episode 42. Every code sample below was run with Python 3.13, NumPy 2.5.3, pandas 3.0.5, scikit-learn 1.9.1, SciPy and Optuna 5.0.0; the output shown is the real output (the timings in seconds change from run to run). Kiran's orders are made up for teaching, which lets us generate 5,000 "next month" orders that nobody tunes on: they tell us how good each choice really is.
The big idea: hyperparameters are the settings you choose before training. Tuning means searching for good ones, and the first rule is to choose with cross-validation on the training rows and touch the test set once: picking the best of 50 settings by test score reported 0.900, while next month's orders said 0.861. Grid search tries every combination of listed values; it explodes with more hyperparameters and repeats the same few values of the one that matters. Random search draws every value from a range, so the same budget tries many more distinct values. Bayesian optimisation (Optuna's TPE sampler) learns from past trials where good settings are. Pruning, successive halving and Hyperband stop spending on settings that are clearly losing.
Sources: J. Bergstra and Y. Bengio, "Random Search for Hyper-Parameter Optimization", JMLR 13:281-305, 2012 (jmlr.org: "randomly chosen trials are more efficient for hyper-parameter optimization than trials on a grid"; "only a few of the hyper-parameters really matter"); L. Li, K. Jamieson, G. DeSalvo, A. Rostamizadeh and A. Talwalkar, "Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization", JMLR 18(185):1-52, 2018 (jmlr.org); the docstrings of the installed Optuna 5.0.0 (TPESampler, MedianPruner, HyperbandPruner; create_study uses TPE and the median pruner by default) and scikit-learn 1.9.1 (HalvingRandomSearchCV is experimental and must be enabled).
Words you need first
- Parameter vs hyperparameter: parameters are learned by training (weights, tree splits); hyperparameters are set before training (learning rate, number of leaves) (Episode 8).
- Gradient boosting,
HistGradientBoostingClassifier, learning rate,max_leaf_nodes,min_samples_leaf,l2_regularization: Episodes 35-36 and 21.booster()in the code is this model with 50 boosting rounds, so every search runs quickly. - Train / test split, data leakage: Episode 15. Cross-validation (CV), folds: Episode 26. Pipeline: Episode 17.
The problem: tuning on the test set
Kiran's late-delivery data (made up): 2,000 orders to train on, 300 to test, and 5,000 orders from "next month" that nobody tunes on. The obvious attempt: try 50 random settings, score each on the test set, keep the best.
import numpy as np, pandas as pd, time
def orders(n, seed): # Kiran's orders (made up)
r = np.random.default_rng(seed)
X = pd.DataFrame({"distance_km": r.uniform(.5, 9, n), "hour": r.integers(0, 24, n), "raining": r.integers(0,2,n),
"prep_min": r.normal(15, 5, n), "rider_trips": r.poisson(6, n), "items": r.integers(1, 8, n)})
s = (.4 * X.distance_km + 1.2 * np.cos((X.hour - 13) * np.pi / 12) + 1.5 * X.raining * (X.distance_km > 5)
+ .1 * X.prep_min - .15 * X.rider_trips)
return X, (s + r.normal(0, .9, n) > 3.2).astype(int) # 1 = late
X, y = orders(2300, 42); tr, te = slice(0, 2000), slice(2000, None) # 2,000 train · 300 test
X_next, y_next = orders(5000, 7) # next month: 5,000 orders nobody tuned on
from sklearn.ensemble import HistGradientBoostingClassifier as HGB # gradient boosting, episode 35
def booster(**p): return HGB(max_iter=50, random_state=0, **p) # 50 boosting rounds: quick to train
rng = np.random.default_rng(0); best = (0, None, None)
for trial in range(50): # MISTAKE: pick by TEST score
p = dict(learning_rate=10 ** rng.uniform(-2, 0), max_leaf_nodes=int(rng.integers(2, 64)),
min_samples_leaf=int(rng.integers(1, 100)))
m = booster(**p).fit(X[tr], y[tr]); score = m.score(X[te], y[te])
if score > best[0]: best = (score, p, m)
print("default settings: test", round(booster().fit(X[tr], y[tr]).score(X[te], y[te]), 3))
print("best of 50 tries: test", round(best[0], 3), " next month", round(best[2].score(X_next, y_next), 3))
Output:
default settings: test 0.883
best of 50 tries: test 0.9 next month 0.861
Default settings: test 0.883. Best of 50: test 0.900. Next month: 0.861. The 0.900 was fake.
Why? Writing the same 300-question paper 50 times with different study plans and reporting only the best score: part of that best score is skill, part is luck (those particular questions suited that attempt). Choosing the best of many on a small test set picks the luckiest. Once the test set helps you choose, it is no longer a fair exam: this is data leakage (Episode 15).
The rule: choose using only the training rows, scored with cross-validation; score the test set once at the end and believe that number. Two terms: the search space is the set of values you allow for each hyperparameter; a trial (or candidate) is one setting that gets tried and scored.
Grid search
List a few values per hyperparameter and try every combination: 4 learning rates × 4 leaf counts = 16 settings. The combinations form a grid, like a multiplication table. With 3-fold CV that is 48 fits; GridSearchCV then refits the best setting on all training rows (refit=True by default).
import numpy as np, pandas as pd, time
def orders(n, seed): # Kiran's orders (made up)
r = np.random.default_rng(seed)
X = pd.DataFrame({"distance_km": r.uniform(.5, 9, n), "hour": r.integers(0, 24, n), "raining": r.integers(0,2,n),
"prep_min": r.normal(15, 5, n), "rider_trips": r.poisson(6, n), "items": r.integers(1, 8, n)})
s = (.4 * X.distance_km + 1.2 * np.cos((X.hour - 13) * np.pi / 12) + 1.5 * X.raining * (X.distance_km > 5)
+ .1 * X.prep_min - .15 * X.rider_trips)
return X, (s + r.normal(0, .9, n) > 3.2).astype(int) # 1 = late
X, y = orders(2300, 42); tr, te = slice(0, 2000), slice(2000, None) # 2,000 train · 300 test
X_next, y_next = orders(5000, 7) # next month: 5,000 orders nobody tuned on
from sklearn.ensemble import HistGradientBoostingClassifier as HGB # gradient boosting, episode 35
def booster(**p): return HGB(max_iter=50, random_state=0, **p) # 50 boosting rounds: quick to train
from sklearn.model_selection import GridSearchCV
space = {"learning_rate": [0.01, 0.03, 0.1, 0.3], "max_leaf_nodes": [3, 7, 15, 31]} # 4 x 4 = 16 settings
t = time.time(); grid = GridSearchCV(booster(), space, cv=3).fit(X[tr], y[tr]) # 16 x 3 folds = 48 fits
print("best:", grid.best_params_, " CV", round(grid.best_score_, 3), f" {time.time() - t:.1f} s")
r = pd.DataFrame(grid.cv_results_) # every setting's mean CV score
print(r.pivot_table("mean_test_score", "param_learning_rate", "param_max_leaf_nodes").round(3))
print("test", round(grid.score(X[te], y[te]), 3), " next month", round(grid.score(X_next, y_next), 3))
Output:
best: {'learning_rate': 0.3, 'max_leaf_nodes': 3} CV 0.855 1.7 s
param_max_leaf_nodes 3 7 15 31
param_learning_rate
0.01 0.768 0.810 0.832 0.836
0.03 0.798 0.827 0.846 0.844
0.10 0.838 0.852 0.849 0.847
0.30 0.855 0.844 0.838 0.831
test 0.887 next month 0.863
- Best: learning rate 0.3 with 3 leaves, CV 0.855; next month 0.863, an honest estimate.
- The table shows the learning rate changes the score far more (0.768 → 0.855) than the leaf count.
- The best sits on the edge of the grid (largest learning rate, fewest leaves). When the best is on the edge, the real best may lie outside: widen the space (Practice 1).
Random search
Grid search has two problems:
- It explodes. 5 hyperparameters × 5 values = 3,125 settings; with 3 folds, 9,375 fits.
- It wastes trials. Our 16 settings tried only 4 different learning rates, the hyperparameter that mattered most. Bergstra and Bengio (2012) found that usually only a few hyperparameters really matter, and that random trials are more efficient than a grid.
Random search draws every value at random from a range (a distribution):
randint(2, 64)draws whole numbers from 2 to 63.loguniform(0.005, 0.5)draws on a log scale: 0.005 → 0.05 is as likely as 0.05 → 0.5. For a learning rate, "ten times bigger" matters, not "plus 0.1".n_iteris how many random settings to try.
import numpy as np, pandas as pd, time
def orders(n, seed): # Kiran's orders (made up)
r = np.random.default_rng(seed)
X = pd.DataFrame({"distance_km": r.uniform(.5, 9, n), "hour": r.integers(0, 24, n), "raining": r.integers(0,2,n),
"prep_min": r.normal(15, 5, n), "rider_trips": r.poisson(6, n), "items": r.integers(1, 8, n)})
s = (.4 * X.distance_km + 1.2 * np.cos((X.hour - 13) * np.pi / 12) + 1.5 * X.raining * (X.distance_km > 5)
+ .1 * X.prep_min - .15 * X.rider_trips)
return X, (s + r.normal(0, .9, n) > 3.2).astype(int) # 1 = late
X, y = orders(2300, 42); tr, te = slice(0, 2000), slice(2000, None) # 2,000 train · 300 test
X_next, y_next = orders(5000, 7) # next month: 5,000 orders nobody tuned on
from sklearn.ensemble import HistGradientBoostingClassifier as HGB # gradient boosting, episode 35
def booster(**p): return HGB(max_iter=50, random_state=0, **p) # 50 boosting rounds: quick to train
from sklearn.model_selection import RandomizedSearchCV; from scipy.stats import loguniform, randint
space = {"learning_rate": loguniform(0.005, 0.5), "max_leaf_nodes": randint(2, 64), # ranges, not lists
"min_samples_leaf": randint(1, 100), "l2_regularization": loguniform(1e-3, 10)}
t = time.time(); rs = RandomizedSearchCV(booster(), space, n_iter=16, cv=3, random_state=0).fit(X[tr], y[tr])
lr = pd.Series(rs.cv_results_["param_learning_rate"], dtype=float)
print("16 settings, 48 fits,", f"{time.time() - t:.1f} s; different learning rates tried:", lr.nunique())
print("best CV", round(rs.best_score_, 3), " learning_rate", round(rs.best_params_["learning_rate"], 3))
print("test", round(rs.score(X[te], y[te]), 3), " next month", round(rs.score(X_next, y_next), 3))
Output:
16 settings, 48 fits, 2.0 s; different learning rates tried: 16
best CV 0.854 learning_rate 0.135
test 0.887 next month 0.861
With the same 48 fits, random search tried 16 different learning rates (the grid tried 4) while searching four hyperparameters instead of two, and reached the same level (next month 0.861).
| Search | Settings | Fits | Distinct learning rates | Next month |
|---|---|---|---|---|
| Best of 50 by test score (wrong) | 50 | 50 | 50 | 0.861 (claimed 0.900) |
| Grid search | 16 | 48 | 4 | 0.863 |
| Random search | 16 | 48 | 16 | 0.861 |
| Optuna TPE | 30 | 90 | 30 | 0.86 |
| Successive halving | 18 → 6 → 2 | mostly on small samples | 18 | 0.86 |
Bayesian optimisation with Optuna
Random search forgets everything between trials. Perfecting a chai recipe, you would not pick the next recipe blindly: you look at your notes ("the good cups had more ginger"), make the next cup near the good ones, and now and then try something new.
- A surrogate model is a cheap model of "which settings score well", learned from past trials.
- Bayesian optimisation uses the surrogate to choose each next trial, balancing exploitation (near the good ones) and exploration (somewhere new).
- Optuna is an open-source hyperparameter optimisation library. You write an objective function that receives a
trial, asks it to suggest values (suggest_float,suggest_int,log=Truefor a log scale), trains, and returns a score. A study is one whole search (direction="maximize",n_trials). - Optuna's default sampler is TPE (tree-structured Parzen estimator): it splits past trials into the best ones and the rest, fits a density to each, and suggests values that are common among the best and rare among the rest (it maximises the ratio l(x) / g(x)).
import numpy as np, pandas as pd, time
def orders(n, seed): # Kiran's orders (made up)
r = np.random.default_rng(seed)
X = pd.DataFrame({"distance_km": r.uniform(.5, 9, n), "hour": r.integers(0, 24, n), "raining": r.integers(0,2,n),
"prep_min": r.normal(15, 5, n), "rider_trips": r.poisson(6, n), "items": r.integers(1, 8, n)})
s = (.4 * X.distance_km + 1.2 * np.cos((X.hour - 13) * np.pi / 12) + 1.5 * X.raining * (X.distance_km > 5)
+ .1 * X.prep_min - .15 * X.rider_trips)
return X, (s + r.normal(0, .9, n) > 3.2).astype(int) # 1 = late
X, y = orders(2300, 42); tr, te = slice(0, 2000), slice(2000, None) # 2,000 train · 300 test
X_next, y_next = orders(5000, 7) # next month: 5,000 orders nobody tuned on
from sklearn.ensemble import HistGradientBoostingClassifier as HGB # gradient boosting, episode 35
def booster(**p): return HGB(max_iter=50, random_state=0, **p) # 50 boosting rounds: quick to train
import optuna; optuna.logging.set_verbosity(optuna.logging.WARNING) # quiet: no line per trial
from sklearn.model_selection import cross_val_score
def objective(trial): # one trial = one setting, scored by CV
p = dict(learning_rate=trial.suggest_float("learning_rate", 0.005, 0.5, log=True),
max_leaf_nodes=trial.suggest_int("max_leaf_nodes", 2, 64),
min_samples_leaf=trial.suggest_int("min_samples_leaf", 1, 100))
return cross_val_score(booster(**p), X[tr], y[tr], cv=3).mean()
study = optuna.create_study(direction="maximize", sampler=optuna.samplers.TPESampler(seed=0))
t = time.time(); study.optimize(objective, n_trials=30); v = study.trials_dataframe().value
print(f"30 trials, {time.time() - t:.1f} s. best CV after 5 / 15 / 30 trials:",
[float(v[:k].max().round(3)) for k in (5, 15, 30)])
print("trials scoring 0.85+: first 10:", (v[:10] >= .85).sum(), " last 10:", (v[-10:] >= .85).sum())
best = booster(**study.best_params).fit(X[tr], y[tr])
print("best:", {k: round(v, 3) for k, v in study.best_params.items()})
print("next month:", round(best.score(X_next, y_next), 3))
Output:
30 trials, 3.4 s. best CV after 5 / 15 / 30 trials: [0.851, 0.855, 0.855]
trials scoring 0.85+: first 10: 3 last 10: 5
best: {'learning_rate': 0.106, 'max_leaf_nodes': 18, 'min_samples_leaf': 27}
next month: 0.86
The best CV rose 0.851 → 0.855 as trials went on, and TPE learned where to look: 3 good trials (CV ≥ 0.85) among the first 10, 5 among the last 10. Next month: 0.86. On a small, fast problem like this every honest method lands near 0.86; Bayesian optimisation pays off when each trial is expensive (a model that trains for an hour).
Pruning: stop hopeless trials early
Many trials are obviously bad halfway through. In cricket trials, a batter out first ball twice in the nets doesn't get all ten overs. Stopping a bad trial early is pruning.
- The trial trains fold by fold and, after each fold, calls
trial.report(score_so_far, step). Those are its intermediate values. trial.should_prune()asks the pruner; if it says yes,raise optuna.TrialPruned()ends the trial.MedianPrunerprunes when the trial's best intermediate result is worse than the median of earlier trials at the same step.n_startup_trials=5means the first 5 trials always run in full, so there is something to compare with.
import numpy as np, pandas as pd, time
def orders(n, seed): # Kiran's orders (made up)
r = np.random.default_rng(seed)
X = pd.DataFrame({"distance_km": r.uniform(.5, 9, n), "hour": r.integers(0, 24, n), "raining": r.integers(0,2,n),
"prep_min": r.normal(15, 5, n), "rider_trips": r.poisson(6, n), "items": r.integers(1, 8, n)})
s = (.4 * X.distance_km + 1.2 * np.cos((X.hour - 13) * np.pi / 12) + 1.5 * X.raining * (X.distance_km > 5)
+ .1 * X.prep_min - .15 * X.rider_trips)
return X, (s + r.normal(0, .9, n) > 3.2).astype(int) # 1 = late
X, y = orders(2300, 42); tr, te = slice(0, 2000), slice(2000, None) # 2,000 train · 300 test
X_next, y_next = orders(5000, 7) # next month: 5,000 orders nobody tuned on
from sklearn.ensemble import HistGradientBoostingClassifier as HGB # gradient boosting, episode 35
def booster(**p): return HGB(max_iter=50, random_state=0, **p) # 50 boosting rounds: quick to train
import optuna; optuna.logging.set_verbosity(optuna.logging.WARNING)
from sklearn.model_selection import StratifiedKFold; folds = list(StratifiedKFold(3).split(X[tr], y[tr]))
def objective(trial):
m = booster(learning_rate=trial.suggest_float("learning_rate", 0.001, 0.5, log=True),
max_leaf_nodes=trial.suggest_int("max_leaf_nodes", 2, 64))
scores = []
for step, (a, b) in enumerate(folds): # report after every fold ...
scores.append(m.fit(X[tr].iloc[a], y[tr].iloc[a]).score(X[tr].iloc[b], y[tr].iloc[b]))
trial.report(np.mean(scores), step)
if trial.should_prune(): raise optuna.TrialPruned() # ... and stop a hopeless trial early
return np.mean(scores)
pruner = optuna.pruners.MedianPruner(n_startup_trials=5) # below the median at this fold? stop
st = optuna.create_study(direction="maximize", sampler=optuna.samplers.TPESampler(seed=0), pruner=pruner)
t = time.time(); st.optimize(objective, n_trials=20); fits = sum(len(t_.intermediate_values) for t_ in st.trials)
pruned = sum(t_.state == optuna.trial.TrialState.PRUNED for t_ in st.trials)
print(f"pruned {pruned} of 20 trials fits {fits} of 60 best CV {st.best_value:.3f} {time.time() - t:.1f} s")
Output:
pruned 8 of 20 trials fits 44 of 60 best CV 0.857 2.3 s
8 of 20 trials pruned, 44 fits instead of 60, and the best CV is still 0.857.
Successive halving and Hyperband
Successive halving is a tournament (auditions, semi-final, final). Give many settings a small resource (here training rows; it can also be boosting rounds or epochs), keep the best fraction, and give the survivors more. With factor=3: 18 settings on 200 rows → best 6 on 600 rows → best 2 on 1,800 rows. The risk: a setting that starts slowly but finishes strongly is dropped early, and you must choose how many settings to start with versus how much budget each gets.
Hyperband (Li et al., 2018) handles that trade-off by running several successive-halving brackets with different balances: many settings with tiny budgets, few settings with big budgets, and some in between. Optuna offers it as optuna.pruners.HyperbandPruner.
import numpy as np, pandas as pd, time
def orders(n, seed): # Kiran's orders (made up)
r = np.random.default_rng(seed)
X = pd.DataFrame({"distance_km": r.uniform(.5, 9, n), "hour": r.integers(0, 24, n), "raining": r.integers(0,2,n),
"prep_min": r.normal(15, 5, n), "rider_trips": r.poisson(6, n), "items": r.integers(1, 8, n)})
s = (.4 * X.distance_km + 1.2 * np.cos((X.hour - 13) * np.pi / 12) + 1.5 * X.raining * (X.distance_km > 5)
+ .1 * X.prep_min - .15 * X.rider_trips)
return X, (s + r.normal(0, .9, n) > 3.2).astype(int) # 1 = late
X, y = orders(2300, 42); tr, te = slice(0, 2000), slice(2000, None) # 2,000 train · 300 test
X_next, y_next = orders(5000, 7) # next month: 5,000 orders nobody tuned on
from sklearn.ensemble import HistGradientBoostingClassifier as HGB # gradient boosting, episode 35
def booster(**p): return HGB(max_iter=50, random_state=0, **p) # 50 boosting rounds: quick to train
from scipy.stats import loguniform, randint
try: from sklearn.model_selection import HalvingRandomSearchCV
except ImportError as e: print("ImportError:", str(e)[:105], "...")
from sklearn.experimental import enable_halving_search_cv # switch on the experimental feature
from sklearn.model_selection import HalvingRandomSearchCV
space = {"learning_rate": loguniform(0.005, 0.5), "max_leaf_nodes": randint(2, 64),
"min_samples_leaf": randint(1, 100)}
t = time.time(); hs = HalvingRandomSearchCV(booster(), space, n_candidates=18, factor=3,
cv=3, min_resources=200, random_state=0).fit(X[tr], y[tr])
print("candidates per round:", hs.n_candidates_, " rows per round:", hs.n_resources_, f" {time.time() - t:.1f} s")
print("best CV", round(hs.best_score_, 3), " next month", round(hs.score(X_next, y_next), 3))
Output:
ImportError: HalvingRandomSearchCV is experimental and the API might change without any deprecation cycle. To use it, ...
candidates per round: [18, 6, 2] rows per round: [200, 600, 1800] 1.6 s
best CV 0.853 next month 0.86
ImportError: HalvingRandomSearchCV is experimental and the API might change without any deprecation cycle. Experimental means it can change without warning, so you opt in with from sklearn.experimental import enable_halving_search_cv first. Then: 18 → 6 → 2 candidates on 200 → 600 → 1,800 rows, next month 0.86.
The three pruners side by side (from Practice 4):
NopPruner pruned 0 of 20 fits 60 of 60 best CV 0.857
MedianPruner pruned 8 of 20 fits 44 of 60 best CV 0.857
HyperbandPruner pruned 5 of 20 fits 55 of 60 best CV 0.857
Tuning a pipeline: parameter names
import numpy as np, pandas as pd, time
def orders(n, seed): # Kiran's orders (made up)
r = np.random.default_rng(seed)
X = pd.DataFrame({"distance_km": r.uniform(.5, 9, n), "hour": r.integers(0, 24, n), "raining": r.integers(0,2,n),
"prep_min": r.normal(15, 5, n), "rider_trips": r.poisson(6, n), "items": r.integers(1, 8, n)})
s = (.4 * X.distance_km + 1.2 * np.cos((X.hour - 13) * np.pi / 12) + 1.5 * X.raining * (X.distance_km > 5)
+ .1 * X.prep_min - .15 * X.rider_trips)
return X, (s + r.normal(0, .9, n) > 3.2).astype(int) # 1 = late
X, y = orders(2300, 42); tr, te = slice(0, 2000), slice(2000, None) # 2,000 train · 300 test
X_next, y_next = orders(5000, 7) # next month: 5,000 orders nobody tuned on
from sklearn.ensemble import HistGradientBoostingClassifier as HGB # gradient boosting, episode 35
def booster(**p): return HGB(max_iter=50, random_state=0, **p) # 50 boosting rounds: quick to train
from sklearn.model_selection import GridSearchCV; from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
model = make_pipeline(StandardScaler(), booster())
try: GridSearchCV(model, {"learning_rate": [0.03, 0.1]}, cv=3).fit(X[tr], y[tr]) # MISTAKE: plain name
except ValueError as e: msg = " ".join(str(e).split()); print("ValueError:", msg[:56], "..."); print(msg[-72:])
print([k for k in model.get_params() if "learning" in k]) # the real name: step__parameter
g = GridSearchCV(model, {"histgradientboostingclassifier__learning_rate": [0.03, 0.1]}, cv=3).fit(X[tr], y[tr])
print("works:", g.best_params_)
Output:
ValueError: Invalid parameter 'learning_rate' for estimator Pipeline ...
Valid parameters are: ['memory', 'steps', 'transform_input', 'verbose'].
['histgradientboostingclassifier__learning_rate']
works: {'histgradientboostingclassifier__learning_rate': 0.1}
ValueError: Invalid parameter 'learning_rate' for estimator Pipeline(...). Valid parameters are: ['memory', 'steps', 'transform_input', 'verbose']. Inside a pipeline, a parameter's name is step name + two underscores + parameter name: histgradientboostingclassifier__learning_rate. model.get_params() lists every valid name.
Common mistakes
| Mistake | Exact message / symptom | Why | Fix |
|---|---|---|---|
| Choose settings by test score | test 0.900, next month 0.861 | Picks the luckiest setting | CV on train, test once |
| Best value on the grid's edge | learning_rate 0.3 with 3 leaves | The real best may be outside | Widen the space |
| Linear range for a learning rate | most draws near the top of the range | Its effect is multiplicative | loguniform / log=True |
| Importing halving search directly | ImportError: HalvingRandomSearchCV is experimental ... |
Opt-in feature | enable_halving_search_cv first |
| Plain name in a pipeline | ValueError: Invalid parameter 'learning_rate' for estimator Pipeline ... |
Names need the step | step__parameter |
Reporting best_score_ as the final score |
slightly optimistic | It was chosen as the best of many | Test once, or nested CV (Episode 26) |
| Too many trials on small data | CV keeps rising, next month doesn't | Tuning to the CV noise | Fewer trials, more data, or nested CV |
Mini project: a fair tuning report
Write tune_all(X_train, y_train, X_test, y_test) that runs grid search, random search (same number of fits), Optuna (30 trials) and halving search on the same model, then prints for each: number of fits, best CV score, chosen settings, and the test score computed once at the end. Add nested cross-validation (Episode 26: an outer CV loop around the whole search) to estimate how good the tuning procedure is, not just one tuned model.
Practice
1. Widen the grid around the old edge (learning rates up to 1.0, fewer leaves).
Solution
import numpy as np, pandas as pd, time
def orders(n, seed): # Kiran's orders (made up)
r = np.random.default_rng(seed)
X = pd.DataFrame({"distance_km": r.uniform(.5, 9, n), "hour": r.integers(0, 24, n), "raining": r.integers(0,2,n),
"prep_min": r.normal(15, 5, n), "rider_trips": r.poisson(6, n), "items": r.integers(1, 8, n)})
s = (.4 * X.distance_km + 1.2 * np.cos((X.hour - 13) * np.pi / 12) + 1.5 * X.raining * (X.distance_km > 5)
+ .1 * X.prep_min - .15 * X.rider_trips)
return X, (s + r.normal(0, .9, n) > 3.2).astype(int) # 1 = late
X, y = orders(2300, 42); tr, te = slice(0, 2000), slice(2000, None) # 2,000 train · 300 test
X_next, y_next = orders(5000, 7) # next month: 5,000 orders nobody tuned on
from sklearn.ensemble import HistGradientBoostingClassifier as HGB # gradient boosting, episode 35
def booster(**p): return HGB(max_iter=50, random_state=0, **p) # 50 boosting rounds: quick to train
from sklearn.model_selection import GridSearchCV
space = {"learning_rate": [0.1, 0.3, 0.6, 1.0], "max_leaf_nodes": [2, 3, 4, 7]} # wider, around the old edge
grid = GridSearchCV(booster(), space, cv=3).fit(X[tr], y[tr])
print("best:", grid.best_params_, " CV", round(grid.best_score_, 3), " next month", round(grid.score(X_next, y_next), 3))
Output:
best: {'learning_rate': 0.3, 'max_leaf_nodes': 3} CV 0.855 next month 0.863
The wider grid confirms the same best (0.3, 3 leaves): it was on the edge of the old grid but is not beaten outside it.
2. Random search with 64 trials instead of 16: does it help?
Solution
import numpy as np, pandas as pd, time
def orders(n, seed): # Kiran's orders (made up)
r = np.random.default_rng(seed)
X = pd.DataFrame({"distance_km": r.uniform(.5, 9, n), "hour": r.integers(0, 24, n), "raining": r.integers(0,2,n),
"prep_min": r.normal(15, 5, n), "rider_trips": r.poisson(6, n), "items": r.integers(1, 8, n)})
s = (.4 * X.distance_km + 1.2 * np.cos((X.hour - 13) * np.pi / 12) + 1.5 * X.raining * (X.distance_km > 5)
+ .1 * X.prep_min - .15 * X.rider_trips)
return X, (s + r.normal(0, .9, n) > 3.2).astype(int) # 1 = late
X, y = orders(2300, 42); tr, te = slice(0, 2000), slice(2000, None) # 2,000 train · 300 test
X_next, y_next = orders(5000, 7) # next month: 5,000 orders nobody tuned on
from sklearn.ensemble import HistGradientBoostingClassifier as HGB # gradient boosting, episode 35
def booster(**p): return HGB(max_iter=50, random_state=0, **p) # 50 boosting rounds: quick to train
from sklearn.model_selection import RandomizedSearchCV; from scipy.stats import loguniform, randint
space = {"learning_rate": loguniform(0.005, 0.5), "max_leaf_nodes": randint(2, 64),
"min_samples_leaf": randint(1, 100), "l2_regularization": loguniform(1e-3, 10)}
for n in [16, 64]:
rs = RandomizedSearchCV(booster(), space, n_iter=n, cv=3, random_state=0).fit(X[tr], y[tr])
print(f"n_iter {n:>2}: best CV {rs.best_score_:.3f} next month {rs.score(X_next, y_next):.3f}")
Output:
n_iter 16: best CV 0.854 next month 0.861
n_iter 64: best CV 0.859 next month 0.861
Best CV rose from 0.854 to 0.859, but next month stayed 0.861. More trials found a setting that fits the CV folds slightly better, not a better model: after a point you are tuning to noise.
3. Add l2_regularization (log scale) to the Optuna search.
Solution
import numpy as np, pandas as pd, time
def orders(n, seed): # Kiran's orders (made up)
r = np.random.default_rng(seed)
X = pd.DataFrame({"distance_km": r.uniform(.5, 9, n), "hour": r.integers(0, 24, n), "raining": r.integers(0,2,n),
"prep_min": r.normal(15, 5, n), "rider_trips": r.poisson(6, n), "items": r.integers(1, 8, n)})
s = (.4 * X.distance_km + 1.2 * np.cos((X.hour - 13) * np.pi / 12) + 1.5 * X.raining * (X.distance_km > 5)
+ .1 * X.prep_min - .15 * X.rider_trips)
return X, (s + r.normal(0, .9, n) > 3.2).astype(int) # 1 = late
X, y = orders(2300, 42); tr, te = slice(0, 2000), slice(2000, None) # 2,000 train · 300 test
X_next, y_next = orders(5000, 7) # next month: 5,000 orders nobody tuned on
from sklearn.ensemble import HistGradientBoostingClassifier as HGB # gradient boosting, episode 35
def booster(**p): return HGB(max_iter=50, random_state=0, **p) # 50 boosting rounds: quick to train
import optuna; optuna.logging.set_verbosity(optuna.logging.WARNING)
from sklearn.model_selection import cross_val_score
def objective(trial):
p = dict(learning_rate=trial.suggest_float("learning_rate", 0.005, 0.5, log=True),
max_leaf_nodes=trial.suggest_int("max_leaf_nodes", 2, 64),
min_samples_leaf=trial.suggest_int("min_samples_leaf", 1, 100),
l2_regularization=trial.suggest_float("l2_regularization", 1e-3, 10, log=True))
return cross_val_score(booster(**p), X[tr], y[tr], cv=3).mean()
study = optuna.create_study(direction="maximize", sampler=optuna.samplers.TPESampler(seed=0))
study.optimize(objective, n_trials=40)
print("best CV", round(study.best_value, 3), {k: round(v, 3) for k, v in study.best_params.items()})
print("next month", round(booster(**study.best_params).fit(X[tr], y[tr]).score(X_next, y_next), 3))
Output:
best CV 0.856 {'learning_rate': 0.19, 'max_leaf_nodes': 30, 'min_samples_leaf': 84, 'l2_regularization': 3.659}
next month 0.857
The best CV is 0.856 with a fairly strong L2 penalty, and next month is 0.857: the same plateau. On this data, every reasonable setting is within about 0.005 of the best; the next gains would come from better features, not more tuning.
4. Compare no pruning, the median pruner and the Hyperband pruner.
Solution
import numpy as np, pandas as pd, time
def orders(n, seed): # Kiran's orders (made up)
r = np.random.default_rng(seed)
X = pd.DataFrame({"distance_km": r.uniform(.5, 9, n), "hour": r.integers(0, 24, n), "raining": r.integers(0,2,n),
"prep_min": r.normal(15, 5, n), "rider_trips": r.poisson(6, n), "items": r.integers(1, 8, n)})
s = (.4 * X.distance_km + 1.2 * np.cos((X.hour - 13) * np.pi / 12) + 1.5 * X.raining * (X.distance_km > 5)
+ .1 * X.prep_min - .15 * X.rider_trips)
return X, (s + r.normal(0, .9, n) > 3.2).astype(int) # 1 = late
X, y = orders(2300, 42); tr, te = slice(0, 2000), slice(2000, None) # 2,000 train · 300 test
X_next, y_next = orders(5000, 7) # next month: 5,000 orders nobody tuned on
from sklearn.ensemble import HistGradientBoostingClassifier as HGB # gradient boosting, episode 35
def booster(**p): return HGB(max_iter=50, random_state=0, **p) # 50 boosting rounds: quick to train
import optuna; optuna.logging.set_verbosity(optuna.logging.WARNING)
from sklearn.model_selection import StratifiedKFold; folds = list(StratifiedKFold(3).split(X[tr], y[tr]))
def objective(trial):
m = booster(learning_rate=trial.suggest_float("learning_rate", 0.001, 0.5, log=True),
max_leaf_nodes=trial.suggest_int("max_leaf_nodes", 2, 64))
scores = []
for step, (a, b) in enumerate(folds):
scores.append(m.fit(X[tr].iloc[a], y[tr].iloc[a]).score(X[tr].iloc[b], y[tr].iloc[b]))
trial.report(np.mean(scores), step)
if trial.should_prune(): raise optuna.TrialPruned()
return np.mean(scores)
for pruner in [optuna.pruners.NopPruner(), optuna.pruners.MedianPruner(n_startup_trials=5),
optuna.pruners.HyperbandPruner(min_resource=1, max_resource=3)]:
st = optuna.create_study(direction="maximize", sampler=optuna.samplers.TPESampler(seed=0), pruner=pruner)
st.optimize(objective, n_trials=20); fits = sum(len(t_.intermediate_values) for t_ in st.trials)
pruned = sum(t_.state == optuna.trial.TrialState.PRUNED for t_ in st.trials)
print(f"{type(pruner).__name__:<15} pruned {pruned:>2} of 20 fits {fits:>2} of 60 best CV {st.best_value:.3f}")
Output:
NopPruner pruned 0 of 20 fits 60 of 60 best CV 0.857
MedianPruner pruned 8 of 20 fits 44 of 60 best CV 0.857
HyperbandPruner pruned 5 of 20 fits 55 of 60 best CV 0.857
Both pruners found the same best CV as running everything (0.857) with fewer fits. With only 3 steps per trial, Hyperband's brackets have little room to work; it shines when the resource has many steps (hundreds of boosting rounds or training epochs).
Quick quiz
GridSearchCV with 3 hyperparameters of 4 values each and cv=5: how many fits, not counting the final refit? Answer: 320. 4 × 4 × 4 = 64 settings, each fitted on 5 folds, plus one refit of the best setting on all training rows.
Interview questions
- Parameter vs hyperparameter? Learned by training vs chosen before training.
- Why not tune on the test set? Choosing the best of many test scores selects luck; the test score is then optimistic (0.900 vs 0.861).
- Grid search vs random search? Grid tries every combination (explodes, repeats values); random draws from ranges, so the same budget explores more values of the important hyperparameters (Bergstra & Bengio, 2012).
- Why a log scale for the learning rate or regularisation strength? Their effect is multiplicative; a log scale spreads trials evenly over orders of magnitude.
- What is Bayesian optimisation? Using a surrogate model of past trials to choose the next trial, balancing exploration and exploitation.
- How does TPE choose the next value? It models the densities of good and bad trials and picks values likely under the good one and unlikely under the bad one.
- What is pruning in Optuna? Stopping a trial early when its intermediate results are worse than earlier trials (median pruner) via
trial.reportandshould_prune. - Explain successive halving and Hyperband. Many candidates on small budgets, keep the best fraction, increase the budget; Hyperband runs several such brackets with different start sizes.
- How do you name a hyperparameter inside a pipeline?
stepname__parameter. - How do you report a tuned model's performance honestly? Test once after tuning, or nested cross-validation.
Glossary
| Term | Meaning, with an example |
|---|---|
| Hyperparameter | A setting chosen before training: learning_rate=0.1 |
| Parameter | A value learned by training: a tree's split thresholds |
| Hyperparameter tuning | Searching for good hyperparameter values |
| Search space | The values allowed for each hyperparameter: learning rate 0.005 to 0.5 |
| Trial (candidate) | One setting that is tried and scored |
| Test-set leakage in tuning | Choosing settings by test score, which makes that score optimistic |
Grid search (GridSearchCV) |
Try every combination of listed values: 4 × 4 = 16 settings |
| Combinatorial explosion | Settings multiply: 5 hyperparameters × 5 values = 3,125 |
Random search (RandomizedSearchCV) |
Draw each value at random from a range, n_iter times |
| Distribution | A range plus how values are drawn from it: randint, loguniform |
Log scale (loguniform, log=True) |
Equal chances per order of magnitude: 0.005→0.05 as likely as 0.05→0.5 |
best_params_, best_score_ |
The winning setting and its mean CV score |
| Refit | Training the best setting again on all training rows |
| Bayesian optimisation | Choosing each next trial using a model of the past trials |
| Surrogate model | A cheap model that guesses how well a setting will score |
| Exploration vs exploitation | Trying new regions vs refining near the best so far |
| Optuna | Open-source hyperparameter optimisation library |
| Objective function | The function Optuna maximises: suggest values, train, return a CV score |
| Study | One whole Optuna search: all its trials |
| TPE (tree-structured Parzen estimator) | Optuna's default sampler: favours values common among the best trials |
| Pruning | Stopping a hopeless trial early |
| Intermediate value | A score reported during a trial with trial.report |
MedianPruner |
Prunes a trial whose result is below the median of earlier trials at the same step |
| Resource (budget) | What a trial consumes: training rows, boosting rounds, epochs |
| Successive halving | Many candidates on a small budget, keep the best fraction, give them more |
factor |
How strongly successive halving cuts each round (3 = keep the best third) |
| Hyperband | Several successive-halving brackets with different trade-offs (Li et al., 2018) |
| Experimental feature | An API that may change without warning, enabled explicitly |
step__parameter |
How to name a hyperparameter inside a pipeline |
| Nested cross-validation | An outer CV around the whole tuning process, for an honest estimate |
Summary
- Choose hyperparameters with cross-validation on training rows; test once (tuning on the test set reported 0.900 for a 0.861 model).
- Grid search: every combination; explodes and repeats values. Random search: same budget, many more distinct values.
- Bayesian optimisation (Optuna TPE) learns where good settings are; it pays off when trials are expensive.
- Pruning, successive halving and Hyperband spend less on losing settings.
- Inside a pipeline, use
step__parameternames. - Past a point, more tuning only fits the CV noise; better features and more data matter more.
Next episode: k-means from scratch, our first unsupervised algorithm: clustering without labels.