Machine Learning Full Course · Episode 37. Every code sample below was run with Python 3.13, NumPy 2.5.3 and scikit-learn 1.9.1; the output shown is the real output (fit times will differ on your machine). Sunita's 8,000 tiffin customers are made up for teaching; the mini project and one practice question use datasets bundled with scikit-learn.
The big idea: when several good models disagree, a second model can learn whom to trust. The first models are the base learners (level 0); their predictions become the input columns of a meta-learner (level 1). This is stacking (stacked generalisation, Wolpert 1992). The obvious way, training the meta-learner on predictions for the rows the base models were trained on, fails: models that memorised those rows look perfect there, so the meta-learner trusts them blindly (test 0.828, worse than every single model). The fix is out-of-fold predictions: every training row is predicted by a model that never saw it (test 0.852, better than every single model). scikit-learn's StackingClassifier does all of it. Blending is the cheaper version with one holdout set. Stacking only helps when the models are diverse (make different mistakes), and it is worth it only when a small gain pays for extra training time, slower predictions and more complexity.
Sources for the facts in this episode: the scikit-learn StackingClassifier documentation (defaults for cv, final_estimator, stack_method, passthrough; added in version 0.22), Wolpert, D. H. (1992), "Stacked generalization", Neural Networks 5(2), 241-259 (citation), and the Netflix Prize article.
Words you need first
- Model, training, prediction, feature, label: Episode 1.
- Logistic regression and
predict_proba(a predicted probability for each class): Episodes 22-23 and 17. - k-nearest neighbours (kNN): predict from the k most similar training rows (Episode 27).
- Random forest: many decorrelated trees that vote (Episode 34).
- Overfitting: a model that memorises its training rows and does worse on new rows (Episode 20).
- Train / test split, holdout, leakage: Episode 15.
- k-fold cross-validation: split the data into k parts, train on k-1, test on the remaining one, k times (Episode 26).
- Ensemble, hard and soft voting: several models combined into one answer (Episode 33).
- Scaling (
StandardScaler),Pipeline/make_pipeline: Episodes 16-17. - Correlation: a number from -1 to 1 saying how closely two lists move together (Episode 11).
Three models disagree
Real-life example. Sunita wants to know which customers will renew next month. She asks three advisers: her accountant adds the numbers up in a straight line (logistic regression), her neighbour says "customers like this one usually renew" (k-nearest neighbours), and her delivery team has a hundred rules of thumb and votes (random forest). Usually they agree. When they don't, whom should she believe?
Data: 8,000 made-up customers with six number columns (missed meals, rating, distance in km, months as a customer, price in rupees, calls to support) and a hidden made-up rule that mixes straight-line effects with two "if this and that" effects. 5,000 rows for training, 3,000 for the test.
import numpy as np; rng = np.random.default_rng(37); n = 8000 # Sunita's customers (made up)
X = np.column_stack([rng.poisson(3,n), rng.uniform(1,5,n), rng.uniform(0,12,n), rng.integers(1,36,n),
rng.uniform(60,160,n), rng.poisson(2,n)]) # missed, rating, km, months, price, calls
s = X @ [-.5,.6,-.15,.04,-.02,-.3] + 2*(X[:,1]>3.5)*(X[:,2]<4) - 1.5*(X[:,0]>5)*(X[:,3]<6) # hidden rule
y = (s + rng.normal(0, .8, n) > np.median(s)).astype(int) # 1 = renews next month
X_tr, X_te, y_tr, y_te = X[:5000], X[5000:], y[:5000], y[5000:]
from sklearn.linear_model import LogisticRegression as LR
from sklearn.neighbors import KNeighborsClassifier as KNN
from sklearn.ensemble import RandomForestClassifier as RF; from sklearn.preprocessing import StandardScaler as SS
from sklearn.pipeline import make_pipeline as mp; pr = lambda m, A: m.predict_proba(A)[:, 1] # P(renews)
base = [("logreg", mp(SS(), LR())), ("knn", mp(SS(), KNN(25))),
("forest", RF(100, min_samples_leaf=3, random_state=0))]
P = []
for name, m in base:
m.fit(X_tr, y_tr); P.append(m.predict(X_te))
print(f"{name:<7} train {m.score(X_tr, y_tr):.3f} test {m.score(X_te, y_te):.3f}")
P = np.array(P); right = (P == y_te)
print(f"the 3 models disagree on {(P.min(0) != P.max(0)).mean():.1%} of test customers")
print(f"at least one model is right on {right.any(0).mean():.1%}")
Output:
logreg train 0.845 test 0.850
knn train 0.862 test 0.839
forest train 0.963 test 0.846
the 3 models disagree on 11.8% of test customers
at least one model is right on 90.1%
The models are close (0.850, 0.839, 0.846) but they disagree on 11.8% of test customers, and at least one is right on 90.1%. The best single model gets 85.0%, so the right answer is often "in the room". Picking only the best model throws that information away.
Base learners and a meta-learner
Real-life example. Three teachers each predict whether a student will pass. Their three predictions go to the principal, who over the years compares them with the real results and learns whom to trust, and how much.
| School | Machine learning |
|---|---|
| Three teachers predict a result | Base learners (level 0): the first models |
| Their predictions go to the principal | Predictions as features: one column per base model |
| The principal learns whom to trust | Meta-learner (level 1): a model trained on those columns |
- An ensemble is several models giving one combined answer (Episode 33). Averaging and voting combine them with fixed, equal trust.
- Stacking (also called stacked generalisation, described by David Wolpert in Neural Networks, 1992) learns the combination. For one customer the base learners might say
[0.81, 0.62, 0.90](probability of renewing); the meta-learner turns those three numbers into one final answer. - "Meta" means "one level above": the meta-learner is a model about the other models' predictions.
The obvious way fails
The obvious way: fit the three models on the training rows, predict those same training rows, and train a logistic regression on the predictions.
import numpy as np; rng = np.random.default_rng(37); n = 8000 # Sunita's customers (made up)
X = np.column_stack([rng.poisson(3,n), rng.uniform(1,5,n), rng.uniform(0,12,n), rng.integers(1,36,n),
rng.uniform(60,160,n), rng.poisson(2,n)]) # missed, rating, km, months, price, calls
s = X @ [-.5,.6,-.15,.04,-.02,-.3] + 2*(X[:,1]>3.5)*(X[:,2]<4) - 1.5*(X[:,0]>5)*(X[:,3]<6) # hidden rule
y = (s + rng.normal(0, .8, n) > np.median(s)).astype(int) # 1 = renews next month
X_tr, X_te, y_tr, y_te = X[:5000], X[5000:], y[:5000], y[5000:]
from sklearn.linear_model import LogisticRegression as LR
from sklearn.neighbors import KNeighborsClassifier as KNN
from sklearn.ensemble import RandomForestClassifier as RF; from sklearn.preprocessing import StandardScaler as SS
from sklearn.pipeline import make_pipeline as mp; pr = lambda m, A: m.predict_proba(A)[:, 1] # P(renews)
base = [("logreg", mp(SS(), LR())), ("knn", mp(SS(), KNN(25))),
("forest", RF(100, min_samples_leaf=3, random_state=0))]
Z_tr = np.column_stack([pr(m.fit(X_tr, y_tr), X_tr) for _, m in base]) # predictions on TRAIN rows
Z_te = np.column_stack([pr(m, X_te) for _, m in base])
meta = LR().fit(Z_tr, y_tr) # learn whom to trust
print("weights (logreg, knn, forest):", meta.coef_.round(2))
print(f"meta-learner test accuracy: {meta.score(Z_te, y_te):.3f}")
f = base[2][1]; print(f"forest: train {f.score(X_tr, y_tr):.3f} test {f.score(X_te, y_te):.3f}")
Output:
weights (logreg, knn, forest): [[-2.33 -3.7 17.64]]
meta-learner test accuracy: 0.828
forest: train 0.963 test 0.846
The meta-learner gives the forest a weight of 17.64 and even bets against the others (negative weights). Test accuracy 0.828: worse than every single model.
Why it failed.
- The meta-learner learned from predictions on the training rows.
- The base models were trained on those exact rows. The forest scores 0.963 there (it has memorised them) but only 0.846 on new rows: overfitting (Episode 20).
- So on the training rows the forest looks almost perfect and the meta-learner learns "trust the forest". This is leakage (Episode 15): the answers leaked into the meta-learner's inputs. It is a principal judging teachers on exam papers whose answers they had already seen.
- On new customers the forest is only average, so the learned trust is wrong.
We need predictions that each model made on rows it never saw during training, for every training row.
Out-of-fold predictions

Split the 5,000 training rows into 5 parts (folds) of 1,000. Round 1: train on parts 2-5, predict part 1. Round 2: train on parts 1, 3, 4, 5, predict part 2. After 5 rounds every row has exactly one prediction from a model that never saw it. These are out-of-fold (OOF) predictions. sklearn.model_selection.cross_val_predict does the 5 rounds.
For the test set, each base model is then refitted once on all training rows, and its test predictions go into the meta-learner.
import numpy as np; rng = np.random.default_rng(37); n = 8000 # Sunita's customers (made up)
X = np.column_stack([rng.poisson(3,n), rng.uniform(1,5,n), rng.uniform(0,12,n), rng.integers(1,36,n),
rng.uniform(60,160,n), rng.poisson(2,n)]) # missed, rating, km, months, price, calls
s = X @ [-.5,.6,-.15,.04,-.02,-.3] + 2*(X[:,1]>3.5)*(X[:,2]<4) - 1.5*(X[:,0]>5)*(X[:,3]<6) # hidden rule
y = (s + rng.normal(0, .8, n) > np.median(s)).astype(int) # 1 = renews next month
X_tr, X_te, y_tr, y_te = X[:5000], X[5000:], y[:5000], y[5000:]
from sklearn.linear_model import LogisticRegression as LR
from sklearn.neighbors import KNeighborsClassifier as KNN
from sklearn.ensemble import RandomForestClassifier as RF; from sklearn.preprocessing import StandardScaler as SS
from sklearn.pipeline import make_pipeline as mp; pr = lambda m, A: m.predict_proba(A)[:, 1] # P(renews)
base = [("logreg", mp(SS(), LR())), ("knn", mp(SS(), KNN(25))),
("forest", RF(100, min_samples_leaf=3, random_state=0))]
from sklearn.model_selection import cross_val_predict as cvp
Z_tr = np.column_stack([cvp(m, X_tr, y_tr, cv=5, method="predict_proba")[:, 1]
for _, m in base]) # out-of-fold: never seen its own row
Z_te = np.column_stack([pr(m.fit(X_tr, y_tr), X_te) for _, m in base]) # then refit on all train rows
meta = LR().fit(Z_tr, y_tr)
print("weights (logreg, knn, forest):", meta.coef_.round(2))
print(f"meta-learner test accuracy: {meta.score(Z_te, y_te):.3f}")
Output:
weights (logreg, knn, forest): [[2.78 1.82 2.7 ]]
meta-learner test accuracy: 0.852
Now the weights are sensible (2.78, 1.82, 2.70: everyone gets a say) and test accuracy is 0.852, better than every single model and 72 more test customers right than the naive meta-learner. The gain over the best single model (0.850) is 0.2 percentage points: honest and typical. Stacking polishes good models; it does not rescue bad ones.
StackingClassifier
StackingClassifier(estimators=[("name", model), ...], final_estimator=None, cv=None,
stack_method="auto", passthrough=False)
| Parameter | What it does (from the scikit-learn docs) |
|---|---|
estimators |
The base learners, as a list of (name, model) pairs. Names must be unique. |
final_estimator |
The meta-learner. None (default) means a LogisticRegression. |
cv |
How the out-of-fold predictions are made. None (default) means 5-fold, via cross_val_predict; for a classifier the folds are stratified (StratifiedKFold, each fold keeps the same share of each class). "prefit" uses already-fitted base models without refitting (risky if they were fitted on the same data). |
stack_method |
Which predictions to pass up. "auto" (default) tries predict_proba, then decision_function, then predict. For binary classification with probabilities, one of the two columns is dropped because they always add up to 1. |
passthrough |
False (default): the meta-learner sees predictions only. True: it also sees the original columns. |
After fitting, estimators_ holds the base models refitted on all training rows and final_estimator_ holds the meta-learner. StackingClassifier was added in scikit-learn 0.22; StackingRegressor is the regression version.
Now add two weaker models, Gaussian naive Bayes (Episode 28) and a fully grown decision tree (Episode 30, which memorises its training rows), and compare a plain average of the probabilities (soft voting, Episode 33) with stacking:
import numpy as np; rng = np.random.default_rng(37); n = 8000 # Sunita's customers (made up)
X = np.column_stack([rng.poisson(3,n), rng.uniform(1,5,n), rng.uniform(0,12,n), rng.integers(1,36,n),
rng.uniform(60,160,n), rng.poisson(2,n)]) # missed, rating, km, months, price, calls
s = X @ [-.5,.6,-.15,.04,-.02,-.3] + 2*(X[:,1]>3.5)*(X[:,2]<4) - 1.5*(X[:,0]>5)*(X[:,3]<6) # hidden rule
y = (s + rng.normal(0, .8, n) > np.median(s)).astype(int) # 1 = renews next month
X_tr, X_te, y_tr, y_te = X[:5000], X[5000:], y[:5000], y[5000:]
from sklearn.linear_model import LogisticRegression as LR
from sklearn.neighbors import KNeighborsClassifier as KNN
from sklearn.ensemble import RandomForestClassifier as RF; from sklearn.preprocessing import StandardScaler as SS
from sklearn.pipeline import make_pipeline as mp; pr = lambda m, A: m.predict_proba(A)[:, 1] # P(renews)
base = [("logreg", mp(SS(), LR())), ("knn", mp(SS(), KNN(25))),
("forest", RF(100, min_samples_leaf=3, random_state=0))]
from sklearn.ensemble import StackingClassifier
from sklearn.naive_bayes import GaussianNB; from sklearn.tree import DecisionTreeClassifier
more = base + [("bayes", GaussianNB()), ("tree", DecisionTreeClassifier(random_state=0))] # 2 weaker models
avg = np.mean([pr(m.fit(X_tr, y_tr), X_te) for _, m in more], axis=0) # plain average (soft voting)
st = StackingClassifier(more, cv=5).fit(X_tr, y_tr) # final_estimator: LogisticRegression
print(f"5 models: average {((avg > .5) == y_te).mean():.3f} stack {st.score(X_te, y_te):.3f}")
for (name, _), w in zip(more, st.final_estimator_.coef_[0]): print(f" weight {name:<7}{w:5.2f}")
Output:
5 models: average 0.843 stack 0.850
weight logreg 1.81
weight knn 1.52
weight forest 2.58
weight bayes 1.82
weight tree 0.11
The average falls to 0.843 because it trusts every model equally, including the overfit tree. Stacking keeps 0.850 and gives the tree a weight of just 0.11: it learned to almost ignore it.
Blending
Next problem: cost. With 5 folds, every base model is trained 6 times (5 for the out-of-fold predictions, 1 on all rows). With big models that is slow.
Real-life example. Keep one mock exam aside. The teachers never see it while they prepare; the principal judges them only on that paper.
Blending splits the training rows into part A and a holdout. The base models are trained once on part A; the meta-learner is trained on their predictions for the holdout only.
import numpy as np; rng = np.random.default_rng(37); n = 8000 # Sunita's customers (made up)
X = np.column_stack([rng.poisson(3,n), rng.uniform(1,5,n), rng.uniform(0,12,n), rng.integers(1,36,n),
rng.uniform(60,160,n), rng.poisson(2,n)]) # missed, rating, km, months, price, calls
s = X @ [-.5,.6,-.15,.04,-.02,-.3] + 2*(X[:,1]>3.5)*(X[:,2]<4) - 1.5*(X[:,0]>5)*(X[:,3]<6) # hidden rule
y = (s + rng.normal(0, .8, n) > np.median(s)).astype(int) # 1 = renews next month
X_tr, X_te, y_tr, y_te = X[:5000], X[5000:], y[:5000], y[5000:]
from sklearn.linear_model import LogisticRegression as LR
from sklearn.neighbors import KNeighborsClassifier as KNN
from sklearn.ensemble import RandomForestClassifier as RF; from sklearn.preprocessing import StandardScaler as SS
from sklearn.pipeline import make_pipeline as mp; pr = lambda m, A: m.predict_proba(A)[:, 1] # P(renews)
base = [("logreg", mp(SS(), LR())), ("knn", mp(SS(), KNN(25))),
("forest", RF(100, min_samples_leaf=3, random_state=0))]
import time; from sklearn.ensemble import StackingClassifier
X_a, y_a, X_h, y_h = X_tr[:3500], y_tr[:3500], X_tr[3500:], y_tr[3500:] # 3,500 fit · 1,500 holdout
t = time.perf_counter(); fit = [m.fit(X_a, y_a) for _, m in base] # base models: once, on part A
Z = lambda A: np.column_stack([pr(m, A) for m in fit]); meta = LR().fit(Z(X_h), y_h) # meta: holdout only
print(f"blending: test {meta.score(Z(X_te), y_te):.3f} fit {time.perf_counter() - t:.1f} s"); t = time.perf_counter()
st = StackingClassifier(base, cv=5).fit(X_tr, y_tr)
print(f"stacking: test {st.score(X_te, y_te):.3f} fit {time.perf_counter() - t:.1f} s")
Output:
blending: test 0.850 fit 0.4 s
stacking: test 0.852 fit 2.0 s
Blending: 0.850 in about 0.4 s. Stacking: 0.852 in about 2 s (times vary between runs and machines).
| Stacking | Blending | |
|---|---|---|
| Meta-learner trains on | out-of-fold predictions for all training rows | predictions for a holdout only |
| Base-model fits | k + 1 per model | 1 per model |
| Data use | every row used for both levels | holdout rows never train the base models; the meta-learner sees only the holdout |
| Leak risk | low if done with OOF (easy to get wrong by hand) | very low: simple to do right |
| Use when | time allows; the usual default | very big data, very slow models, quick experiments |
Model diversity
Real-life example. A cooking contest judged by three cooks from the same kitchen gets three votes but really one opinion. A chef, a food critic and a regular customer notice different things. In an ensemble, model diversity means the models make different mistakes.
import numpy as np; rng = np.random.default_rng(37); n = 8000 # Sunita's customers (made up)
X = np.column_stack([rng.poisson(3,n), rng.uniform(1,5,n), rng.uniform(0,12,n), rng.integers(1,36,n),
rng.uniform(60,160,n), rng.poisson(2,n)]) # missed, rating, km, months, price, calls
s = X @ [-.5,.6,-.15,.04,-.02,-.3] + 2*(X[:,1]>3.5)*(X[:,2]<4) - 1.5*(X[:,0]>5)*(X[:,3]<6) # hidden rule
y = (s + rng.normal(0, .8, n) > np.median(s)).astype(int) # 1 = renews next month
X_tr, X_te, y_tr, y_te = X[:5000], X[5000:], y[:5000], y[5000:]
from sklearn.linear_model import LogisticRegression as LR
from sklearn.neighbors import KNeighborsClassifier as KNN
from sklearn.ensemble import RandomForestClassifier as RF; from sklearn.preprocessing import StandardScaler as SS
from sklearn.pipeline import make_pipeline as mp; pr = lambda m, A: m.predict_proba(A)[:, 1] # P(renews)
base = [("logreg", mp(SS(), LR())), ("knn", mp(SS(), KNN(25))),
("forest", RF(100, min_samples_leaf=3, random_state=0))]
same = [(f"forest{i}", RF(100, min_samples_leaf=3, random_state=i)) for i in range(3)] # 3 near-copies
for label, est in [("3 forests", same), ("3 diverse", base)]:
fit = [m.fit(X_tr, y_tr) for _, m in est]
r = np.corrcoef([pr(m, X_te) for m in fit])[np.triu_indices(3, 1)].mean() # how alike?
P = np.array([m.predict(X_te) for m in fit]); right = (P == y_te)
print(f"{label}: correlation {r:.3f} disagree on {(P.min(0) != P.max(0)).mean():.1%}")
print(f" best single {right.mean(1).max():.3f} at least one right {right.any(0).mean():.1%}")
Output:
3 forests: correlation 0.993 disagree on 4.2%
best single 0.846 at least one right 86.5%
3 diverse: correlation 0.949 disagree on 11.8%
best single 0.850 at least one right 90.1%
Three forests that differ only in their random seed have prediction correlation 0.993: they disagree on only 4.2% of customers, and at least one is right on 86.5%, barely above the best forest (84.6%). Three different model families have correlation 0.949, disagree on 11.8%, and at least one is right on 90.1%. A meta-learner can only choose between answers that are in the room; near-copies bring the same answer three times. (When we stacked three such forests while preparing this episode, the stack scored no better than the best single forest.)
How to get diversity:
- Mix model families: linear models, neighbours, trees, boosting, SVMs.
- Give models different features or different preprocessing.
- Measure it: the correlation between their predictions; close to 1 means near-copies.
- Diversity is not the same as being bad. Each base model should be decent on its own.
- Keep the meta-learner simple (logistic regression or ridge). It sees only a few columns, and a complex meta-learner can overfit the predictions all over again.
When ensembling is worth it

(The y-axis starts at 0.81, so the bars exaggerate the differences.)
Worth it when:
- Competitions: tiny gains decide the ranking. The Netflix Prize, a US$1,000,000 contest, was won in September 2009 by the team BellKor's Pragmatic Chaos, 10.06% better than Netflix's own Cinematch system; the leading teams usually used an ensemble of many methods (SVD, k-nearest neighbours, neural networks and more).
- Business problems where every 0.1% is worth real money: fraud detection, ad clicks, credit risk.
Costs:
- Training: k + 1 fits for every base model, plus the meta-learner.
- Prediction: every request runs all the models, so answers are slower and cost more to serve.
- Complexity: harder to explain, debug, monitor and retrain.
Rule of thumb: first improve the features (Episodes 38-40) and tune one strong model (Episode 42). Stack at the end, when the last bit of accuracy really pays.
Common mistakes
The live mistake from the video:
import numpy as np; rng = np.random.default_rng(37); n = 8000 # Sunita's customers (made up)
X = np.column_stack([rng.poisson(3,n), rng.uniform(1,5,n), rng.uniform(0,12,n), rng.integers(1,36,n),
rng.uniform(60,160,n), rng.poisson(2,n)]) # missed, rating, km, months, price, calls
s = X @ [-.5,.6,-.15,.04,-.02,-.3] + 2*(X[:,1]>3.5)*(X[:,2]<4) - 1.5*(X[:,0]>5)*(X[:,3]<6) # hidden rule
y = (s + rng.normal(0, .8, n) > np.median(s)).astype(int) # 1 = renews next month
X_tr, X_te, y_tr, y_te = X[:5000], X[5000:], y[:5000], y[5000:]
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.ensemble import StackingClassifier
st = StackingClassifier([LogisticRegression(), KNeighborsClassifier()]) # forgot the names
try: st.fit(X_tr, y_tr)
except TypeError as e: print("TypeError:", e)
Output:
TypeError: 'LogisticRegression' object is not iterable
| Mistake | What you see | Why | Fix |
|---|---|---|---|
| Meta-learner trained on predictions for the base models' own training rows | test 0.828, forest weight 17.64 | The base models had seen those rows (leakage) | Out-of-fold predictions (cross_val_predict, or StackingClassifier) |
| Models passed without names | TypeError: 'LogisticRegression' object is not iterable |
estimators must be (name, model) pairs |
[("lr", LogisticRegression()), ...] |
| Two models with the same name | ValueError: Names provided are not unique: ['lr', 'lr'] |
Names are used as keys | Give every model a unique name |
stack_method="predict_proba" with a model that has no probabilities |
ValueError: Underlying estimator svm does not implement the method predict_proba. |
e.g. LinearSVC has only decision_function |
Leave stack_method="auto" |
| Stacking near-copies of one model | 3 forests disagree on only 4.2% of customers | They make the same mistakes | Mix model families |
| Using the test set to train or tune the meta-learner | a test score that is too good | The test set is no longer unseen | Out-of-fold or a holdout carved from the training rows |
passthrough=True with an unscaled logistic-regression meta-learner |
ConvergenceWarning: lbfgs failed to converge |
Raw columns are on very different scales | final_estimator=make_pipeline(StandardScaler(), LogisticRegression()) |
Mini project: stacking on the breast cancer dataset
Stack a scaled logistic regression, a scaled SVM (Episode 32) and a random forest on scikit-learn's bundled breast cancer dataset (569 tumours, 30 measurements, benign or malignant), and compare all four with 5-fold cross-validation. SVC has no predict_proba by default, so stack_method="auto" uses its decision_function.
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import cross_val_score, StratifiedKFold
from sklearn.linear_model import LogisticRegression; from sklearn.svm import SVC
from sklearn.ensemble import RandomForestClassifier, StackingClassifier
from sklearn.pipeline import make_pipeline; from sklearn.preprocessing import StandardScaler
X, y = load_breast_cancer(return_X_y=True)
base = [("logreg", make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))),
("svm", make_pipeline(StandardScaler(), SVC(random_state=0))), # no predict_proba: auto uses decision_function
("forest", RandomForestClassifier(300, random_state=0))]
cv = StratifiedKFold(5, shuffle=True, random_state=0)
for name, m in base + [("stack", StackingClassifier(base))]:
s = cross_val_score(m, X, y, cv=cv)
print(f"{name:<7} accuracy {s.mean():.3f} ± {s.std():.3f}")
Output:
logreg accuracy 0.979 ± 0.014
svm accuracy 0.977 ± 0.007
forest accuracy 0.958 ± 0.017
stack accuracy 0.982 ± 0.012
The stack is the best on average (0.982), but the spread (± 0.012) is about as large as the gain, so the honest conclusion is "at least as good as the best single model". Try it with passthrough=True and with a RandomForestClassifier as the meta-learner, and compare.
Practice
1. Add passthrough=True to Sunita's stack. Does it help?
Solution
With the default logistic-regression meta-learner, the raw columns are unscaled and you get ConvergenceWarning: lbfgs failed to converge (the score is 0.852 either way). Scale inside the meta-learner:
import numpy as np; rng = np.random.default_rng(37); n = 8000 # Sunita's customers (made up)
X = np.column_stack([rng.poisson(3,n), rng.uniform(1,5,n), rng.uniform(0,12,n), rng.integers(1,36,n),
rng.uniform(60,160,n), rng.poisson(2,n)]) # missed, rating, km, months, price, calls
s = X @ [-.5,.6,-.15,.04,-.02,-.3] + 2*(X[:,1]>3.5)*(X[:,2]<4) - 1.5*(X[:,0]>5)*(X[:,3]<6) # hidden rule
y = (s + rng.normal(0, .8, n) > np.median(s)).astype(int) # 1 = renews next month
X_tr, X_te, y_tr, y_te = X[:5000], X[5000:], y[:5000], y[5000:]
from sklearn.linear_model import LogisticRegression as LR
from sklearn.neighbors import KNeighborsClassifier as KNN
from sklearn.ensemble import RandomForestClassifier as RF; from sklearn.preprocessing import StandardScaler as SS
from sklearn.pipeline import make_pipeline as mp; pr = lambda m, A: m.predict_proba(A)[:, 1] # P(renews)
base = [("logreg", mp(SS(), LR())), ("knn", mp(SS(), KNN(25))),
("forest", RF(100, min_samples_leaf=3, random_state=0))]
from sklearn.ensemble import StackingClassifier
meta = mp(SS(), LR()) # raw columns are on different scales: scale inside the meta-learner
for p in [False, True]:
st = StackingClassifier(base, final_estimator=meta, cv=5, passthrough=p).fit(X_tr, y_tr)
print(f"passthrough={p}: test {st.score(X_te, y_te):.3f}")
Output:
passthrough=False: test 0.852
passthrough=True: test 0.853
A tiny gain (0.852 → 0.853): the base models already capture most of what the raw columns say.
2. Use StackingRegressor on scikit-learn's diabetes dataset (442 patients, disease progression after one year). Compare R² with 5-fold cross-validation.
Solution
from sklearn.datasets import load_diabetes
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.linear_model import RidgeCV; from sklearn.neighbors import KNeighborsRegressor
from sklearn.ensemble import RandomForestRegressor, StackingRegressor
from sklearn.pipeline import make_pipeline; from sklearn.preprocessing import StandardScaler
X, y = load_diabetes(return_X_y=True)
base = [("ridge", RidgeCV()), ("knn", make_pipeline(StandardScaler(), KNeighborsRegressor(15))),
("forest", RandomForestRegressor(200, min_samples_leaf=5, random_state=0))]
for name, m in base + [("stack", StackingRegressor(base))]:
r2 = cross_val_score(m, X, y, cv=5, scoring="r2")
print(f"{name:<7} R2 {r2.mean():.3f} ± {r2.std():.3f}")
Output:
ridge R2 0.480 ± 0.047
knn R2 0.452 ± 0.063
forest R2 0.442 ± 0.053
stack R2 0.485 ± 0.053
The stack (R² 0.485) edges out the best single model (ridge, 0.480). For StackingRegressor the default meta-learner is RidgeCV. R² is from Episode 19: the share of the target's variation the model explains.
3. Blend Sunita's models with a 1,000-row and a 2,500-row holdout. What changes?
Solution
import numpy as np; rng = np.random.default_rng(37); n = 8000 # Sunita's customers (made up)
X = np.column_stack([rng.poisson(3,n), rng.uniform(1,5,n), rng.uniform(0,12,n), rng.integers(1,36,n),
rng.uniform(60,160,n), rng.poisson(2,n)]) # missed, rating, km, months, price, calls
s = X @ [-.5,.6,-.15,.04,-.02,-.3] + 2*(X[:,1]>3.5)*(X[:,2]<4) - 1.5*(X[:,0]>5)*(X[:,3]<6) # hidden rule
y = (s + rng.normal(0, .8, n) > np.median(s)).astype(int) # 1 = renews next month
X_tr, X_te, y_tr, y_te = X[:5000], X[5000:], y[:5000], y[5000:]
from sklearn.linear_model import LogisticRegression as LR
from sklearn.neighbors import KNeighborsClassifier as KNN
from sklearn.ensemble import RandomForestClassifier as RF; from sklearn.preprocessing import StandardScaler as SS
from sklearn.pipeline import make_pipeline as mp; pr = lambda m, A: m.predict_proba(A)[:, 1] # P(renews)
base = [("logreg", mp(SS(), LR())), ("knn", mp(SS(), KNN(25))),
("forest", RF(100, min_samples_leaf=3, random_state=0))]
for h in [1000, 2500]:
X_a, y_a, X_h, y_h = X_tr[:-h], y_tr[:-h], X_tr[-h:], y_tr[-h:]
fit = [m.fit(X_a, y_a) for _, m in base]
Z = lambda A: np.column_stack([pr(m, A) for m in fit])
meta = LR().fit(Z(X_h), y_h)
print(f"holdout {h}: base models see {len(X_a)} rows, blend test {meta.score(Z(X_te), y_te):.3f}")
Output:
holdout 1000: base models see 4000 rows, blend test 0.852
holdout 2500: base models see 2500 rows, blend test 0.851
A bigger holdout gives the meta-learner more rows but leaves fewer for the base models. Here both work about equally well; with less data, the trade-off hurts more.
4. Stack HistGradientBoostingClassifier (Episode 36's histogram boosting, built into scikit-learn) with the logistic regression and kNN.
Solution
import numpy as np; rng = np.random.default_rng(37); n = 8000 # Sunita's customers (made up)
X = np.column_stack([rng.poisson(3,n), rng.uniform(1,5,n), rng.uniform(0,12,n), rng.integers(1,36,n),
rng.uniform(60,160,n), rng.poisson(2,n)]) # missed, rating, km, months, price, calls
s = X @ [-.5,.6,-.15,.04,-.02,-.3] + 2*(X[:,1]>3.5)*(X[:,2]<4) - 1.5*(X[:,0]>5)*(X[:,3]<6) # hidden rule
y = (s + rng.normal(0, .8, n) > np.median(s)).astype(int) # 1 = renews next month
X_tr, X_te, y_tr, y_te = X[:5000], X[5000:], y[:5000], y[5000:]
from sklearn.linear_model import LogisticRegression as LR
from sklearn.neighbors import KNeighborsClassifier as KNN
from sklearn.ensemble import RandomForestClassifier as RF; from sklearn.preprocessing import StandardScaler as SS
from sklearn.pipeline import make_pipeline as mp; pr = lambda m, A: m.predict_proba(A)[:, 1] # P(renews)
base = [("logreg", mp(SS(), LR())), ("knn", mp(SS(), KNN(25))),
("forest", RF(100, min_samples_leaf=3, random_state=0))]
from sklearn.ensemble import StackingClassifier, HistGradientBoostingClassifier as HGB
est = [base[0], base[1], ("boost", HGB(random_state=0))]
for name, m in est: print(f"{name:<7} test {m.fit(X_tr, y_tr).score(X_te, y_te):.3f}")
st = StackingClassifier(est, cv=5).fit(X_tr, y_tr)
print(f"stack test {st.score(X_te, y_te):.3f} weights {st.final_estimator_.coef_.round(2)}")
Output:
logreg test 0.850
knn test 0.839
boost test 0.850
stack test 0.854 weights [[2.58 2.47 1.78]]
The stack (0.854) beats both 0.850 models: boosting and logistic regression make different mistakes.
5. On paper: with cv=5, how many times is each base model fitted inside StackingClassifier.fit? And with blending?
Solution
Stacking: 5 fits for the out-of-fold predictions + 1 refit on all training rows = 6 per base model (plus 1 fit of the meta-learner). Blending: 1 per base model (plus the meta-learner).
Quick quiz
The meta-learner needs base-model predictions for its training rows. Where should they come from?
A. The same rows the models were trained on · B. Out-of-fold predictions · C. The test set · D. Random numbers
Answer: B. A is the leak we saw live (test 0.828). C uses up the test set, so the final score is no longer honest.
Interview questions
- What is stacking, and how does it differ from voting? Voting combines base models with fixed (usually equal) weights; stacking trains a meta-learner on the base models' predictions, so it learns how much to trust each one, and can even ignore a weak model.
- Why must the meta-learner be trained on out-of-fold predictions? Predictions on rows a model was trained on are over-optimistic (it may have memorised them). A meta-learner trained on them over-trusts the most overfit model. OOF predictions show how each model behaves on unseen data.
- Stacking vs blending? Stacking uses k-fold OOF predictions for every training row (k + 1 fits per model, all data used). Blending uses a single holdout (1 fit per model, faster, simpler, but spends data).
- What makes a good set of base learners? Each one decent on its own, and diverse: different families, features or preprocessing, so their errors are not strongly correlated.
- Why is the meta-learner usually simple? It has only a few input columns; a complex meta-learner can overfit the predictions. Logistic regression (classification) or ridge (regression) are typical defaults.
- What does
passthrough=Truedo? It gives the meta-learner the original features next to the predictions. It can help when some raw feature tells the meta-learner when to trust which model; scale them if the meta-learner is linear. - When would you not stack in production? When the gain is tiny compared with the extra training time, slower predictions, serving cost, and harder debugging and explanation.
Glossary
| Term | Meaning | Example from this episode |
|---|---|---|
| Ensemble | Several models whose predictions are combined into one answer | Averaging the five models' probabilities |
| Stacking (stacked generalisation) | Training a second model on the predictions of several first models | Logistic regression on top of logreg, kNN and forest: 0.852 |
| Base learner (level 0) | One of the first models in a stack | The random forest |
| Meta-learner (level 1) | The model trained on the base learners' predictions | final_estimator_ of StackingClassifier |
| Meta | "One level above"; a model about other models | The principal judging the teachers |
| Predictions as features | Each base model's predicted probability becomes one input column of the meta-learner | [0.81, 0.62, 0.90] for one customer |
| Fold | One of the k equal parts in k-fold cross-validation | 1,000 of the 5,000 training rows |
| Out-of-fold (OOF) prediction | A prediction for a row made by a model trained on the other folds, which never saw that row | cross_val_predict(m, X_tr, y_tr, cv=5, method="predict_proba") |
cross_val_predict |
scikit-learn function returning one out-of-fold prediction per row | Used for the meta-learner's training columns |
| Leakage | Information about the answers reaching a model where it should not | Meta-learner trained on train-row predictions: test 0.828 |
StackingClassifier / StackingRegressor |
scikit-learn's built-in stacking, with OOF predictions done for you | StackingClassifier(base, cv=5) |
estimators |
The list of (name, model) base learners |
[("logreg", ...), ("knn", ...), ("forest", ...)] |
final_estimator |
The meta-learner parameter; default LogisticRegression (classifier), RidgeCV (regressor) |
final_estimator=make_pipeline(StandardScaler(), LogisticRegression()) |
stack_method |
Which base-model output is passed up: predict_proba, decision_function or predict; "auto" tries them in that order |
SVC in the mini project uses decision_function |
passthrough |
Whether the meta-learner also sees the original columns | passthrough=True: 0.853 with a scaled meta-learner |
| Blending | Stacking with one holdout instead of k folds | Base models on 3,500 rows, meta-learner on 1,500: 0.850 |
| Holdout | Rows set aside and not used to train a model, so they can judge it | The 1,500-row blending holdout |
| Soft voting | Averaging the predicted probabilities of several models | Average of 5 models: 0.843 |
| Model diversity | Base models making different mistakes | Correlation 0.949 (diverse) vs 0.993 (three forests) |
| Correlation of predictions | How closely two models' predicted probabilities move together, from -1 to 1 | 0.993 means near-copies |
| Overfitting | Memorising the training rows and doing worse on new rows | Forest: train 0.963, test 0.846 |
| Netflix Prize | A US$1,000,000 recommendation contest won in 2009 by BellKor's Pragmatic Chaos with an ensemble of many models | Why stacking became famous |
Summary
- When good models disagree, the right answer is often among them (at least one right on 90.1% vs 85.0% for the best model).
- Stacking: base learners (level 0) → their predictions → a meta-learner (level 1).
- Training the meta-learner on predictions for rows the base models were trained on leaks: it trusts the most overfit model (0.828).
- Use out-of-fold predictions (
cross_val_predict): 0.852, better than every single model. StackingClassifierdoes it all:estimatorsas(name, model)pairs,final_estimator(default logistic regression),cv(default 5 folds),stack_method="auto",passthrough.- Unlike averaging, stacking learns to ignore a weak model (tree weight 0.11).
- Blending = one holdout: faster (1 fit per model), slightly weaker, spends data.
- Stack diverse models (different families); near-copies give no gain. Keep the meta-learner simple.
- Worth it for contests and when tiny gains pay; otherwise improve features and one model first.
Next episode: Feature engineering 1: numerical features (binning, log and power transforms, interactions, ratios and aggregations).