SyntaxPilot
Courses/Machine Learning/Lesson 26

Machine Learning · Lesson 26 of 90 · 14 min

Cross-validation

k-fold cross-validationstratified k-foldleave-one-out (LOOCV)group k-foldtime series splitcross_val_scorecross_validatemean and spread of scoresnested cross-validationrepeated k-fold

Machine Learning Full Course · Episode 26. Every code sample below was run with Python 3.13, NumPy 2.5.3 and scikit-learn 1.9.1; the output shown is the real output (timings vary by machine). All data (loans, clinic visits, chai sales) is made up for teaching, except the mini project, which uses the breast-cancer dataset bundled with scikit-learn.

The big idea: one train/test split gives one score, and that score depends on luck: which rows happened to land in the test set. Cross-validation tests the model on several different splits and averages: in k-fold cross-validation every row is tested exactly once. Then the split must match reality: stratified folds keep class shares, group folds keep related rows together, time-series splits never train on the future, and nested cross-validation keeps hyperparameter tuning from grading its own homework.

Words you need first

  • Training set / test set: rows the model learns from / rows used only to measure it (Episode 15).
  • Holdout: setting aside one test set once.
  • Score: a number measuring performance, e.g. accuracy (share correct), recall (share of positives found), R² (Episode 19: 1 is perfect, 0 is as good as predicting the mean, negative is worse than the mean).
  • Standard deviation (std): how far values typically sit from their mean (Episode 10).
  • Hyperparameter: a setting we choose rather than the model learning it (Episode 8), e.g. C or the number of neighbours.
  • Data leakage: information from the test data sneaking into training, making scores look better than reality (Episode 15).
  • k-nearest neighbours (kNN): a model that predicts from the most similar stored rows. Used twice below as a model that "remembers"; built from scratch in Episode 27.

The problem: one split is luck

Real-life example: judging a new batsman by one innings. Out for 4, or 90 not out: one innings is mostly luck (the pitch, the bowler, a dropped catch). Selectors look at many innings: the average and how much it varies.

300 made-up loan applications, 4 features, 1 = repaid. Same model, same data, only the random_state of the split changes:

import numpy as np
from sklearn.linear_model import LogisticRegression
rng = np.random.default_rng(26); n = 300             # made-up loan applications, 1 = repaid
X = rng.normal(0, 1, (n, 4))
y = (X[:, 0] + 0.8 * X[:, 1] - 0.5 * X[:, 2] + rng.normal(0, 1.2, n) > 0).astype(int)
from sklearn.model_selection import train_test_split
for seed in range(8):
    X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=seed)
    acc = LogisticRegression().fit(X_tr, y_tr).score(X_te, y_te)
    print(f"split with random_state={seed}: test accuracy {acc:.3f}")

Output:

split with random_state=0: test accuracy 0.700
split with random_state=1: test accuracy 0.717
split with random_state=2: test accuracy 0.633
split with random_state=3: test accuracy 0.767
split with random_state=4: test accuracy 0.750
split with random_state=5: test accuracy 0.817
split with random_state=6: test accuracy 0.733
split with random_state=7: test accuracy 0.683
  • From 0.633 to 0.817: 18 percentage points apart, from the split alone.
  • With 60 test rows, each row is worth 1/60 ≈ 1.7 percentage points, so a few easy or hard rows move the score a lot. Small datasets make this worse.
  • A holdout also wastes data: 20% of the rows never help training.

K-fold cross-validation

Real-life example: instead of one big exam, a teacher splits the syllabus into 5 units and gives 5 unit tests. Every chapter is in exactly one unit; the report card shows the average and how much the tests varied.

K-fold cross-validation: split the rows into k equal parts called folds. Run k rounds: in each, one fold is the test set and the other k − 1 folds train the model. Report the mean and standard deviation of the k scores.

round 1:  [test ] [train] [train] [train] [train]
round 2:  [train] [test ] [train] [train] [train]
round 3:  [train] [train] [test ] [train] [train]
round 4:  [train] [train] [train] [test ] [train]
round 5:  [train] [train] [train] [train] [test ]
# (first 5 lines: the same loan data as above)
from sklearn.model_selection import cross_val_score, KFold
kf = KFold(n_splits=5, shuffle=True, random_state=0)
for i, (tr, te) in enumerate(kf.split(X)):
    print(f"fold {i + 1}: train on {len(tr)} rows, test on {len(te)}: {te[:6]} ...")
scores = cross_val_score(LogisticRegression(), X, y, cv=kf)
print("5 fold scores:", scores.round(3))
print(f"mean {scores.mean():.3f}  std {scores.std():.3f}")

Output:

fold 1: train on 240 rows, test on 60: [ 5  7  8 12 15 22] ...
fold 2: train on 240 rows, test on 60: [18 20 21 27 33 37] ...
fold 3: train on 240 rows, test on 60: [ 2  3  4 10 13 14] ...
fold 4: train on 240 rows, test on 60: [ 0  6 11 23 35 36] ...
fold 5: train on 240 rows, test on 60: [ 1  9 17 25 28 31] ...
5 fold scores: [0.7   0.767 0.7   0.667 0.767]
mean 0.720  std 0.040
  • kf.split(X) yields pairs of index arrays (train rows, test rows). The test rows never repeat across folds.
  • cross_val_score(model, X, y, cv=...) clones the model, fits it k times and returns k scores (accuracy for classifiers by default; change with scoring="recall", "f1", "roc_auc", "r2", ...).
  • Honest answer: 0.720 ± 0.040. Two models whose means differ by less than about one std are basically tied.
  • shuffle=True mixes the rows first (with a random_state for reproducibility). Without it, fold 1 is simply the first 60 rows.
  • Typical k: 5 or 10. Cost: k fits instead of one. Larger k → bigger training sets, but each test fold is smaller and the scores are noisier (Practice 2).
  • Cross-validation judges; it doesn't produce the final model. After choosing, refit the chosen model once on all the training data.

Stratified k-fold

The problem: another 300 loans, now predicting default (not repaying): only 33 defaulted, about 1 in 9, imbalanced data. And the file came sorted by label, a common side effect of exports.

import numpy as np, textwrap
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score, KFold, StratifiedKFold
rng = np.random.default_rng(6); n = 300               # made-up loans, 1 = defaulted (rare)
X = rng.normal(0, 1, (n, 4))
y = (X[:, 0] + rng.normal(0, 0.8, n) > 1.6).astype(int)
order = np.argsort(y); X, y = X[order], y[order]       # the file came sorted by label
print("defaults:", y.sum(), "of", n)
for name, cv in [("KFold          ", KFold(5)), ("StratifiedKFold", StratifiedKFold(5))]:
    print(name, "defaults in each test fold:", [int(y[te].sum()) for _, te in cv.split(X, y)])
tr, te = list(KFold(5).split(X))[4]                    # the last plain fold
try:
    LogisticRegression().fit(X[tr], y[tr])
except ValueError as e:
    print(textwrap.fill(f"ValueError: {e}", 60))
s = cross_val_score(LogisticRegression(), X, y, cv=StratifiedKFold(5), scoring="recall")
print("stratified recall:", s.round(2), f" mean {s.mean():.2f}")

Output:

defaults: 33 of 300
KFold           defaults in each test fold: [0, 0, 0, 0, 33]
StratifiedKFold defaults in each test fold: [6, 6, 7, 7, 7]
ValueError: This solver needs samples of at least 2 classes
in the data, but the data contains only one class:
np.int64(0)
stratified recall: [0.33 0.33 0.43 0.29 0.  ]  mean 0.28
  • Plain KFold(5) on sorted data puts all 33 defaults in the last fold. That round trains on rows with no defaults at all, so logistic regression refuses: the ValueError above. The first four rounds have no defaults to measure recall on.
  • If you run cross_val_score(..., cv=KFold(5), scoring="recall") directly, scikit-learn prints UndefinedMetricWarning: Recall is ill-defined and being set to 0.0 due to no true samples for the empty folds and a FitFailedWarning ("1 fits failed out of a total of 5") for the last one, and returns [0. 0. 0. 0. nan].
  • Stratified k-fold builds folds that each keep the same class shares as the whole dataset: 6–7 defaults per fold (about 11%).
  • Good news: cross_val_score(classifier, X, y, cv=5) (an integer cv with a classifier) already uses StratifiedKFold. The trap is building plain KFold yourself.
  • For classification: StratifiedKFold(n_splits=5, shuffle=True, random_state=0).

Leave-one-out, cross_validate and repeated k-fold

  • Leave-one-out cross-validation (LOOCV): k = n. Each round leaves out a single row and tests on it. n fits; each score is 0 or 1 (right or wrong). Uses the data maximally; expensive; only for very small datasets.
  • cross_validate: like cross_val_score, but takes several metrics at once, can return training scores (return_train_score=True) and records fit_time and score_time. Returns a dictionary.
  • Repeated k-fold (RepeatedKFold, RepeatedStratifiedKFold): reshuffle and run k-fold several times (e.g. 10 repeats → 50 scores) and average. A steadier mean, at n_repeats times the cost.
# (first 5 lines: the same 300 repaid/not-repaid loans as above)
import time
from sklearn.model_selection import cross_val_score, cross_validate, LeaveOneOut, RepeatedStratifiedKFold
t = time.time(); s = cross_val_score(LogisticRegression(), X, y, cv=LeaveOneOut())
print(f"LOOCV: {len(s)} fits, mean {s.mean():.3f}, first scores {s[:6]}")
r = cross_validate(LogisticRegression(), X, y, cv=5, scoring=["accuracy", "f1"], return_train_score=True)
print("keys:", *sorted(r)[:3]); print("     ", *sorted(r)[3:])
a, b, f = r["train_accuracy"].mean(), r["test_accuracy"].mean(), r["test_f1"].mean()
print(f"train acc {a:.3f}  test acc {b:.3f}  test f1 {f:.3f}")
rep = RepeatedStratifiedKFold(n_splits=5, n_repeats=10, random_state=0)
s = cross_val_score(LogisticRegression(), X, y, cv=rep)
print(f"repeated 5-fold x 10: {len(s)} scores, mean {s.mean():.3f}, std {s.std():.3f}")

Output:

LOOCV: 300 fits, mean 0.737, first scores [1. 1. 1. 1. 1. 0.]
keys: fit_time score_time test_accuracy
      test_f1 train_accuracy train_f1
train acc 0.751  test acc 0.740  test f1 0.763
repeated 5-fold x 10: 50 scores, mean 0.742, std 0.046
  • Training accuracy 0.751 vs test 0.740: close, so the model is not overfitting (Episode 20). A big gap would mean it memorises.
  • All three methods agree: about 74%. Not 63%, not 82%.

Group k-fold

Real-life example: if exam questions are copied from the homework, students who memorised the answers score 95% without learning anything. The honest test uses unseen questions.

The problem: a clinic with 60 patients, 10 visits each. Visits of the same patient look alike. The label is random per patient, so there is nothing to learn: an honest score must be about 50%.

import numpy as np
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import cross_val_score, KFold, GroupKFold
rng = np.random.default_rng(3)                        # made-up clinic: 60 patients, 10 visits each
patient = np.repeat(np.arange(60), 10)
look = rng.normal(0, 1, (60, 5))                      # each patient's own typical readings
X = look[patient] + rng.normal(0, 0.3, (600, 5))      # each visit: close to that patient's typical
y = rng.integers(0, 2, 60)[patient]                   # label: random per patient, NOT learnable
model = KNeighborsClassifier(n_neighbors=1)           # a model that remembers rows (next episode)
s1 = cross_val_score(model, X, y, cv=KFold(5, shuffle=True, random_state=0))
s2 = cross_val_score(model, X, y, cv=GroupKFold(5), groups=patient)
print(f"KFold (same patient in train and test): {s1.mean():.3f}")
print(f"GroupKFold (patients kept apart):       {s2.mean():.3f}")

Output:

KFold (same patient in train and test): 0.938
GroupKFold (patients kept apart):       0.538
  • Shuffled KFold scores 0.938 on unlearnable labels: each test visit has 9 "sister" visits of the same patient in training, so a 1-nearest-neighbour model just recognises the patient. That is leakage between related rows.
  • Group k-fold puts every row of a group (all visits of one patient) in the same fold, so the model is always tested on new patients: 0.538, a coin toss, the truth.
  • A group is any set of rows that belong together: visits of a patient, orders of a customer, photos of one person, readings from one machine, sentences from one document.
  • Use it whenever the real model will meet new groups. Other group splitters: StratifiedGroupKFold (groups + class balance), LeaveOneGroupOut, GroupShuffleSplit.

Time series split

The problem: a chai stall records cups sold for 120 days; business keeps growing. Shuffled folds would train on day 80 and test on day 79: using the future to predict the past. In real life you only have the past.

import numpy as np
from sklearn.neighbors import KNeighborsRegressor
from sklearn.model_selection import cross_val_score, KFold, TimeSeriesSplit
rng = np.random.default_rng(1); days = np.arange(120)  # made-up chai stall: 120 days of cups sold
cups = 100 + 1.5 * days + 15 * np.sin(days / 7) + rng.normal(0, 5, 120)   # the stall keeps growing
X = days.reshape(-1, 1)
for i, (tr, te) in enumerate(TimeSeriesSplit(n_splits=4).split(X)):
    print(f"split {i + 1}: train days {tr[0]}-{tr[-1]}, test days {te[0]}-{te[-1]}")
model = KNeighborsRegressor(n_neighbors=3)
s1 = cross_val_score(model, X, cups, cv=KFold(4, shuffle=True, random_state=0), scoring="r2")
s2 = cross_val_score(model, X, cups, cv=TimeSeriesSplit(n_splits=4), scoring="r2")
print(f"shuffled KFold R2 {s1.mean():.3f}   (peeks at the future)")
print(f"TimeSeriesSplit R2 {s2.mean():.3f}  (honest: only the past)")

Output:

split 1: train days 0-23, test days 24-47
split 2: train days 0-47, test days 48-71
split 3: train days 0-71, test days 72-95
split 4: train days 0-95, test days 96-119
shuffled KFold R2 0.990   (peeks at the future)
TimeSeriesSplit R2 -6.413  (honest: only the past)
  • TimeSeriesSplit uses an expanding window: the training block grows and the test block always comes after it.
  • Shuffled k-fold: R² 0.990, looks perfect. Time series split: −6.413, far worse than predicting the average. The 3-nearest-days model can only average days it has seen (it interpolates); it cannot follow growth into new days (extrapolate). Shuffled folds hid that completely.
  • Options: TimeSeriesSplit(n_splits=..., max_train_size=...) for a sliding window, gap=... to leave days between train and test (when the label is known only later).
  • Anything with dates (prices, sales, sensor readings, user activity): use a time-based split and never shuffle.

Nested cross-validation

The problem: 80 rows with coin-flip labels (nothing to learn). We try 15 values (the odd numbers 1 to 29) of a hyperparameter (the number of neighbours) and keep the one with the best cross-validation score: that is hyperparameter tuning, and GridSearchCV automates it. Is the best score honest? We picked it because it was the highest: like the luckiest of 15 batsmen.

import numpy as np
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import GridSearchCV, cross_val_score, StratifiedKFold
rng = np.random.default_rng(8); n = 80                 # pure noise: 80 rows, coin-flip labels
X = rng.normal(0, 1, (n, 10)); y = rng.integers(0, 2, n)
grid = {"n_neighbors": list(range(1, 31, 2))}          # try 15 settings: 1, 3, ..., 29
inner = StratifiedKFold(5, shuffle=True, random_state=0)
search = GridSearchCV(KNeighborsClassifier(), grid, cv=inner).fit(X, y)
print(f"best {search.best_params_}: CV score {search.best_score_:.3f}  <- looks skilful")
outer = StratifiedKFold(5, shuffle=True, random_state=1)
nested = cross_val_score(GridSearchCV(KNeighborsClassifier(), grid, cv=inner), X, y, cv=outer)
print(f"nested CV score {nested.mean():.3f}  <- honest: the labels are coin flips")

Output:

best {'n_neighbors': 25}: CV score 0.625  <- looks skilful
nested CV score 0.500  <- honest: the labels are coin flips
  • GridSearchCV(model, grid, cv=...): tries every combination in grid, cross-validates each, keeps the best (best_params_, best_score_) and refits it on all the data.
  • best_score_ = 0.625 on coin flips: optimistic, because it was selected for being high. The more settings you try, the more optimistic it gets.
  • Nested cross-validation: two loops.
  • The inner loop chooses: inside each outer training set, GridSearchCV picks a setting with its own cross-validation.
  • The outer loop judges: each chosen model is tested on an outer fold the tuning never saw.
  • In code: put the grid search inside cross_val_score. Result 0.500: the truth.
  • Alternative: tune with cross-validation on the training data, then evaluate once on a separate test set you never touched.

Which splitter when?

Situation Splitter
Classification, independent rows StratifiedKFold(5 or 10, shuffle=True) (default for cv=int with a classifier)
Regression, independent rows KFold(5 or 10, shuffle=True)
Very small dataset LeaveOneOut() or RepeatedStratifiedKFold
Scores noisy between runs RepeatedKFold / RepeatedStratifiedKFold
Several rows per person/customer/machine GroupKFold / StratifiedGroupKFold with groups=
Rows ordered in time TimeSeriesSplit (no shuffling)
Tuning hyperparameters and reporting performance Nested CV, or tune with CV + a separate untouched test set
After choosing Refit the chosen pipeline on all training data

Always put preprocessing (scalers, encoders, imputers) inside a Pipeline (Episode 17), so each fold fits them on its own training part only.

Common mistakes

Mistake What you see Why Fix
Trusting one train/test split 0.633 … 0.817 depending on random_state One sample is luck K-fold; report mean ± std
Plain KFold on data sorted by label ValueError: This solver needs samples of at least 2 classes in the data, but the data contains only one class: np.int64(0) A fold holds all of one class StratifiedKFold(shuffle=True)
Scoring recall on folds with no positives UndefinedMetricWarning: Recall is ill-defined and being set to 0.0 due to no true samples Nothing to recall Stratify
Same patient in train and test 0.938 on unlearnable labels Group leakage GroupKFold(groups=...)
Shuffling dated data R² 0.990 vs −6.413 honestly Trains on the future TimeSeriesSplit
Reporting best_score_ of a grid search 0.625 on coin flips Selected for being high Nested CV or an untouched test set
Scaling before splitting Slightly optimistic scores Test-fold statistics leak into training Scaler inside a Pipeline
Forgetting groups= with GroupKFold ValueError: The 'groups' parameter should not be None. Splitter needs IDs Pass groups=ids
Using the CV models as the final model Five slightly different models CV only judges Refit once on all training data

Mini project: cross-validate a full pipeline on a real dataset

scikit-learn ships a small medical dataset (569 tumours, 30 measurements, 1 = benign). Cross-validate a scaler + logistic regression pipeline with several metrics, then tune C with nested CV.

from sklearn.datasets import load_breast_cancer
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_validate, StratifiedKFold, GridSearchCV, cross_val_score
X, y = load_breast_cancer(return_X_y=True)            # 569 tumours, 30 measurements, bundled with scikit-learn
print(X.shape, "positives (benign = 1):", y.sum())
pipe = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))
cv = StratifiedKFold(5, shuffle=True, random_state=0)
r = cross_validate(pipe, X, y, cv=cv, scoring=["accuracy", "recall", "roc_auc"], return_train_score=True)
for k in ["train_accuracy", "test_accuracy", "test_recall", "test_roc_auc"]:
    print(f"{k:<15} {r[k].mean():.3f} ± {r[k].std():.3f}")
grid = GridSearchCV(pipe, {"logisticregression__C": [0.01, 0.1, 1, 10, 100]}, cv=cv)
nested = cross_val_score(grid, X, y, cv=StratifiedKFold(5, shuffle=True, random_state=1))
print(f"nested CV accuracy (C tuned inside) {nested.mean():.3f} ± {nested.std():.3f}")

Output:

(569, 30) positives (benign = 1): 357
train_accuracy  0.988 ± 0.003
test_accuracy   0.979 ± 0.014
test_recall     0.992 ± 0.007
test_roc_auc    0.995 ± 0.006
nested CV accuracy (C tuned inside) 0.975 ± 0.012
  • The scaler sits inside the pipeline, so every fold scales with its own training statistics.
  • Pipeline hyperparameters are named <step name>__<parameter> (logisticregression__C).
  • Nested accuracy (0.975) is the honest estimate including the tuning step.

Practice

  1. Summarise the 50 repeated-CV scores of the loan model: min, max, the middle 90%, and plot a histogram.
  2. Compare 5-fold and 10-fold cross-validation on the 300 loans. What happens to the mean and the std?
  3. Rerun the clinic example with 200 patients and only 3 visits each. How big is the leak now?
  4. Tune over 100 settings (k = 1..50, uniform or distance weights) on the coin-flip data. Best score vs nested score?
Solution 1
# (first 5 lines: the same loan data as above)
import matplotlib.pyplot as plt
from sklearn.model_selection import cross_val_score, RepeatedStratifiedKFold
s = cross_val_score(LogisticRegression(), X, y, cv=RepeatedStratifiedKFold(n_splits=5, n_repeats=10, random_state=0))
print(f"min {s.min():.3f}  max {s.max():.3f}  mean {s.mean():.3f}  std {s.std():.3f}")
print("middle 90% of scores:", np.percentile(s, [5, 95]).round(3))
plt.hist(s, bins=12); plt.xlabel("fold accuracy"); plt.ylabel("count"); plt.savefig("repeated_cv.png")

Output:

min 0.650  max 0.833  mean 0.742  std 0.046
middle 90% of scores: [0.667 0.817]

Single folds range from 0.650 to 0.833, just like the single splits at the start; their average, 0.742, is stable.

Solution 2
# (first 5 lines: the same loan data as above)
from sklearn.model_selection import cross_val_score, StratifiedKFold
for k in [5, 10]:
    s = cross_val_score(LogisticRegression(), X, y, cv=StratifiedKFold(k, shuffle=True, random_state=0))
    print(f"{k:>2}-fold: {k} scores, mean {s.mean():.3f}, std {s.std():.3f}, test rows per fold {n // k}")

Output:

 5-fold: 5 scores, mean 0.740, std 0.056, test rows per fold 60
10-fold: 10 scores, mean 0.740, std 0.083, test rows per fold 30

Same mean; with 10 folds each test fold has only 30 rows, so individual fold scores jump around more (larger std). The mean is not less reliable: it averages more folds.

Solution 3
import numpy as np
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import cross_val_score, KFold, GroupKFold
rng = np.random.default_rng(3)                        # made-up clinic: 200 patients, 3 visits each
patient = np.repeat(np.arange(200), 3)
look = rng.normal(0, 1, (200, 5))
X = look[patient] + rng.normal(0, 0.3, (600, 5))
y = rng.integers(0, 2, 200)[patient]                  # still not learnable
model = KNeighborsClassifier(n_neighbors=1)
s1 = cross_val_score(model, X, y, cv=KFold(5, shuffle=True, random_state=0))
s2 = cross_val_score(model, X, y, cv=GroupKFold(5), groups=patient)
print(f"KFold {s1.mean():.3f}   GroupKFold {s2.mean():.3f}")

Output:

KFold 0.777   GroupKFold 0.512

Fewer sister visits per patient means a smaller leak (0.777 instead of 0.938), but still a big lie on random labels.

Solution 4
import numpy as np
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import GridSearchCV, cross_val_score, StratifiedKFold
rng = np.random.default_rng(8); n = 80                 # pure noise: 80 rows, coin-flip labels
X = rng.normal(0, 1, (n, 10)); y = rng.integers(0, 2, n)
grid = {"n_neighbors": list(range(1, 51)), "weights": ["uniform", "distance"]}   # 100 settings
inner = StratifiedKFold(5, shuffle=True, random_state=0)
search = GridSearchCV(KNeighborsClassifier(), grid, cv=inner).fit(X, y)
print(f"best of 100: {search.best_params_}, CV score {search.best_score_:.3f}")
nested = cross_val_score(GridSearchCV(KNeighborsClassifier(), grid, cv=inner), X, y,
                         cv=StratifiedKFold(5, shuffle=True, random_state=1))
print(f"nested CV score {nested.mean():.3f}")

Output:

best of 100: {'n_neighbors': 25, 'weights': 'uniform'}, CV score 0.625
nested CV score 0.512

The best-of-100 score still claims 62.5%; nested CV stays near 50%.

Quick quiz

predict tomorrow's electricity use for a city,
from 3 years of hourly meter readings
which cross-validation splitter?

A) TimeSeriesSplit B) StratifiedKFold C) KFold(shuffle=True) D) LeaveOneOut

Answer

A) TimeSeriesSplit. The deployed model only ever knows the past, so test it on later hours than it trained on. Shuffled folds would peek at the future and look far too good; stratification is for class labels, not a numeric target in time.

Interview questions

  1. Why use cross-validation instead of one train/test split? One split's score depends on which rows landed in the test set (0.633 to 0.817 here); k-fold averages over k tests and uses every row for both training and testing.
  2. What does k-fold cross-validation return? k scores, one per fold; report their mean and standard deviation.
  3. What is stratified k-fold and when is it needed? Folds that preserve class proportions; needed for classification, especially with imbalanced classes.
  4. LOOCV pros and cons? Uses almost all data for training each time; needs n fits and gives 0/1 per-row scores; only for tiny datasets.
  5. When do you need GroupKFold? When several rows come from the same entity (patient, user, device) and the model will be used on new entities.
  6. Why not shuffle time-series data in CV? It trains on the future and tests on the past, which inflates scores; use TimeSeriesSplit (expanding window).
  7. Why is GridSearchCV.best_score_ optimistic? It is the maximum over many noisy estimates, chosen for being high.
  8. What is nested cross-validation? An inner CV loop for tuning inside an outer CV loop for evaluation, so the evaluated folds never influence the tuning.
  9. After cross-validation, which model do you deploy? Refit the chosen configuration on all training data.
  10. Where should preprocessing go during CV? Inside a Pipeline, so each fold fits it on its own training part (no leakage).

Glossary

Term Meaning Example
Holdout One test set, set aside once 80/20 split
Cross-validation (CV) Testing a model on several train/test splits and averaging 5-fold
Fold One of the k equal parts the rows are split into 60 rows each
K-fold cross-validation k rounds, each fold tests once, the rest train mean 0.720 ± 0.040
KFold scikit-learn splitter for plain k-fold KFold(5, shuffle=True)
cross_val_score Runs CV and returns one score per fold [0.7 0.767 0.7 0.667 0.767]
Mean and spread of scores Average of fold scores and their standard deviation 0.720 and 0.040
Stratified k-fold Folds that keep the class shares of the whole dataset defaults [6, 6, 7, 7, 7]
Imbalanced data One class much rarer than the other 33 defaults in 300
Leave-one-out (LOOCV) k = n: each round tests on a single row 300 fits
cross_validate CV with several metrics, train scores and timings keys fit_time, test_f1, ...
Repeated k-fold k-fold run several times with reshuffling 5 × 10 = 50 scores
Group Rows that belong together visits of one patient
Group k-fold Keeps each group entirely in one fold 0.538 vs 0.938
Group leakage A group split across train and test, so the model recognises it 0.938 on random labels
Time series split Train on earlier data, test on later data (expanding window) days 0–95 → 96–119
Expanding window The training period grows with each split 24, 48, 72, 96 days
Interpolate / extrapolate Predict inside / outside the range seen in training kNN can't extrapolate growth
R² 1 − (model's squared error / mean predictor's squared error) 0.990 vs −6.413
Hyperparameter tuning Choosing settings like k or C by validation scores 30 values of k
GridSearchCV Tries every setting in a grid with CV and keeps the best best_score_ 0.625
Selection bias (optimism) The best of many noisy scores is too high 0.625 on coin flips
Nested cross-validation Inner CV tunes, outer CV judges 0.500
Inner / outer loop The tuning loop / the evaluation loop cv=inner, cv=outer
Refit Training the chosen configuration on all training data after CV

Summary

  • One split is luck: the same model scored 0.633 to 0.817 depending only on the seed.
  • K-fold tests every row once and reports mean ± std (0.720 ± 0.040); refit the chosen model on all data afterwards.
  • Stratified folds keep class shares (sorted data with plain KFold crashed with "only one class").
  • LOOCV, cross_validate and repeated k-fold all agreed on about 74%.
  • GroupKFold stops related rows leaking (0.938 → 0.538 on random labels); TimeSeriesSplit stops training on the future (R² 0.990 → −6.413).
  • Nested CV keeps tuning honest: best_score_ said 0.625 on coin flips, nested CV said 0.500.

Next episode: k-nearest neighbours: lazy learning, kNN from scratch, distance metrics, choosing k, distance weighting, scaling, the curse of dimensionality, KD-trees and ball trees.