SyntaxPilot
Courses/Machine Learning/Lesson 15

Machine Learning · Lesson 15 of 90 · 12 min

Train, validation and test splits

training setvalidation settest setholdoutgeneralisationi.i.d. assumptionrandom splitstratified splitgroup-aware splittime-based splitdata leakagetarget leakagetrain-test contamination

Machine Learning Full Course · Episode 15. Every code sample below was run with Python 3.13, NumPy 2.5.3, pandas 3.0.5 and scikit-learn 1.9.1; the output shown is the real output. All data is made up for teaching.

The big idea: a model's score on the data it learned from tells you almost nothing. What matters is how it does on data it has never seen. So you hide part of your data, and you protect that hidden part from every kind of leak.

Memorising vs learning

A student memorises last year's question paper. Tested on that same paper: 100/100. In the real exam, with new questions on the same topics, he fails. Models do exactly this.

We use a decision tree, a model that learns a chain of yes/no questions about the features (built from scratch in Episode 29), and score it with the mean absolute error (MAE): for each flat, |prediction − true rent|, averaged. Lower is better; it's in rupees.

import numpy as np, pandas as pd
from sklearn.tree import DecisionTreeRegressor
def flats(n, seed):                            # made-up flats, same recipe
    rng = np.random.default_rng(seed)
    area = rng.normal(950, 300, n).clip(350, 2500).round()
    bhk = np.clip(area // 450, 1, 3)
    rent = (area * 20 * rng.lognormal(0, .25, n)).round(-2)
    return pd.DataFrame({"area": area, "bhk": bhk}), rent

X, y = flats(300, seed=15)
model = DecisionTreeRegressor(random_state=0).fit(X, y)
mae = np.abs(model.predict(X) - y).mean()       # mean absolute error
print("error on the flats it learned from:", round(mae))
X_new, y_new = flats(300, seed=99)              # 300 flats it never saw
new_mae = np.abs(model.predict(X_new) - y_new).mean()
print("error on brand-new flats:", round(new_mae))

Output:

error on the flats it learned from: 775
error on brand-new flats: 5496

flats(n, seed) makes listings with the same recipe each time; a different seed gives different flats. On its own 300 flats the tree is off by only ₹775. On 300 new flats from the same recipe: ₹5,496, seven times worse. It memorised the noise of its training flats.

Generalisation and the holdout

  • Generalisation = performing well on data the model has never seen. It's the whole point of machine learning.
  • You can't get "brand-new flats" on demand, so you hide part of the data you have:
  • the training set: the model learns from it;
  • the test set: you judge the model on it, like a teacher who keeps the real exam locked away.
  • Keeping a part aside like this is a holdout.
import numpy as np, pandas as pd
from sklearn.tree import DecisionTreeRegressor
def flats(n, seed):                            # made-up flats, same recipe
    rng = np.random.default_rng(seed)
    area = rng.normal(950, 300, n).clip(350, 2500).round()
    bhk = np.clip(area // 450, 1, 3)
    rent = (area * 20 * rng.lognormal(0, .25, n)).round(-2)
    return pd.DataFrame({"area": area, "bhk": bhk}), rent
from sklearn.model_selection import train_test_split
X, y = flats(300, seed=15)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=0)
print("train:", len(X_tr), " test:", len(X_te))
for depth in [None, 3]:
    m = DecisionTreeRegressor(max_depth=depth, random_state=0).fit(X_tr, y_tr)
    tr = np.abs(m.predict(X_tr) - y_tr).mean()
    te = np.abs(m.predict(X_te) - y_te).mean()
    print("max_depth", depth, " train error", round(tr), " test error", round(te))

Output:

train: 240  test: 60
max_depth None  train error 701  test error 5081
max_depth 3  train error 3609  test error 3833
  • train_test_split(X, y, test_size=0.2, random_state=0) shuffles the rows and puts 20% in the test set. random_state fixes the shuffle so you get the same split every run (reproducibility, Episode 2).
  • It returns four pieces: X_train, X_test, y_train, y_test.
  • The full-depth tree: train error 701, test error 5,081: the test set catches the memorising.
  • max_depth=3 (at most 3 questions deep): train 3,609, test 3,833. Worse on training but far better on the test. A small gap between training and test error means the model learned something general (Episode 20 explores this gap: overfitting).

Why one test set isn't enough: peeking

To pick the best depth, the obvious way is to try many and keep the one with the lowest test error. But then the test set helped choose the model: it's no longer unseen. How bad can that be? An extreme case where there is nothing to learn:

import numpy as np
rng = np.random.default_rng(15)
y_test = rng.integers(0, 2, 50)            # 50 test labels: pure coin flips
y_new = rng.integers(0, 2, 1000)           # 1,000 future labels

def model(s):                              # model number s: guesses randomly
    return np.random.default_rng(100 + s).integers(0, 2, 1050)

scores = [(model(s)[:50] == y_test).mean() for s in range(200)]
best = int(np.argmax(scores))
print("best of 200 on the test set: model", best, "accuracy", scores[best])
print("the same model on new data:  accuracy", (model(best)[50:] == y_new).mean())

Output:

best of 200 on the test set: model 61 accuracy 0.7
the same model on new data:  accuracy 0.497
  • Accuracy = the fraction of predictions that are right (Episode 24 covers it in depth).
  • The labels are coin flips, so every "model" is really 50% accurate. Yet the best of 200 on a 50-row test set scores 0.7. On 1,000 new labels: 0.497.
  • Choosing among many options by test score finds the luckiest option, not the best. With real models the effect is smaller but always in the same direction: the reported score is too optimistic.

The validation set: three sets, three jobs

Set Job Typical share
Training set learn the model's parameters (the numbers inside it) 60%
Validation set choose hyperparameters (settings you pick, like max_depth, Episode 8), compare models and features 20%
Test set the final grade, looked at once, at the very end 20%

With very large datasets (millions of rows), 1 to 2% each for validation and test is plenty. Cross-validation (Episode 26) reuses the data more efficiently when it's small.

import numpy as np, pandas as pd
from sklearn.tree import DecisionTreeRegressor
def flats(n, seed):                            # made-up flats, same recipe
    rng = np.random.default_rng(seed)
    area = rng.normal(950, 300, n).clip(350, 2500).round()
    bhk = np.clip(area // 450, 1, 3)
    rent = (area * 20 * rng.lognormal(0, .25, n)).round(-2)
    return pd.DataFrame({"area": area, "bhk": bhk}), rent
from sklearn.model_selection import train_test_split
def mae(m, X, y): return round(np.abs(m.predict(X) - y).mean())
X, y = flats(500, seed=15)
X_rest, X_te, y_rest, y_te = train_test_split(X, y, test_size=0.2, random_state=0)
X_tr, X_va, y_tr, y_va = train_test_split(X_rest, y_rest, test_size=0.25, random_state=0)
print("train", len(X_tr), " validation", len(X_va), " test", len(X_te))
for d in [1, 2, 3, 5, 8, None]:
    m = DecisionTreeRegressor(max_depth=d, random_state=0).fit(X_tr, y_tr)
    print("max_depth", d, " validation error", mae(m, X_va, y_va))
final = DecisionTreeRegressor(max_depth=2, random_state=0).fit(X_tr, y_tr)
print("depth 2, test error (looked at ONCE):", mae(final, X_te, y_te))

Output:

train 300  validation 100  test 100
max_depth 1  validation error 3981
max_depth 2  validation error 3556
max_depth 3  validation error 3723
max_depth 5  validation error 3734
max_depth 8  validation error 4597
max_depth None  validation error 4967
depth 2, test error (looked at ONCE): 4312
  • Two calls: 20% off for test, then 25% of the remaining 80% (= 20% of the total) for validation: 300 / 100 / 100.
  • Depth 2 wins on validation (3,556). The test error, looked at once, is 4,312: higher, which is normal, because we picked depth 2 partly for its luck on validation. The test score is the honest number you report.

The i.i.d. assumption

A random split is fair only if the rows are i.i.d.: independent and identically distributed.

  • Identically distributed: every row comes from the same process (a well-stirred pot of dal: every spoon tastes like the pot).
  • Independent: one row tells you nothing extra about another (one spoon doesn't change the next).

When i.i.d. breaks, one spoon lies. Three common cases, and a fix for each: rare classes (stratify), groups of related rows (group split), time (time-based split).

Stratified split: keep the class proportions

import numpy as np
from sklearn.model_selection import train_test_split
y = np.array([1] * 5 + [0] * 95)       # 100 loans, 5 defaults (label 1)
X = np.arange(100).reshape(-1, 1)      # a dummy feature
for seed in range(5):
    *_, y_te = train_test_split(X, y, test_size=0.2, random_state=seed)
    print("random split, seed", seed, "-> defaults in test:", y_te.sum())
*_, y_te = train_test_split(X, y, test_size=0.2, random_state=0, stratify=y)
print("stratified split    -> defaults in test:", y_te.sum())

Output:

random split, seed 0 -> defaults in test: 1
random split, seed 1 -> defaults in test: 0
random split, seed 2 -> defaults in test: 3
random split, seed 3 -> defaults in test: 0
random split, seed 4 -> defaults in test: 1
stratified split    -> defaults in test: 1
  • 100 loans, 5 defaults (a default = a borrower who fails to repay; label 1). A 20% test set should hold 1.
  • Random splits put 1, 0, 3, 0 and 1 defaults in the test set. With 0 you can't check whether the model catches defaulters at all.
  • A stratified split keeps each class in the same proportion in every part (stratify=y): exactly 1 every time. Use it for every classification problem, especially with rare classes.
  • *_, y_te = ... is Python's way of keeping only the last returned value.

60 buildings with 5 flats each. Flats in one building share a location and have similar rents: they are not independent. The real job is predicting rent in a new building.

import numpy as np, pandas as pd
from sklearn.tree import DecisionTreeRegressor
from sklearn.model_selection import train_test_split, GroupShuffleSplit
rng = np.random.default_rng(15)
building = np.repeat(np.arange(60), 5)         # 60 buildings x 5 flats each
lat, lon = rng.random(60), rng.random(60)      # where each building is
level = rng.normal(20000, 6000, 60)            # each building's price level
X = pd.DataFrame({"lat": lat[building], "lon": lon[building]})
y = level[building] + rng.normal(0, 1000, 300)

def err(tr, te):
    m = DecisionTreeRegressor(random_state=0).fit(X.iloc[tr], y[tr])
    return round(np.abs(m.predict(X.iloc[te]) - y[te]).mean())

tr, te = train_test_split(np.arange(300), test_size=0.2, random_state=0)
print("random split error:", err(tr, te))
tr, te = next(GroupShuffleSplit(test_size=0.2, random_state=0).split(X, y, building))
print("group split error: ", err(tr, te))
print("buildings in both train and test:", len(set(building[tr]) & set(building[te])))

Output:

random split error: 864
group split error:  7394
buildings in both train and test: 0
  • Random split: error 864, because the other four flats of each test building were in training, so the tree just "looked up" the building.
  • Group split (GroupShuffleSplit, with the building id as the group): every building is entirely in train or entirely in test. Error 7,394, over eight times worse, and honest: for a new building this model is weak.
  • Test information leaking into training through related or duplicate rows is train-test contamination. Other examples: the same patient's scans on both sides, the same customer's orders, near-duplicate photos. Also see GroupKFold (Episode 26).

Time-based split: train on the past, test on the future

import numpy as np, pandas as pd
from sklearn.tree import DecisionTreeRegressor
from sklearn.model_selection import train_test_split
rng = np.random.default_rng(15)
month = np.repeat(np.arange(36), 10)           # 3 years, 10 listings a month
area = rng.normal(950, 300, 360).clip(350, 2500).round()
rent = area * (15 + 0.5 * month) * rng.lognormal(0, .15, 360)   # rents rise
X = pd.DataFrame({"month": month, "area": area})

def err(tr, te):
    m = DecisionTreeRegressor(min_samples_leaf=5, random_state=0)
    m.fit(X[tr], rent[tr])
    return round(np.abs(m.predict(X[te]) - rent[te]).mean())

idx = np.arange(360)
a, b = train_test_split(idx, test_size=60, random_state=0)
print("random split error:", err(np.isin(idx, a), np.isin(idx, b)))
print("time split error:  ", err(month < 30, month >= 30))

Output:

random split error: 3190
time split error:   4693
  • Rents rise every month; we want to predict future rents.
  • Random split: 3,190. The model saw rows from months 31 and 35 while being tested on months 32 and 34: it peeked at the future.
  • Time-based split (train months 0 to 29, test 30 to 35): 4,693, the honest number. It also shows the real problem: a tree can't predict values higher than anything it has seen.
  • With time-ordered data, always split by time (TimeSeriesSplit in Episode 26; forecasting in Episodes 50 to 52).

Data leakage and target leakage

Data leakage = information reaching the model during training that it won't have when it's used for real. Like an exam paper with the answers printed in it: everyone scores 100, and the score means nothing.

  • Target leakage: a feature that secretly contains (or is computed from) the label.
  • Train-test contamination: test information getting into training (the group and time examples above, or cleaning statistics computed on all rows).
import numpy as np, pandas as pd
from sklearn.tree import DecisionTreeRegressor
from sklearn.model_selection import train_test_split
rng = np.random.default_rng(15)
area = rng.normal(950, 300, 300).clip(350, 2500).round()
rent = (area * 20 * rng.lognormal(0, .25, 300)).round(-2)
X = pd.DataFrame({"area": area, "brokerage": rent})   # fee = 1 month's rent
X_tr, X_te, y_tr, y_te = train_test_split(X, rent, test_size=0.2, random_state=0)
for cols in [["area", "brokerage"], ["area"]]:
    m = DecisionTreeRegressor(max_depth=4, random_state=0).fit(X_tr[cols], y_tr)
    print(cols, "test error:", round(np.abs(m.predict(X_te[cols]) - y_te).mean()))

Output:

['area', 'brokerage'] test error: 618
['area'] test error: 4044

The brokerage fee is usually one month's rent, paid after the deal. With it: error 618, "brilliant". Without it: 4,044, the honest error. When your cousin asks for a fair rent, there is no brokerage yet. If a feature makes a model suspiciously good, ask: will I know this at prediction time?

The anti-leakage checklist

  1. Split first, before cleaning, scaling, imputing or feature engineering.
  2. Learn every statistic (medians for imputing, means and scales, percentiles for capping, category lists) from the training set only, then apply it to validation and test (Episode 16: fit vs transform; Episode 17: pipelines).
  3. For every feature, ask: known at prediction time?
  4. Look for duplicates and groups across the split → group split.
  5. Time data: train on the past, test on the future.

Common mistakes

Mistake Symptom (from today's code) Why Fix
scoring on training data 775 → 5,496 on new flats memorising hold out a test set
tuning on the test set 0.7 on test → 0.497 on new data picks the lucky option validation set; test once
random split with a rare class 0 defaults in the test set luck stratify=y
random split with groups 864 vs 7,394 train-test contamination GroupShuffleSplit / GroupKFold
random split with time 3,190 vs 4,693 peeking at the future time-based split
a feature known only later 618 vs 4,044 target leakage known at prediction time?
imputing/scaling before splitting optimistic test score test info in training split first; fit on train only
forgetting random_state a different score every run a different shuffle fix random_state

A real error you'll meet when stratifying with too few examples of a class:

import numpy as np
from sklearn.model_selection import train_test_split
y = np.array([1] + [0] * 99)            # only ONE example of class 1
train_test_split(np.zeros((100, 1)), y, test_size=0.2, stratify=y)
ValueError: The least populated classes in y have only 1 member, which is too few. The minimum number of groups for any class cannot be less than 2. Classes with too few members are: [1]

Mini project: an honest evaluation, end to end

Split first, fill missing areas with the training median, choose the depth on validation, report the test error once:

import numpy as np, pandas as pd
from sklearn.tree import DecisionTreeRegressor
from sklearn.model_selection import train_test_split
# Mini project: an honest evaluation, end to end (split first, clean with train-only numbers)
rng = np.random.default_rng(7); n = 600
area = rng.normal(950, 300, n).clip(350, 2500).round()
rent = (area * 20 * rng.lognormal(0, .25, n)).round(-2)
df = pd.DataFrame({"area": area, "bhk": np.clip(area // 450, 1, 3), "rent": rent})
df.loc[rng.random(n) < 0.15, "area"] = np.nan           # some missing areas

X, y = df[["area", "bhk"]], df["rent"]
X_rest, X_te, y_rest, y_te = train_test_split(X, y, test_size=0.2, random_state=0)
X_tr, X_va, y_tr, y_va = train_test_split(X_rest, y_rest, test_size=0.25, random_state=0)

fill = X_tr["area"].median()                            # learned on TRAIN only
X_tr, X_va, X_te = (d.fillna({"area": fill}) for d in (X_tr, X_va, X_te))
print("median used for filling (train only):", fill)

def mae(m, X, y): return round(np.abs(m.predict(X) - y).mean())
scores = {}
for d in [2, 3, 4, 6, None]:
    m = DecisionTreeRegressor(max_depth=d, random_state=0).fit(X_tr, y_tr)
    scores[d] = mae(m, X_va, y_va)
print("validation errors:", scores)
best = min(scores, key=scores.get)
final = DecisionTreeRegressor(max_depth=best, random_state=0).fit(X_tr, y_tr)
print("best depth:", best, " test error (once):", mae(final, X_te, y_te))

Output:

median used for filling (train only): 902.5
validation errors: {2: 3967, 3: 4078, 4: 4053, 6: 4302, None: 4900}
best depth: 2  test error (once): 4388

Practice

  1. Split 1,000 rows 70 / 15 / 15 with two calls of train_test_split.
  2. Repeat the peeking demo with 2,000 models.
Solutions 1 and 2
import numpy as np
from sklearn.model_selection import train_test_split
# Homework 1: a 70 / 15 / 15 split with two calls
X = np.arange(1000).reshape(-1, 1); y = np.arange(1000)
X_rest, X_te, y_rest, y_te = train_test_split(X, y, test_size=0.15, random_state=0)
X_tr, X_va, y_tr, y_va = train_test_split(X_rest, y_rest, test_size=0.15 / 0.85, random_state=0)
print(len(X_tr), len(X_va), len(X_te))

Output:

700 150 150

The second call takes 15% of the total out of the remaining 85%, so its test_size is 0.15 / 0.85.

import numpy as np
# Homework 2: the peeking demo with 2,000 models
rng = np.random.default_rng(15)
y_test = rng.integers(0, 2, 50)
y_new = rng.integers(0, 2, 1000)
def model(s): return np.random.default_rng(100 + s).integers(0, 2, 1050)
scores = [(model(s)[:50] == y_test).mean() for s in range(2000)]
best = int(np.argmax(scores))
print("best of 2,000 on the test set:", scores[best])
print("same model on new data:", (model(best)[50:] == y_new).mean())

Output:

best of 2,000 on the test set: 0.72
same model on new data: 0.533

More options → a luckier winner on the test set (0.72), still no better than a coin on new data.

  1. Find the leak: a fraud model uses the feature refund_issued.
  2. Time-split with the last 3 months as the test set.
Solutions 3 and 4
  1. A refund is issued after a transaction is found to be fraudulent, so refund_issued is computed from the label: target leakage. At prediction time (the moment of payment) it doesn't exist yet. Remove it.
import numpy as np, pandas as pd
from sklearn.tree import DecisionTreeRegressor
# Homework 4: time split with the last 3 months as the test set
rng = np.random.default_rng(15)
month = np.repeat(np.arange(36), 10)
area = rng.normal(950, 300, 360).clip(350, 2500).round()
rent = area * (15 + 0.5 * month) * rng.lognormal(0, .15, 360)
X = pd.DataFrame({"month": month, "area": area})
tr, te = month < 33, month >= 33
m = DecisionTreeRegressor(min_samples_leaf=5, random_state=0).fit(X[tr], rent[tr])
print("train months 0-32, test months 33-35, error:", round(np.abs(m.predict(X[te]) - rent[te]).mean()))

Output:

train months 0-32, test months 33-35, error: 3287

Testing on the nearer future (3 months) gives a smaller error (3,287) than 6 months ahead (4,693): the further ahead you predict, the harder it gets. Always state the horizon you tested.

Quick quiz

import numpy as np
from sklearn.model_selection import train_test_split

y = np.array([1] * 10 + [0] * 90)
X = np.zeros((100, 1))
*_, y_te = train_test_split(X, y, test_size=0.3,
                           random_state=7, stratify=y)
print(len(y_te), y_te.sum())

A) 30 3 B) 30 10 C) 70 7 D) 30 0

Answer

A) 30 3. test_size=0.3 gives 30 test rows, and stratify=y keeps 10% ones in every part: 3 of 30.

Interview questions

  1. Why can't you evaluate a model on its training data? It can memorise; training error underestimates the error on new data. We care about generalisation.
  2. What are the training, validation and test sets for? Learn parameters; choose hyperparameters and models; give one final unbiased estimate.
  3. Why is tuning on the test set a problem? Selecting by test score overfits to the test set; the reported score becomes optimistic.
  4. What is the i.i.d. assumption and when does it fail? Rows independent and from the same distribution; fails with groups (patients, users, buildings), time series and distribution shift.
  5. When do you use a stratified split? Classification, especially with imbalanced classes, so every split keeps the class proportions.
  6. Give an example of train-test contamination. The same patient's images in train and test; near-duplicate rows; a scaler fitted on all data.
  7. What is target leakage? How do you detect it? A feature derived from the label or only known after it; suspiciously high scores, features with huge importance, and asking "known at prediction time?"
  8. How would you split data for a model that forecasts next month's sales? By time: train on earlier months, validate and test on later months (walk-forward validation).

Glossary

Term Meaning Example
Decision tree A model that learns a chain of yes/no questions DecisionTreeRegressor
Mean absolute error (MAE) Average of |prediction − truth| ₹775 vs ₹5,496
Accuracy Fraction of predictions that are right 0.7
Generalisation Performing well on unseen data test error
Training set Data the model learns from 240 flats
Test set Hidden data for the final grade, used once 60 flats
Holdout Keeping part of the data aside for evaluation 20% test
train_test_split scikit-learn function that shuffles and splits test_size=0.2
random_state Seed that fixes the shuffle random_state=0
Parameters Numbers the model learns the tree's split points
Hyperparameters Settings you choose before training max_depth
Validation set Data for choosing hyperparameters and models 100 flats
Peeking Letting the test set influence choices best of 200 guessers
i.i.d. Independent and identically distributed rows a well-stirred pot
Independent One row tells nothing extra about another separate spoons
Identically distributed All rows from the same process same pot
Random split Rows assigned at random train_test_split
Stratified split Keeps class proportions in every part stratify=y
Default (loan) A borrower failing to repay label 1
Group-aware split All rows of a group on one side GroupShuffleSplit
Group A set of related rows one building
Time-based split Train on the past, test on the future months 0 to 29 / 30 to 35
Data leakage Training on information unavailable at prediction time brokerage fee
Target leakage A feature computed from (or after) the label brokerage = rent
Train-test contamination Test information getting into training same building on both sides
Prediction time The moment the model is used for real when your cousin asks

Summary

  • Training error lies (775 vs 5,496): judge models on data they've never seen (generalisation).
  • Holdout: train_test_split; validation set to choose, test set to grade once (choosing by test score found a "70% accurate" coin flip).
  • Random splits assume i.i.d. rows. Stratify classes, split by group (864 vs 7,394), split by time (3,190 vs 4,693).
  • Data leakage: target leakage (a brokerage fee made the error 618 instead of 4,044) and train-test contamination. Split first and learn every statistic from the training set only.

Next episode: Feature scaling and encoding: standardisation (z-score), min-max, robust scaling, one-hot encoding, the dummy variable trap, ordinal encoding, unknown categories, and fit vs transform (fit on train only).