These are the notes for AI episode 6. Every score in episode 5 was measured on data the model might already have seen. Here you build a model with zero error that is useless on its first new day, then invent the fix: generalization, overfitting and underfitting, the training / validation / test split, and how to avoid data leakage. Every Python example below runs as-is in the SyntaxPilot compiler, and the outputs shown are real.
The problem: Rohan memorises last year's paper
| Rohan's exam season | What it means |
|---|---|
| He learns every answer of last year's paper by heart | Memorising, not understanding |
| Practice test (the same paper): 100% | Perfect on questions he has seen |
| The real exam: new numbers, same ideas | Questions he has never seen |
| He fails | He never learned the method |
Now let's build Rohan as a machine learning model.
Demo 1: a model that memorises
# Raju's notebook: (temperature °C, cups of chai sold)
notebook = [(14, 95), (16, 87), (18, 94), (22, 82),
(24, 68), (28, 60), (30, 60), (34, 59)]
memory = dict(notebook) # the "model" memorises every day
def predict(temp):
return memory[temp]
errors = [abs(predict(t) - cups) for t, cups in notebook]
print("error on the notebook days:", sum(errors) / len(errors))
print("tomorrow, 17 °C ->", predict(17))
Output:
error on the notebook days: 0.0
Traceback (most recent call last):
File "main.py", line 13, in <module>
print("tomorrow, 17 °C ->", predict(17))
~~~~~~~^^^^
File "main.py", line 8, in predict
return memory[temp]
~~~~~~^^^^^^
KeyError: 17
Zero error on the notebook days, and no answer at all for the first new day. A model's job is not to remember the past; it is to answer new questions well. Doing well on data you have never seen is called generalization.
Patching the memoriser: copy the nearest day?
If 17 °C is missing, copy the closest notebook day: 16 °C → 87 cups. But look at the notebook: 18 °C sold 94 cups, more than 16 °C did. Hotter, yet more chai? That is noise: random ups and downs (a wedding nearby, a rainy evening) that are not part of the pattern and won't repeat. A model that copies days copies the noise too.
Demo 2: three models, two tests
train = [(14, 95), (16, 87), (18, 94), (22, 82),
(24, 68), (28, 60), (30, 60), (34, 59)]
new_days = [(17, 89), (21, 83), (25, 78), (31, 60)] # never seen
def average(t): # model 1: one number for every day
return sum(c for _, c in train) / len(train)
def nearest(t): # model 2: copy the closest notebook day
return min(train, key=lambda day: abs(day[0] - t))[1]
def line(t): # model 3: a straight line
return 124 - 2.0 * t
def mae(model, days):
return sum(abs(model(t) - c) for t, c in days) / len(days)
for model in [average, nearest, line]:
print(f"{model.__name__:8} train error {mae(model, train):5.2f}"
f" new-day error {mae(model, new_days):5.2f}")
Output:
average train error 13.88 new-day error 9.69
nearest train error 0.00 new-day error 3.25
line train error 4.62 new-day error 2.00
(mae is the mean absolute error from episode 4: the average gap in cups. The line 124 - 2.0 × temp is the one Raju learned in episode 4.)
| Model | Train error | New-day error | Diagnosis |
|---|---|---|---|
nearest |
0.00 | 3.25 | Overfitting: memorised the noise |
average |
13.88 | 9.69 | Underfitting: too simple to learn "hot days sell less" |
line |
4.62 | 2.00 | Good fit: learned the trend, ignored the noise |
Overfitting and underfitting (three students)
| In the classroom | In machine learning | Signs |
|---|---|---|
| Rohan the memoriser: 100% practice, fails the exam | Overfitting | low training error, much higher error on new data |
| Guesses "C" every time: bad at both | Underfitting | high error everywhere |
| Learns the method: good at both | Good fit | both errors low and close together |
How to read the two errors together:
train error very low, new-data error much higher → OVERFITTING (big gap)
train error high, new-data error high → UNDERFITTING
train error low, new-data error low → GOOD FIT (small gap)
Holding data back: training, validation and test sets
How do we measure new-data error before the real new days arrive? Do what a good teacher does:
| At school | In machine learning | Used for |
|---|---|---|
| Textbook exercises | Training set | the model learns from it |
| Class test | Validation set | comparing models and choosing settings |
| Final exam, sealed until the end | Test set | one honest score, used once |
| A leaked exam paper | Data leakage | gives a fake, too-good score |
Why not reuse the validation set as the final score? After you have chosen your model by it, it is no longer new to you, so its score is slightly optimistic.
all data ──shuffle──> [ train 16 days | validation 4 | test 4 ]
learn choose report once
Demo 3: train, validation, test
import random
days = [(12, 99), (13, 101), (14, 95), (15, 92), (16, 87), (17, 89), (18, 94), (19, 88),
(20, 89), (21, 83), (22, 82), (23, 79), (24, 68), (25, 78), (26, 75), (27, 72),
(28, 60), (29, 57), (30, 60), (31, 60), (32, 62), (33, 58), (34, 59), (35, 51)]
random.seed(1)
random.shuffle(days) # mix hot and cool days first
train, val, test = days[:16], days[16:20], days[20:]
print("train:", len(train), " validation:", len(val), " test:", len(test))
def mae(model, data):
return sum(abs(model(t) - c) for t, c in data) / len(data)
def nearest(t):
return min(train, key=lambda d: abs(d[0] - t))[1]
def line(t):
return 124 - 2.0 * t
for model in [nearest, line]:
print(f"{model.__name__:8} validation error {mae(model, val):.2f}")
best = min([nearest, line], key=lambda m: mae(m, val))
# ...and open the TEST set only once, at the very end
print("chosen:", best.__name__, " test error:", round(mae(best, test), 2))
Output:
train: 16 validation: 4 test: 4
nearest validation error 6.50
line validation error 2.50
chosen: line test error: 3.75
Validation picks the line. The honest test error (3.75) is a little worse than the validation error, which is normal. This is the number you report. random.seed(1) makes the shuffle repeatable, so you and your teammates get the same split.
Demo 4: data leakage gives a fake perfect score
import random
days = [(12, 99), (13, 101), (14, 95), (15, 92), (16, 87), (17, 89), (18, 94), (19, 88),
(20, 89), (21, 83), (22, 82), (23, 79), (24, 68), (25, 78), (26, 75), (27, 72),
(28, 60), (29, 57), (30, 60), (31, 60), (32, 62), (33, 58), (34, 59), (35, 51)]
random.seed(1)
random.shuffle(days)
test = days[20:]
train = days # oops: the test days are inside train too
def nearest(t):
return min(train, key=lambda d: abs(d[0] - t))[1]
print("test error:", sum(abs(nearest(t) - c) for t, c in test) / len(test))
Output:
test error: 0.0
Perfect, and completely fake: the model just looked up the answers. If a test score looks too good to be true, check for leakage first. Leakage also happens in subtler ways: the same customer in both train and test, or a feature that secretly contains the answer (for example "refund issued" when predicting "will this order be returned?").
How to fix overfitting (and underfitting)
| Fix | Why it helps |
|---|---|
| More data | noise averages out; it is much harder to memorise 10,000 days than 8 |
| A simpler model | a line can't memorise every bump, so it must learn the trend |
| Regularization | during training the model pays a penalty for being too complicated |
| Early stopping | watch validation error while training; stop when it starts rising (episode 7 shows training step by step) |
For underfitting, do the opposite: a more flexible model, better features, or training longer.
Demo 5: a lucky validation set
Change only the shuffle seed in Demo 3, from random.seed(1) to random.seed(2):
import random
days = [(12, 99), (13, 101), (14, 95), (15, 92), (16, 87), (17, 89), (18, 94), (19, 88),
(20, 89), (21, 83), (22, 82), (23, 79), (24, 68), (25, 78), (26, 75), (27, 72),
(28, 60), (29, 57), (30, 60), (31, 60), (32, 62), (33, 58), (34, 59), (35, 51)]
random.seed(2)
random.shuffle(days) # mix hot and cool days first
train, val, test = days[:16], days[16:20], days[20:]
print("train:", len(train), " validation:", len(val), " test:", len(test))
def mae(model, data):
return sum(abs(model(t) - c) for t, c in data) / len(data)
def nearest(t):
return min(train, key=lambda d: abs(d[0] - t))[1]
def line(t):
return 124 - 2.0 * t
for model in [nearest, line]:
print(f"{model.__name__:8} validation error {mae(model, val):.2f}")
best = min([nearest, line], key=lambda m: mae(m, val))
# ...and open the TEST set only once, at the very end
print("chosen:", best.__name__, " test error:", round(mae(best, test), 2))
Output:
train: 16 validation: 4 test: 4
nearest validation error 2.50
line validation error 3.25
chosen: nearest test error: 4.25
Now validation picks the overfitting model, and its test error (4.25) is worse than the line would have scored. Four validation days is a tiny exam: a few easy questions and a memoriser looks brilliant. One small validation set gives a noisy verdict.
Demo 6: k-fold cross-validation
Give every day a turn in the class test. Keep the test days sealed, split the other 20 days into 5 folds of 4, let each fold be the validation set once (the other 16 days train), and average.
import random
days = [(12, 99), (13, 101), (14, 95), (15, 92), (16, 87), (17, 89), (18, 94), (19, 88),
(20, 89), (21, 83), (22, 82), (23, 79), (24, 68), (25, 78), (26, 75), (27, 72),
(28, 60), (29, 57), (30, 60), (31, 60), (32, 62), (33, 58), (34, 59), (35, 51)]
random.seed(2)
random.shuffle(days)
pool, test = days[:20], days[20:] # the test days stay sealed
def mae(model, data):
return sum(abs(model(t) - c) for t, c in data) / len(data)
totals = {"nearest": 0, "line": 0}
for k in range(5): # 5 folds: each 4 days takes a turn
val = pool[k * 4:(k + 1) * 4]
train = pool[:k * 4] + pool[(k + 1) * 4:]
nearest = lambda t: min(train, key=lambda d: abs(d[0] - t))[1]
line = lambda t: 124 - 2.0 * t
totals["nearest"] += mae(nearest, val) / 5
totals["line"] += mae(line, val) / 5
for name, err in totals.items():
print(f"{name:8} average validation error {err:.2f}")
Output:
nearest average validation error 4.45
line average validation error 3.50
With the same unlucky seed, the line wins again. It wins for seeds 1 to 5 as well (line 3.15 / 3.50 / 2.90 / 3.25 / 3.40 vs nearest 4.65 / 4.45 / 4.70 / 3.80 / 5.00). This is k-fold cross-validation (here k = 5). It costs k training runs, but the choice no longer depends on luck. Small data? Cross-validate.
Glossary
| Term | Meaning |
|---|---|
| Generalization | Doing well on data the model has never seen |
| Memorising | Storing the training examples instead of learning a pattern |
| Noise | Random ups and downs in data that are not part of the pattern |
| Overfitting | Learning the training data too closely, noise included: low train error, high new-data error |
| Underfitting | A model too simple to learn the pattern: high error everywhere |
| Good fit | Low error on both training and new data |
| Training set | The data the model learns from |
| Validation set | Held-back data used to compare models and choose settings |
| Test set | Held-back data used once, at the end, for an honest score |
| Data leakage | Test information sneaking into training, giving fake good scores |
| Shuffle / seed | Mixing rows randomly before splitting; a seed makes the mix repeatable |
| Regularization | A training penalty for complexity, to reduce overfitting |
| Early stopping | Stopping training when validation error starts to rise |
| Cross-validation | Rotating which part of the data is the validation set, then averaging |
Common mistakes
| Mistake | Code | Error |
|---|---|---|
| A lookup instead of a model | memory[17] (17 °C never seen) |
KeyError: 17 |
| Test rows inside training (leakage) | train = days |
no error, test error 0.0 |
| Splitting sorted data | last 20% of days sorted by temperature as test | no error, but the test is only the hottest days |
| Choosing a model by test error | comparing 5 models on the test set | no error, an optimistic score |
| Judging by training error | nearest has train error 0.00 |
no error, picks the overfitting model |
Fixes: learn a pattern, not a table; split first and keep the test set apart; shuffle (with a seed) before splitting; choose on validation, report test once.
Practice
- Change
random.seed(1)torandom.seed(2)in Demo 3. Which model does validation choose now? - Add the
averagemodel to Demo 3. What is its validation error? - Split the sorted days without shuffling:
train, val, test = days[:16], days[16:20], days[20:]. Which days are in the test set? What's the hottest training day? - Explain overfitting to a friend with your own exam story.
Solutions
1. With seed 2 the validation errors are nearest 2.50 and line 3.25, so validation picks nearest. With only 4 validation days, luck matters a lot: this is why Demo 6 uses cross-validation. Validation errors for seeds 1 to 5:
seed 1: average 9.28 nearest 6.50 line 2.50
seed 2: average 16.94 nearest 2.50 line 3.25
seed 3: average 12.72 nearest 5.75 line 1.25
seed 4: average 13.25 nearest 3.50 line 4.25
seed 5: average 13.00 nearest 5.75 line 3.50
2. With seed 1, average has a validation error of 9.28: underfitting, by far the worst.
3.
days = [(12, 99), (13, 101), (14, 95), (15, 92), (16, 87), (17, 89), (18, 94),
(19, 88), (20, 89), (21, 83), (22, 82), (23, 79), (24, 68), (25, 78),
(26, 75), (27, 72), (28, 60), (29, 57), (30, 60), (31, 60), (32, 62),
(33, 58), (34, 59), (35, 51)]
train, val, test = days[:16], days[16:20], days[20:]
print("test days:", test)
print("hottest training day:", max(t for t, _ in train), "°C")
test days: [(32, 62), (33, 58), (34, 59), (35, 51)]
hottest training day: 27 °C
The test set holds only the four hottest days, and the model never trained above 27 °C. The test doesn't look like real new data.
4. Example: learning the answers to a cricket quiz app's 50 questions by heart, then failing when the app adds new questions.
Quick quiz 1
A photo classifier: training accuracy 99%, validation accuracy 61%. What's going on?
- A. Underfitting
- B. Overfitting
- C. A good fit
- D. Data leakage
Answer
B, overfitting. Near-perfect on training photos, much worse on new ones: the big gap is the classic sign.
Quick quiz 2
You're choosing between 5 models. Which data should you compare them on?
- A. The training set
- B. The validation set
- C. The test set
- D. All the data together
Answer
B, the validation set. Training error rewards memorising, and the test set stays sealed until the very end.
Interview questions
- What is overfitting? How do you detect it, and name three ways to reduce it.
- What is underfitting, and how is it fixed?
- Why do we need a validation set as well as a test set?
- What is data leakage? Give two examples.
- Why shuffle before splitting, and why set a random seed?
- What is k-fold cross-validation, and when is it useful? Why must the test set stay out of the folds?
Summary
- The goal of a model is generalization: doing well on data it has never seen.
- A memorising model had 0 error and crashed on the first new day (
KeyError: 17). - Overfitting = low training error, high new-data error (memorised noise). Underfitting = high error everywhere.
- Train to learn, validate to choose, test once at the end.
- Data leakage gives fake perfect scores: keep the test set apart, and shuffle before splitting.
- A tiny validation set can pick the wrong model by luck: use k-fold cross-validation.
- Fix overfitting with more data, simpler models, regularization and early stopping.
Next episode: so far we trained models by trying every possible line. That works with two numbers, but real models have millions. How a model improves: loss, gradient descent and the learning rate.