SyntaxPilot
Courses/Machine Learning/Lesson 20

Machine Learning · Lesson 20 of 90 · 11 min

Bias, variance and overfitting

underfittingoverfittingbiasvariancebias-variance trade-offirreducible error (noise)model complexitycapacitytraining vs validation errorlearning curvesvalidation curves

Machine Learning Full Course · Episode 20. Every code sample below was run with Python 3.13, NumPy 2.5.3, scikit-learn 1.9.1 and matplotlib 3.11.2; the output shown is the real output. All data is made up for teaching.

The big idea: a model can fail in two opposite ways. Too simple, and it is wrong in the same way every time (high bias, underfitting). Too flexible, and it memorises the noise in its training data and changes wildly with every new sample (high variance, overfitting). The expected error on new data is bias² + variance + noise, and your job is to find the model capacity where that sum is smallest. Two plots tell you where you stand: the validation curve and the learning curve.

Three students, one exam

Student Strategy Practice paper Real exam ML name
1 "the answer is always C" bad bad underfitting: too simple to capture the pattern
2 memorises last year's paper, typos included 100% fails new questions overfitting: memorised the training data, noise and all
3 understands the chapters good good a good fit that generalises (Episode 15)

The chai stall experiment

Made-up data: cups of chai sold against temperature. Cold days sell more, with a gentle wiggle; every day also has random noise (standard deviation 8 cups). We have 20 days to learn from (training set) and 200 new days to check on (validation set, Episode 15). Temperature is scaled to −1..1 so high powers stay small. We fit polynomials (Episode 19) of degree 1, 4 and 15:

import numpy as np, matplotlib.pyplot as plt
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline
from sklearn.metrics import mean_squared_error as mse
def truth(t): return 150 - 4 * t + 25 * np.sin(t / 5)              # made-up recipe
rng = np.random.default_rng(21)
def sample(n):                     # n days: temperature scaled to -1..1, cups sold (noise sd 8)
    t = rng.uniform(5, 40, n)
    return ((t - 22.5) / 17.5).reshape(-1, 1), truth(t) + rng.normal(0, 8, n)
def poly(d): return make_pipeline(PolynomialFeatures(d), LinearRegression())
X_tr, y_tr = sample(20); X_va, y_va = sample(200)          # 20 days to learn, 200 new days
grid = np.linspace(-1, 1, 300).reshape(-1, 1)
fig, ax = plt.subplots(1, 3, figsize=(15, 4.4), sharey=True)
for a, d in zip(ax, [1, 4, 15]):
    m = poly(d).fit(X_tr, y_tr)
    print(f"degree {d:2}: train MSE {mse(y_tr, m.predict(X_tr)):6.1f}   val MSE {mse(y_va, m.predict(X_va)):10.1f}")
    a.scatter(X_tr, y_tr); a.plot(grid, m.predict(grid), "r"); a.set(title=f"degree {d}", ylim=(0, 220))
plt.savefig("fits.png", dpi=110, bbox_inches="tight")

Output:

degree  1: train MSE  365.7   val MSE      360.2
degree  4: train MSE   25.0   val MSE       65.0
degree 15: train MSE    8.6   val MSE  4280828.2

Degree 1, 4 and 15 fits

Degree Train MSE Validation MSE Diagnosis
1 365.7 360.2 both high: underfitting
4 25.0 65.0 both low, close: good fit
15 8.6 4,280,828.2 train tiny, validation enormous: overfitting

Reading training vs validation error

  • Underfitting: training error high, validation error high.
  • Overfitting: training error low, validation error much higher: a big gap.
  • Good fit: both low and close together.

Irreducible error

The noise has standard deviation 8, so variance 8² = 64. Nobody can predict a surprise wedding or a cricket match, so no model can beat an average squared error of 64 on new days. That floor is the irreducible error (also called noise or Bayes error). Degree 15's training MSE of 8.6 is far below the floor: impossible honestly, so it memorised the noise.

Bias and variance

Throwing darts:

Darts Meaning Model
tight group, left of the bullseye consistently off high bias: wrong on average because its assumptions are too simple
scattered all over, centred on average unstable high variance: predictions change a lot with a different training set
tight group on the bullseye the goal low bias, low variance

To "throw" a model many times, train it on many different training sets drawn from the same source, and look at its predictions at fixed test points:

  • Bias² = (average prediction − truth)², averaged over test points.
  • Variance = how much the predictions spread around their own average.
  • For squared error: expected test MSE = bias² + variance + noise.
import numpy as np, matplotlib.pyplot as plt
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline
from sklearn.metrics import mean_squared_error as mse
def truth(t): return 150 - 4 * t + 25 * np.sin(t / 5)              # made-up recipe
rng = np.random.default_rng(21)
def sample(n):                     # n days: temperature scaled to -1..1, cups sold (noise sd 8)
    t = rng.uniform(5, 40, n)
    return ((t - 22.5) / 17.5).reshape(-1, 1), truth(t) + rng.normal(0, 8, n)
def poly(d): return make_pipeline(PolynomialFeatures(d), LinearRegression())
X_tr, y_tr = sample(20); X_va, y_va = sample(200)          # 20 days to learn, 200 new days
grid = np.linspace(-1, 1, 50).reshape(-1, 1); t = grid.ravel() * 17.5 + 22.5   # 50 test days
for d in [1, 4, 7]:
    preds = np.array([poly(d).fit(*sample(20)).predict(grid) for _ in range(300)])  # 300 training sets
    bias2 = np.mean((preds.mean(axis=0) - truth(t)) ** 2); var = np.mean(preds.var(axis=0))
    test = np.mean((preds - truth(t) - rng.normal(0, 8, preds.shape)) ** 2)
    print(f"degree {d}: bias² {bias2:.1f}   variance {var:.1f}")
    print(f"   sum + noise 64 = {bias2 + var + 64:.1f}   real test MSE {test:.1f}")

Output:

degree 1: bias² 332.9   variance 50.4
   sum + noise 64 = 447.3   real test MSE 450.0
degree 4: bias² 1.6   variance 51.5
   sum + noise 64 = 117.1   real test MSE 119.0
degree 7: bias² 377.6   variance 57589.8
   sum + noise 64 = 58031.4   real test MSE 58029.5
  • Degree 1: bias² 332.9, variance 50.4. Mostly bias: underfitting in numbers.
  • Degree 4: bias² 1.6, variance 51.5: both small.
  • Degree 7: variance 57,589.8: overfitting in numbers. (Its bias² of 377.6 is the average of wild curves, not a systematic shape error.)
  • The decomposition checks out: 447.3 vs a measured 450.0, 117.1 vs 119.0, 58,031.4 vs 58,029.5.

Capacity, complexity and the trade-off

  • Capacity: how complicated a shape a model is able to fit.
  • Model complexity knobs: anything that changes capacity: polynomial degree, tree depth (Episode 29), number of features, number of neural-network layers, regularisation strength (Episode 21, reversed).
  • More capacity: bias ↓, variance ↑. Less capacity: bias ↑, variance ↓.
  • Bias-variance trade-off: with a fixed amount of data you can't push both to zero; pick the capacity with the lowest bias² + variance, which is not the one with the lowest training error.

Validation curves

Turn the knob step by step and record both errors:

import numpy as np, matplotlib.pyplot as plt
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline
from sklearn.metrics import mean_squared_error as mse
def truth(t): return 150 - 4 * t + 25 * np.sin(t / 5)              # made-up recipe
rng = np.random.default_rng(21)
def sample(n):                     # n days: temperature scaled to -1..1, cups sold (noise sd 8)
    t = rng.uniform(5, 40, n)
    return ((t - 22.5) / 17.5).reshape(-1, 1), truth(t) + rng.normal(0, 8, n)
def poly(d): return make_pipeline(PolynomialFeatures(d), LinearRegression())
X_tr, y_tr = sample(20); X_va, y_va = sample(200)          # 20 days to learn, 200 new days
degrees, trs, vas = [1, 2, 3, 4, 5, 6, 8, 10, 12, 15], [], []
for d in degrees:                                                          # turn the knob
    m = poly(d).fit(X_tr, y_tr); trs.append(mse(y_tr, m.predict(X_tr))); vas.append(mse(y_va, m.predict(X_va)))
    print(f"degree {d:2}: train {trs[-1]:6.1f}  validation {vas[-1]:10.1f}")
plt.figure(figsize=(10, 4.8)); plt.yscale("log"); plt.xlabel("polynomial degree (capacity)")
plt.plot(degrees, trs, "o-", label="train"); plt.plot(degrees, vas, "s-", label="validation")
plt.legend(); plt.savefig("validation_curve.png", dpi=110, bbox_inches="tight")

Output:

degree  1: train  365.7  validation      360.2
degree  2: train   62.4  validation      194.2
degree  3: train   61.5  validation      227.1
degree  4: train   25.0  validation       65.0
degree  5: train   24.8  validation       63.7
degree  6: train   24.2  validation       95.3
degree  8: train   22.5  validation      815.0
degree 10: train   14.4  validation    47092.4
degree 12: train    9.0  validation   589485.4
degree 15: train    8.6  validation  4280828.2

Validation curve

  • Training error only goes down as capacity grows.
  • Validation error is a U: falls, bottoms out at degree 5 (63.7, right at the noise floor of 64), then climbs to millions.
  • Left side = underfitting (high bias). Right side = overfitting (high variance). Pick the bottom.

A validation curve is error (or score) against a complexity knob, for train and validation. scikit-learn's validation_curve does the same with cross-validation (Episode 26), averaging several splits:

import numpy as np
from sklearn.model_selection import validation_curve
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline
def truth(t): return 150 - 4 * t + 25 * np.sin(t / 5)
rng = np.random.default_rng(21)
t = rng.uniform(5, 40, 60); X = ((t - 22.5) / 17.5).reshape(-1, 1)
y = truth(t) + rng.normal(0, 8, 60)
model = make_pipeline(PolynomialFeatures(), LinearRegression())
degrees = [1, 2, 4, 6, 10]
train, val = validation_curve(model, X, y, param_name="polynomialfeatures__degree",
                              param_range=degrees, cv=5, scoring="neg_mean_squared_error")
for d, tr, va in zip(degrees, -train.mean(axis=1), -val.mean(axis=1)):
    print(f"degree {d:2}: train {tr:8.1f}  validation {va:10.1f}")

Output:

degree  1: train    359.1  validation      381.0
degree  2: train    113.4  validation      131.3
degree  4: train     50.4  validation       59.8
degree  6: train     47.9  validation       60.8
degree 10: train     43.6  validation       67.0

(scikit-learn maximises scores, so errors come back negated: neg_mean_squared_error. The parameter name is <step name>__<parameter>.)

Learning curves

"Should I note down 300 more days?" Train on growing amounts of data and watch both errors:

import numpy as np, matplotlib.pyplot as plt
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline
from sklearn.metrics import mean_squared_error as mse
def truth(t): return 150 - 4 * t + 25 * np.sin(t / 5)              # made-up recipe
rng = np.random.default_rng(21)
def sample(n):                     # n days: temperature scaled to -1..1, cups sold (noise sd 8)
    t = rng.uniform(5, 40, n)
    return ((t - 22.5) / 17.5).reshape(-1, 1), truth(t) + rng.normal(0, 8, n)
def poly(d): return make_pipeline(PolynomialFeatures(d), LinearRegression())
X_tr, y_tr = sample(20); X_va, y_va = sample(200)          # 20 days to learn, 200 new days
def scores(d, n):                                      # average over 30 repeats
    runs = [sample(n) for _ in range(30)]; models = [poly(d).fit(X, y) for X, y in runs]
    return (np.mean([mse(y, m.predict(X)) for (X, y), m in zip(runs, models)]),
            np.mean([mse(y_va, m.predict(X_va)) for m in models]))
fig, ax = plt.subplots(1, 2, figsize=(13, 4.6), sharey=True); sizes = [20, 40, 80, 160, 320]
for a, d in zip(ax, [1, 8]):
    tr, va = zip(*[scores(d, n) for n in [20, 40, 80, 160, 320]])
    print(f"degree {d}  train:", np.round(tr).astype(int), "\n     validation:", np.round(va).astype(int))
    a.plot(sizes, tr, "o-", label="train"); a.plot(sizes, va, "s-", label="validation")
    a.set(title=f"degree {d}", xlabel="training days", yscale="log"); a.legend()
plt.savefig("learning_curves.png", dpi=110, bbox_inches="tight")

Output:

degree 1  train: [375 380 382 383 394] 
     validation: [397 383 362 357 355]
degree 8  train: [34 47 61 60 62] 
     validation: [8026  744   70   64   59]

Learning curves for degree 1 and degree 8

  • Degree 1 (high bias): the curves meet quickly, but high (about 360-400) and stay there from 20 to 320 days. More data is wasted; add capacity or better features.
  • Degree 8 (high variance): a huge gap at first (train 34, validation 8,026), closing as data grows (320 days: 62 vs 59, at the noise floor). More data helps.

A learning curve is error against training-set size, for train and validation. scikit-learn's helper is learning_curve:

import numpy as np
from sklearn.model_selection import learning_curve
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline
def truth(t): return 150 - 4 * t + 25 * np.sin(t / 5)
rng = np.random.default_rng(21)
t = rng.uniform(5, 40, 400); X = ((t - 22.5) / 17.5).reshape(-1, 1)
y = truth(t) + rng.normal(0, 8, 400)
for d in [1, 8]:
    sizes, train, val = learning_curve(make_pipeline(PolynomialFeatures(d), LinearRegression()), X, y,
                                       train_sizes=[20, 40, 80, 160, 320], cv=5, shuffle=True,
                                       random_state=0, scoring="neg_mean_squared_error")
    print(f"degree {d}: train", np.round(-train.mean(axis=1)).astype(int),
          " validation", np.round(-val.mean(axis=1)).astype(int))

Output:

degree 1: train [289 321 346 364 374]  validation [444 435 387 376 376]
degree 8: train [37 54 54 59 64]  validation [186985  57817     82     69     67]

Diagnosis table

High bias (underfitting) High variance (overfitting)
Symptom train high, validation high, close train low, validation much higher
Learning curve curves meet early, high big gap that shrinks with data
More data won't help helps
Fix more capacity (higher degree, deeper tree, more features), better features, less regularisation more data, less capacity, fewer features, regularisation (Episode 21), ensembles (Episodes 33-34), early stopping

Common mistakes

Mistake Real symptom Why Fix
picking the model with the lowest training error degree 15: train 8.6, validation 4,280,828.2 memorised noise pick by validation error
being proud of a training error below the noise 8.6 < 64 irreducible error is a floor compare with the noise level / a baseline
collecting more data for an underfit model degree 1 stays near 380 from 20 to 320 days bias, not variance add capacity or features
tuning the degree on the test set an optimistic test score the test set leaked into your choices tune on validation (or CV), test once at the end
one validation split for a small dataset the "best" degree changes with the split validation error is itself noisy cross-validation (Episode 26)
high powers of raw inputs numbers like 40¹⁵ ≈ 10²⁴ huge, badly conditioned features scale inputs first (we used −1..1)

Mini project: a model doctor

A function that fits a model on growing training sizes and prints a diagnosis:

import numpy as np
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline
from sklearn.metrics import mean_squared_error as mse

def doctor(make_model, sample, X_va, y_va, noise_var, sizes=(20, 80, 320), repeats=20):
    rows = []
    for n in sizes:
        tr = va = 0
        for _ in range(repeats):
            X, y = sample(n); m = make_model().fit(X, y)
            tr += mse(y, m.predict(X)) / repeats; va += mse(y_va, m.predict(X_va)) / repeats
        rows.append((n, tr, va)); print(f"  {n:3} rows: train {tr:9.1f}  validation {va:9.1f}")
    n, tr, va = rows[-1]
    if va > 2 * noise_var and va - tr < 0.5 * va:
        print("  diagnosis: HIGH BIAS - add capacity or features; more data won't help")
    elif rows[0][2] > 2 * rows[-1][2]:
        print("  diagnosis: HIGH VARIANCE at small sizes - more data, less capacity or regularisation")
    else:
        print("  diagnosis: close to the noise floor - good fit")

def truth(t): return 150 - 4 * t + 25 * np.sin(t / 5)
rng = np.random.default_rng(21)
def sample(n):
    t = rng.uniform(5, 40, n)
    return ((t - 22.5) / 17.5).reshape(-1, 1), truth(t) + rng.normal(0, 8, n)
X_va, y_va = sample(500)
for d in [1, 4, 8]:
    print(f"degree {d}:")
    doctor(lambda: make_pipeline(PolynomialFeatures(d), LinearRegression()), sample, X_va, y_va, noise_var=64)

Output:

degree 1:
   20 rows: train     345.4  validation     452.2
   80 rows: train     382.4  validation     386.2
  320 rows: train     386.6  validation     373.7
  diagnosis: HIGH BIAS - add capacity or features; more data won't help
degree 4:
   20 rows: train      43.6  validation      93.7
   80 rows: train      58.4  validation      68.3
  320 rows: train      63.6  validation      65.5
  diagnosis: close to the noise floor - good fit
degree 8:
   20 rows: train      39.8  validation   92125.1
   80 rows: train      56.1  validation      77.0
  320 rows: train      62.3  validation      66.0
  diagnosis: HIGH VARIANCE at small sizes - more data, less capacity or regularisation

Practice

  1. Repeat the validation curve with 60 training days instead of 20. Where is the best degree now?
  2. Measure bias² and variance for degrees 2, 3 and 5.
  3. Raise the noise standard deviation to 20. What happens to the irreducible error and to the best degree?
  4. Draw learning curves (as numbers) for degree 4.
Solution 1
import numpy as np, matplotlib.pyplot as plt
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline
from sklearn.metrics import mean_squared_error as mse
def truth(t): return 150 - 4 * t + 25 * np.sin(t / 5)              # made-up recipe
rng = np.random.default_rng(21)
def sample(n):                     # n days: temperature scaled to -1..1, cups sold (noise sd 8)
    t = rng.uniform(5, 40, n)
    return ((t - 22.5) / 17.5).reshape(-1, 1), truth(t) + rng.normal(0, 8, n)
def poly(d): return make_pipeline(PolynomialFeatures(d), LinearRegression())
X_tr, y_tr = sample(20); X_va, y_va = sample(200)          # 20 days to learn, 200 new days
X_tr, y_tr = sample(60)                             # 60 days this time
for d in [1, 2, 3, 4, 5, 6, 8, 10, 12, 15]:
    m = poly(d).fit(X_tr, y_tr)
    print(f"degree {d:2}: train {mse(y_tr, m.predict(X_tr)):6.1f}  validation {mse(y_va, m.predict(X_va)):9.1f}")

Output:

degree  1: train  435.0  validation     361.4
degree  2: train  140.6  validation     117.8
degree  3: train  116.0  validation      96.0
degree  4: train   59.6  validation      61.0
degree  5: train   58.5  validation      62.6
degree  6: train   53.8  validation      73.1
degree  8: train   53.6  validation      73.7
degree 10: train   53.0  validation      74.1
degree 12: train   50.7  validation      78.3
degree 15: train   47.9  validation     103.3

With three times more data, high degrees overfit far less: the right side of the U is much lower and flatter, because more data reduces variance.

Solution 2
import numpy as np, matplotlib.pyplot as plt
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline
from sklearn.metrics import mean_squared_error as mse
def truth(t): return 150 - 4 * t + 25 * np.sin(t / 5)              # made-up recipe
rng = np.random.default_rng(21)
def sample(n):                     # n days: temperature scaled to -1..1, cups sold (noise sd 8)
    t = rng.uniform(5, 40, n)
    return ((t - 22.5) / 17.5).reshape(-1, 1), truth(t) + rng.normal(0, 8, n)
def poly(d): return make_pipeline(PolynomialFeatures(d), LinearRegression())
X_tr, y_tr = sample(20); X_va, y_va = sample(200)          # 20 days to learn, 200 new days
grid = np.linspace(-1, 1, 50).reshape(-1, 1); t = grid.ravel() * 17.5 + 22.5
for d in [2, 3, 5]:
    preds = np.array([poly(d).fit(*sample(20)).predict(grid) for _ in range(300)])
    bias2 = np.mean((preds.mean(axis=0) - truth(t)) ** 2); var = np.mean(preds.var(axis=0))
    print(f"degree {d}: bias² {bias2:6.1f}  variance {var:7.1f}  total {bias2 + var + 64:7.1f}")

Output:

degree 2: bias²   59.8  variance    29.9  total   153.7
degree 3: bias²   60.0  variance    95.0  total   219.0
degree 5: bias²    2.4  variance   256.0  total   322.4

Bias falls as the degree rises (the wiggle needs a few powers), while variance rises; degree 4 from the episode sits near the sweet spot.

Solution 3
import numpy as np
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline
from sklearn.metrics import mean_squared_error as mse
def truth(t): return 150 - 4 * t + 25 * np.sin(t / 5)
rng = np.random.default_rng(21)
def sample(n, sd):
    t = rng.uniform(5, 40, n)
    return ((t - 22.5) / 17.5).reshape(-1, 1), truth(t) + rng.normal(0, sd, n)
for sd in [8, 20]:
    X_tr, y_tr = sample(20, sd); X_va, y_va = sample(2000, sd)
    vals = {d: mse(y_va, make_pipeline(PolynomialFeatures(d), LinearRegression()).fit(X_tr, y_tr).predict(X_va))
            for d in [1, 2, 3, 4, 5, 6]}
    best = min(vals, key=vals.get)
    print(f"noise sd {sd}: floor {sd ** 2}, validation", {d: round(v) for d, v in vals.items()}, "best degree", best)

Output:

noise sd 8: floor 64, validation {1: 407, 2: 164, 3: 181, 4: 71, 5: 71, 6: 85} best degree 5
noise sd 20: floor 400, validation {1: 874, 2: 675, 3: 1902, 4: 3657, 5: 24917, 6: 7946} best degree 2

The floor rises from 64 to 400 (sd²). With more noise, the same 20 days carry less information, so variance grows and the best degree usually drops: simpler models win when data is noisy.

Solution 4
import numpy as np, matplotlib.pyplot as plt
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline
from sklearn.metrics import mean_squared_error as mse
def truth(t): return 150 - 4 * t + 25 * np.sin(t / 5)              # made-up recipe
rng = np.random.default_rng(21)
def sample(n):                     # n days: temperature scaled to -1..1, cups sold (noise sd 8)
    t = rng.uniform(5, 40, n)
    return ((t - 22.5) / 17.5).reshape(-1, 1), truth(t) + rng.normal(0, 8, n)
def poly(d): return make_pipeline(PolynomialFeatures(d), LinearRegression())
X_tr, y_tr = sample(20); X_va, y_va = sample(200)          # 20 days to learn, 200 new days
for n in [20, 40, 80, 160, 320]:
    tr = va = 0
    for _ in range(30):
        X, y = sample(n); m = poly(4).fit(X, y)
        tr += mse(y, m.predict(X)) / 30; va += mse(y_va, m.predict(X_va)) / 30
    print(f"{n:3} days: train {tr:6.1f}  validation {va:6.1f}")

Output:

 20 days: train   44.1  validation   95.9
 40 days: train   62.0  validation   71.3
 80 days: train   59.5  validation   64.0
160 days: train   63.9  validation   60.9
320 days: train   63.1  validation   60.1

Degree 4 starts with a small gap, and both curves settle near the noise floor of 64: low bias and, with enough data, low variance.

Quick quiz

train MSE:       4.1
validation MSE: 96.3
noise variance: about 60

What is the problem, and what will help?

A) High variance: more data or less capacity B) High bias: more capacity C) Perfect fit: ship it D) Irreducible error: nothing helps

Answer

A. Training error far below the noise floor and a big gap to validation = overfitting (high variance). More data, less capacity, or regularisation.

Interview questions

  1. Define bias and variance. Bias: error from wrong assumptions; the gap between the average prediction (over training sets) and the truth. Variance: how much predictions change when the training set changes.
  2. State the bias-variance decomposition. For squared error: E[(y − ŷ)²] = bias² + variance + irreducible noise σ².
  3. How do you detect overfitting? Low training error with much higher validation error; a validation curve turning up; a learning curve with a large gap.
  4. Will more data fix underfitting? No. Learning curves converge at a high error; add capacity or features.
  5. Name ways to reduce variance. More data, simpler models, feature selection, regularisation, bagging/ensembles, early stopping, data augmentation.
  6. What is irreducible error? Noise in the target that no model can predict from the features; a floor on expected error.
  7. Validation curve vs learning curve? Validation curve: score vs a hyperparameter (complexity). Learning curve: score vs training-set size.
  8. Why not tune hyperparameters on the test set? The choice adapts to the test data, so the test score becomes optimistic; tune on validation or cross-validation and keep the test set for one final check.

Glossary

Term Meaning Example
Underfitting Model too simple; high error on train and new data degree 1: 365.7 / 360.2
Overfitting Model memorises training noise; low train, high new-data error degree 15: 8.6 / 4,280,828.2
Generalisation Doing well on data not seen in training degree 4
Training error Error on the data the model learned from train MSE
Validation error Error on held-back data used for choices validation MSE
Gap Validation error − training error large = high variance
Noise Random part of the target the features can't explain sd 8 cups
Irreducible error The noise variance: a floor no model can beat 64
Bias Error of the average prediction vs the truth (too-simple assumptions) degree 1: bias² 332.9
Variance Spread of predictions across training sets degree 7: 57,589.8
Bias-variance decomposition Expected MSE = bias² + variance + noise 117.1 vs 119.0
Bias-variance trade-off Lowering one usually raises the other degree up: bias ↓, variance ↑
Capacity How complicated a shape a model can fit degree 15 > degree 4
Model complexity The settings that control capacity degree, depth, features
Hyperparameter A setting you choose, not learned from data degree
Validation curve Train and validation error vs a complexity knob U-shaped
Learning curve Train and validation error vs training-set size gap closes for degree 8
validation_curve scikit-learn helper, with cross-validation param_name="polynomialfeatures__degree"
learning_curve scikit-learn helper, with cross-validation train_sizes=[20, ..., 320]
neg_mean_squared_error MSE with a minus sign, so higher is better negate to read it
Log scale An axis where each step multiplies by 10 10, 100, 1,000

Summary

  • Underfitting = high error everywhere (high bias); overfitting = low training error, much higher validation error (high variance).
  • Expected test error = bias² + variance + irreducible noise; our measured numbers matched (117.1 vs 119.0).
  • Capacity trades bias for variance; choose it at the bottom of the validation curve (degree 5, 63.7, at the noise floor).
  • Learning curves tell you whether more data helps: no for high bias (curves meet high), yes for high variance (gap closes).

Next episode: regularisation: keep a high-capacity model but calm its weights with ridge, lasso and elastic net.