Machine Learning Full Course · Episode 63. Every code sample below was run with Python 3.13 and PyTorch 2.15.0 (CPU, one thread, as in the video's compiler). The output shown is the real output. Long trainings can differ slightly in the last digit on other machines or thread counts.
The big idea: a big network can memorise a small dataset, including its wrong labels: 100% training accuracy, worse validation accuracy. Regularisation is anything that makes memorising harder or stops it in time, so the model generalises. For deep networks the toolkit is: dropout (randomly switch off neurons while training), weight decay (keep weights small), data augmentation (train on slightly changed copies), early stopping (stop when validation stops improving, keep the best weights), batch normalisation and layer normalisation (standardise values inside the network; batch norm also adds a little noise), and label smoothing (don't reward 100% confidence).
Sources: Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever and Ruslan Salakhutdinov, "Dropout: A Simple Way to Prevent Neural Networks from Overfitting", JMLR 15(56):1929−1958, 2014 (jmlr.org). Sergey Ioffe and Christian Szegedy, "Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift", 2015 (arXiv 1502.03167). Jimmy Lei Ba, Jamie Ryan Kiros and Geoffrey E. Hinton, "Layer Normalization", 2016 (arXiv 1607.06450). Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas and Aleksander Madry, "How Does Batch Normalization Help Optimization?", NeurIPS 2018 (proceedings).
Words you need first
- Overfitting (Episode 20): great on training data, poor on new data.
- Generalisation: doing well on examples the model never saw.
- Training / validation / test set (Episode 15): learn on training; measure while building on validation; report once on test.
- Regularisation (Episode 21): any technique that reduces overfitting.
- Label noise: training labels that are wrong (here 15% are flipped on purpose).
- Logits (Episode 59): a network's raw scores before sigmoid/softmax.
- Cross-entropy (Episodes 12 and 23): the classification loss
−log p(right class). model.train()/model.eval()(Episode 59): switch layers like dropout and batch norm between training and evaluation behaviour.state_dict: a dictionary of all of a model's weights, used to save and load them.
The problem: a big network memorises noisy labels
import torch, torch.nn as nn
torch.manual_seed(63)
def make(n): # the circle, but 15% of labels are wrong (noise)
X = torch.rand(n, 2) * 4 - 2; y = (X.pow(2).sum(1) < 1.5).float()
flip = torch.rand(n) < 0.15; y[flip] = 1 - y[flip]
return X, y.unsqueeze(1)
Xtr, ytr = make(100); Xva, yva = make(1000) # only 100 training examples
acc = lambda m, X, y: ((m(X) > 0).float() == y).float().mean().item()
torch.manual_seed(0)
model = nn.Sequential(nn.Linear(2, 256), nn.ReLU(), nn.Linear(256, 256), nn.ReLU(), nn.Linear(256, 1))
opt, loss_fn = torch.optim.Adam(model.parameters(), lr=0.01), nn.BCEWithLogitsLoss()
for epoch in range(1, 1001):
loss = loss_fn(model(Xtr), ytr); opt.zero_grad(); loss.backward(); opt.step()
if epoch in (50, 200, 1000):
tr, va = acc(model, Xtr, ytr), acc(model, Xva, yva)
print(f"epoch {epoch:4}: train acc {tr:.0%} validation acc {va:.1%}")
Output:
epoch 50: train acc 83% validation acc 77.6%
epoch 200: train acc 92% validation acc 74.0%
epoch 1000: train acc 100% validation acc 72.9%
The network has 2·256 + 256 + 256·256 + 256 + 256 + 1 = 66,817 parameters for 100 examples. Training accuracy climbs to 100%, but 15 of the 100 labels are wrong, so a perfect score means it learned the mistakes; validation falls from 77.6% to 72.9%. About 85% is the best any model can score on this validation set, because 15% of its labels are flipped too.
Dropout
A cricket coach sends random players to sit out at every practice; nobody can rely on one star, so the team gets stronger; on match day the full XI plays.
- Training: each value coming out of the layer is set to 0 with probability
p; a new random choice every forward pass. - Inverted dropout scaling: the surviving values are multiplied by
1/(1 − p)so the average signal stays the same (×2 forp = 0.5). - Evaluation: nothing is dropped or scaled.
model.eval()turns dropout off;model.train()turns it back on. - Why it works: neurons can't co-adapt (depend on specific partners), and training samples many "thinned" networks whose average the full network approximates at test time (Srivastava et al., 2014).
import torch, torch.nn as nn
torch.manual_seed(63)
drop = nn.Dropout(p=0.5) # each value is zeroed with probability 0.5
x = torch.ones(10)
drop.train(); print("train:", drop(x).tolist())
print("train:", drop(x).tolist()) # a new random pattern each call
drop.eval(); print("eval: ", drop(x).tolist())
big = torch.ones(100_000); drop.train()
print("average kept value in train mode:", round(drop(big).mean().item(), 3))
Output:
train: [2.0, 0.0, 0.0, 0.0, 2.0, 2.0, 2.0, 2.0, 0.0, 2.0]
train: [0.0, 2.0, 0.0, 0.0, 0.0, 2.0, 2.0, 0.0, 2.0, 2.0]
eval: [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
average kept value in train mode: 0.996
Weight decay and data augmentation
- Weight decay (Episode 61's AdamW): shrink every weight by a small fraction each step; small weights give a smoother, simpler function.
- Data augmentation (Episode 60): train on randomly changed copies. Here each training point is moved by Gaussian noise with std 0.2, new noise every epoch, so no exact point can be memorised. For images: flips, crops, colour changes; for audio: noise, time shifts.
import torch, torch.nn as nn
torch.manual_seed(63)
def make(n): # the circle, with 15% of labels flipped
X = torch.rand(n, 2) * 4 - 2; y = (X.pow(2).sum(1) < 1.5).float()
flip = torch.rand(n) < 0.15; y[flip] = 1 - y[flip]
return X, y.unsqueeze(1)
Xtr, ytr = make(100); Xva, yva = make(1000)
def train(name, p_drop=0.0, wd=0.0, noise=0.0):
torch.manual_seed(0); L, D = nn.Linear, lambda: nn.Dropout(p_drop)
model = nn.Sequential(L(2, 256), nn.ReLU(), D(), L(256, 256), nn.ReLU(), D(), L(256, 1))
opt, loss_fn = torch.optim.AdamW(model.parameters(), 0.01, weight_decay=wd), nn.BCEWithLogitsLoss()
for epoch in range(1000):
model.train(); Xb = Xtr + noise * torch.randn_like(Xtr) # dropout ON; jiggle points
loss = loss_fn(model(Xb), ytr); opt.zero_grad(); loss.backward(); opt.step()
model.eval(); acc = lambda X, y: ((model(X) > 0).float() == y).float().mean().item() # dropout OFF
print(f"{name:20} train {acc(Xtr, ytr):4.0%} validation {acc(Xva, yva):.1%}")
train("nothing"); train("dropout 0.5", p_drop=0.5); train("weight decay 1.0", wd=1.0)
train("augmentation 0.2", noise=0.2); train("all three", p_drop=0.5, wd=1.0, noise=0.2)
Output:
nothing train 100% validation 72.9%
dropout 0.5 train 88% validation 75.1%
weight decay 1.0 train 87% validation 76.8%
augmentation 0.2 train 84% validation 80.2%
all three train 77% validation 80.4%
| Run | Train | Validation |
|---|---|---|
| nothing | 100% | 72.9% |
| dropout 0.5 | 88% | 75.1% |
| weight decay 1.0 | 87% | 76.8% |
| augmentation 0.2 | 84% | 80.2% |
| all three | 77% | 80.4% |
Lower training accuracy with higher validation accuracy is the sign that the model stopped memorising noise.
Early stopping and patience
Chai gets better as it boils, then turns bitter; you taste as you go and serve the best moment.
- After every epoch, compute the validation loss.
- If it is the best so far, save a copy of the weights (
copy.deepcopy(model.state_dict())). - Patience: the number of epochs to wait without improvement before stopping (the loss wobbles, so one bad epoch is not enough).
- When patience runs out, stop and load the best weights back.
import torch, torch.nn as nn, copy
torch.manual_seed(63)
def make(n): # the same noisy circle
X = torch.rand(n, 2) * 4 - 2; y = (X.pow(2).sum(1) < 1.5).float()
flip = torch.rand(n) < 0.15; y[flip] = 1 - y[flip]
return X, y.unsqueeze(1)
Xtr, ytr = make(100); Xva, yva = make(1000)
torch.manual_seed(0)
model = nn.Sequential(nn.Linear(2, 256), nn.ReLU(), nn.Linear(256, 256), nn.ReLU(), nn.Linear(256, 1))
opt, loss_fn = torch.optim.Adam(model.parameters(), lr=0.01), nn.BCEWithLogitsLoss()
best, patience, waited = float("inf"), 20, 0
for epoch in range(1, 1001):
loss = loss_fn(model(Xtr), ytr); opt.zero_grad(); loss.backward(); opt.step()
with torch.no_grad(): val = loss_fn(model(Xva), yva).item()
if val < best: best, best_epoch, state, waited = val, epoch, copy.deepcopy(model.state_dict()), 0
else: waited += 1 # no improvement this epoch
if waited == patience: break # 20 epochs without a new best: stop
model.load_state_dict(state); acc = ((model(Xva) > 0).float() == yva).float().mean().item()
print(f"stopped at epoch {epoch}; best val loss {best:.3f} at epoch {best_epoch}\naccuracy {acc:.1%}")
Output:
stopped at epoch 45; best val loss 0.539 at epoch 25
accuracy 79.6%
Stopped at epoch 45 instead of 1,000, with 79.6% instead of 72.9%: cheaper and better. Caution: the validation set chose the epoch, so its score is slightly optimistic; report the final number on a separate test set (see the mini project). copy.deepcopy matters: state_dict() returns references to the live tensors, which keep changing as training continues.
Batch normalisation
Inside a deep network, each layer's outputs can drift to any scale as earlier layers learn. Batch normalisation (BatchNorm) standardises inside the network:
for each feature (column), over the current batch:
x̂ = (x − batch mean) / √(batch variance + ε)
y = γ · x̂ + β # γ (gamma) and β (beta): a learned scale and shift
γandβlet the network undo the normalisation if that helps.- Running statistics: during training it also keeps moving averages of the mean and variance (
momentum=0.1: each batch moves them 10% of the way). Eval mode uses these, so a prediction doesn't depend on the other examples in the batch. - PyTorch:
nn.BatchNorm1d(features)for vectors,nn.BatchNorm2d(channels)for images (Episode 65). Usually placed after a Linear/Conv layer, before the activation. - Effects: allows higher learning rates, less sensitivity to initialisation, and a mild regularising noise (each example's output depends on its random batch-mates).
import torch, torch.nn as nn
x = torch.tensor([[1.0, 100.0], [2.0, 300.0], [3.0, 500.0], [4.0, 700.0]]) # 4 examples, 2 features
bn = nn.BatchNorm1d(2) # normalise each FEATURE (column) over the batch
r = lambda t: [round(v, 2) for v in t.tolist()]
bn.train(); out = bn(x).detach()
for j in (0, 1): print(f"train, feature {j + 1}:", r(out[:, j]))
print("means:", r(out.mean(0)), " stds:", r(out.std(0, unbiased=False)))
print("running mean after 1 batch:", r(bn.running_mean))
for _ in range(99): bn(x) # 99 more training batches
print("running mean after 100: ", r(bn.running_mean))
bn.eval(); out = bn(x).detach() # eval: uses the running statistics
for j in (0, 1): print(f"eval, feature {j + 1}:", r(out[:, j]))
bn.train()
try:
bn(torch.tensor([[1.0, 100.0]])) # a batch of ONE example in train mode
except ValueError as e:
print("ValueError:", e)
Output:
train, feature 1: [-1.34, -0.45, 0.45, 1.34]
train, feature 2: [-1.34, -0.45, 0.45, 1.34]
means: [0.0, 0.0] stds: [1.0, 1.0]
running mean after 1 batch: [0.25, 40.0]
running mean after 100: [2.5, 399.99]
eval, feature 1: [-1.16, -0.39, 0.39, 1.16]
eval, feature 2: [-1.16, -0.39, 0.39, 1.16]
ValueError: Expected more than 1 value per channel when training, got input size torch.Size([1, 2])
- Both columns become −1.34 … 1.34 with mean 0 and std 1: the hundreds and the single digits are on the same scale.
- Running mean after one batch is
0.1 × (2.5, 400) = (0.25, 40); after 100 batches it is the true mean (2.5, 400). - Eval output (−1.16 …) is close to, not identical to, train output (−1.34 …): the running variance uses the unbiased estimate (divides by n − 1 = 3), while training uses the batch's own variance (divides by n = 4). With realistic batch sizes the difference is tiny.
- A batch of one example in train mode:
ValueError: Expected more than 1 value per channel when training, got input size torch.Size([1, 2]).
Layer normalisation
Layer normalisation (LayerNorm): normalise each example over its own features, instead of each feature over the batch.
| BatchNorm | LayerNorm | |
|---|---|---|
| Statistics over | the batch, per feature (down each column) | the features, per example (across each row) |
| Batch size 1 | fails in training | works |
| Train vs eval | different (batch stats vs running stats) | identical computation |
| Typical use | CNNs (images) | transformers, RNNs |
Ba, Kiros and Hinton (2016) note that layer norm does the same computation at training and test time and can be applied to recurrent networks.
import torch, torch.nn as nn
x = torch.tensor([[2.0, 4.0, 9.0], [10.0, 20.0, 45.0], [1.0, 1.0, 1.0]]) # 3 examples, 3 features
ln = nn.LayerNorm(3) # normalise each EXAMPLE (row) over its own features
out = ln(x).detach()
for i in range(3): print(f"row {i + 1}:", [round(v, 3) for v in out[i].tolist()])
print("row means:", out.mean(1).round(decimals=3).tolist())
print("one example alone:", [round(v, 3) for v in ln(x[:1]).detach()[0].tolist()])
Output:
row 1: [-1.019, -0.34, 1.359]
row 2: [-1.019, -0.34, 1.359]
row 3: [0.0, 0.0, 0.0]
row means: [0.0, 0.0, 0.0]
one example alone: [-1.019, -0.34, 1.359]
Rows 1 and 2 (the second is 5 × the first) give the same output: LayerNorm ignores an example's overall scale. The all-ones row has zero variance, so it becomes all zeros (ε prevents dividing by zero).
Label smoothing
Cross-entropy keeps rewarding more confidence, all the way to 100%, even when some labels are wrong. Label smoothing with ε = 0.1 makes the target 90% the right class + 10% spread evenly over all K classes (so the right class gets 0.9 + 0.1/K).
import torch, torch.nn as nn
target = torch.tensor([0]) # the right class is class 0 (of 3)
for logits in ([4.0, 0.0, 0.0], [10.0, 0.0, 0.0], [30.0, 0.0, 0.0]): # more and more sure
z = torch.tensor([logits])
hard = nn.CrossEntropyLoss()(z, target).item()
soft = nn.CrossEntropyLoss(label_smoothing=0.1)(z, target).item()
p = torch.softmax(z, 1)[0, 0].item()
print(f"z={logits} p {p:.5f} loss {hard:.4f} smoothed {soft:.4f}")
Output:
z=[4.0, 0.0, 0.0] p 0.96466 loss 0.0360 smoothed 0.3026
z=[10.0, 0.0, 0.0] p 0.99991 loss 0.0001 smoothed 0.6668
z=[30.0, 0.0, 0.0] p 1.00000 loss 0.0000 smoothed 2.0000
Plain loss falls to 0 as the logit grows; the smoothed loss is lowest at a moderate confidence and rises again (0.30 → 0.67 → 2.00), so the model is not pushed to extreme logits. In PyTorch: nn.CrossEntropyLoss(label_smoothing=0.1).
The internal covariate shift debate
- Internal covariate shift: the change in a layer's input distribution during training, because the layers before it keep updating (Episode 82 covers ordinary covariate shift between training and live data).
- Ioffe and Szegedy (2015) proposed BatchNorm as a way to reduce internal covariate shift; the phrase is in the paper's title. They argued the shift slows training, forcing lower learning rates and careful initialisation.
- Santurkar, Tsipras, Ilyas and Madry (NeurIPS 2018) argued that keeping layer inputs stable contributes little to BatchNorm's success, and that BatchNorm instead makes the optimisation landscape significantly smoother, so gradients are more predictive and stable, allowing faster training.
- Lesson: a method can work very well before we fully understand why. Treat an explanation as a hypothesis and ask for the experiment.
Where each tool goes in a training loop
model = nn.Sequential(nn.Linear(2, 256), nn.BatchNorm1d(256), nn.ReLU(), nn.Dropout(0.5),
nn.Linear(256, 256), nn.BatchNorm1d(256), nn.ReLU(), nn.Dropout(0.5), nn.Linear(256, 1))
opt = torch.optim.AdamW(model.parameters(), lr=0.01, weight_decay=1.0) # weight decay
loss_fn = nn.BCEWithLogitsLoss() # for multiclass: nn.CrossEntropyLoss(label_smoothing=0.1)
for epoch in range(max_epochs):
model.train() # dropout ON, batch norm uses batch statistics
... augmented batches -> loss -> backward -> step
model.eval() # dropout OFF, batch norm uses running statistics
... validation loss -> early stopping check
Common mistakes
| Mistake | Exact message / symptom | Why | Fix |
|---|---|---|---|
| Trusting 100% training accuracy | validation 72.9% | memorised the noise | watch validation |
Forgetting model.eval() |
predictions change from call to call | dropout still active | model.eval() before measuring |
Forgetting model.train() after evaluating |
dropout silently off for the rest of training | mode is sticky | model.train() at the start of each epoch |
| BatchNorm with batch size 1 in training | ValueError: Expected more than 1 value per channel when training, got input size torch.Size([1, 2]) |
no spread to compute | bigger batches, drop_last=True, or LayerNorm |
| Keeping the last weights | the "bitter chai" (72.9%) | past the best epoch | early stopping + load best |
best_state = model.state_dict() without a copy |
"best" weights change as training goes on | references, not a copy | copy.deepcopy(...) |
| Reporting the early-stopping validation score | an optimistic number | validation chose the epoch | a separate test set |
| Augmenting validation data | a noisy, unfair score | only training data is augmented | fixed validation data |
Mini project: everything together, with a test set
import torch, torch.nn as nn, copy
torch.manual_seed(63)
def make(n):
X = torch.rand(n, 2) * 4 - 2; y = (X.pow(2).sum(1) < 1.5).float()
flip = torch.rand(n) < 0.15; y[flip] = 1 - y[flip]
return X, y.unsqueeze(1)
Xtr, ytr = make(100); Xva, yva = make(1000); Xte, yte = make(1000) # train / validation / TEST
torch.manual_seed(0)
model = nn.Sequential(nn.Linear(2, 256), nn.ReLU(), nn.Dropout(0.5),
nn.Linear(256, 256), nn.ReLU(), nn.Dropout(0.5), nn.Linear(256, 1))
opt = torch.optim.AdamW(model.parameters(), lr=0.01, weight_decay=1.0)
loss_fn = nn.BCEWithLogitsLoss()
acc = lambda X, y: ((model(X) > 0).float() == y).float().mean().item()
best, waited, patience = float("inf"), 0, 50
for epoch in range(1, 2001):
model.train()
loss = loss_fn(model(Xtr + 0.2 * torch.randn_like(Xtr)), ytr); opt.zero_grad(); loss.backward(); opt.step()
model.eval()
with torch.no_grad(): val = loss_fn(model(Xva), yva).item()
if val < best: best, best_epoch, state, waited = val, epoch, copy.deepcopy(model.state_dict()), 0
else:
waited += 1
if waited == patience: break
model.load_state_dict(state); model.eval()
with torch.no_grad():
print(f"stopped at epoch {epoch}, best epoch {best_epoch}")
print(f"train {acc(Xtr, ytr):.1%} | validation {acc(Xva, yva):.1%} | test {acc(Xte, yte):.1%}")
Output:
stopped at epoch 126, best epoch 76
train 78.0% | validation 82.2% | test 81.1%
Dropout, weight decay, augmentation and early stopping together; the test set (never used for any decision) gives the honest final number.
Practice
1. Try dropout p = 0.1, 0.3, 0.5 and 0.7.
Solution
Replace the train(...) calls in the comparison program with for p in (0.1, 0.3, 0.5, 0.7): train(f"dropout {p}", p_drop=p). Output:
dropout 0.1 train 95% validation 74.4%
dropout 0.3 train 93% validation 74.7%
dropout 0.5 train 88% validation 75.1%
dropout 0.7 train 85% validation 78.9%
More dropout lowers training accuracy. On this tiny, noisy problem validation kept improving up to p = 0.7 (78.9%); on most problems very high dropout eventually underfits, so tune p on validation data.
2. Add early stopping to the "all three" run.
Solution
import torch, torch.nn as nn, copy
torch.manual_seed(63)
def make(n):
X = torch.rand(n, 2) * 4 - 2; y = (X.pow(2).sum(1) < 1.5).float()
flip = torch.rand(n) < 0.15; y[flip] = 1 - y[flip]
return X, y.unsqueeze(1)
Xtr, ytr = make(100); Xva, yva = make(1000)
torch.manual_seed(0)
model = nn.Sequential(nn.Linear(2, 256), nn.ReLU(), nn.Dropout(0.5),
nn.Linear(256, 256), nn.ReLU(), nn.Dropout(0.5), nn.Linear(256, 1))
opt, loss_fn = torch.optim.AdamW(model.parameters(), lr=0.01, weight_decay=1.0), nn.BCEWithLogitsLoss()
best, waited, patience = float("inf"), 0, 20
for epoch in range(1, 1001):
model.train()
loss = loss_fn(model(Xtr + 0.2 * torch.randn_like(Xtr)), ytr); opt.zero_grad(); loss.backward(); opt.step()
model.eval()
with torch.no_grad(): val = loss_fn(model(Xva), yva).item()
if val < best: best, best_epoch, state, waited = val, epoch, copy.deepcopy(model.state_dict()), 0
else:
waited += 1
if waited == patience: break
model.load_state_dict(state); model.eval()
with torch.no_grad():
print(f"stopped at epoch {epoch}, best epoch {best_epoch}, validation acc {((model(Xva) > 0).float() == yva).float().mean().item():.1%}")
Output:
stopped at epoch 61, best epoch 41, validation acc 81.2%
Note model.train() for the training step and model.eval() for the validation check, because the model has dropout.
3. Put nn.BatchNorm1d(256) after each hidden Linear layer.
Solution
In the comparison program change the model to nn.Sequential(nn.Linear(2, 256), nn.BatchNorm1d(256), nn.ReLU(), nn.Dropout(p_drop), nn.Linear(256, 256), nn.BatchNorm1d(256), nn.ReLU(), nn.Dropout(p_drop), nn.Linear(256, 1)). Output:
BatchNorm only train 97% validation 73.2%
BatchNorm + all three train 76% validation 78.5%
BatchNorm alone barely helps here (73.2% vs 72.9%), and with the other three tools it gives 78.5%, a little below 80.4% without it. The whole training set (100 examples) is one batch, so the batch statistics are the same every step and add no regularising noise; BatchNorm's main benefit is faster, steadier optimisation in deeper networks.
4. Try patience 5 and 50. What changes?
Solution
Patience 5:
stopped at epoch 8; best val loss 0.603 at epoch 3
validation accuracy with the best weights: 64.9%
Patience 50:
stopped at epoch 75; best val loss 0.539 at epoch 25
validation accuracy with the best weights: 79.6%
Patience 5 stopped at epoch 8 on a temporary wobble, with only 64.9%: too impatient. Patience 50 trained to epoch 75 but found the same best epoch (25) as patience 20, so the result is identical (79.6%): extra patience costs time, not accuracy, because we load the best weights back.
Quick quiz
nn.Dropout(p=0.2) in train mode on torch.ones(5). A value that is not dropped becomes?
Answer
1.25 = 1 / (1 − 0.2), so the average stays 1.
More questions:
- Does dropout run in
model.eval()?Answer
No: in eval mode the layer passes values through unchanged. - What does patience mean in early stopping?
Answer
How many epochs without a new best validation loss we wait before stopping. - BatchNorm normalises over what?
Answer
Each feature, over the examples in the batch. - LayerNorm normalises over what?
Answer
Each example, over its own features. - With 3 classes and
label_smoothing=0.1, what is the target for the right class?Answer
0.9 + 0.1/3 ≈ 0.933 (and 0.033 for each other class).
Interview questions
- How does dropout work at train vs test time? Train: zero each unit with probability
pand scale the rest by1/(1 − p); test: use all units with no scaling (inverted dropout). - Why does BatchNorm behave differently in eval mode? It switches from the current batch's statistics to running averages collected in training, so predictions don't depend on batch-mates.
- BatchNorm vs LayerNorm, and why do transformers use LayerNorm? LayerNorm is per example, independent of batch size and identical at train and test; it suits variable-length sequences and small batches.
- What is early stopping, and what is the catch? Stop when validation loss stops improving and keep the best weights; the validation set becomes part of model selection, so use a separate test set.
- What is label smoothing for? It prevents over-confident predictions (and very large logits), often improving calibration and generalisation, especially with noisy labels.
- Is "reducing internal covariate shift" why BatchNorm works? Disputed: the original paper said so; Santurkar et al. (2018) found stable layer inputs explain little and pointed to a smoother loss landscape.
Glossary
| Term | Meaning (plain words + example) |
|---|---|
| Regularisation | Any technique that reduces overfitting: dropout, weight decay |
| Generalisation | Doing well on unseen data: validation 80.4% |
| Label noise | Wrong labels in the data: 15% flipped |
| Memorisation | Fitting each training example, noise included: 100% train accuracy |
| Dropout | Randomly zeroing units with probability p during training: nn.Dropout(0.5) |
| Inverted dropout scaling | Multiplying kept values by 1/(1 − p): 1 → 2 for p = 0.5 |
| Co-adaptation | Units relying on specific other units; dropout breaks this |
| Train mode / eval mode | model.train() / model.eval(): dropout and batch norm behave differently |
| Weight decay | Shrinking weights a little each step: AdamW(weight_decay=1.0) |
| Data augmentation | Training on randomly changed copies: points + noise 0.2 |
| Early stopping | Stopping when validation stops improving, keeping the best weights |
| Patience | Epochs to wait without improvement before stopping: 20 |
state_dict |
A dictionary of a model's weights, for saving and loading |
| Deep copy | A full independent copy (copy.deepcopy), not a reference |
| Test set | Data used only once, for the final honest score |
| Batch normalisation (BatchNorm) | Standardising each feature over the batch, then γ·x̂ + β |
| γ (gamma), β (beta) | BatchNorm/LayerNorm's learned scale and shift |
| Running statistics | Moving averages of mean and variance used in eval mode |
momentum (BatchNorm) |
How far running stats move per batch: 0.1 |
| ε (epsilon) | A tiny number added to the variance to avoid dividing by zero |
| Layer normalisation (LayerNorm) | Standardising each example over its own features: nn.LayerNorm(3) |
| Label smoothing | Target = 1 − ε on the right class + ε spread over all classes |
| Overconfidence | Predicting near-100% probabilities even when wrong |
| Internal covariate shift | A layer's input distribution changing as earlier layers learn |
| Loss landscape smoothness | How gently the loss and its gradient change; smoother = steadier training |
Summary
- A big network on little, noisy data memorises it: 100% train, 72.9% validation.
- Dropout (train only, with
1/(1 − p)scaling), weight decay and augmentation make memorising harder: together 80.4%. - Early stopping with patience keeps the best epoch's weights; confirm on a test set.
- BatchNorm normalises each feature over the batch (running stats in eval; fails on a batch of one in training); LayerNorm normalises each example and works for any batch size.
- Label smoothing stops rewarding 100% confidence.
- Why BatchNorm works is still debated: reduced internal covariate shift (2015) vs a smoother loss landscape (2018).
Next episode: Training tricks and debugging: overfitting one batch on purpose, reading loss curves, NaN losses, mixed precision, gradient accumulation and checkpoints.