SyntaxPilot
Courses/Machine Learning/Lesson 64
Video coming soonThis lesson’s video is being made

Start with the notes below; the video appears here as soon as it is on YouTube.

Machine Learning · Lesson 64 of 90

Training tricks and debugging

overfitting a single batchreading loss curvesNaN lossesmixed precision (FP16, BF16)automatic mixed precision (AMP)gradient accumulationsaving and loading state_dictcheckpointingreproducibility settingsGPU memory and batch size

Machine Learning Full Course · Episode 64. Every code sample below was run with Python 3.13 and PyTorch 2.15.0 (CPU, one thread, as in the video's compiler). The output shown is the real output. Programs without a seed print different numbers on every run (that is the point of the reproducibility section).

The big idea: most training failures don't crash. The code runs and the numbers are just bad. So you need tests that tell "my code is broken" apart from "the problem is hard": overfit a single batch (a working pipeline must memorise 8 examples), check the starting loss, and read the loss curve. Then the tools for when it is the machine that is the limit: NaN hunting, mixed precision (16-bit numbers for speed and memory), gradient accumulation (a big batch in small pieces), checkpoints (save and resume everything) and seeds (repeatable runs).

Sources: Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh and Hao Wu, "Mixed Precision Training", ICLR 2018 (arXiv 1710.03740). Shibo Wang and Pankaj Kanwar, "BFloat16: The secret to high performance on Cloud TPUs", Google Cloud blog, 23 August 2019 (cloud.google.com). PyTorch documentation, "Automatic Mixed Precision examples" (docs.pytorch.org). Andrej Karpathy's 2019 list of common training mistakes, number one: "you didn't try to overfit a single batch first" (quoted at notesbylex.com); see also his blog post "A Recipe for Training Neural Networks", 25 April 2019.

Words you need first

  • Loss (Episode 8): one number that says how wrong the model is; training pushes it down.
  • Epoch, batch, step (Episode 59): an epoch is one pass over all the training data; a batch is a small group of examples; one step = one optimizer update on one batch.
  • Logit (Episode 59): the raw score before sigmoid/softmax. For yes/no, a logit above 0 means "yes".
  • ReLU (Episode 56): max(0, x): keeps positives, turns every negative into 0.
  • Learning rate (Episode 8): how big each update step is.
  • Validation set (Episode 15): data kept aside to measure the model while you build it.
  • state_dict (Episode 59): a dictionary of a model's learned tensors, by name.
  • Bug / debugging: a bug is a mistake in the code; debugging is finding and fixing it.

The problem: a model that is stuck

A small network on 2,000 examples (10 features, a yes/no label). It trains without any error, but the loss stops falling:

import torch, torch.nn as nn
torch.manual_seed(64)
X = torch.randn(2000, 10); y = (X[:, 0] + X[:, 1] * X[:, 2] > 0).float().unsqueeze(1)
torch.manual_seed(0)
model = nn.Sequential(nn.Linear(10, 64), nn.ReLU(), nn.Linear(64, 1), nn.ReLU())
opt, loss_fn = torch.optim.Adam(model.parameters(), lr=0.01), nn.BCEWithLogitsLoss()
for epoch in range(1, 201):
    for i in range(0, 2000, 100):
        xb, yb = X[i:i + 100], y[i:i + 100]
        loss = loss_fn(model(xb), yb); opt.zero_grad(); loss.backward(); opt.step()
    if epoch in (1, 50, 200):
        acc = ((model(X) > 0).float() == y).float().mean().item()
        print(f"epoch {epoch:3}: loss {loss.item():.3f}   accuracy {acc:.1%}")

Output:

epoch   1: loss 0.546   accuracy 81.1%
epoch  50: loss 0.377   accuracy 98.9%
epoch 200: loss 0.368   accuracy 99.6%

The loss barely moves between epoch 50 and 200. Is the data too noisy? The model too small? The learning rate wrong? Train longer? Guessing fails: there are too many suspects, every guess costs a full training run, and a bug in a training loop usually doesn't crash.

Overfit a single batch

Real-life example: a cook preparing a wedding for 500 guests finds the food tastes wrong. Instead of blaming the pots, the stove or the ingredients, they cook one plate with full attention. If even one plate tastes wrong, the recipe is broken. If one plate is perfect, scale up.

The test:

  1. Take one small batch, e.g. 8 examples, and train only on it, again and again.
  2. A working model + loss + optimizer can memorise 8 examples easily: the loss must go to about 0.
  3. If it can't, the bug is in the code (model, loss, optimizer, labels), not in the amount of data or the model size.
  4. Also check the starting loss. A model that guesses 50/50 on 2 classes has loss ln 2 ≈ 0.693; on C balanced classes, ln C (2.303 for 10 classes). A very different start means something is wrong before training begins.
import torch, torch.nn as nn
torch.manual_seed(64)
X = torch.randn(2000, 10); y = (X[:, 0] + X[:, 1] * X[:, 2] > 0).float().unsqueeze(1)
xb, yb = X[:8], y[:8]                                   # ONE tiny batch of 8 examples
for last in (nn.ReLU(), nn.Identity()):                 # with the extra ReLU, then without it
    torch.manual_seed(0)
    model = nn.Sequential(nn.Linear(10, 64), nn.ReLU(), nn.Linear(64, 1), last)
    opt, loss_fn = torch.optim.Adam(model.parameters(), lr=0.01), nn.BCEWithLogitsLoss()
    print(type(last).__name__, "start loss", round(loss_fn(model(xb), yb).item(), 3))
    for step in range(500):
        loss = loss_fn(model(xb), yb); opt.zero_grad(); loss.backward(); opt.step()
    print("   after 500 steps: loss", round(loss.item(), 4))
    print("   logits:", model(xb).detach().view(-1).round().int().tolist())
print("labels:   ", yb.view(-1).int().tolist())

Output:

ReLU start loss 0.667
   after 500 steps: loss 0.26
   logits: [9, 0, 12, 11, 0, 11, 0, 10]
Identity start loss 0.626
   after 500 steps: loss 0.0
   logits: [9, -10, 13, 11, -11, 11, -10, 10]
labels:    [1, 0, 1, 1, 0, 1, 0, 1]
  • Both start near 0.693: the start looks healthy.
  • With the final ReLU, 500 steps on just 8 examples end at 0.26: it cannot memorise 8 examples, so the code is broken.
  • The logits show why: every example with label 0 has logit exactly 0.0. ReLU turns every negative number into 0, so the model can never say a confident "no"; the best it can do for those is p = 0.5, loss ln 2. Three of the eight labels are 0: 3/8 × 0.693 = 0.26. The numbers match exactly.
  • Without the extra ReLU (nn.Identity() = do nothing), the loss is 0.0 and the "no" examples get negative logits.

The fix in the real model: delete the last ReLU.

import torch, torch.nn as nn
torch.manual_seed(64)
X = torch.randn(2000, 10); y = (X[:, 0] + X[:, 1] * X[:, 2] > 0).float().unsqueeze(1)
torch.manual_seed(0)
model = nn.Sequential(nn.Linear(10, 64), nn.ReLU(), nn.Linear(64, 1))
opt, loss_fn = torch.optim.Adam(model.parameters(), lr=0.01), nn.BCEWithLogitsLoss()
for epoch in range(1, 201):
    for i in range(0, 2000, 100):
        xb, yb = X[i:i + 100], y[i:i + 100]
        loss = loss_fn(model(xb), yb); opt.zero_grad(); loss.backward(); opt.step()
    if epoch in (1, 50, 200):
        acc = ((model(X) > 0).float() == y).float().mean().item()
        print(f"epoch {epoch:3}: loss {loss.item():.3f}   accuracy {acc:.1%}")

Output:

epoch   1: loss 0.378   accuracy 84.7%
epoch  50: loss 0.009   accuracy 99.9%
epoch 200: loss 0.000   accuracy 100.0%

Not a hard problem at all: one wrong word, found in a minute.

Reading loss curves

Real-life example: a doctor doesn't look at one temperature reading; the shape of the fever chart over time tells the illness, and each shape has its own treatment. A loss curve is the loss measured every few epochs (training and validation), and its shape tells you what is wrong.

import torch, torch.nn as nn
torch.manual_seed(64)
X = torch.randn(2000, 10); y = (X[:, 0] + X[:, 1] * X[:, 2] > 0).float().unsqueeze(1)
print("loss on all the data, every 5 epochs:")
for lr in (0.00001, 0.01, 1.0):                         # too small, about right, too big
    torch.manual_seed(0)
    model = nn.Sequential(nn.Linear(10, 64), nn.ReLU(), nn.Linear(64, 1))
    opt, loss_fn = torch.optim.Adam(model.parameters(), lr=lr), nn.BCEWithLogitsLoss()
    curve = []
    for epoch in range(1, 31):
        for i in range(0, 2000, 100):
            loss = loss_fn(model(X[i:i+100]), y[i:i+100]); opt.zero_grad(); loss.backward(); opt.step()
        if epoch % 5 == 0: curve.append(round(loss_fn(model(X), y).item(), 2))
    print(f"lr {lr:<7}: {curve}")

Output:

loss on all the data, every 5 epochs:
lr 1e-05  : [0.67, 0.67, 0.66, 0.66, 0.66, 0.66]
lr 0.01   : [0.12, 0.08, 0.06, 0.04, 0.03, 0.03]
lr 1.0    : [1.23, 0.87, 0.35, 0.42, 0.4, 0.37]
Shape What it means What to do
Flat from the start (lr 0.00001: 0.67 → 0.66) learning rate far too small, or a bug raise the lr; run the one-batch test
Smooth fall that levels off (lr 0.01: 0.12 → 0.03) healthy keep going, or stop when it levels
Bouncing, or rising (lr 1.0: 1.23 → 0.35 → 0.42 → 0.40) learning rate too big lower it; warmup; gradient clipping (Episode 62)
Training falls, validation rises overfitting regularise; early stopping (Episode 63)
Shoots up to inf, then nan diverged see the next section
Starts far from ln C broken initialisation, labels or loss check the data, the last layer and the loss

Plot both curves (training and validation) on the same chart, with a log scale on the y-axis when values span several orders of magnitude.

NaN losses

inf (infinity): a float32 holds numbers only up to about 3.4 × 10³⁸; anything bigger becomes inf. NaN ("not a number") is the result of a calculation with no answer: inf − inf, 0 × inf, 0 / 0. NaN spreads: anything combined with NaN is NaN, like a calculator showing "Error" for every later sum. One NaN in a gradient, and after a single update every weight is NaN.

Cause 1: a learning rate far too big for the size of the inputs.

import torch, torch.nn as nn
torch.manual_seed(64)
X = torch.randn(100, 3) * 10; y = X @ torch.tensor([2.0, -1.0, 0.5]) + 3   # a simple regression
w = torch.zeros(3, requires_grad=True)
for step in range(1, 29):                              # learning rate 0.1: far too big here
    loss = ((X @ w - y) ** 2).mean(); loss.backward()
    with torch.no_grad(): w -= 0.1 * w.grad; w.grad.zero_()
    if step % 4 == 0: print(f"step {step:2}: loss {loss.item():.3g}   w {w.tolist()[0]:.3g}")

Output:

step  4: loss 1.98e+11   w -5.81e+05
step  8: loss 5.64e+22   w -2.56e+11
step 12: loss 1.69e+34   w -1.31e+17
step 16: loss inf   w -7.04e+22
step 20: loss inf   w -3.84e+28
step 24: loss inf   w -2.11e+34
step 28: loss nan   w nan

Each step overshoots further, the loss reaches inf at step 16, then the weight itself overflows and the sums become inf − inf: NaN at step 28. Training is dead.

Cause 2: maths with no answer, such as log(0).

import torch, torch.nn as nn
z, label = torch.tensor([40.0]), torch.tensor([0.0])   # a very confident WRONG logit
p = torch.sigmoid(z); print("sigmoid(40) =", p.item())  # float32 rounds it to exactly 1
print("manual  -log(1 - p):", (-torch.log(1 - p)).item())
print("BCEWithLogitsLoss:  ", nn.BCEWithLogitsLoss()(z, label).item())
print("0 * inf =", (torch.tensor(0.0) * torch.tensor(float("inf"))).item())
w = torch.tensor([1.0, 2.0], requires_grad=True)
loss = (w * torch.tensor([0.0, float("inf")])).sum(); loss.backward()
print("loss:", loss.item(), "| finite?", torch.isfinite(loss).item(), "| grad:", w.grad.tolist())

Output:

sigmoid(40) = 1.0
manual  -log(1 - p): inf
BCEWithLogitsLoss:   40.0
0 * inf = nan
loss: inf | finite? False | grad: [0.0, inf]

sigmoid(40) is really 1 − 4.2 × 10⁻¹⁸, but float32 cannot store that difference, so it rounds to exactly 1.0, 1 − p is 0, and −log(0) = inf. BCEWithLogitsLoss works from the logit with a numerically stable formula (max(z, 0) − z·y + log(1 + e^(−|z|))) and gives the right answer, 40. torch.isfinite tells you a loss has gone bad.

NaN checklist

  1. Learning rate too big → lower it, add warmup, clip gradients.
  2. Inputs not scaled → standardise them (Episode 16).
  3. log(0), sqrt of a negative, division by 0 → use built-in losses on logits (BCEWithLogitsLoss, CrossEntropyLoss), add a small eps where you must divide.
  4. Bad data → torch.isnan(X).any(), torch.isinf(X).any() before training.
  5. Find the first bad step: if not torch.isfinite(loss): raise ... and print the inputs, outputs and gradient norms at that step. torch.autograd.set_detect_anomaly(True) makes backward report which operation produced the NaN (slow; for debugging only).
  6. In FP16 mixed precision: overflow above 65,504 → use GradScaler or BF16.

Mixed precision: FP16, BF16 and AMP

Precision is how many digits a number format keeps. Three formats matter:

Format Bytes Layout (sign, exponent, mantissa bits) Largest value About how many digits
FP32 (float32, single precision) 4 1, 8, 23 about 3.4 × 10³⁸ 7
FP16 (float16, half precision) 2 1, 5, 10 65,504 3
BF16 (bfloat16, "brain float") 2 1, 8, 7 about 3.39 × 10³⁸ 2 to 3

The exponent bits decide the range (how big or small a number can be); the mantissa bits decide the digits. BF16 was conceived at Google Brain (hence the name): it keeps FP32's 8 exponent bits, so it has the same range as FP32, at the cost of digits.

import torch
for dt in (torch.float32, torch.float16, torch.bfloat16):    # size and largest value
    size, top = torch.tensor(0, dtype=dt).element_size(), torch.finfo(dt).max
    print(f"{str(dt)[6:]:9} {size} bytes   max {top:.5g}")
for dt in (torch.float32, torch.float16, torch.bfloat16):
    t = lambda v: torch.tensor(v, dtype=dt).item()           # store v in this format, read it back
    print(f"{str(dt)[6:]:9} 70000->{t(70000.0):<6g} 1e-8->{t(1e-8):<6.2g} 1.001->{t(1.001):.6g}")
g = torch.tensor(1e-8)                                       # a tiny gradient
print("tiny gradient stored in float16:", g.half().item())
print("x 65536, float16, then / 65536: ", (g * 65536).half().float().item() / 65536)

Output:

float32   4 bytes   max 3.4028e+38
float16   2 bytes   max 65504
bfloat16  2 bytes   max 3.3895e+38
float32   70000->70000  1e-8->1e-08  1.001->1.001
float16   70000->inf    1e-8->0      1.001->1.00098
bfloat16  70000->70144  1e-8->1e-08  1.001->1
tiny gradient stored in float16: 0.0
x 65536, float16, then / 65536:  9.997165761888027e-09
  • FP16: 70,000 overflows to inf, 1e-8 underflows to 0 (too small to store).
  • BF16: 70,000 fits (as 70,144: only about 3 digits), 1e-8 survives, but 1.001 becomes exactly 1.
  • Loss scaling: a tiny gradient vanishes in FP16, but multiplied by 65,536 first, stored, and divided back in FP32 it survives (9.997e-09).

Real-life example: rough work and a fair copy in an exam. Quick rough work in pencil, with few digits (the 16-bit forward and backward pass: fast, little memory), but the final answer in the fair copy with every digit (a master copy of the weights in FP32, so millions of tiny updates still add up). Loss scaling is like weighing saffron: one strand is too light for the kitchen scale, so you weigh 100 strands and divide.

Mixed precision = 16-bit maths where it is safe, FP32 where it matters. Micikevicius et al. (ICLR 2018) introduced it with exactly these two ideas, FP32 master weights and loss scaling, and reported nearly 2× lower memory use. Automatic mixed precision (AMP) chooses the precision per operation for you: inside torch.autocast, matrix multiplies and convolutions run in 16 bits; sensitive operations (softmax, sums, losses, normalisation) run in FP32.

import torch, torch.nn as nn
print("GPU available:", torch.cuda.is_available())         # this machine: CPU only
torch.manual_seed(0)
layer, x = nn.Linear(512, 512), torch.randn(64, 512)
with torch.autocast(device_type="cpu", dtype=torch.bfloat16):  # automatic mixed precision
    out = layer(x)                                         # matmul runs in bfloat16
    s = out.float().softmax(1)                             # we keep softmax in float32
name = lambda t: str(t.dtype)[6:]
print("weights:", name(layer.weight), "| output:", name(out), "| softmax:", name(s))
ref = layer(x)                                              # the same layer in full float32
print("biggest difference vs float32:", round((out.float() - ref).abs().max().item(), 4))
print("output bytes:", out.nelement() * out.element_size(), "vs", ref.nelement() * ref.element_size())

Output:

GPU available: False
weights: float32 | output: bfloat16 | softmax: float32
biggest difference vs float32: 0.0077
output bytes: 65536 vs 131072

On a CPU, autocast uses BF16. The weights stay FP32 (the fair copy), the layer's output is BF16, the biggest difference from full FP32 is 0.0077 (the price of fewer digits), and the output takes exactly half the bytes.

On an NVIDIA GPU with FP16, the standard loop from the PyTorch documentation (not run in the video: that machine has no GPU):

model = Net().cuda()                         # weights stay float32
opt = torch.optim.AdamW(model.parameters(), lr=1e-3)
scaler = torch.amp.GradScaler("cuda")       # one scaler for the whole run
for xb, yb in loader:
    opt.zero_grad()
    with torch.autocast(device_type="cuda", dtype=torch.float16):
        loss = loss_fn(model(xb.cuda()), yb.cuda())
    scaler.scale(loss).backward()           # multiply the loss, then backward
    scaler.step(opt)                        # unscale; skip the step if inf/nan
    scaler.update()                         # adjust the scale for next time
  • Only the forward pass and the loss go inside autocast; backward() is called outside it.
  • scaler.step divides the gradients back and skips the update if they contain inf or nan (then scaler.update lowers the scale).
  • To clip gradients with a scaler, call scaler.unscale_(opt) before clip_grad_norm_.
  • With BF16 (dtype=torch.bfloat16) you usually don't need a GradScaler, because BF16 has FP32's range.

GPU memory and batch size

Real-life example: a lift that carries at most 8 people. The lift operator always rides (fixed cost); the rest is for passengers (the batch). Too many passengers and the alarm beeps: PyTorch stops with an out-of-memory error.

import torch, torch.nn as nn
model = nn.Sequential(nn.Linear(1024, 4096), nn.ReLU(), nn.Linear(4096, 4096), nn.ReLU(),
                      nn.Linear(4096, 10))
n = sum(p.numel() for p in model.parameters())
MB = lambda b: f"{b / 2**20:,.0f} MB"
print(f"parameters: {n:,}")
print("weights      (4 bytes each):", MB(n * 4))
print("+ gradients  (4 bytes each):", MB(n * 4))
print("+ Adam m and v (8 bytes)   :", MB(n * 8), "| total", MB(n * 16))
for batch in (32, 256, 2048):                     # activations grow with the batch size
    x = torch.randn(batch, 1024); acts = [x]
    for layer in model: acts.append(layer(acts[-1]))
    print(f"batch {batch:5}: activations kept for backward ~", MB(sum(a.numel() * 4 for a in acts)))

Output:

parameters: 21,020,682
weights      (4 bytes each): 80 MB
+ gradients  (4 bytes each): 80 MB
+ Adam m and v (8 bytes)   : 160 MB | total 321 MB
batch    32: activations kept for backward ~ 2 MB
batch   256: activations kept for backward ~ 17 MB
batch  2048: activations kept for backward ~ 136 MB
  • Fixed cost with Adam in FP32: 16 bytes per parameter (4 weights + 4 gradients + 8 for Adam's m and v, Episode 61). 21 million parameters = 321 MB before any data.
  • Activations (every layer's output, kept for the backward pass) grow in proportion to the batch size: 8 times the batch, 8 times the activation memory.
  • When memory runs out: lower the batch size first; then mixed precision (half the activation bytes), gradient accumulation (below), and, for very deep models, activation checkpointing (torch.utils.checkpoint: recompute activations in backward instead of storing them).
  • On a GPU, torch.cuda.memory_allocated() and torch.cuda.max_memory_allocated() report what PyTorch is using.

Gradient accumulation

Problem: you want a batch of 64 (steadier gradients, the learning rate you tuned), but only 16 fit in memory. Real-life example: a small auto-rickshaw makes four trips to deliver what a truck carries in one.

  • .backward() adds to .grad; it never replaces it. So run 4 micro-batches of 16, call backward() after each, and take one optimizer step after the 4th.
  • Divide each micro-batch loss by 4, so the summed gradient equals the average over all 64.
import torch, torch.nn as nn
torch.manual_seed(64)
X, y = torch.randn(64, 10), torch.randn(64, 1)
torch.manual_seed(0); model = nn.Sequential(nn.Linear(10, 32), nn.ReLU(), nn.Linear(32, 1))
loss_fn = nn.MSELoss()
loss_fn(model(X), y).backward()                          # A: one big batch of 64
big = model[0].weight.grad.clone(); model.zero_grad()
for i in range(0, 64, 16):                               # B: 4 micro-batches of 16
    loss = loss_fn(model(X[i:i + 16]), y[i:i + 16]) / 4  # divide by the number of steps
    loss.backward()                                      # .grad ADDS UP across backward() calls
acc = model[0].weight.grad
print("same gradient?", torch.allclose(big, acc, atol=1e-6), "| max difference:", f"{(big - acc).abs().max():.1e}")
model.zero_grad()
for i in range(0, 64, 16): loss_fn(model(X[i:i + 16]), y[i:i + 16]).backward()  # forgot / 4
print("without / 4 the gradient is", round((model[0].weight.grad / big).mean().item(), 2), "times too big")

Output:

same gradient? True | max difference: 2.2e-08
without / 4 the gradient is 4.0 times too big

The accumulated gradient equals the big-batch gradient up to float rounding (2.2e-08). Forget the division and it is exactly 4× too big, as if the learning rate were 4× higher. (Layers that use batch statistics, like BatchNorm, still see batches of 16, so accumulation is not perfectly identical for them.)

The loop:

accum = 4
for i, (xb, yb) in enumerate(loader):
    loss = loss_fn(model(xb), yb) / accum
    loss.backward()
    if (i + 1) % accum == 0:
        opt.step()
        opt.zero_grad()

Checkpoints: saving and resuming training

Real-life example: a save point in a mobile game stores the level, the coins and the power-ups, so when the phone dies you continue from level 9, not level 1.

  • model.state_dict(): an ordered dictionary of every learned tensor, by name.
  • torch.save(obj, "file.pt") writes any dictionary of tensors to a file; torch.load("file.pt") reads it.
  • model.load_state_dict(d) copies the numbers into a new model of exactly the same shape.
  • A checkpoint stores everything needed to continue: model, optimizer (Adam's m, v and step count), epoch, and the scheduler and GradScaler if you use them.
import torch, torch.nn as nn
torch.manual_seed(64); X, y = torch.randn(256, 10), torch.randn(256, 1)
def setup():
    torch.manual_seed(0); m = nn.Sequential(nn.Linear(10, 32), nn.ReLU(), nn.Linear(32, 1))
    return m, torch.optim.Adam(m.parameters(), lr=0.01)
def train(m, opt, epochs):
    for _ in range(epochs):
        loss = nn.MSELoss()(m(X), y); opt.zero_grad(); loss.backward(); opt.step()
    return round(loss.item(), 5)
m, opt = setup(); print("uninterrupted, 20 epochs:", train(m, opt, 20))
m, opt = setup(); train(m, opt, 10)                     # 10 epochs... then the laptop dies
torch.save({"epoch": 10, "model": m.state_dict(), "optim": opt.state_dict()}, "ckpt.pt")
m2, opt2 = setup()                                      # a fresh process: brand-new objects
ck = torch.load("ckpt.pt"); m2.load_state_dict(ck["model"]); opt2.load_state_dict(ck["optim"])
print("resumed from epoch", ck["epoch"], "-> 20:", train(m2, opt2, 10))
print("keys:", list(m2.state_dict().keys()))

Output:

uninterrupted, 20 epochs: 0.69934
resumed from epoch 10 -> 20: 0.69934
keys: ['0.weight', '0.bias', '2.weight', '2.bias']

The resumed run ends at exactly the same loss as the uninterrupted one. The ReLU (position 1) has nothing to learn, so it has no entry in the state dict.

Two mistakes:

import torch, torch.nn as nn, textwrap
torch.manual_seed(64); X, y = torch.randn(256, 10), torch.randn(256, 1)
def setup():
    torch.manual_seed(0); m = nn.Sequential(nn.Linear(10, 32), nn.ReLU(), nn.Linear(32, 1))
    return m, torch.optim.Adam(m.parameters(), lr=0.01)
def train(m, opt, epochs):
    for _ in range(epochs):
        loss = nn.MSELoss()(m(X), y); opt.zero_grad(); loss.backward(); opt.step()
    return round(loss.item(), 5)
m, opt = setup(); train(m, opt, 10); torch.save(m.state_dict(), "model_only.pt")
m2, opt2 = setup(); m2.load_state_dict(torch.load("model_only.pt"))  # optimizer NOT saved
print("model only, resumed -> 20:", train(m2, opt2, 10))
wider = nn.Sequential(nn.Linear(10, 64), nn.ReLU(), nn.Linear(64, 1))
try: wider.load_state_dict(torch.load("model_only.pt"))
except RuntimeError as e:                                  # the first two lines, wrapped to fit
    print("RuntimeError:")
    for line in str(e).splitlines()[:2]: print(textwrap.fill(line.strip(), 60, subsequent_indent="    "))

Output:

model only, resumed -> 20: 0.65894
RuntimeError:
Error(s) in loading state_dict for Sequential:
size mismatch for 0.weight: copying a param with shape
    torch.Size([32, 10]) from checkpoint, the shape in
    current model is torch.Size([64, 10]).
  • Model only, no optimizer: a fresh Adam has forgotten its moving averages and step count, so the next 10 epochs take different steps (0.65894 instead of 0.69934). It is no longer the same training.
  • Different architecture: RuntimeError: Error(s) in loading state_dict for Sequential: size mismatch for 0.weight: copying a param with shape torch.Size([32, 10]) from checkpoint, the shape in current model is torch.Size([64, 10]). Rebuild exactly the same model before loading. (load_state_dict(d, strict=False) only skips missing or extra keys; it never fixes a size mismatch.)

Good habits: save every epoch (or every N steps) to a file with the epoch in its name, keep the best one separately (early stopping, Episode 63), write to a temporary file and rename it so a crash mid-save doesn't corrupt the checkpoint, and save model.state_dict(), not the whole model object (pickled classes break when your code changes). Only torch.load files you trust.

Reproducibility

To know whether your change improved the score or you just got lucky, a run must be reproducible: run it again, get exactly the same numbers. Randomness hides in starting weights, data shuffling, dropout and augmentation. A seed fixes a random number generator's sequence (Episode 2); Python's random, NumPy and PyTorch each have their own. Real-life example: a recipe with exact grams instead of "salt to taste".

import random, numpy as np, torch, torch.nn as nn
def run(seed=None):
    if seed is not None:                                 # seed EVERY source of randomness
        random.seed(seed); np.random.seed(seed); torch.manual_seed(seed)
    model = nn.Linear(4, 1)                              # random starting weights
    order = torch.randperm(8).tolist()                   # random shuffle of the data
    noise = np.random.rand()                             # random augmentation
    return round(model.weight[0, 0].item(), 4), order[:4], round(noise, 3)
print("no seed :", run()); print("no seed :", run())
print("seed 42 :", run(42)); print("seed 42 :", run(42))
torch.use_deterministic_algorithms(True)                 # error if an op has no deterministic version
torch.backends.cudnn.benchmark = False                   # GPU: don't auto-pick fastest (varying) kernels
print("deterministic mode:", torch.are_deterministic_algorithms_enabled())

Output:

no seed : (0.2835, [4, 3, 6, 5], 0.834)
no seed : (-0.1998, [2, 7, 3, 4], 0.096)
seed 42 : (0.3823, [7, 6, 4, 3], 0.375)
seed 42 : (0.3823, [7, 6, 4, 3], 0.375)
deterministic mode: True

(The two "no seed" lines are different on every run; the seeded lines are always the same.)

Reproducibility settings:

Setting What it does
random.seed(s), np.random.seed(s), torch.manual_seed(s) fix the three generators (torch.manual_seed also seeds CUDA)
torch.use_deterministic_algorithms(True) use repeatable versions of operations, or raise an error if one has none
torch.backends.cudnn.benchmark = False on NVIDIA GPUs, don't auto-pick (possibly different) fastest kernels each run
DataLoader(..., generator=torch.Generator().manual_seed(s), worker_init_fn=...) repeatable shuffling and worker randomness (Episode 60)
same library versions, same hardware, same thread count floating-point results can differ in the last digits across versions and machines

Deterministic mode can be slower. Use it to debug and compare; for final results, report the mean and spread over several seeds.

Common mistakes

Mistake Exact message / symptom Why Fix
ReLU after the last layer loss stuck at 0.26 on 8 examples, logits exactly 0.0 can't output negatives remove it; overfit one batch to find such bugs
-log(sigmoid(z)) by hand loss inf, then nan sigmoid(40) rounds to 1.0 BCEWithLogitsLoss on logits
Big lr on unscaled inputs loss 1.98e+11 → inf → nan each step overshoots lower lr, standardise, clip
Trusting only accuracy 99.6% accuracy while the loss is stuck at 0.368 accuracy hides confidence problems watch the loss too
FP16 without a scaler gradients underflow to 0, or overflow to inf FP16 range is 6e-8 to 65,504 GradScaler, or BF16
Accumulating without / N gradient 4.0 times too big .grad adds up divide each loss by N
zero_grad() inside the accumulation loop only the last micro-batch counts it wipes the sum zero only after step()
Saving only the model a different run: 0.65894 vs 0.69934 optimizer state lost save optimizer, scheduler, scaler, epoch
Loading into another shape size mismatch for 0.weight: ... torch.Size([32, 10]) from checkpoint, the shape in current model is torch.Size([64, 10]) different architecture rebuild the same model
Seeding only torch runs still differ NumPy / random not seeded seed all three

Mini project: a training loop with every safety net

Sanity check of the starting loss, resume from a checkpoint if one exists, BF16 autocast, a NaN guard, gradient accumulation with clipping, a checkpoint every epoch:

import os, random, numpy as np, torch, torch.nn as nn
def seed_everything(s):
    random.seed(s); np.random.seed(s); torch.manual_seed(s)
seed_everything(64)
X = torch.randn(2000, 10); y = (X[:, 0] + X[:, 1] * X[:, 2] > 0).float().unsqueeze(1)
Xtr, ytr, Xva, yva = X[:1600], y[:1600], X[1600:], y[1600:]
model = nn.Sequential(nn.Linear(10, 64), nn.ReLU(), nn.Linear(64, 1))
opt, loss_fn = torch.optim.AdamW(model.parameters(), lr=0.01), nn.BCEWithLogitsLoss()
# 1. sanity checks before the real run
print("start loss:", round(loss_fn(model(Xtr), ytr).item(), 3), "(ln 2 = 0.693)")
CK, ACCUM, MICRO = "run.pt", 4, 25            # effective batch = 4 x 25 = 100
start = 0
if os.path.exists(CK):                         # 2. resume if a checkpoint exists
    ck = torch.load(CK); model.load_state_dict(ck["model"]); opt.load_state_dict(ck["optim"]); start = ck["epoch"]
for epoch in range(start, 30):
    model.train(); perm = torch.randperm(1600)
    for i in range(0, 1600, MICRO):
        idx = perm[i:i + MICRO]
        with torch.autocast("cpu", dtype=torch.bfloat16):          # 3. mixed precision
            logits = model(Xtr[idx])
        loss = loss_fn(logits.float(), ytr[idx]) / ACCUM           # loss in float32
        if not torch.isfinite(loss): raise RuntimeError(f"non-finite loss at epoch {epoch}")   # 4. NaN guard
        loss.backward()
        if (i // MICRO + 1) % ACCUM == 0:                           # 5. gradient accumulation
            nn.utils.clip_grad_norm_(model.parameters(), 1.0); opt.step(); opt.zero_grad()
    model.eval()
    with torch.no_grad(): val = loss_fn(model(Xva), yva).item(); acc = ((model(Xva) > 0).float() == yva).float().mean().item()
    torch.save({"epoch": epoch + 1, "model": model.state_dict(), "optim": opt.state_dict()}, CK)   # 6. checkpoint
    if (epoch + 1) % 10 == 0: print(f"epoch {epoch + 1}: val loss {val:.3f}  val acc {acc:.1%}")
os.remove(CK)

Output:

start loss: 0.711 (ln 2 = 0.693)
epoch 10: val loss 0.112  val acc 95.7%
epoch 20: val loss 0.083  val acc 96.7%
epoch 30: val loss 0.086  val acc 95.7%

The start loss 0.711 is close to ln 2, as it should be. Kill the program in the middle and run it again: it continues from the last saved epoch (the program deletes run.pt only when it finishes).

Practice

1. Put the ReLU back in the stuck model and try a bigger learning rate, 0.1. Does it fix it?

Solution
import torch, torch.nn as nn
torch.manual_seed(64)
X = torch.randn(2000, 10); y = (X[:, 0] + X[:, 1] * X[:, 2] > 0).float().unsqueeze(1)
torch.manual_seed(0)
model = nn.Sequential(nn.Linear(10, 64), nn.ReLU(), nn.Linear(64, 1), nn.ReLU())
opt, loss_fn = torch.optim.Adam(model.parameters(), lr=0.1), nn.BCEWithLogitsLoss()
for epoch in range(1, 201):
    for i in range(0, 2000, 100):
        xb, yb = X[i:i + 100], y[i:i + 100]
        loss = loss_fn(model(xb), yb); opt.zero_grad(); loss.backward(); opt.step()
    if epoch in (1, 50, 200):
        acc = ((model(X) > 0).float() == y).float().mean().item()
        print(f"epoch {epoch:3}: loss {loss.item():.3f}   accuracy {acc:.1%}")

Output:

epoch   1: loss 0.529   accuracy 82.6%
epoch  50: loss 0.395   accuracy 92.7%
epoch 200: loss 0.395   accuracy 93.8%

No. The loss is still stuck (0.395, even worse than 0.368). A bigger learning rate cannot fix a model that is unable to output negative logits. Tuning hyperparameters around a bug wastes time; the one-batch test finds it.

2. Train the exploding regression with learning rate 0.001.

Solution
import torch, torch.nn as nn
torch.manual_seed(64)
X = torch.randn(100, 3) * 10; y = X @ torch.tensor([2.0, -1.0, 0.5]) + 3   # a simple regression
w = torch.zeros(3, requires_grad=True)
for step in range(1, 201):                              # learning rate 0.001
    loss = ((X @ w - y) ** 2).mean(); loss.backward()
    with torch.no_grad(): w -= 0.001 * w.grad; w.grad.zero_()
    if step in (1, 5, 10, 20, 200): print(f"step {step:2}: loss {loss.item():.3g}   w {w.tolist()[0]:.3g}")

Output:

step  1: loss 670   w 0.457
step  5: loss 74.7   w 1.43
step 10: loss 13.9   w 1.82
step 20: loss 9   w 1.98
step 200: loss 8.94   w 2

The weight converges to 2 (the true value) and the loss falls smoothly. It stops at 8.94 instead of 0 because the model has no bias term and the data has + 3: the leftover error is about 3² = 9. Add a bias (b) and it goes to almost 0.

3. Compute 0.1 + 0.2 in float32, float16 and bfloat16.

Solution
import torch
for dt in (torch.float32, torch.float16, torch.bfloat16):
    a, b = torch.tensor(0.1, dtype=dt), torch.tensor(0.2, dtype=dt)
    print(f"{str(dt)[6:]:9} 0.1 + 0.2 = {(a + b).item()}")

Output:

float32   0.1 + 0.2 = 0.30000001192092896
float16   0.1 + 0.2 = 0.2998046875
bfloat16  0.1 + 0.2 = 0.30078125

None of them is exactly 0.3 (0.1 and 0.2 have no exact binary form). FP16 keeps about 3 digits, BF16 only 2 to 3. That is why sums of many small updates are done in FP32.

4. Add a learning-rate scheduler (halve the lr every 5 epochs) to the checkpoint and resume.

Solution
import torch, torch.nn as nn
torch.manual_seed(64); X, y = torch.randn(256, 10), torch.randn(256, 1)
def setup():
    torch.manual_seed(0); m = nn.Sequential(nn.Linear(10, 32), nn.ReLU(), nn.Linear(32, 1))
    opt = torch.optim.Adam(m.parameters(), lr=0.01)
    return m, opt, torch.optim.lr_scheduler.StepLR(opt, step_size=5, gamma=0.5)   # halve lr every 5 epochs
def train(m, opt, sched, epochs):
    for _ in range(epochs):
        loss = nn.MSELoss()(m(X), y); opt.zero_grad(); loss.backward(); opt.step(); sched.step()
    return round(loss.item(), 5), opt.param_groups[0]["lr"]
m, opt, sch = setup(); print("uninterrupted:", train(m, opt, sch, 20))
m, opt, sch = setup(); train(m, opt, sch, 10)
torch.save({"model": m.state_dict(), "optim": opt.state_dict(), "sched": sch.state_dict()}, "ck.pt")
m2, opt2, sch2 = setup(); ck = torch.load("ck.pt")
m2.load_state_dict(ck["model"]); opt2.load_state_dict(ck["optim"]); sch2.load_state_dict(ck["sched"])
print("resumed      :", train(m2, opt2, sch2, 10))
m3, opt3, sch3 = setup(); m3.load_state_dict(ck["model"]); opt3.load_state_dict(ck["optim"])   # scheduler NOT restored
print("no scheduler :", train(m3, opt3, sch3, 10))

Output:

uninterrupted: (0.78164, 0.000625)
resumed      : (0.78164, 0.000625)
no scheduler : (0.77804, 0.000625)

Saving and loading the scheduler's state_dict too gives exactly the uninterrupted result. Without it, the run is different (0.77804 instead of 0.78164).

5. Your 10-class classifier starts at loss 9.7. What do you suspect?

Solution

A healthy start is about ln 10 = 2.303. 9.7 means the initial predictions are very confidently wrong: huge initial logits (bad initialisation or unscaled inputs, e.g. raw pixels 0..255), or labels that don't match the outputs. Check the input scale, the last layer, and that the labels are class indices 0..9.

Quick quiz

Gradient accumulation over 8 micro-batches. What goes in place of ???

accum = 8
for i, (xb, yb) in enumerate(loader):
    loss = loss_fn(model(xb), yb) / accum
    loss.backward()
    if (i + 1) % accum == 0:
        ???
Answer

opt.step() then opt.zero_grad(): after 8 micro-batches the summed gradient is complete; step once, then clear .grad for the next 8.

More questions:

  1. What should the loss be at step 0 for a balanced 4-class problem?
    AnswerAbout ln 4 ≈ 1.386.
  2. Which format has the bigger range, FP16 or BF16?
    AnswerBF16: it has FP32's 8 exponent bits (max about 3.4 × 10³⁸); FP16's max is 65,504.
  3. What does scaler.step(opt) do when the gradients contain inf?
    AnswerIt skips the optimizer step (and scaler.update() then lowers the scale).
  4. Why save the optimizer's state_dict in a checkpoint?
    AnswerOptimizers like Adam keep state (moving averages, step count); without it the resumed run takes different steps.
  5. What is 0 * inf?
    Answernan.

Interview questions

  1. Your network's loss doesn't decrease. How do you debug it? Check the starting loss against ln C; overfit one small batch (must reach ~0); check data and labels (visualise a few, check shapes and ranges); check the last layer and loss pairing (logits into CrossEntropyLoss/BCEWithLogitsLoss, no extra activation); check zero_grad(), model.train(), that the optimizer has the model's parameters; then tune the learning rate.
  2. What causes NaN losses and how do you find them? Too-large learning rates and unscaled inputs (divergence), log(0) and similar, bad input data, FP16 overflow. Guard with torch.isfinite(loss), use stable built-in losses, gradient clipping, set_detect_anomaly(True) to locate the operation.
  3. FP16 vs BF16? Both 2 bytes. FP16: 5 exponent + 10 mantissa bits, more digits, small range (max 65,504), needs loss scaling. BF16: 8 exponent + 7 mantissa bits, FP32's range, fewer digits, usually no scaling.
  4. What is mixed precision training and why is it safe? 16-bit forward/backward for speed and memory; FP32 master weights so small updates aren't lost; loss scaling (FP16) so small gradients don't underflow; autocast keeps sensitive operations in FP32.
  5. How does gradient accumulation work, and when is it not equivalent to a bigger batch? Sum gradients of N micro-batches (each loss divided by N), step once. Not equivalent for batch-dependent layers (BatchNorm sees the smaller batch).
  6. What goes into a checkpoint? Model and optimizer state_dicts, epoch/step, scheduler, GradScaler, best metric, and ideally the random generator states and config.
  7. How do you make PyTorch training reproducible? Seed random, NumPy and torch; use_deterministic_algorithms(True); cudnn.benchmark = False; seeded DataLoader generator and workers; same versions and hardware.

Glossary

Term Meaning (plain words + example)
Debugging Finding and fixing a mistake in code: the extra ReLU
Overfitting a single batch Training on one small batch to check the pipeline can memorise it: 8 examples, loss → 0
Sanity check A quick test that must pass if the code is right: start loss ≈ ln 2
Starting loss The loss before any training; for C balanced classes about ln C
Loss curve The loss plotted over epochs or steps, for training and validation
Plateau A flat part of the loss curve where it stops improving
Divergence The loss growing instead of shrinking, often to inf
inf (infinity) The value a float takes when a number is too big to store: above about 3.4 × 10³⁸ in FP32
NaN (not a number) The result of a calculation with no answer: 0 × inf, inf − inf, 0 / 0
Overflow A number too big for its format, becoming inf: 70,000 in FP16
Underflow A number too small for its format, becoming 0: 1e-8 in FP16
Numerically stable Computed in a way that avoids overflow/underflow: BCEWithLogitsLoss
torch.isfinite True if a value is not inf and not nan
Precision How many digits a number format keeps
Exponent bits / mantissa bits The bits that set the range / the digits of a floating-point number
FP32 (float32, single precision) 4-byte float, PyTorch's default
FP16 (float16, half precision) 2-byte float, max 65,504, about 3 digits
BF16 (bfloat16, brain float) 2-byte float from Google Brain with FP32's range, about 2-3 digits
Mixed precision 16-bit maths where safe, FP32 where it matters
Master weights The FP32 copy of the weights that receives the updates
Loss scaling Multiplying the loss (so the gradients) by a big factor to avoid underflow, then dividing back
Automatic mixed precision (AMP) PyTorch choosing the precision per operation: torch.autocast + GradScaler
torch.autocast A context in which eligible operations run in 16 bits
GradScaler Does dynamic loss scaling and skips steps with inf/nan gradients
GPU, CUDA A graphics processor that runs maths in parallel; CUDA is NVIDIA's platform for it (.cuda())
GPU memory The GPU's own fixed-size memory; must hold weights, gradients, optimizer state and activations
Activations Each layer's outputs, kept for the backward pass; grow with batch size
Out of memory (OOM) The error when the GPU can't fit what you asked for
Batch size Examples per step: 32, 256, 2048
Micro-batch One small piece of a bigger batch in gradient accumulation
Gradient accumulation Summing gradients over several micro-batches before one optimizer step
Effective batch size Micro-batch size × accumulation steps: 4 × 16 = 64
torch.save / torch.load Write / read tensors and dictionaries of tensors to/from a file
load_state_dict Copy saved tensors into a model or optimizer of the same shape
Checkpoint A saved file with everything needed to resume: model, optimizer, epoch, scheduler, scaler
Checkpointing Saving checkpoints regularly during training
Size mismatch The error when a saved tensor's shape differs from the model's
Reproducibility Getting exactly the same numbers when you run again
Seed A starting value that fixes a random generator's sequence: torch.manual_seed(42)
Deterministic algorithm An operation that gives bit-identical results every run
cuDNN benchmark NVIDIA cuDNN timing several kernels and picking the fastest, which can vary between runs

Summary

  • Most training bugs don't crash. Overfit one batch: a working pipeline memorises 8 examples (loss → 0). Ours couldn't (0.26 = 3/8 × ln 2) because of a ReLU after the last layer.
  • Check the starting loss (ln C) and read the loss curve: flat = lr too small or a bug; smooth = healthy; bouncing = lr too big; train down, validation up = overfitting.
  • NaN comes from divergence (big lr, unscaled inputs) and maths with no answer (log(0)); use stable built-in losses and a torch.isfinite guard.
  • FP16 has digits but little range (65,504); BF16 has FP32's range but few digits. Mixed precision = 16-bit maths + FP32 master weights + loss scaling; AMP = torch.autocast (+ GradScaler for FP16).
  • Memory = 16 bytes per parameter with Adam + activations that grow with the batch. Gradient accumulation fakes a big batch: divide each loss by N, step once.
  • Checkpoints save model, optimizer, epoch, scheduler and scaler; resuming then gives exactly the same result. Seeds and deterministic settings make runs repeatable.

Next episode: CNNs 1: convolutions: why a plain network fails on images, and how sliding one small kernel over a picture fixes it.