Machine Learning Full Course · Episode 62. Every code sample below was run with Python 3.13 and PyTorch 2.15.0 (CPU). The output shown is the real output.
The big idea: two things decide whether a network trains well before any clever architecture: the learning rate over time and the starting weights. One fixed learning rate is a compromise: early training wants big steps, the end wants small ones, so we use a schedule (step decay, exponential decay, cosine annealing, warmup, one-cycle). The LR finder picks the starting value in one short run. Starting weights with the wrong spread make a deep network's signal explode (to inf/nan) or vanish (to 0) before the first update; Xavier (for tanh/sigmoid) and He (for ReLU) initialisation choose the spread from the layer size. Gradient clipping caps the occasional giant gradient, and leaky ReLU / GELU avoid dead ReLU neurons.
Sources: Xavier Glorot and Yoshua Bengio, "Understanding the difficulty of training deep feedforward neural networks", AISTATS 2010, proposed a new initialisation scheme and showed the logistic sigmoid is unsuited to deep networks with random initialisation (PMLR 9:249-256). Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun, "Delving Deep into Rectifiers", 2015, derived an initialisation that takes rectifiers (ReLU) into account and trains very deep models from scratch (arXiv 1502.01852). Leslie N. Smith, "Cyclical Learning Rates for Training Neural Networks", 2015, describes estimating reasonable learning-rate bounds by increasing the learning rate during a short run (arXiv 1506.01186). Dan Hendrycks and Kevin Gimpel, "Gaussian Error Linear Units (GELUs)", 2016, define GELU as x·Φ(x) (arXiv 1606.08415).
Words you need first
- Learning rate (
lr): the optimiser's step size (Episodes 8 and 61). - Epoch, batch, iteration: one pass over the data; a small group of examples; one optimiser step (Episode 59).
- Hyperparameter: a setting we choose, such as
lr(Episode 8). - Activation function / activation: the non-linear function after a layer (ReLU, tanh, sigmoid) and its output (Episode 56).
- Standard deviation (std, Episode 10): how spread out numbers are around their mean.
- Normal distribution
N(0, σ²): the bell curve;torch.randndraws fromN(0, 1). - Gradient norm: the length of the whole gradient vector,
√(Σ g²)(Episode 3's L2 norm). - Backpropagation (Episode 57): gradients flow from the last layer back to the first, multiplied layer by layer.
The problem: one fixed learning rate
import torch, torch.nn as nn
from torch.utils.data import TensorDataset, DataLoader
torch.manual_seed(62)
X = torch.rand(1000, 2) * 4 - 2; y = (X.pow(2).sum(1) < 1).float().unsqueeze(1) # the circle
loader = DataLoader(TensorDataset(X, y), batch_size=50, shuffle=True)
for lr in (0.0001, 0.01, 0.1, 3.0): # one fixed learning rate for all 30 epochs
torch.manual_seed(0)
model = nn.Sequential(nn.Linear(2, 32), nn.ReLU(), nn.Linear(32, 32), nn.ReLU(), nn.Linear(32, 1))
opt, loss_fn = torch.optim.Adam(model.parameters(), lr=lr), nn.BCEWithLogitsLoss()
for epoch in range(30):
for xb, yb in loader:
loss = loss_fn(model(xb), yb); opt.zero_grad(); loss.backward(); opt.step()
acc = ((model(X) > 0).float() == y).float().mean().item()
print(f"lr={lr:<7} final loss {loss_fn(model(X), y).item():.3f} accuracy {acc:.1%}")
Output:
lr=0.0001 final loss 0.411 accuracy 80.1%
lr=0.01 final loss 0.031 accuracy 99.5%
lr=0.1 final loss 0.073 accuracy 97.6%
lr=3.0 final loss 0.471 accuracy 80.1%
80.1% is what "always predict outside the circle" scores. lr = 0.0001 barely moved; lr = 3.0 jumped around and collapsed to that baseline; 0.01 was best (99.5%); 0.1 was good but rougher (97.6%). No single value is right for the whole run: big steps help early, small steps help at the end, like driving fast on the main road and slowly into a parking spot.
Learning-rate schedules
A learning-rate schedule sets lr as a function of the epoch (or step). In PyTorch: torch.optim.lr_scheduler, and call sched.step() after opt.step().
| Schedule | Rule | PyTorch |
|---|---|---|
| Step decay | multiply lr by γ (gamma) every N epochs: a staircase |
StepLR(opt, step_size=10, gamma=0.1) |
| Exponential decay | multiply by γ every epoch | ExponentialLR(opt, gamma=0.9) |
| Cosine annealing | lr_t = lr_min + (lr_max − lr_min)·(1 + cos(π·t/T))/2 |
CosineAnnealingLR(opt, T_max=30) |
| Warmup | start small and grow for the first few epochs, then decay | LinearLR + SequentialLR |
| One-cycle policy | warm up to max_lr, then anneal to almost 0, in one cycle |
OneCycleLR(opt, max_lr=0.1, total_steps=...) |
- Annealing is a metalwork word: cooling slowly so the metal settles; here the step size cools down.
- Warmup: at the start the weights are random and gradients (and Adam's
v̂) are unreliable, so we begin gently.
import torch
from torch.optim.lr_scheduler import (StepLR, ExponentialLR, CosineAnnealingLR,
LinearLR, SequentialLR, OneCycleLR)
def show(name, make_sched, epochs=30):
opt = torch.optim.SGD([torch.zeros(1, requires_grad=True)], lr=0.1)
sched, lrs = make_sched(opt), []
for epoch in range(epochs):
lrs.append(opt.param_groups[0]["lr"]) # the learning rate used in this epoch
opt.step(); sched.step() # train one epoch ... then move the schedule on
print(f"{name:9}", " ".join(f"{lrs[e]:.3f}" for e in (0, 4, 9, 10, 19, 20, 29)))
print(f"{'epoch:':9}", " ".join(f"{e + 1:>5}" for e in (0, 4, 9, 10, 19, 20, 29)))
show("step", lambda o: StepLR(o, step_size=10, gamma=0.1)) # x0.1 every 10 epochs
show("exp", lambda o: ExponentialLR(o, gamma=0.9)) # x0.9 every epoch
show("cosine", lambda o: CosineAnnealingLR(o, T_max=30)) # half a cosine wave down to 0
show("warmup", lambda o: SequentialLR(o, [LinearLR(o, start_factor=0.1, total_iters=5),
CosineAnnealingLR(o, T_max=25)], milestones=[5]))
show("onecycle", lambda o: OneCycleLR(o, max_lr=0.1, total_steps=30))
Output (the learning rate used in epochs 1, 5, 10, 11, 20, 21 and 30):
epoch: 1 5 10 11 20 21 30
step 0.100 0.100 0.100 0.010 0.010 0.001 0.001
exp 0.100 0.066 0.039 0.035 0.014 0.012 0.005
cosine 0.100 0.096 0.079 0.075 0.030 0.025 0.000
warmup 0.010 0.082 0.094 0.090 0.041 0.035 0.000
onecycle 0.004 0.052 0.099 0.098 0.046 0.039 0.000
Step decay drops ×0.1 at epochs 11 and 21; exponential shrinks smoothly to 0.005; cosine stays high, then falls faster, ending at ~0 (0.00027); warmup rises 0.010 → 0.082 over 5 epochs, then follows a cosine; one-cycle starts at 0.004, peaks at ~0.1 around epoch 10 (30% of the run by default) and ends at ~0.
Does a schedule help?
import torch, torch.nn as nn
from torch.utils.data import TensorDataset, DataLoader
torch.manual_seed(62)
X = torch.rand(1000, 2) * 4 - 2; y = (X.pow(2).sum(1) < 1).float().unsqueeze(1)
loader = DataLoader(TensorDataset(X, y), batch_size=50, shuffle=True) # 20 batches per epoch
def train(schedule):
torch.manual_seed(0)
model = nn.Sequential(nn.Linear(2, 32), nn.ReLU(), nn.Linear(32, 32), nn.ReLU(), nn.Linear(32, 1))
opt, loss_fn = torch.optim.Adam(model.parameters(), lr=0.1), nn.BCEWithLogitsLoss()
sched = schedule(opt) if schedule else None
for epoch in range(30):
for xb, yb in loader:
loss = loss_fn(model(xb), yb); opt.zero_grad(); loss.backward(); opt.step()
if sched: sched.step() # OneCycleLR moves every BATCH
acc = ((model(X) > 0).float() == y).float().mean().item()
return f"final loss {loss_fn(model(X), y).item():.3f} accuracy {acc:.1%}"
print("fixed lr=0.1: ", train(None))
from torch.optim.lr_scheduler import OneCycleLR
print("one-cycle, max 0.1:", train(lambda o: OneCycleLR(o, max_lr=0.1, total_steps=30 * 20)))
Output:
fixed lr=0.1: final loss 0.073 accuracy 97.6%
one-cycle, max 0.1: final loss 0.014 accuracy 99.9%
Same network, same peak learning rate: one-cycle gives 99.9% and a loss five times lower than fixed lr = 0.1 (97.6%). OneCycleLR is stepped every batch, so total_steps = epochs × batches per epoch = 30 × 20.
The learning-rate finder (LR range test)
Trying ten learning rates means ten full trainings. The LR finder does one short run, raising lr a little on every batch, and watches the loss: flat while lr is too small, falling in the useful range, exploding when too big. Pick a value inside the falling region, well below the explosion.
import torch, torch.nn as nn
torch.manual_seed(62)
X = torch.rand(1000, 2) * 4 - 2; y = (X.pow(2).sum(1) < 1).float().unsqueeze(1)
model = nn.Sequential(nn.Linear(2, 32), nn.ReLU(), nn.Linear(32, 32), nn.ReLU(), nn.Linear(32, 1))
loss_fn, opt = nn.BCEWithLogitsLoss(), torch.optim.Adam(model.parameters())
smooth, best, start_lr = None, float("inf"), None # LR finder: a bigger lr on every batch
for i in range(56):
lr = 10 ** (-5 + i / 8) # 0.00001 ... 10, x10 every 8 batches
for g in opt.param_groups: g["lr"] = lr
idx = torch.randint(0, 1000, (100,))
loss = loss_fn(model(X[idx]), y[idx]); opt.zero_grad(); loss.backward(); opt.step()
smooth = loss.item() if smooth is None else 0.7 * smooth + 0.3 * loss.item() # less jumpy
if i == 0: first = smooth
if start_lr is None and smooth < 0.9 * first: start_lr = lr # loss really starts to fall
best = min(best, smooth)
if i % 8 == 0: print(f"lr {lr:<8.5g} smoothed loss {smooth:.3f}")
if smooth > 4 * best: print(f"lr {lr:<8.5g} smoothed loss {smooth:.3f} <- exploded, stop"); break
print(f"learning starts near lr {start_lr:.2g}, blows up near lr {lr:.2g}")
Output:
lr 1e-05 smoothed loss 0.624
lr 0.0001 smoothed loss 0.621
lr 0.001 smoothed loss 0.611
lr 0.01 smoothed loss 0.493
lr 0.1 smoothed loss 0.465
lr 1 smoothed loss 0.400
lr 5.6234 smoothed loss 5.354 <- exploded, stop
learning starts near lr 0.0042, blows up near lr 5.6
Too small below ~0.001 (the loss stays at 0.61–0.62), learning starts near 0.004, the loss explodes near 5.6. A value around 0.01 is inside the useful range, and our first experiment agrees (0.01 gave 99.5%). The smoothed loss is a moving average (0.7·old + 0.3·new) so one lucky or unlucky batch doesn't fool us.
Initialisation: exploding and vanishing signals
Weight initialisation: choosing the weights before training. A message passed along 50 people: if everyone repeats it louder, the end is a roar (exploding); softer, and the end hears nothing (vanishing). Every layer should keep the spread about the same.
- Exploding gradients: gradients grow layer after layer until they are huge,
infornan. - Vanishing gradients: gradients shrink layer after layer until the first layers get (almost) zero and stop learning.
nan("not a number"): the result ofinf − inf,0·inf, etc. Once ananappears it spreads everywhere.- Saturation: tanh/sigmoid outputs stuck near ±1 (or 0/1), where their slope is almost 0.
import torch, torch.nn as nn
def check(name, init, act):
torch.manual_seed(62)
layers = [nn.Linear(256, 256, bias=False) for _ in range(50)] # 50 layers deep
for L in layers: init(L.weight)
h = x = torch.randn(512, 256)
for L in layers: h = act(L(h))
h.pow(2).mean().backward() # any loss will do
grad = layers[0].weight.grad.norm().item()
print(f"{name:22} last std {h.std().item():<9.3g} first grad {grad:.3g}")
check("normal std=1, none", lambda w: nn.init.normal_(w, std=1.0), lambda h: h)
check("normal std=0.01, tanh", lambda w: nn.init.normal_(w, std=0.01), torch.tanh)
check("normal std=1, tanh", lambda w: nn.init.normal_(w, std=1.0), torch.tanh)
check("Xavier, tanh", nn.init.xavier_normal_, torch.tanh)
check("Xavier, ReLU", nn.init.xavier_normal_, torch.relu)
check("He (Kaiming), ReLU", lambda w: nn.init.kaiming_normal_(w, nonlinearity="relu"), torch.relu)
Output (50 layers of 256 → 256, 512 random inputs):
normal std=1, none last std nan first grad nan
normal std=0.01, tanh last std 1.34e-40 first grad 0
normal std=1, tanh last std 0.974 first grad inf
Xavier, tanh last std 0.0927 first grad 0.00449
Xavier, ReLU last std 1.64e-08 first grad 4e-16
He (Kaiming), ReLU last std 0.551 first grad 0.318
std = 1, no activation: each layer multiplies the spread by about √256 = 16, so values overflow toinfand thennan.std = 0.01, tanh: each layer shrinks the spread to about 0.16×; after 50 layers the activations are 1e-40 and the first layer's gradient is exactly 0 (vanished).std = 1, tanh: the forward pass looks fine (0.974, but saturated at ±1), yet the gradient explodes toinf.
Xavier (Glorot) and He (Kaiming) initialisation
Fan-in: the number of inputs to a layer; fan-out: its number of outputs. For nn.Linear(256, 256) both are 256.
| Name | std of the normal distribution | Use with | PyTorch |
|---|---|---|---|
| Xavier / Glorot | √(2 / (fan_in + fan_out)) = 0.0625 here |
tanh, sigmoid | nn.init.xavier_normal_(w) |
| He / Kaiming | √(2 / fan_in) = 0.088 here |
ReLU (and leaky ReLU) | nn.init.kaiming_normal_(w, nonlinearity="relu") |
- Why the 2 in He? ReLU sets about half of its inputs to 0, which halves the signal's energy; doubling the variance makes up for it.
- There are also
_uniform_versions with the same spread.nn.Linearalready uses a Kaiming-uniform default; deep custom networks often set the initialisation on purpose.
Results from the same program: Xavier + tanh keeps a live signal (0.0927, gradient 0.00449); Xavier + ReLU fades (1.64e-08) because ReLU halves the signal each layer; He + ReLU stays healthy (0.551 at the last layer, gradient 0.318 at the first).
Gradient clipping
Even with good initialisation, one unusual batch can produce a giant gradient. Gradient clipping (by norm): if the gradient norm is above max_norm, scale the whole gradient down to length max_norm. The direction stays the same; only the length is capped, like a speed governor on a bus.
loss.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0) # between backward and step
opt.step()
import torch
x = torch.tensor([1.0, 2.0, 3.0, 100.0]); y = 2 * x # true rule y = 2x; one huge input (100)
for clip in (None, 1.0):
w = torch.tensor([0.0], requires_grad=True); opt = torch.optim.SGD([w], lr=0.01)
for step in range(1, 301):
loss = ((w * x - y) ** 2).mean(); opt.zero_grad(); loss.backward()
before = w.grad.norm().item()
if clip: torch.nn.utils.clip_grad_norm_([w], max_norm=clip) # shrink the gradient if too long
opt.step()
if step in (1, 2, 3, 300):
after = w.grad.norm().item()
label = "clipped" if clip else "plain"
print(f"{label:7} step {step:3} grad {before:9.4g} -> {after:9.4g} w {w.item():.4g}")
Output:
plain step 1 grad 1.001e+04 -> 1.001e+04 w 100.1
plain step 2 grad 4.914e+05 -> 4.914e+05 w -4814
plain step 3 grad 2.411e+07 -> 2.411e+07 w 2.363e+05
plain step 300 grad nan -> nan w nan
clipped step 1 grad 1.001e+04 -> 1 w 0.01
clipped step 2 grad 9964 -> 1 w 0.02
clipped step 3 grad 9914 -> 1 w 0.03
clipped step 300 grad 31.66 -> 1 w 1.996
Without clipping, the input 100 makes the first gradient 10,014; w jumps to 100.1, then −4,814, then 236,300, and is nan by step 300. With clipping every step has length 1 (here lr·1 = 0.01), so w walks steadily to 1.996 ≈ 2. clip_grad_norm_ returns the norm before clipping, handy for logging. There is also clip_grad_value_ (clip each element), less common.
Better activations: leaky ReLU and GELU
- ReLU
max(0, z)has gradient 0 for everyz < 0(PyTorch also uses 0 atz = 0). - Dying ReLU / dead neuron: if a neuron's input is negative for every example (say after a big update pushed its bias far down), it outputs 0, gets gradient 0 and never recovers.
- Leaky ReLU:
zifz > 0, else0.01·z(the slope 0.01 isnegative_slope): a little gradient always flows. - GELU (Gaussian error linear unit):
z · Φ(z), whereΦis the standard normal CDF (Episode 10). Smooth, slightly negative for small negativez, widely used in transformer models.
import torch, torch.nn as nn
z = torch.tensor([-3.0, -1.0, -0.5, 0.0, 0.5, 1.0, 3.0], requires_grad=True)
for name, f in [("ReLU", nn.ReLU()), ("Leaky", nn.LeakyReLU(0.01)), ("GELU", nn.GELU())]:
out = f(z); z.grad = None; out.sum().backward()
print(f"{name:5}", [round(v, 3) for v in out.tolist()]) # f(z)
print(" grad", [round(v, 3) for v in z.grad.tolist()]) # slope
torch.manual_seed(62) # a "dead" neuron: its bias pushed far negative
neuron = nn.Linear(2, 1); nn.init.constant_(neuron.bias, -10.0)
x = torch.randn(1000, 2)
for name, f in [("ReLU", nn.ReLU()), ("Leaky", nn.LeakyReLU(0.01))]:
neuron.zero_grad(); f(neuron(x)).sum().backward()
active, g = (neuron(x) > 0).sum().item(), neuron.weight.grad.abs().sum().item()
print(f"{name:5} on 1000 inputs: active {active}, weight grad {g:.3g}")
Output:
ReLU [0.0, 0.0, 0.0, 0.0, 0.5, 1.0, 3.0]
grad [0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 1.0]
Leaky [-0.03, -0.01, -0.005, 0.0, 0.5, 1.0, 3.0]
grad [0.01, 0.01, 0.01, 0.01, 1.0, 1.0, 1.0]
GELU [-0.004, -0.159, -0.154, 0.0, 0.346, 0.841, 2.996]
grad [-0.012, -0.083, 0.133, 0.5, 0.867, 1.083, 1.012]
ReLU on 1000 inputs: active 0, weight grad 0
Leaky on 1000 inputs: active 0, weight grad 1.22
ReLU's gradient is 0 for the four inputs ≤ 0; leaky ReLU keeps 0.01; GELU dips to −0.159 at −1 and curves smoothly. The neuron with bias −10 is never active; with ReLU its weight gradient is exactly 0 (dead), with leaky ReLU it is 1.22, so training can still move it.
Common mistakes
| Mistake | Exact message / symptom | Why | Fix |
|---|---|---|---|
One fixed lr for the whole run |
97.6% vs 99.9% with one-cycle | can't take small steps at the end | a schedule |
sched.step() before opt.step() |
UserWarning: Detected call of lr_scheduler.step() before optimizer.step()... Failure to do this will result in PyTorch skipping the first value of the learning rate schedule. |
wrong order | opt.step() then sched.step() |
Stepping OneCycleLR once per epoch with total_steps = batches |
the schedule ends after a few epochs | it counts calls | match total_steps to how often you call it |
std = 1 weights in a deep network |
nan, first-layer grad inf |
exploding | Xavier / He |
Tiny weights (std = 0.01) |
activations 1.34e-40, grad 0 | vanishing | Xavier / He |
| Xavier with ReLU in a deep net | last-layer std 1.64e-08 | ReLU halves the signal | He (Kaiming) |
| No clipping with rare huge inputs | w = nan |
one giant step | clip_grad_norm_ |
| Dead ReLU neurons | weight grad 0 | always-negative input | leaky ReLU / GELU, lower lr, good init |
Mini project: put it all together
A deeper network (6 hidden layers), He initialisation, AdamW, a warmup + cosine schedule stepped per batch (LambdaLR multiplies the base lr by the function's value), and gradient clipping:
import torch, torch.nn as nn, math
from torch.utils.data import TensorDataset, DataLoader
torch.manual_seed(62)
X = torch.rand(1000, 2) * 4 - 2; y = (X.pow(2).sum(1) < 1).float().unsqueeze(1)
loader = DataLoader(TensorDataset(X, y), batch_size=50, shuffle=True)
torch.manual_seed(0)
layers = [nn.Linear(2, 64), nn.ReLU()]
for _ in range(5): layers += [nn.Linear(64, 64), nn.ReLU()] # 6 hidden layers
model = nn.Sequential(*layers, nn.Linear(64, 1))
for m in model:
if isinstance(m, nn.Linear): nn.init.kaiming_normal_(m.weight, nonlinearity="relu"); nn.init.zeros_(m.bias)
opt = torch.optim.AdamW(model.parameters(), lr=0.01, weight_decay=0.01)
total, warm = 30 * len(loader), 2 * len(loader) # 600 steps, 40 of warmup
sched = torch.optim.lr_scheduler.LambdaLR(opt, lambda s: (s + 1) / warm if s < warm
else 0.5 * (1 + math.cos(math.pi * (s - warm) / (total - warm))))
loss_fn, biggest = nn.BCEWithLogitsLoss(), 0.0
for epoch in range(1, 31):
for xb, yb in loader:
loss = loss_fn(model(xb), yb); opt.zero_grad(); loss.backward()
biggest = max(biggest, nn.utils.clip_grad_norm_(model.parameters(), 1.0).item())
opt.step(); sched.step()
if epoch in (1, 2, 10, 30):
acc = ((model(X) > 0).float() == y).float().mean().item()
print(f"epoch {epoch:2}: lr {opt.param_groups[0]['lr']:.5f} loss {loss_fn(model(X), y).item():.3f} acc {acc:.1%}")
print(f"largest gradient norm before clipping: {biggest:.2f}")
Output:
epoch 1: lr 0.00525 loss 0.319 acc 80.5%
epoch 2: lr 0.01000 loss 0.144 acc 94.9%
epoch 10: lr 0.00812 loss 0.018 acc 99.4%
epoch 30: lr 0.00000 loss 0.003 acc 100.0%
largest gradient norm before clipping: 4.18
The learning rate warms up to 0.01 by the end of epoch 2, then follows a cosine down to 0; the model reaches 100% on the circle, and the largest gradient norm seen (4.18) was clipped to 1.
Practice
1. Plot the cosine and one-cycle learning-rate curves with matplotlib.
Solution
import torch, matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
from torch.optim.lr_scheduler import CosineAnnealingLR, OneCycleLR
def curve(make, steps=100):
opt = torch.optim.SGD([torch.zeros(1, requires_grad=True)], lr=0.1); s = make(opt); out = []
for _ in range(steps): out.append(opt.param_groups[0]["lr"]); opt.step(); s.step()
return out
cos = curve(lambda o: CosineAnnealingLR(o, T_max=100))
one = curve(lambda o: OneCycleLR(o, max_lr=0.1, total_steps=100))
plt.plot(cos, label="cosine annealing"); plt.plot(one, label="one-cycle")
plt.xlabel("step"); plt.ylabel("learning rate"); plt.legend(); plt.savefig("lr_curves.png")
print("peak of one-cycle at step", one.index(max(one)), "| saved lr_curves.png")
Output:
peak of one-cycle at step 29 | saved lr_curves.png
The PNG shows cosine falling from 0.1 to 0 like half a wave, and one-cycle rising for the first 30% and then falling.
2. Run the LR finder with SGD instead of Adam.
Solution
Change the optimiser line to opt = torch.optim.SGD(model.parameters(), lr=1e-5). Output:
lr 1e-05 smoothed loss 0.624
lr 0.0001 smoothed loss 0.622
lr 0.001 smoothed loss 0.623
lr 0.01 smoothed loss 0.610
lr 0.1 smoothed loss 0.585
lr 1 smoothed loss 0.493
lr 10 smoothed loss 0.538
lr 23.714 smoothed loss 2.340 <- exploded, stop
learning starts near lr 0.32, blows up near lr 24
SGD's useful range sits at larger learning rates than Adam's: its step is lr × gradient, while Adam's is about lr.
3. Use GELU in the 50-layer test. Which initialisation works?
Solution
Add check("Xavier, GELU", nn.init.xavier_normal_, nn.functional.gelu) and check("He, GELU", lambda w: nn.init.kaiming_normal_(w, nonlinearity="relu"), nn.functional.gelu). Output:
normal std=1, none last std nan first grad nan
normal std=0.01, tanh last std 1.34e-40 first grad 0
normal std=1, tanh last std 0.974 first grad inf
Xavier, tanh last std 0.0927 first grad 0.00449
Xavier, ReLU last std 1.64e-08 first grad 4e-16
He (Kaiming), ReLU last std 0.551 first grad 0.318
Xavier, GELU last std 1.63e-15 first grad 0
He, GELU last std 3.36e-06 first grad 8.61e-10
Neither keeps a 50-layer GELU network fully alive: GELU passes less of the signal than ReLU for inputs near zero, so even He fades (3.36e-06), though far less than Xavier (1.63e-15). Very deep GELU networks also rely on normalisation layers (Episode 63) and residual connections (Episode 66).
4. Add clip_grad_norm_ to Episode 61's race and print the largest gradient norm seen.
Solution
Inside the batch loop: n = torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0); biggest = max(biggest, n.item()) between backward() and step(). Output:
epoch: 1 5 10 20
SGD 0.647 0.537 0.493 0.475 max grad norm 0.68
SGD+momentum 0.508 0.467 0.457 0.399 max grad norm 0.66
AdaGrad 0.502 0.470 0.462 0.443 max grad norm 0.64
RMSprop 0.436 0.114 0.065 0.109 max grad norm 1.24
Adam 0.476 0.418 0.055 0.029 max grad norm 1.07
On this small, well-scaled problem the gradient norm stays below 1 for SGD, momentum and AdaGrad, so clipping never acts; it only trims RMSprop (max 1.24) and Adam (max 1.07), and RMSprop's last loss moves from 0.138 to 0.109. Clipping is insurance, not a speed-up.
Quick quiz
nn.Linear(400, 100) followed by ReLU. What std does He initialisation use?
Answer
√(2/400) ≈ 0.071. He uses fan-in only (400). √(2/500) ≈ 0.063 is Xavier's formula.
More questions:
- Which schedule multiplies
lrby 0.1 every 10 epochs?Answer
Step decay,StepLR(opt, step_size=10, gamma=0.1). - Why warm up?
Answer
Early gradients from random weights are large and unreliable; a smalllrat first avoids wrecking the model, then it rises. - In the LR finder, where do you pick
lr?Answer
Inside the region where the loss falls fastest, well below where it explodes. - What does
clip_grad_norm_change: the direction or the length?Answer
Only the length (when it's abovemax_norm). - What is a dead ReLU neuron?
Answer
One whose input is negative for all examples, so its output and gradient are 0 and it never learns again.
Interview questions
- Why decay the learning rate? Large steps make fast progress but bounce around the minimum; smaller steps later let the parameters settle into it.
- Cosine annealing vs step decay? Both lower
lrover time; cosine is smooth with no hand-picked milestones, step decay drops suddenly at chosen epochs. - What is the one-cycle policy? Increase
lrto a maximum, then decrease it to nearly zero over a single cycle (often with momentum moving the opposite way); it frequently trains faster. - Explain vanishing and exploding gradients. Backpropagation multiplies by each layer's weights and activation slopes; if those factors are consistently below 1 the gradient shrinks exponentially with depth, above 1 it grows exponentially.
- Xavier vs He? Xavier keeps the variance with
2/(fan_in+fan_out), suited to tanh/sigmoid; He uses2/fan_inbecause ReLU zeroes half its inputs. - Besides initialisation, what fights vanishing/exploding gradients? ReLU-family activations, normalisation layers (Episode 63), residual connections (Episode 66), gradient clipping, and LSTM gates for sequences (Episode 70).
Glossary
| Term | Meaning (plain words + example) |
|---|---|
| Learning-rate schedule | A plan that changes lr during training |
| Scheduler | The PyTorch object that applies a schedule: StepLR(opt, ...) |
| Step decay | Multiply lr by γ every N epochs: 0.1 → 0.01 → 0.001 |
| γ (gamma) | The decay factor: 0.1 or 0.9 |
| Exponential decay | Multiply lr by γ every epoch: 0.1, 0.09, 0.081 … |
| Cosine annealing | lr follows half a cosine wave down to ~0 |
| Annealing | Cooling slowly; here, lowering the step size gradually |
| Warmup | Starting with a small lr and raising it for the first epochs |
| One-cycle policy | Warm up to max_lr, then anneal to ~0, once: OneCycleLR |
| LR finder (LR range test) | One short run with lr growing each batch, to find the useful range |
| Smoothed loss | A moving average of the loss: 0.7·old + 0.3·new |
| Weight initialisation | Choosing the starting weights: nn.init.kaiming_normal_(w) |
| Fan-in / fan-out | A layer's number of inputs / outputs: 256 / 256 |
| Xavier (Glorot) initialisation | std √(2/(fan_in+fan_out)), for tanh and sigmoid |
| He (Kaiming) initialisation | std √(2/fan_in), for ReLU |
| Exploding gradients | Gradients growing layer by layer to huge values / inf |
| Vanishing gradients | Gradients shrinking layer by layer to ~0 |
nan |
"Not a number": the result of inf − inf; it spreads through every later calculation |
| Saturation | tanh/sigmoid stuck at their flat ends, slope ≈ 0 |
| Gradient norm | The length of the gradient vector: 10,014 |
| Gradient clipping | Scaling the gradient down when its norm exceeds max_norm |
max_norm |
The clipping limit: 1.0 |
| Dying ReLU (dead neuron) | A ReLU unit that always outputs 0 and gets 0 gradient |
| Leaky ReLU | z for z > 0, else 0.01·z |
negative_slope |
Leaky ReLU's slope below 0: 0.01 |
| GELU | Gaussian error linear unit, z·Φ(z): smooth, used in transformers |
| Φ (Phi) | The standard normal cumulative distribution function (CDF) |
Summary
- One fixed learning rate is a compromise; schedules start big and finish small: step decay, exponential, cosine annealing, warmup, one-cycle.
- The LR finder finds the useful range in one short run.
- Bad starting weights make a deep network explode or vanish before training starts.
- Xavier for tanh/sigmoid, He (Kaiming) for ReLU.
- Gradient clipping caps rare giant gradients; leaky ReLU and GELU avoid dead neurons.
Next episode: Regularisation for deep networks: dropout, weight decay, early stopping, data augmentation, batch norm, layer norm and label smoothing.