Machine Learning Full Course · Episode 66. Every code sample below was run with Python 3.13, PyTorch 2.15.0 and torchvision 0.29 (CPU, one thread, as in the video's compiler). The output shown is the real output; training times depend on the machine. MNIST is loaded from /data/mnist.pt (Episode 65 notes show the one-line torchvision download).
The big idea: CNN architectures grew by solving each other's problems. LeNet (1998) set the pattern conv → pool → conv → pool → classifier. AlexNet (2012) made it big (60 million parameters) with ReLU, GPUs and dropout and won ImageNet by a huge margin. VGG (2014) used only small 3 × 3 kernels, stacked deep. Then deeper plain networks got worse: gradients vanish on the way back, and even with batch norm deeper nets degrade. ResNet (2015) fixed it with the residual (skip) connection out = relu(F(x) + x), training 152 layers. 1 × 1 convolutions change channel counts cheaply, and global average pooling replaces the huge, position-tied classifier head.
Sources: Yann LeCun, Léon Bottou, Yoshua Bengio and Patrick Haffner, "Gradient-Based Learning Applied to Document Recognition", Proceedings of the IEEE 86(11):2278−2324, 1998 (bottou.org). Alex Krizhevsky, Ilya Sutskever and Geoffrey Hinton, "ImageNet Classification with Deep Convolutional Neural Networks", NeurIPS 2012 (abstract). Karen Simonyan and Andrew Zisserman, "Very Deep Convolutional Networks for Large-Scale Image Recognition", 2014 (arXiv 1409.1556). Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun, "Deep Residual Learning for Image Recognition", 2015 (arXiv 1512.03385, paper text). Min Lin, Qiang Chen and Shuicheng Yan, "Network In Network", 2013 (arXiv 1312.4400). CIFAR-10 (Alex Krizhevsky's page).
Words you need first
- Convolution, kernel, feature map, pooling, receptive field (Episode 65).
- Architecture: the design of a network: which layers, in what order, with what sizes.
- ImageNet / ILSVRC: a huge labelled photo collection and its yearly contest (ImageNet Large Scale Visual Recognition Challenge); the classification task had 1,000 classes and about 1.2 million training images.
- Top-5 error: a prediction counts as wrong if the right label is not among the model's 5 highest-scored classes.
- Depth: the number of layers with weights.
- Fully connected layer: another name for
nn.Linear: every input connects to every output. - Backpropagation and the chain rule (Episode 57): gradients flow backwards, multiplied layer by layer.
- Batch normalisation (Episode 63): standardising each channel over the batch inside the network.
- Data augmentation (Episode 63): training on randomly changed copies of the data.
The story: four buildings
Real-life example: four buildings, each solving the last one's problem. LeNet is a small, careful house; AlexNet a much bigger building made possible by new materials (ReLU, dropout, graphics cards); VGG a tall tower of identical small bricks; ResNet a skyscraper that only works because it has lifts (skip connections), so nothing has to climb every stair.
| Year | Network | Key idea | Size |
|---|---|---|---|
| 1998 | LeNet-5 | conv → pool → conv → pool → fully connected; 32 × 32 handwritten characters | 61,706 parameters (modern version below) |
| 2012 | AlexNet | 5 conv + 3 fully connected, ReLU ("non-saturating neurons"), GPU convolution, dropout | 60 million parameters, 650,000 neurons |
| 2014 | VGG | only 3 × 3 kernels, 16-19 weight layers | VGG-16: 138 million parameters |
| 2015 | ResNet | residual (skip) connections, BN after each conv, up to 152 layers | ResNet-152: 60 million parameters |
LeNet-5 (1998)
Described by LeCun, Bottou, Bengio and Haffner in 1998 for reading handwritten characters (32 × 32 input). Layout: C1 conv (6 kernels 5 × 5) → S2 pooling → C3 conv (16 kernels 5 × 5) → S4 pooling → C5 (120) → F6 (84) → 10 outputs. It used tanh-style activations and average-style pooling.
import torch, torch.nn as nn
lenet = nn.Sequential( # LeNet-5 (LeCun et al., 1998), modern PyTorch
nn.Conv2d(1, 6, 5), nn.Tanh(), nn.AvgPool2d(2), # C1: 6 maps 28x28 S2: 14x14
nn.Conv2d(6, 16, 5), nn.Tanh(), nn.AvgPool2d(2), # C3: 16 maps 10x10 S4: 5x5
nn.Flatten(), nn.Linear(16 * 5 * 5, 120), nn.Tanh(), # C5: 120
nn.Linear(120, 84), nn.Tanh(), nn.Linear(84, 10)) # F6: 84, output: 10 digits
x = torch.randn(1, 1, 32, 32) # LeNet-5 reads 32 x 32 images
for layer in lenet:
x = layer(x)
if not isinstance(layer, nn.Tanh):
n = sum(p.numel() for p in layer.parameters())
print(f"{type(layer).__name__:9} -> {str(tuple(x.shape)):16} parameters {n:6,}")
print("total parameters:", f"{sum(p.numel() for p in lenet.parameters()):,}")
Output:
Conv2d -> (1, 6, 28, 28) parameters 156
AvgPool2d -> (1, 6, 14, 14) parameters 0
Conv2d -> (1, 16, 10, 10) parameters 2,416
AvgPool2d -> (1, 16, 5, 5) parameters 0
Flatten -> (1, 400) parameters 0
Linear -> (1, 120) parameters 48,120
Linear -> (1, 84) parameters 10,164
Linear -> (1, 10) parameters 850
total parameters: 61,706
The two convolutions have about 2,500 parameters; the fully connected layers about 59,000. This simplified modern version connects every C3 kernel to all 6 maps and uses a plain linear output; the original connected C3 kernels to subsets of maps, had trainable subsampling layers and a different output layer, so its parameter count differs.
AlexNet (2012): the ImageNet moment
From the paper's abstract (Krizhevsky, Sutskever, Hinton, NeurIPS 2012):
- trained on 1.2 million images of the ImageNet LSVRC-2010 contest, 1,000 classes;
- 60 million parameters and 650,000 neurons; five convolutional layers, some followed by max pooling, and three fully connected layers ending in a 1000-way softmax;
- "to make training faster, we used non-saturating neurons and a very efficient GPU implementation of the convolution operation" (non-saturating neurons = ReLU, Episode 56);
- dropout in the fully connected layers to reduce overfitting (Episode 63);
- a variant won ILSVRC-2012 with a top-5 test error of 15.3%, against 26.2% for the second-best entry.
VGG (2014): small kernels, deep stacks
Simonyan and Zisserman used only 3 × 3 kernels and increased depth to 16-19 weight layers; in the ImageNet Challenge 2014 their team took first place in localisation and second in classification.
Why 3 × 3? Two 3 × 3 convolutions in a row have a 5 × 5 receptive field, three have 7 × 7 (Episode 65), with fewer weights and an extra ReLU between them (more non-linearity).
import torch.nn as nn, torchvision.models as models
count = lambda m: sum(p.numel() for p in m.parameters())
for name in ("alexnet", "vgg16", "resnet18", "resnet50", "resnet152"):
m = getattr(models, name)(weights=None) # the architecture only, random weights
convs = sum(isinstance(l, nn.Conv2d) for l in m.modules())
print(f"{name:9} conv layers {convs:3} parameters {count(m):12,}")
C = 64 # VGG's trick: stack small 3 x 3 kernels
one5, two3 = count(nn.Conv2d(C, C, 5)), 2 * count(nn.Conv2d(C, C, 3))
one7, three3 = count(nn.Conv2d(C, C, 7)), 3 * count(nn.Conv2d(C, C, 3))
print(f"see 5x5: one 5x5 conv {one5:,} | two 3x3 convs {two3:,}")
print(f"see 7x7: one 7x7 conv {one7:,} | three 3x3 convs {three3:,}")
Output:
alexnet conv layers 5 parameters 61,100,840
vgg16 conv layers 13 parameters 138,357,544
resnet18 conv layers 20 parameters 11,689,512
resnet50 conv layers 53 parameters 25,557,032
resnet152 conv layers 155 parameters 60,192,808
see 5x5: one 5x5 conv 102,464 | two 3x3 convs 73,856
see 7x7: one 7x7 conv 200,768 | three 3x3 convs 110,784
(weights=None builds the architecture with random weights; nothing is downloaded. The conv-layer counts for ResNets include the 1 × 1 convolutions on their shortcut paths.) AlexNet and VGG keep most of their parameters in the fully connected layers; ResNet-152 is much deeper than VGG-16 but has fewer than half its parameters, and He et al. report 11.3 billion FLOPs for ResNet-152 against 15.3 billion for VGG-16.
Going deeper: vanishing gradients
Eight layers (AlexNet) to nineteen (VGG) helped, so why not 50 or 100? In backpropagation the gradient travels back through every layer and is multiplied by each layer's local derivative (chain rule). If many of those factors are below 1, the gradient shrinks towards 0 before it reaches the first layers: the vanishing gradient problem. Those layers then barely learn.
import torch, torch.nn as nn
torch.manual_seed(66); x, y = torch.randn(32, 8, 14, 14), torch.randint(0, 10, (32,))
class Block(nn.Module):
def __init__(self, skip):
super().__init__(); self.skip = skip
self.conv = nn.Conv2d(8, 8, 3, padding=1)
def forward(self, x):
out = torch.relu(self.conv(x))
return x + out if self.skip else out
for depth in (2, 10, 30, 50):
for skip in (False, True):
torch.manual_seed(0)
blocks = [Block(skip) for _ in range(depth)]
net = nn.Sequential(*blocks, nn.AdaptiveAvgPool2d(1), nn.Flatten(), nn.Linear(8, 10))
nn.functional.cross_entropy(net(x), y).backward()
g = net[0].conv.weight.grad.norm().item()
print(f"depth {depth:2} {'with skip' if skip else 'plain '}: gradient at layer 1 = {g:.2e}")
Output:
depth 2 plain : gradient at layer 1 = 1.91e-02
depth 2 with skip: gradient at layer 1 = 7.10e-02
depth 10 plain : gradient at layer 1 = 3.78e-06
depth 10 with skip: gradient at layer 1 = 6.96e-01
depth 30 plain : gradient at layer 1 = 1.09e-15
depth 30 with skip: gradient at layer 1 = 1.54e+02
depth 50 plain : gradient at layer 1 = 0.00e+00
depth 50 with skip: gradient at layer 1 = 2.04e+04
A plain stack of 50 blocks delivers exactly zero gradient to its first layer. With the skip (x + out) the gradient never vanishes; here it even grows with depth, which is why real ResNets put batch normalisation inside every block.
The degradation problem. He et al. (2015) found that even with batch normalisation, deeper plain networks got worse: on CIFAR-10 a 56-layer plain network had higher training error (and so test error) than a 20-layer one (their Figure 1). That is not overfitting: it is worse on the training data itself. It's strange, because the deeper network could copy the shallower one and make its extra layers compute the identity (output = input); but learning "do nothing" turns out to be hard for a stack of convolutions and ReLUs.
ResNet: residual (skip) connections
Real-life example: fifty students pass a message down a line by whispering, and it's garbled by the end (a plain deep network). Now pass the written note along too: the original always arrives (the skip connection). Each student only writes a small correction on it, or passes it on unchanged (the residual).
output = relu( F(x) + x ) F = conv → BN → ReLU → conv → BN
- The block's desired output is
H(x); the layers learn only the residualF(x) = H(x) − x, what to add tox. - "Do nothing" is now easy:
F's weights near 0 giveF(x) ≈ 0, output≈ x(after the ReLU). - The
+ xpath carries the gradient straight back through every block (the derivative ofxwith respect to itself is 1). - An identity shortcut adds no parameters and no computation. When
Fchanges the number of channels or the size, the shortcut uses a 1 × 1 convolution (with stride) to match shapes. - He et al. used batch normalisation right after each convolution, trained networks up to 152 layers ("8× deeper than VGG nets"), reached 3.57% top-5 error on the ImageNet test set with an ensemble, and won 1st place in ILSVRC 2015 classification.
A residual block in PyTorch:
class Residual(nn.Module): # out = relu(x + F(x)), same shape in and out
def __init__(self, c):
super().__init__(); conv = lambda: nn.Conv2d(c, c, 3, padding=1, bias=False)
self.f = nn.Sequential(conv(), nn.BatchNorm2d(c), nn.ReLU(), conv(), nn.BatchNorm2d(c))
def forward(self, x): return torch.relu(x + self.f(x))
(bias=False because the batch norm right after has its own shift.)
1 × 1 convolutions
A 1 × 1 kernel looks at one pixel across all its channels. So it is the same small linear layer applied at every pixel: it mixes channels and changes their number cheaply, e.g. 256 → 64 (a bottleneck, used in ResNet-50 and deeper). The idea comes from Network in Network (Lin, Chen and Yan, 2013), whose "micro networks" slide over the input like convolutions.
Global average pooling (GAP)
The flatten + linear head is where LeNet, AlexNet and VGG keep most of their weights, and it is tied to positions again. Global average pooling averages each whole feature map into one number: 64 maps → 64 numbers → a tiny linear layer. Where the pattern was no longer matters, and it works for any input size. Network in Network proposed it as easier to interpret and less prone to overfitting than fully connected layers. In PyTorch: nn.AdaptiveAvgPool2d(1) (then nn.Flatten()).
import torch, torch.nn as nn
torch.manual_seed(66)
x = torch.randn(1, 256, 14, 14) # 256 feature maps of 14 x 14
squeeze = nn.Conv2d(256, 64, kernel_size=1) # a 1 x 1 convolution
n = sum(p.numel() for p in squeeze.parameters())
print("1x1 conv:", tuple(x.shape), "->", tuple(squeeze(x).shape))
print("1x1 conv parameters:", n)
pixel = x[0, :, 3, 5] # the 256 numbers at ONE pixel
same = squeeze.weight.view(64, 256) @ pixel + squeeze.bias
print("same as a Linear layer on each pixel?", torch.allclose(same, squeeze(x)[0, :, 3, 5], atol=1e-5))
gap = nn.AdaptiveAvgPool2d(1) # global average pooling
f = squeeze(x)
print("global average pooling:", tuple(f.shape), "->", tuple(gap(f).shape))
print("first value = mean of map 0?", torch.allclose(gap(f)[0, 0, 0, 0], f[0, 0].mean()))
old = nn.Linear(64 * 14 * 14, 10); new = nn.Linear(64, 10)
count = lambda m: f"{sum(p.numel() for p in m.parameters()):,}"
print("classifier head: Flatten + Linear", count(old), "| GAP + Linear", count(new))
for side in (14, 20, 33): # any map size gives one number per map
print(f"GAP on {side} x {side} maps ->", tuple(gap(torch.randn(1, 64, side, side)).shape))
Output:
1x1 conv: (1, 256, 14, 14) -> (1, 64, 14, 14)
1x1 conv parameters: 16448
same as a Linear layer on each pixel? True
global average pooling: (1, 64, 14, 14) -> (1, 64, 1, 1)
first value = mean of map 0? True
classifier head: Flatten + Linear 125,450 | GAP + Linear 650
GAP on 14 x 14 maps -> (1, 64, 1, 1)
GAP on 20 x 20 maps -> (1, 64, 1, 1)
GAP on 33 x 33 maps -> (1, 64, 1, 1)
Training a small ResNet on MNIST
Plan: conv 16 + BN + ReLU → residual (16) → max pool (28 → 14) → 1 × 1 conv (16 → 32) → residual (32) → max pool (14 → 7) → residual (32) → GAP (32 numbers) → linear (10). Data augmentation: each training batch is moved by a random −3..3 pixels across and down (torch.roll). It trains for about 35 seconds on one CPU thread, longer than the video compiler's time limit, so the video shows its real output on a slide.
import torch, torch.nn as nn, time
torch.manual_seed(66); d = torch.load("/data/mnist.pt")
X = d["train_x"][:12000].float().div(255).unsqueeze(1); y = d["train_y"][:12000]
Xt = d["test_x"][:2000].float().div(255).unsqueeze(1); yt = d["test_y"][:2000]
class Residual(nn.Module): # out = relu(x + F(x))
def __init__(self, c):
super().__init__(); conv = lambda: nn.Conv2d(c, c, 3, padding=1, bias=False)
self.f = nn.Sequential(conv(), nn.BatchNorm2d(c), nn.ReLU(), conv(), nn.BatchNorm2d(c))
def forward(self, x): return torch.relu(x + self.f(x))
torch.manual_seed(0)
net = nn.Sequential(nn.Conv2d(1, 16, 3, padding=1, bias=False), nn.BatchNorm2d(16), nn.ReLU(),
Residual(16), nn.MaxPool2d(2), nn.Conv2d(16, 32, 1), Residual(32), nn.MaxPool2d(2),
Residual(32), nn.AdaptiveAvgPool2d(1), nn.Flatten(), nn.Linear(32, 10))
opt = torch.optim.Adam(net.parameters(), lr=1e-3); t = time.time()
for epoch in range(3):
net.train()
for i in range(0, 12000, 64):
dx, dy = torch.randint(-3, 4, (2,)).tolist() # augmentation: move -3..3 pixels
xb = torch.roll(X[i:i + 64], shifts=(dx, dy), dims=(3, 2))
loss = nn.functional.cross_entropy(net(xb), y[i:i + 64])
opt.zero_grad(); loss.backward(); opt.step()
net.eval(); acc = lambda X: f"{(net(X).argmax(1) == yt).float().mean().item():.1%}"
with torch.no_grad():
n = sum(p.numel() for p in net.parameters())
print("parameters:", f"{n:,}", "| training:", round(time.time() - t), "s")
print("test accuracy:", acc(Xt))
print("moved 1, 2, 3 pixels right:", *[acc(torch.roll(Xt, k, dims=3)) for k in (1, 2, 3)])
Output:
parameters: 42,842 | training: 35 s
test accuracy: 91.9%
moved 1, 2, 3 pixels right: 91.3% 90.6% 89.3%
The same network without augmentation:
import torch, torch.nn as nn, time
torch.manual_seed(66); d = torch.load("/data/mnist.pt")
X = d["train_x"][:12000].float().div(255).unsqueeze(1); y = d["train_y"][:12000]
Xt = d["test_x"][:2000].float().div(255).unsqueeze(1); yt = d["test_y"][:2000]
class Residual(nn.Module): # out = relu(x + F(x))
def __init__(self, c):
super().__init__(); conv = lambda: nn.Conv2d(c, c, 3, padding=1, bias=False)
self.f = nn.Sequential(conv(), nn.BatchNorm2d(c), nn.ReLU(), conv(), nn.BatchNorm2d(c))
def forward(self, x): return torch.relu(x + self.f(x))
torch.manual_seed(0)
net = nn.Sequential(nn.Conv2d(1, 16, 3, padding=1, bias=False), nn.BatchNorm2d(16), nn.ReLU(),
Residual(16), nn.MaxPool2d(2), nn.Conv2d(16, 32, 1), Residual(32), nn.MaxPool2d(2),
Residual(32), nn.AdaptiveAvgPool2d(1), nn.Flatten(), nn.Linear(32, 10))
opt = torch.optim.Adam(net.parameters(), lr=1e-3); t = time.time()
for epoch in range(3):
net.train()
for i in range(0, 12000, 64):
xb = X[i:i + 64] # no augmentation
loss = nn.functional.cross_entropy(net(xb), y[i:i + 64])
opt.zero_grad(); loss.backward(); opt.step()
net.eval(); acc = lambda X: f"{(net(X).argmax(1) == yt).float().mean().item():.1%}"
with torch.no_grad():
n = sum(p.numel() for p in net.parameters())
print("parameters:", f"{n:,}", "| training:", round(time.time() - t), "s")
print("test accuracy:", acc(Xt))
print("moved 1, 2, 3 pixels right:", *[acc(torch.roll(Xt, k, dims=3)) for k in (1, 2, 3)])
Output:
parameters: 42,842 | training: 71 s
test accuracy: 88.3%
moved 1, 2, 3 pixels right: 87.9% 86.7% 84.9%
| Model (12,000 training digits, 3 epochs, CPU) | Weights | Test | Moved 3 px |
|---|---|---|---|
| MLP (Episode 65) | 101,770 | 89.5% | 44.0% |
| First CNN (Episode 65) | 9,098 | 91.0% | 58.3% |
| ResNet + GAP, no augmentation | 42,842 | 88.3% | 84.9% |
| ResNet + GAP + augmentation | 42,842 | 91.9% | 89.3% |
Global average pooling makes the network far less tied to positions (84.9% at 3 pixels even without augmentation); augmentation adds the rest. (At this small depth the skip connections are not what makes the difference: see practice 3.) On all 60,000 digits for 5 epochs (practice 4) it reaches 98.6%, and 98.3% on digits moved 3 pixels.
CIFAR-10 is the usual next step: 60,000 colour images of 32 × 32 in 10 classes (airplane, automobile, bird, cat, deer, dog, frog, horse, ship, truck), 50,000 for training and 10,000 for testing. The same code works with nn.Conv2d(3, 16, ...) for 3 input channels, loaded with torchvision.datasets.CIFAR10("data", download=True, transform=transforms.ToTensor()); real colour photos are much harder than digits, so expect lower accuracy and use more epochs, more channels and augmentation such as random crops and horizontal flips.
Common mistakes
| Mistake | Exact message / symptom | Why | Fix |
|---|---|---|---|
| Stacking more plain layers | gradient at layer 1: 0.00e+00 at 50 blocks |
vanishing gradients, degradation | residual connections + batch norm |
Adding x when shapes differ |
RuntimeError: The size of tensor a (16) must match the size of tensor b (32) at non-singleton dimension 1 |
F(x) changed the channels |
1 × 1 conv (with stride) on the shortcut |
| Huge flatten + linear head | 125,450 weights for 10 classes, tied to positions | one weight per position | global average pooling |
Expecting weights=None models to know anything |
random predictions | architecture only | train it, or load pretrained weights (Episode 67) |
| Augmenting the test set | a noisy, unfair score | only training data is changed | fixed test data |
| Evaluating a BN network in train mode | predictions depend on the batch | BN uses batch statistics | model.eval() |
| Testing 10,000 images in one forward pass | slow, or out of memory | activations for all at once | evaluate in chunks of e.g. 1,000 |
Practice
1. Count the parameters of resnet34 and vgg19.
Solution
import torchvision.models as models
for name in ("resnet34", "vgg19"):
m = getattr(models, name)(weights=None)
print(f"{name}: {sum(p.numel() for p in m.parameters()):,} parameters")
Output:
resnet34: 21,797,672 parameters
vgg19: 143,667,240 parameters
ResNet-34 has about 22 million parameters, VGG-19 about 144 million: depth is cheap, wide fully connected layers are not.
2. Run the gradient test with 100 blocks.
Solution
import torch, torch.nn as nn
torch.manual_seed(66); x, y = torch.randn(32, 8, 14, 14), torch.randint(0, 10, (32,))
class Block(nn.Module):
def __init__(self, skip):
super().__init__(); self.skip = skip
self.conv = nn.Conv2d(8, 8, 3, padding=1)
def forward(self, x):
out = torch.relu(self.conv(x))
return x + out if self.skip else out
for depth in (50, 100):
for skip in (False, True):
torch.manual_seed(0)
blocks = [Block(skip) for _ in range(depth)]
net = nn.Sequential(*blocks, nn.AdaptiveAvgPool2d(1), nn.Flatten(), nn.Linear(8, 10))
nn.functional.cross_entropy(net(x), y).backward()
g = net[0].conv.weight.grad.norm().item()
print(f"depth {depth:2} {'with skip' if skip else 'plain '}: gradient at layer 1 = {g:.2e}")
Output:
depth 50 plain : gradient at layer 1 = 0.00e+00
depth 50 with skip: gradient at layer 1 = 2.04e+04
depth 100 plain : gradient at layer 1 = 0.00e+00
depth 100 with skip: gradient at layer 1 = 5.29e+08
Plain: exactly zero. With skip and no normalisation the gradient explodes instead (5.29e+08), which is why ResNets use batch norm inside each block.
3. Train our ResNet with the skip connections removed.
Solution
import torch, torch.nn as nn, time
torch.manual_seed(66); d = torch.load("/data/mnist.pt")
X = d["train_x"][:12000].float().div(255).unsqueeze(1); y = d["train_y"][:12000]
Xt = d["test_x"][:2000].float().div(255).unsqueeze(1); yt = d["test_y"][:2000]
class Residual(nn.Module): # now a PLAIN block: relu(F(x))
def __init__(self, c):
super().__init__(); conv = lambda: nn.Conv2d(c, c, 3, padding=1, bias=False)
self.f = nn.Sequential(conv(), nn.BatchNorm2d(c), nn.ReLU(), conv(), nn.BatchNorm2d(c))
def forward(self, x): return torch.relu(self.f(x)) # skip connection REMOVED
torch.manual_seed(0)
net = nn.Sequential(nn.Conv2d(1, 16, 3, padding=1, bias=False), nn.BatchNorm2d(16), nn.ReLU(),
Residual(16), nn.MaxPool2d(2), nn.Conv2d(16, 32, 1), Residual(32), nn.MaxPool2d(2),
Residual(32), nn.AdaptiveAvgPool2d(1), nn.Flatten(), nn.Linear(32, 10))
opt = torch.optim.Adam(net.parameters(), lr=1e-3); t = time.time()
for epoch in range(3):
net.train()
for i in range(0, 12000, 64):
dx, dy = torch.randint(-3, 4, (2,)).tolist() # augmentation: move -3..3 pixels
xb = torch.roll(X[i:i + 64], shifts=(dx, dy), dims=(3, 2))
loss = nn.functional.cross_entropy(net(xb), y[i:i + 64])
opt.zero_grad(); loss.backward(); opt.step()
net.eval(); acc = lambda X: f"{(net(X).argmax(1) == yt).float().mean().item():.1%}"
with torch.no_grad():
n = sum(p.numel() for p in net.parameters())
print("parameters:", f"{n:,}", "| training:", round(time.time() - t), "s")
print("test accuracy:", acc(Xt))
print("moved 1, 2, 3 pixels right:", *[acc(torch.roll(Xt, k, dims=3)) for k in (1, 2, 3)])
Output:
parameters: 42,842 | training: 33 s
test accuracy: 96.2%
moved 1, 2, 3 pixels right: 96.1% 96.4% 96.5%
A surprise worth knowing: with only three blocks (about 8 conv layers) the plain version trains fine, and in this short run it even scores higher (96.2%, and 96.5% on moved digits) than the residual version (91.9%, 89.3%). Skip connections are not magic at small depth; they matter when networks get deep, where plain stacks stop learning (the gradient test above, and He et al.'s 56- vs 20-layer result). Results of 3-epoch runs also move by a few points with the seed, so compare several seeds before concluding. Try 10 extra Residual(32) blocks in both versions and compare again.
4. Train on all 60,000 digits for 5 epochs and test on all 10,000.
Solution
import torch, torch.nn as nn, time
torch.manual_seed(66); d = torch.load("/data/mnist.pt")
X = d["train_x"].float().div(255).unsqueeze(1); y = d["train_y"]
Xt = d["test_x"].float().div(255).unsqueeze(1); yt = d["test_y"]
class Residual(nn.Module): # out = relu(x + F(x))
def __init__(self, c):
super().__init__(); conv = lambda: nn.Conv2d(c, c, 3, padding=1, bias=False)
self.f = nn.Sequential(conv(), nn.BatchNorm2d(c), nn.ReLU(), conv(), nn.BatchNorm2d(c))
def forward(self, x): return torch.relu(x + self.f(x))
torch.manual_seed(0)
net = nn.Sequential(nn.Conv2d(1, 16, 3, padding=1, bias=False), nn.BatchNorm2d(16), nn.ReLU(),
Residual(16), nn.MaxPool2d(2), nn.Conv2d(16, 32, 1), Residual(32), nn.MaxPool2d(2),
Residual(32), nn.AdaptiveAvgPool2d(1), nn.Flatten(), nn.Linear(32, 10))
opt = torch.optim.Adam(net.parameters(), lr=1e-3); t = time.time()
for epoch in range(5):
net.train()
for i in range(0, 60000, 64):
dx, dy = torch.randint(-3, 4, (2,)).tolist() # augmentation: move -3..3 pixels
xb = torch.roll(X[i:i + 64], shifts=(dx, dy), dims=(3, 2))
loss = nn.functional.cross_entropy(net(xb), y[i:i + 64])
opt.zero_grad(); loss.backward(); opt.step()
net.eval() # test in chunks of 1,000 to save memory
acc = lambda X: f"{torch.cat([net(X[i:i + 1000]).argmax(1) for i in range(0, len(X), 1000)]).eq(yt).float().mean().item():.1%}"
with torch.no_grad():
n = sum(p.numel() for p in net.parameters())
print("parameters:", f"{n:,}", "| training:", round(time.time() - t), "s")
print("test accuracy:", acc(Xt))
print("moved 1, 2, 3 pixels right:", *[acc(torch.roll(Xt, k, dims=3)) for k in (1, 2, 3)])
Output:
parameters: 42,842 | training: 308 s
test accuracy: 98.6%
moved 1, 2, 3 pixels right: 98.5% 98.4% 98.3%
5. A 1 × 1 convolution from 512 to 128 channels: how many parameters?
Solution
128 × 512 × 1 × 1 + 128 = 65,664.
Quick quiz
A residual block computes relu(x + F(x)). Training pushes all of F's weights to 0. What does the block output?
Answer
relu(x): F(x) = 0, so the input passes through. Doing (almost) nothing is easy for a residual block, which is why very deep ResNets can be trained.
More questions:
- Why do two stacked 3 × 3 convolutions beat one 5 × 5?
Answer
Same 5 × 5 receptive field, fewer weights (73,856 vs 102,464 at 64 channels), and an extra non-linearity. - Where are most of VGG-16's parameters?
Answer
In its fully connected layers at the end. - What does
nn.AdaptiveAvgPool2d(1)return for input(8, 64, 20, 20)?Answer
(8, 64, 1, 1): one mean per map. - Is the degradation problem overfitting?
Answer
No: the deeper plain network is worse on the training data too. - What did AlexNet's ILSVRC-2012 entry score?
Answer
15.3% top-5 error, against 26.2% for the second-best entry.
Interview questions
- Why do residual connections help? They let a block learn only a correction
F(x)to its input, make the identity easy to represent, and give gradients a direct path back through the network, so very deep networks train (no degradation). - What is the degradation problem? Deeper plain networks having higher training error than shallower ones (He et al.: 56 vs 20 layers on CIFAR-10); it is an optimisation difficulty, not overfitting.
- What does a 1 × 1 convolution do? A per-pixel linear layer across channels: changes channel counts (bottlenecks), mixes channels, adds non-linearity with a following ReLU, at low cost.
- Why global average pooling instead of flatten + dense? Far fewer parameters, less overfitting, less dependence on where features are, and any input size works.
- Walk through the evolution LeNet → AlexNet → VGG → ResNet. Pattern established; scaled up with ReLU, GPUs, dropout and data; standardised on deep stacks of 3 × 3; made very deep training possible with skip connections and batch norm.
- How does a shortcut handle a change of shape? A projection: a 1 × 1 convolution (with the same stride) on the shortcut, usually followed by batch norm.
Glossary
| Term | Meaning (plain words + example) |
|---|---|
| Architecture | The design of a network: layers, order, sizes |
| LeNet-5 | LeCun et al.'s 1998 CNN for 32 × 32 handwritten characters |
| AlexNet | Krizhevsky, Sutskever and Hinton's 2012 CNN: 60 M parameters, ReLU, GPUs, dropout |
| ImageNet / ILSVRC | A huge labelled image collection / its yearly recognition contest |
| Top-5 error | Share of images whose right label is not in the model's 5 best guesses |
| VGG | Simonyan and Zisserman's 2014 CNNs of only 3 × 3 kernels, 16-19 weight layers |
| Fully connected layer | nn.Linear: every input connected to every output |
| Deeper networks | More layers with weights; better up to a point, then plain nets degrade |
| Vanishing gradient | Gradients shrinking towards 0 on their way back through many layers |
| Exploding gradient | Gradients growing huge through many layers |
| Degradation problem | Deeper plain networks with higher training error than shallower ones |
| Identity mapping | Output = input ("do nothing") |
| ResNet | He et al.'s 2015 residual networks, up to 152 layers |
| Residual | What a block adds to its input: F(x) = H(x) − x |
| Residual (skip, shortcut) connection | Adding a block's input to its output: F(x) + x |
| Residual block | relu(F(x) + x) with F = conv-BN-ReLU-conv-BN |
| Projection shortcut | A 1 × 1 conv on the shortcut when shapes change |
| 1 × 1 convolution | A per-pixel linear layer across channels: 256 → 64 channels |
| Bottleneck | Shrinking channels with a 1 × 1 conv before expensive layers |
| Network in Network | Lin, Chen and Yan (2013): micro networks and global average pooling |
| Global average pooling (GAP) | Averaging each feature map to one number: nn.AdaptiveAvgPool2d(1) |
torchvision.models |
Ready-made architectures: resnet18(weights=None) |
weights=None |
Build the architecture with random weights (no pretrained download) |
| Data augmentation (shift) | Moving training images by random pixels: torch.roll |
| MNIST / CIFAR-10 | 70,000 grey 28 × 28 digits / 60,000 colour 32 × 32 images in 10 classes |
| FLOPs | Floating-point operations, a measure of compute: ResNet-152 11.3 billion |
Summary
- LeNet (1998): conv → pool → conv → pool → fully connected; 61,706 parameters, most in the dense layers.
- AlexNet (2012): 60 M parameters, ReLU, GPU convolution, dropout; 15.3% vs 26.2% top-5 at ILSVRC-2012.
- VGG (2014): only 3 × 3 kernels; two 3 × 3 see 5 × 5 with fewer weights; VGG-16 has 138 M parameters.
- Plain deeper networks fail: the gradient at layer 1 of 50 plain blocks is exactly 0, and even with BN deeper plain nets degrade.
- ResNet (2015):
out = relu(F(x) + x); the identity is easy, gradients flow; 152 layers, 3.57% top-5, ILSVRC 2015 winner. - 1 × 1 convolutions change channel counts cheaply; global average pooling shrinks the head from 125,450 to 650 weights and frees it from positions.
- Our small ResNet + GAP + augmentation: 42,842 weights, 91.9%, and 89.3% on digits moved 3 pixels (Episode 65's CNN: 58.3%).
Next episode: Transfer learning: take a network already trained on ImageNet and teach it a new task in minutes.