SyntaxPilot
Courses/Machine Learning/Lesson 66
Video coming soonThis lesson’s video is being made

Start with the notes below; the video appears here as soon as it is on YouTube.

Machine Learning · Lesson 66 of 90

CNNs 2: architectures

LeNetAlexNetVGGdeeper networksResNetresidual (skip) connections1x1 convolutionsglobal average poolingtraining a CNN on MNIST or CIFAR-10

Machine Learning Full Course · Episode 66. Every code sample below was run with Python 3.13, PyTorch 2.15.0 and torchvision 0.29 (CPU, one thread, as in the video's compiler). The output shown is the real output; training times depend on the machine. MNIST is loaded from /data/mnist.pt (Episode 65 notes show the one-line torchvision download).

The big idea: CNN architectures grew by solving each other's problems. LeNet (1998) set the pattern conv → pool → conv → pool → classifier. AlexNet (2012) made it big (60 million parameters) with ReLU, GPUs and dropout and won ImageNet by a huge margin. VGG (2014) used only small 3 × 3 kernels, stacked deep. Then deeper plain networks got worse: gradients vanish on the way back, and even with batch norm deeper nets degrade. ResNet (2015) fixed it with the residual (skip) connection out = relu(F(x) + x), training 152 layers. 1 × 1 convolutions change channel counts cheaply, and global average pooling replaces the huge, position-tied classifier head.

Sources: Yann LeCun, Léon Bottou, Yoshua Bengio and Patrick Haffner, "Gradient-Based Learning Applied to Document Recognition", Proceedings of the IEEE 86(11):2278−2324, 1998 (bottou.org). Alex Krizhevsky, Ilya Sutskever and Geoffrey Hinton, "ImageNet Classification with Deep Convolutional Neural Networks", NeurIPS 2012 (abstract). Karen Simonyan and Andrew Zisserman, "Very Deep Convolutional Networks for Large-Scale Image Recognition", 2014 (arXiv 1409.1556). Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun, "Deep Residual Learning for Image Recognition", 2015 (arXiv 1512.03385, paper text). Min Lin, Qiang Chen and Shuicheng Yan, "Network In Network", 2013 (arXiv 1312.4400). CIFAR-10 (Alex Krizhevsky's page).

Words you need first

  • Convolution, kernel, feature map, pooling, receptive field (Episode 65).
  • Architecture: the design of a network: which layers, in what order, with what sizes.
  • ImageNet / ILSVRC: a huge labelled photo collection and its yearly contest (ImageNet Large Scale Visual Recognition Challenge); the classification task had 1,000 classes and about 1.2 million training images.
  • Top-5 error: a prediction counts as wrong if the right label is not among the model's 5 highest-scored classes.
  • Depth: the number of layers with weights.
  • Fully connected layer: another name for nn.Linear: every input connects to every output.
  • Backpropagation and the chain rule (Episode 57): gradients flow backwards, multiplied layer by layer.
  • Batch normalisation (Episode 63): standardising each channel over the batch inside the network.
  • Data augmentation (Episode 63): training on randomly changed copies of the data.

The story: four buildings

Real-life example: four buildings, each solving the last one's problem. LeNet is a small, careful house; AlexNet a much bigger building made possible by new materials (ReLU, dropout, graphics cards); VGG a tall tower of identical small bricks; ResNet a skyscraper that only works because it has lifts (skip connections), so nothing has to climb every stair.

Year Network Key idea Size
1998 LeNet-5 conv → pool → conv → pool → fully connected; 32 × 32 handwritten characters 61,706 parameters (modern version below)
2012 AlexNet 5 conv + 3 fully connected, ReLU ("non-saturating neurons"), GPU convolution, dropout 60 million parameters, 650,000 neurons
2014 VGG only 3 × 3 kernels, 16-19 weight layers VGG-16: 138 million parameters
2015 ResNet residual (skip) connections, BN after each conv, up to 152 layers ResNet-152: 60 million parameters

LeNet-5 (1998)

Described by LeCun, Bottou, Bengio and Haffner in 1998 for reading handwritten characters (32 × 32 input). Layout: C1 conv (6 kernels 5 × 5) → S2 pooling → C3 conv (16 kernels 5 × 5) → S4 pooling → C5 (120) → F6 (84) → 10 outputs. It used tanh-style activations and average-style pooling.

import torch, torch.nn as nn
lenet = nn.Sequential(                                  # LeNet-5 (LeCun et al., 1998), modern PyTorch
    nn.Conv2d(1, 6, 5), nn.Tanh(), nn.AvgPool2d(2),       # C1: 6 maps 28x28   S2: 14x14
    nn.Conv2d(6, 16, 5), nn.Tanh(), nn.AvgPool2d(2),      # C3: 16 maps 10x10  S4: 5x5
    nn.Flatten(), nn.Linear(16 * 5 * 5, 120), nn.Tanh(),  # C5: 120
    nn.Linear(120, 84), nn.Tanh(), nn.Linear(84, 10))     # F6: 84, output: 10 digits
x = torch.randn(1, 1, 32, 32)                             # LeNet-5 reads 32 x 32 images
for layer in lenet:
    x = layer(x)
    if not isinstance(layer, nn.Tanh):
        n = sum(p.numel() for p in layer.parameters())
        print(f"{type(layer).__name__:9} -> {str(tuple(x.shape)):16} parameters {n:6,}")
print("total parameters:", f"{sum(p.numel() for p in lenet.parameters()):,}")

Output:

Conv2d    -> (1, 6, 28, 28)   parameters    156
AvgPool2d -> (1, 6, 14, 14)   parameters      0
Conv2d    -> (1, 16, 10, 10)  parameters  2,416
AvgPool2d -> (1, 16, 5, 5)    parameters      0
Flatten   -> (1, 400)         parameters      0
Linear    -> (1, 120)         parameters 48,120
Linear    -> (1, 84)          parameters 10,164
Linear    -> (1, 10)          parameters    850
total parameters: 61,706

The two convolutions have about 2,500 parameters; the fully connected layers about 59,000. This simplified modern version connects every C3 kernel to all 6 maps and uses a plain linear output; the original connected C3 kernels to subsets of maps, had trainable subsampling layers and a different output layer, so its parameter count differs.

AlexNet (2012): the ImageNet moment

From the paper's abstract (Krizhevsky, Sutskever, Hinton, NeurIPS 2012):

  • trained on 1.2 million images of the ImageNet LSVRC-2010 contest, 1,000 classes;
  • 60 million parameters and 650,000 neurons; five convolutional layers, some followed by max pooling, and three fully connected layers ending in a 1000-way softmax;
  • "to make training faster, we used non-saturating neurons and a very efficient GPU implementation of the convolution operation" (non-saturating neurons = ReLU, Episode 56);
  • dropout in the fully connected layers to reduce overfitting (Episode 63);
  • a variant won ILSVRC-2012 with a top-5 test error of 15.3%, against 26.2% for the second-best entry.

VGG (2014): small kernels, deep stacks

Simonyan and Zisserman used only 3 × 3 kernels and increased depth to 16-19 weight layers; in the ImageNet Challenge 2014 their team took first place in localisation and second in classification.

Why 3 × 3? Two 3 × 3 convolutions in a row have a 5 × 5 receptive field, three have 7 × 7 (Episode 65), with fewer weights and an extra ReLU between them (more non-linearity).

import torch.nn as nn, torchvision.models as models
count = lambda m: sum(p.numel() for p in m.parameters())
for name in ("alexnet", "vgg16", "resnet18", "resnet50", "resnet152"):
    m = getattr(models, name)(weights=None)               # the architecture only, random weights
    convs = sum(isinstance(l, nn.Conv2d) for l in m.modules())
    print(f"{name:9} conv layers {convs:3}   parameters {count(m):12,}")
C = 64                                                    # VGG's trick: stack small 3 x 3 kernels
one5, two3 = count(nn.Conv2d(C, C, 5)), 2 * count(nn.Conv2d(C, C, 3))
one7, three3 = count(nn.Conv2d(C, C, 7)), 3 * count(nn.Conv2d(C, C, 3))
print(f"see 5x5: one 5x5 conv {one5:,} | two 3x3 convs {two3:,}")
print(f"see 7x7: one 7x7 conv {one7:,} | three 3x3 convs {three3:,}")

Output:

alexnet   conv layers   5   parameters   61,100,840
vgg16     conv layers  13   parameters  138,357,544
resnet18  conv layers  20   parameters   11,689,512
resnet50  conv layers  53   parameters   25,557,032
resnet152 conv layers 155   parameters   60,192,808
see 5x5: one 5x5 conv 102,464 | two 3x3 convs 73,856
see 7x7: one 7x7 conv 200,768 | three 3x3 convs 110,784

(weights=None builds the architecture with random weights; nothing is downloaded. The conv-layer counts for ResNets include the 1 × 1 convolutions on their shortcut paths.) AlexNet and VGG keep most of their parameters in the fully connected layers; ResNet-152 is much deeper than VGG-16 but has fewer than half its parameters, and He et al. report 11.3 billion FLOPs for ResNet-152 against 15.3 billion for VGG-16.

Going deeper: vanishing gradients

Eight layers (AlexNet) to nineteen (VGG) helped, so why not 50 or 100? In backpropagation the gradient travels back through every layer and is multiplied by each layer's local derivative (chain rule). If many of those factors are below 1, the gradient shrinks towards 0 before it reaches the first layers: the vanishing gradient problem. Those layers then barely learn.

import torch, torch.nn as nn
torch.manual_seed(66); x, y = torch.randn(32, 8, 14, 14), torch.randint(0, 10, (32,))
class Block(nn.Module):
    def __init__(self, skip):
        super().__init__(); self.skip = skip
        self.conv = nn.Conv2d(8, 8, 3, padding=1)
    def forward(self, x):
        out = torch.relu(self.conv(x))
        return x + out if self.skip else out
for depth in (2, 10, 30, 50):
    for skip in (False, True):
        torch.manual_seed(0)
        blocks = [Block(skip) for _ in range(depth)]
        net = nn.Sequential(*blocks, nn.AdaptiveAvgPool2d(1), nn.Flatten(), nn.Linear(8, 10))
        nn.functional.cross_entropy(net(x), y).backward()
        g = net[0].conv.weight.grad.norm().item()
        print(f"depth {depth:2} {'with skip' if skip else 'plain    '}: gradient at layer 1 = {g:.2e}")

Output:

depth  2 plain    : gradient at layer 1 = 1.91e-02
depth  2 with skip: gradient at layer 1 = 7.10e-02
depth 10 plain    : gradient at layer 1 = 3.78e-06
depth 10 with skip: gradient at layer 1 = 6.96e-01
depth 30 plain    : gradient at layer 1 = 1.09e-15
depth 30 with skip: gradient at layer 1 = 1.54e+02
depth 50 plain    : gradient at layer 1 = 0.00e+00
depth 50 with skip: gradient at layer 1 = 2.04e+04

A plain stack of 50 blocks delivers exactly zero gradient to its first layer. With the skip (x + out) the gradient never vanishes; here it even grows with depth, which is why real ResNets put batch normalisation inside every block.

The degradation problem. He et al. (2015) found that even with batch normalisation, deeper plain networks got worse: on CIFAR-10 a 56-layer plain network had higher training error (and so test error) than a 20-layer one (their Figure 1). That is not overfitting: it is worse on the training data itself. It's strange, because the deeper network could copy the shallower one and make its extra layers compute the identity (output = input); but learning "do nothing" turns out to be hard for a stack of convolutions and ReLUs.

ResNet: residual (skip) connections

Real-life example: fifty students pass a message down a line by whispering, and it's garbled by the end (a plain deep network). Now pass the written note along too: the original always arrives (the skip connection). Each student only writes a small correction on it, or passes it on unchanged (the residual).

output = relu( F(x) + x )        F = conv → BN → ReLU → conv → BN
  • The block's desired output is H(x); the layers learn only the residual F(x) = H(x) − x, what to add to x.
  • "Do nothing" is now easy: F's weights near 0 give F(x) ≈ 0, output ≈ x (after the ReLU).
  • The + x path carries the gradient straight back through every block (the derivative of x with respect to itself is 1).
  • An identity shortcut adds no parameters and no computation. When F changes the number of channels or the size, the shortcut uses a 1 × 1 convolution (with stride) to match shapes.
  • He et al. used batch normalisation right after each convolution, trained networks up to 152 layers ("8× deeper than VGG nets"), reached 3.57% top-5 error on the ImageNet test set with an ensemble, and won 1st place in ILSVRC 2015 classification.

A residual block in PyTorch:

class Residual(nn.Module):                     # out = relu(x + F(x)), same shape in and out
    def __init__(self, c):
        super().__init__(); conv = lambda: nn.Conv2d(c, c, 3, padding=1, bias=False)
        self.f = nn.Sequential(conv(), nn.BatchNorm2d(c), nn.ReLU(), conv(), nn.BatchNorm2d(c))
    def forward(self, x): return torch.relu(x + self.f(x))

(bias=False because the batch norm right after has its own shift.)

1 × 1 convolutions

A 1 × 1 kernel looks at one pixel across all its channels. So it is the same small linear layer applied at every pixel: it mixes channels and changes their number cheaply, e.g. 256 → 64 (a bottleneck, used in ResNet-50 and deeper). The idea comes from Network in Network (Lin, Chen and Yan, 2013), whose "micro networks" slide over the input like convolutions.

Global average pooling (GAP)

The flatten + linear head is where LeNet, AlexNet and VGG keep most of their weights, and it is tied to positions again. Global average pooling averages each whole feature map into one number: 64 maps → 64 numbers → a tiny linear layer. Where the pattern was no longer matters, and it works for any input size. Network in Network proposed it as easier to interpret and less prone to overfitting than fully connected layers. In PyTorch: nn.AdaptiveAvgPool2d(1) (then nn.Flatten()).

import torch, torch.nn as nn
torch.manual_seed(66)
x = torch.randn(1, 256, 14, 14)                       # 256 feature maps of 14 x 14
squeeze = nn.Conv2d(256, 64, kernel_size=1)           # a 1 x 1 convolution
n = sum(p.numel() for p in squeeze.parameters())
print("1x1 conv:", tuple(x.shape), "->", tuple(squeeze(x).shape))
print("1x1 conv parameters:", n)
pixel = x[0, :, 3, 5]                                 # the 256 numbers at ONE pixel
same = squeeze.weight.view(64, 256) @ pixel + squeeze.bias
print("same as a Linear layer on each pixel?", torch.allclose(same, squeeze(x)[0, :, 3, 5], atol=1e-5))
gap = nn.AdaptiveAvgPool2d(1)                         # global average pooling
f = squeeze(x)
print("global average pooling:", tuple(f.shape), "->", tuple(gap(f).shape))
print("first value = mean of map 0?", torch.allclose(gap(f)[0, 0, 0, 0], f[0, 0].mean()))
old = nn.Linear(64 * 14 * 14, 10); new = nn.Linear(64, 10)
count = lambda m: f"{sum(p.numel() for p in m.parameters()):,}"
print("classifier head: Flatten + Linear", count(old), "| GAP + Linear", count(new))
for side in (14, 20, 33):                             # any map size gives one number per map
    print(f"GAP on {side} x {side} maps ->", tuple(gap(torch.randn(1, 64, side, side)).shape))

Output:

1x1 conv: (1, 256, 14, 14) -> (1, 64, 14, 14)
1x1 conv parameters: 16448
same as a Linear layer on each pixel? True
global average pooling: (1, 64, 14, 14) -> (1, 64, 1, 1)
first value = mean of map 0? True
classifier head: Flatten + Linear 125,450 | GAP + Linear 650
GAP on 14 x 14 maps -> (1, 64, 1, 1)
GAP on 20 x 20 maps -> (1, 64, 1, 1)
GAP on 33 x 33 maps -> (1, 64, 1, 1)

Training a small ResNet on MNIST

Plan: conv 16 + BN + ReLU → residual (16) → max pool (28 → 14) → 1 × 1 conv (16 → 32) → residual (32) → max pool (14 → 7) → residual (32) → GAP (32 numbers) → linear (10). Data augmentation: each training batch is moved by a random −3..3 pixels across and down (torch.roll). It trains for about 35 seconds on one CPU thread, longer than the video compiler's time limit, so the video shows its real output on a slide.

import torch, torch.nn as nn, time
torch.manual_seed(66); d = torch.load("/data/mnist.pt")
X = d["train_x"][:12000].float().div(255).unsqueeze(1); y = d["train_y"][:12000]
Xt = d["test_x"][:2000].float().div(255).unsqueeze(1); yt = d["test_y"][:2000]
class Residual(nn.Module):                                    # out = relu(x + F(x))
    def __init__(self, c):
        super().__init__(); conv = lambda: nn.Conv2d(c, c, 3, padding=1, bias=False)
        self.f = nn.Sequential(conv(), nn.BatchNorm2d(c), nn.ReLU(), conv(), nn.BatchNorm2d(c))
    def forward(self, x): return torch.relu(x + self.f(x))
torch.manual_seed(0)
net = nn.Sequential(nn.Conv2d(1, 16, 3, padding=1, bias=False), nn.BatchNorm2d(16), nn.ReLU(),
                    Residual(16), nn.MaxPool2d(2), nn.Conv2d(16, 32, 1), Residual(32), nn.MaxPool2d(2),
                    Residual(32), nn.AdaptiveAvgPool2d(1), nn.Flatten(), nn.Linear(32, 10))
opt = torch.optim.Adam(net.parameters(), lr=1e-3); t = time.time()
for epoch in range(3):
    net.train()
    for i in range(0, 12000, 64):
        dx, dy = torch.randint(-3, 4, (2,)).tolist()          # augmentation: move -3..3 pixels
        xb = torch.roll(X[i:i + 64], shifts=(dx, dy), dims=(3, 2))
        loss = nn.functional.cross_entropy(net(xb), y[i:i + 64])
        opt.zero_grad(); loss.backward(); opt.step()
net.eval(); acc = lambda X: f"{(net(X).argmax(1) == yt).float().mean().item():.1%}"
with torch.no_grad():
    n = sum(p.numel() for p in net.parameters())
    print("parameters:", f"{n:,}", "| training:", round(time.time() - t), "s")
    print("test accuracy:", acc(Xt))
    print("moved 1, 2, 3 pixels right:", *[acc(torch.roll(Xt, k, dims=3)) for k in (1, 2, 3)])

Output:

parameters: 42,842 | training: 35 s
test accuracy: 91.9%
moved 1, 2, 3 pixels right: 91.3% 90.6% 89.3%

The same network without augmentation:

import torch, torch.nn as nn, time
torch.manual_seed(66); d = torch.load("/data/mnist.pt")
X = d["train_x"][:12000].float().div(255).unsqueeze(1); y = d["train_y"][:12000]
Xt = d["test_x"][:2000].float().div(255).unsqueeze(1); yt = d["test_y"][:2000]
class Residual(nn.Module):                                    # out = relu(x + F(x))
    def __init__(self, c):
        super().__init__(); conv = lambda: nn.Conv2d(c, c, 3, padding=1, bias=False)
        self.f = nn.Sequential(conv(), nn.BatchNorm2d(c), nn.ReLU(), conv(), nn.BatchNorm2d(c))
    def forward(self, x): return torch.relu(x + self.f(x))
torch.manual_seed(0)
net = nn.Sequential(nn.Conv2d(1, 16, 3, padding=1, bias=False), nn.BatchNorm2d(16), nn.ReLU(),
                    Residual(16), nn.MaxPool2d(2), nn.Conv2d(16, 32, 1), Residual(32), nn.MaxPool2d(2),
                    Residual(32), nn.AdaptiveAvgPool2d(1), nn.Flatten(), nn.Linear(32, 10))
opt = torch.optim.Adam(net.parameters(), lr=1e-3); t = time.time()
for epoch in range(3):
    net.train()
    for i in range(0, 12000, 64):
        xb = X[i:i + 64]                                      # no augmentation
        loss = nn.functional.cross_entropy(net(xb), y[i:i + 64])
        opt.zero_grad(); loss.backward(); opt.step()
net.eval(); acc = lambda X: f"{(net(X).argmax(1) == yt).float().mean().item():.1%}"
with torch.no_grad():
    n = sum(p.numel() for p in net.parameters())
    print("parameters:", f"{n:,}", "| training:", round(time.time() - t), "s")
    print("test accuracy:", acc(Xt))
    print("moved 1, 2, 3 pixels right:", *[acc(torch.roll(Xt, k, dims=3)) for k in (1, 2, 3)])

Output:

parameters: 42,842 | training: 71 s
test accuracy: 88.3%
moved 1, 2, 3 pixels right: 87.9% 86.7% 84.9%
Model (12,000 training digits, 3 epochs, CPU) Weights Test Moved 3 px
MLP (Episode 65) 101,770 89.5% 44.0%
First CNN (Episode 65) 9,098 91.0% 58.3%
ResNet + GAP, no augmentation 42,842 88.3% 84.9%
ResNet + GAP + augmentation 42,842 91.9% 89.3%

Global average pooling makes the network far less tied to positions (84.9% at 3 pixels even without augmentation); augmentation adds the rest. (At this small depth the skip connections are not what makes the difference: see practice 3.) On all 60,000 digits for 5 epochs (practice 4) it reaches 98.6%, and 98.3% on digits moved 3 pixels.

CIFAR-10 is the usual next step: 60,000 colour images of 32 × 32 in 10 classes (airplane, automobile, bird, cat, deer, dog, frog, horse, ship, truck), 50,000 for training and 10,000 for testing. The same code works with nn.Conv2d(3, 16, ...) for 3 input channels, loaded with torchvision.datasets.CIFAR10("data", download=True, transform=transforms.ToTensor()); real colour photos are much harder than digits, so expect lower accuracy and use more epochs, more channels and augmentation such as random crops and horizontal flips.

Common mistakes

Mistake Exact message / symptom Why Fix
Stacking more plain layers gradient at layer 1: 0.00e+00 at 50 blocks vanishing gradients, degradation residual connections + batch norm
Adding x when shapes differ RuntimeError: The size of tensor a (16) must match the size of tensor b (32) at non-singleton dimension 1 F(x) changed the channels 1 × 1 conv (with stride) on the shortcut
Huge flatten + linear head 125,450 weights for 10 classes, tied to positions one weight per position global average pooling
Expecting weights=None models to know anything random predictions architecture only train it, or load pretrained weights (Episode 67)
Augmenting the test set a noisy, unfair score only training data is changed fixed test data
Evaluating a BN network in train mode predictions depend on the batch BN uses batch statistics model.eval()
Testing 10,000 images in one forward pass slow, or out of memory activations for all at once evaluate in chunks of e.g. 1,000

Practice

1. Count the parameters of resnet34 and vgg19.

Solution
import torchvision.models as models
for name in ("resnet34", "vgg19"):
    m = getattr(models, name)(weights=None)
    print(f"{name}: {sum(p.numel() for p in m.parameters()):,} parameters")

Output:

resnet34: 21,797,672 parameters
vgg19: 143,667,240 parameters

ResNet-34 has about 22 million parameters, VGG-19 about 144 million: depth is cheap, wide fully connected layers are not.

2. Run the gradient test with 100 blocks.

Solution
import torch, torch.nn as nn
torch.manual_seed(66); x, y = torch.randn(32, 8, 14, 14), torch.randint(0, 10, (32,))
class Block(nn.Module):
    def __init__(self, skip):
        super().__init__(); self.skip = skip
        self.conv = nn.Conv2d(8, 8, 3, padding=1)
    def forward(self, x):
        out = torch.relu(self.conv(x))
        return x + out if self.skip else out
for depth in (50, 100):
    for skip in (False, True):
        torch.manual_seed(0)
        blocks = [Block(skip) for _ in range(depth)]
        net = nn.Sequential(*blocks, nn.AdaptiveAvgPool2d(1), nn.Flatten(), nn.Linear(8, 10))
        nn.functional.cross_entropy(net(x), y).backward()
        g = net[0].conv.weight.grad.norm().item()
        print(f"depth {depth:2} {'with skip' if skip else 'plain    '}: gradient at layer 1 = {g:.2e}")

Output:

depth 50 plain    : gradient at layer 1 = 0.00e+00
depth 50 with skip: gradient at layer 1 = 2.04e+04
depth 100 plain    : gradient at layer 1 = 0.00e+00
depth 100 with skip: gradient at layer 1 = 5.29e+08

Plain: exactly zero. With skip and no normalisation the gradient explodes instead (5.29e+08), which is why ResNets use batch norm inside each block.

3. Train our ResNet with the skip connections removed.

Solution
import torch, torch.nn as nn, time
torch.manual_seed(66); d = torch.load("/data/mnist.pt")
X = d["train_x"][:12000].float().div(255).unsqueeze(1); y = d["train_y"][:12000]
Xt = d["test_x"][:2000].float().div(255).unsqueeze(1); yt = d["test_y"][:2000]
class Residual(nn.Module):                                    # now a PLAIN block: relu(F(x))
    def __init__(self, c):
        super().__init__(); conv = lambda: nn.Conv2d(c, c, 3, padding=1, bias=False)
        self.f = nn.Sequential(conv(), nn.BatchNorm2d(c), nn.ReLU(), conv(), nn.BatchNorm2d(c))
    def forward(self, x): return torch.relu(self.f(x))          # skip connection REMOVED
torch.manual_seed(0)
net = nn.Sequential(nn.Conv2d(1, 16, 3, padding=1, bias=False), nn.BatchNorm2d(16), nn.ReLU(),
                    Residual(16), nn.MaxPool2d(2), nn.Conv2d(16, 32, 1), Residual(32), nn.MaxPool2d(2),
                    Residual(32), nn.AdaptiveAvgPool2d(1), nn.Flatten(), nn.Linear(32, 10))
opt = torch.optim.Adam(net.parameters(), lr=1e-3); t = time.time()
for epoch in range(3):
    net.train()
    for i in range(0, 12000, 64):
        dx, dy = torch.randint(-3, 4, (2,)).tolist()          # augmentation: move -3..3 pixels
        xb = torch.roll(X[i:i + 64], shifts=(dx, dy), dims=(3, 2))
        loss = nn.functional.cross_entropy(net(xb), y[i:i + 64])
        opt.zero_grad(); loss.backward(); opt.step()
net.eval(); acc = lambda X: f"{(net(X).argmax(1) == yt).float().mean().item():.1%}"
with torch.no_grad():
    n = sum(p.numel() for p in net.parameters())
    print("parameters:", f"{n:,}", "| training:", round(time.time() - t), "s")
    print("test accuracy:", acc(Xt))
    print("moved 1, 2, 3 pixels right:", *[acc(torch.roll(Xt, k, dims=3)) for k in (1, 2, 3)])

Output:

parameters: 42,842 | training: 33 s
test accuracy: 96.2%
moved 1, 2, 3 pixels right: 96.1% 96.4% 96.5%

A surprise worth knowing: with only three blocks (about 8 conv layers) the plain version trains fine, and in this short run it even scores higher (96.2%, and 96.5% on moved digits) than the residual version (91.9%, 89.3%). Skip connections are not magic at small depth; they matter when networks get deep, where plain stacks stop learning (the gradient test above, and He et al.'s 56- vs 20-layer result). Results of 3-epoch runs also move by a few points with the seed, so compare several seeds before concluding. Try 10 extra Residual(32) blocks in both versions and compare again.

4. Train on all 60,000 digits for 5 epochs and test on all 10,000.

Solution
import torch, torch.nn as nn, time
torch.manual_seed(66); d = torch.load("/data/mnist.pt")
X = d["train_x"].float().div(255).unsqueeze(1); y = d["train_y"]
Xt = d["test_x"].float().div(255).unsqueeze(1); yt = d["test_y"]
class Residual(nn.Module):                                    # out = relu(x + F(x))
    def __init__(self, c):
        super().__init__(); conv = lambda: nn.Conv2d(c, c, 3, padding=1, bias=False)
        self.f = nn.Sequential(conv(), nn.BatchNorm2d(c), nn.ReLU(), conv(), nn.BatchNorm2d(c))
    def forward(self, x): return torch.relu(x + self.f(x))
torch.manual_seed(0)
net = nn.Sequential(nn.Conv2d(1, 16, 3, padding=1, bias=False), nn.BatchNorm2d(16), nn.ReLU(),
                    Residual(16), nn.MaxPool2d(2), nn.Conv2d(16, 32, 1), Residual(32), nn.MaxPool2d(2),
                    Residual(32), nn.AdaptiveAvgPool2d(1), nn.Flatten(), nn.Linear(32, 10))
opt = torch.optim.Adam(net.parameters(), lr=1e-3); t = time.time()
for epoch in range(5):
    net.train()
    for i in range(0, 60000, 64):
        dx, dy = torch.randint(-3, 4, (2,)).tolist()          # augmentation: move -3..3 pixels
        xb = torch.roll(X[i:i + 64], shifts=(dx, dy), dims=(3, 2))
        loss = nn.functional.cross_entropy(net(xb), y[i:i + 64])
        opt.zero_grad(); loss.backward(); opt.step()
net.eval()                                                    # test in chunks of 1,000 to save memory
acc = lambda X: f"{torch.cat([net(X[i:i + 1000]).argmax(1) for i in range(0, len(X), 1000)]).eq(yt).float().mean().item():.1%}"
with torch.no_grad():
    n = sum(p.numel() for p in net.parameters())
    print("parameters:", f"{n:,}", "| training:", round(time.time() - t), "s")
    print("test accuracy:", acc(Xt))
    print("moved 1, 2, 3 pixels right:", *[acc(torch.roll(Xt, k, dims=3)) for k in (1, 2, 3)])

Output:

parameters: 42,842 | training: 308 s
test accuracy: 98.6%
moved 1, 2, 3 pixels right: 98.5% 98.4% 98.3%

5. A 1 × 1 convolution from 512 to 128 channels: how many parameters?

Solution

128 × 512 × 1 × 1 + 128 = 65,664.

Quick quiz

A residual block computes relu(x + F(x)). Training pushes all of F's weights to 0. What does the block output?

Answer

relu(x): F(x) = 0, so the input passes through. Doing (almost) nothing is easy for a residual block, which is why very deep ResNets can be trained.

More questions:

  1. Why do two stacked 3 × 3 convolutions beat one 5 × 5?
    AnswerSame 5 × 5 receptive field, fewer weights (73,856 vs 102,464 at 64 channels), and an extra non-linearity.
  2. Where are most of VGG-16's parameters?
    AnswerIn its fully connected layers at the end.
  3. What does nn.AdaptiveAvgPool2d(1) return for input (8, 64, 20, 20)?
    Answer(8, 64, 1, 1): one mean per map.
  4. Is the degradation problem overfitting?
    AnswerNo: the deeper plain network is worse on the training data too.
  5. What did AlexNet's ILSVRC-2012 entry score?
    Answer15.3% top-5 error, against 26.2% for the second-best entry.

Interview questions

  1. Why do residual connections help? They let a block learn only a correction F(x) to its input, make the identity easy to represent, and give gradients a direct path back through the network, so very deep networks train (no degradation).
  2. What is the degradation problem? Deeper plain networks having higher training error than shallower ones (He et al.: 56 vs 20 layers on CIFAR-10); it is an optimisation difficulty, not overfitting.
  3. What does a 1 × 1 convolution do? A per-pixel linear layer across channels: changes channel counts (bottlenecks), mixes channels, adds non-linearity with a following ReLU, at low cost.
  4. Why global average pooling instead of flatten + dense? Far fewer parameters, less overfitting, less dependence on where features are, and any input size works.
  5. Walk through the evolution LeNet → AlexNet → VGG → ResNet. Pattern established; scaled up with ReLU, GPUs, dropout and data; standardised on deep stacks of 3 × 3; made very deep training possible with skip connections and batch norm.
  6. How does a shortcut handle a change of shape? A projection: a 1 × 1 convolution (with the same stride) on the shortcut, usually followed by batch norm.

Glossary

Term Meaning (plain words + example)
Architecture The design of a network: layers, order, sizes
LeNet-5 LeCun et al.'s 1998 CNN for 32 × 32 handwritten characters
AlexNet Krizhevsky, Sutskever and Hinton's 2012 CNN: 60 M parameters, ReLU, GPUs, dropout
ImageNet / ILSVRC A huge labelled image collection / its yearly recognition contest
Top-5 error Share of images whose right label is not in the model's 5 best guesses
VGG Simonyan and Zisserman's 2014 CNNs of only 3 × 3 kernels, 16-19 weight layers
Fully connected layer nn.Linear: every input connected to every output
Deeper networks More layers with weights; better up to a point, then plain nets degrade
Vanishing gradient Gradients shrinking towards 0 on their way back through many layers
Exploding gradient Gradients growing huge through many layers
Degradation problem Deeper plain networks with higher training error than shallower ones
Identity mapping Output = input ("do nothing")
ResNet He et al.'s 2015 residual networks, up to 152 layers
Residual What a block adds to its input: F(x) = H(x) − x
Residual (skip, shortcut) connection Adding a block's input to its output: F(x) + x
Residual block relu(F(x) + x) with F = conv-BN-ReLU-conv-BN
Projection shortcut A 1 × 1 conv on the shortcut when shapes change
1 × 1 convolution A per-pixel linear layer across channels: 256 → 64 channels
Bottleneck Shrinking channels with a 1 × 1 conv before expensive layers
Network in Network Lin, Chen and Yan (2013): micro networks and global average pooling
Global average pooling (GAP) Averaging each feature map to one number: nn.AdaptiveAvgPool2d(1)
torchvision.models Ready-made architectures: resnet18(weights=None)
weights=None Build the architecture with random weights (no pretrained download)
Data augmentation (shift) Moving training images by random pixels: torch.roll
MNIST / CIFAR-10 70,000 grey 28 × 28 digits / 60,000 colour 32 × 32 images in 10 classes
FLOPs Floating-point operations, a measure of compute: ResNet-152 11.3 billion

Summary

  • LeNet (1998): conv → pool → conv → pool → fully connected; 61,706 parameters, most in the dense layers.
  • AlexNet (2012): 60 M parameters, ReLU, GPU convolution, dropout; 15.3% vs 26.2% top-5 at ILSVRC-2012.
  • VGG (2014): only 3 × 3 kernels; two 3 × 3 see 5 × 5 with fewer weights; VGG-16 has 138 M parameters.
  • Plain deeper networks fail: the gradient at layer 1 of 50 plain blocks is exactly 0, and even with BN deeper plain nets degrade.
  • ResNet (2015): out = relu(F(x) + x); the identity is easy, gradients flow; 152 layers, 3.57% top-5, ILSVRC 2015 winner.
  • 1 × 1 convolutions change channel counts cheaply; global average pooling shrinks the head from 125,450 to 650 weights and frees it from positions.
  • Our small ResNet + GAP + augmentation: 42,842 weights, 91.9%, and 89.3% on digits moved 3 pixels (Episode 65's CNN: 58.3%).

Next episode: Transfer learning: take a network already trained on ImageNet and teach it a new task in minutes.