SyntaxPilot
Courses/Machine Learning/Lesson 67
Video coming soonThis lesson’s video is being made

Start with the notes below; the video appears here as soon as it is on YouTube.

Machine Learning · Lesson 67 of 90

Transfer learning

pretrained modelImageNettorchvision modelsfeature extractionfine-tuningfreezing and unfreezing layersreplacing the classifier headdiscriminative learning ratesdomain shift

Machine Learning Full Course · Episode 67. Every code sample below was run with Python 3.13, PyTorch 2.15.0 and torchvision 0.29.2 on a CPU. The output shown is the real output; training times depend on the machine. The Beans photos are loaded from /data/beans.pt and the ImageNet weights from /data/resnet18_imagenet.pt (see "Getting the data and weights" below; on your own machine models.resnet18(weights="IMAGENET1K_V1") downloads the same weights).

The big idea: a network trained on a huge dataset (here ResNet-18 on ImageNet) has already learned general visual features (edges, colours, textures, parts). Instead of training from scratch on a small dataset, reuse that pretrained body and train only what is new. Feature extraction keeps the body frozen and trains a new head on its features; fine-tuning also trains the body, gently, with discriminative learning rates. On 1,034 bean-leaf photos: from scratch about 70% validation accuracy, feature extraction 88.0%, fine-tuning 95.5% (93.8% on the test set).

Sources: torchvision's ResNet18_Weights.IMAGENET1K_V1 metadata (printed in the code below: 69.758% top-1, 1,000 classes, 11,689,512 parameters). Jason Yosinski, Jeff Clune, Yoshua Bengio and Hod Lipson, "How transferable are features in deep neural networks?", 2014 (arXiv 1411.1792). Jeremy Howard and Sebastian Ruder, "Universal Language Model Fine-tuning for Text Classification", 2018 (arXiv 1801.06146). The Beans dataset, Makerere AI Lab (Hugging Face). The weights file is torchvision's ResNet-18 ImageNet weights as mirrored at timm/resnet18.tv_in1k.

Words you need first

  • Pretrained model: a network that has already been trained, usually on a much bigger task. Example: ResNet-18 trained on ImageNet's 1.2 million photos.
  • Weights file / state_dict: the saved numbers of a trained network. In PyTorch, net.state_dict() is a dictionary from layer names ("layer1.0.conv1.weight", "fc.bias", ...) to tensors; torch.save writes it and net.load_state_dict(...) puts it back.
  • Body (backbone): the layers that turn a photo into features. In ResNet-18, everything before fc.
  • Feature: a number that describes something about the input. ResNet-18's body gives 512 features per photo.
  • Head (classifier head): the last layer(s) that turn features into class scores. In ResNet-18 it is net.fc = nn.Linear(512, 1000).
  • Transfer learning: reusing what a model learned on one task (the source) for a new task (the target).
  • ImageNet: a huge labelled photo collection; its classification contest used 1,000 classes and about 1.2 million training photos (Episode 66).
  • Normalisation: standardising each colour channel with a mean and standard deviation, here ImageNet's: mean [0.485, 0.456, 0.406], std [0.229, 0.224, 0.225].
  • Validation set / test set (Episode 15): photos held out to choose settings (validation) and to give the final, honest score (test).

The story: a farmer's bean leaves

A farmer grows beans. Some leaves get small brown spots (angular leaf spot), some get rusty patches (bean rust), and catching it early saves the crop. There is no plant expert nearby, but there is a phone: take a photo of a leaf and get an answer. The Beans dataset from the Makerere AI Lab has exactly such photos: 1,034 training, 133 validation and 128 test images in 3 classes (angular_leaf_spot, bean_rust, healthy), 500 × 500 pixels (we resize them to 160 × 160).

Real-life example for transfer learning: an experienced cook joins a new restaurant. Ten years in a big hotel kitchen taught them cutting, frying and tasting (the pretrained body). The new restaurant has a new menu (a new head: 3 bean classes instead of 1,000 ImageNet classes). Nobody trains them for ten years again; a few days with the menu is enough (training a little on small data).

Getting the data and weights

tools/get_beans.py in the course repo downloads the Beans parquet files from Hugging Face, resizes the photos to 160 × 160 and saves /data/beans.pt (train_x, val_x, test_x as uint8 tensors of shape (N, 3, 160, 160), plus labels and class names). It also saves torchvision's ResNet-18 (and ResNet-50) ImageNet weights as state_dict files. On a normal machine you can simply write:

import torchvision.models as models
net = models.resnet18(weights=models.ResNet18_Weights.IMAGENET1K_V1)   # downloads once, then cached

The obvious attempt: train from scratch

ResNet-18 (Episode 66) with random weights and a 3-class head, Adam with learning rate 0.001, batches of 32, 5 epochs.

import torch, torch.nn as nn, torchvision.models as models, time
torch.manual_seed(67); d = torch.load("/data/beans.pt")
mean = torch.tensor([0.485, 0.456, 0.406]).view(1, 3, 1, 1)
std = torch.tensor([0.229, 0.224, 0.225]).view(1, 3, 1, 1)
prep = lambda x: (x.float() / 255 - mean) / std
X, y, Xv, yv = prep(d["train_x"]), d["train_y"], prep(d["val_x"]), d["val_y"]
def acc(net, X, y):
    net.eval()
    with torch.no_grad():
        p = torch.cat([net(X[i:i + 64]).argmax(1) for i in range(0, len(X), 64)])
    return f"{(p == y).float().mean().item():.1%}"
net = models.resnet18(weights=None)              # same design, RANDOM weights
net.fc = nn.Linear(512, 3)                       # 3 outputs: our 3 leaf classes
opt = torch.optim.Adam(net.parameters(), lr=1e-3); t = time.time()
for epoch in range(5):
    net.train(); perm = torch.randperm(len(X))
    for i in range(0, len(X), 32):
        b = perm[i:i + 32]
        loss = nn.functional.cross_entropy(net(X[b]), y[b])
        opt.zero_grad(); loss.backward(); opt.step()
    print(f"epoch {epoch + 1}: train {acc(net, X, y)}  validation {acc(net, Xv, yv)}"
          f"  ({time.time() - t:.0f} s)")

Output:

epoch 1: train 69.7%  validation 66.9%  (74 s)
epoch 2: train 64.9%  validation 55.6%  (139 s)
epoch 3: train 69.1%  validation 69.2%  (178 s)
epoch 4: train 78.3%  validation 75.2%  (219 s)
epoch 5: train 73.5%  validation 69.9%  (257 s)

Validation accuracy jumps around (66.9, 55.6, 69.2, 75.2, 69.9%) and ends near 70%. Why: 11.7 million weights must be learned from 1,034 photos, starting from nothing: the network must first discover edges, colours and textures before it can tell rust from spots. Too few examples make learning noisy and overfitting easy.

Pretrained models: what does ImageNet's ResNet-18 think a leaf is?

import torch, torchvision.models as models
w = models.ResNet18_Weights.IMAGENET1K_V1            # facts that ship with the weights
print("ImageNet top-1:", w.meta["_metrics"]["ImageNet-1K"]["acc@1"], "%",
      "| classes:", len(w.meta["categories"]), "| parameters:", f"{w.meta['num_params']:,}")
net = models.resnet18(weights=None)
net.load_state_dict(torch.load("/data/resnet18_imagenet.pt"))   # the trained weights
net.eval()
d = torch.load("/data/beans.pt")                     # bean-leaf photos, 160 x 160
mean = torch.tensor([0.485, 0.456, 0.406]).view(1, 3, 1, 1)
std = torch.tensor([0.229, 0.224, 0.225]).view(1, 3, 1, 1)
for i in (4, 50, 120):
    x = (d["val_x"][i:i + 1].float() / 255 - mean) / std
    with torch.no_grad():
        p = net(x).softmax(1)[0]
    best = p.argmax().item()
    print(f"{d['classes'][d['val_y'][i]]:17} -> {w.meta['categories'][best]} ({p[best]:.0%})")

Output:

ImageNet top-1: 69.758 % | classes: 1000 | parameters: 11,689,512
angular_leaf_spot -> hippopotamus (61%)
bean_rust         -> head cabbage (58%)
healthy           -> leaf beetle (19%)

The ImageNet head only knows ImageNet's 1,000 classes, and none of them is a bean disease, so it calls our leaves "hippopotamus", "head cabbage" and "leaf beetle". The head is useless for us, but the body may not be.

What the layers learned

Yosinski, Clune, Bengio and Lipson (2014) measured how transferable each layer's features are. Their findings: on the first layers, networks learn features "similar to Gabor filters and color blobs" (edge and colour detectors): general, useful for any photo. The last layers are specific to the original task. Transferring features and then fine-tuning even improved generalisation compared with training from random weights, and transferability drops as the source and target tasks become less similar.

Layers What they detect Transfer?
first (conv1, layer1) edges, colour blobs very general: keep
middle (layer2, layer3) textures, simple parts mostly general
last block (layer4) object parts tied to ImageNet partly specific: maybe retrain
head (fc, 1,000 outputs) ImageNet's classes task-specific: replace

Feature extraction

Feature extraction: use the frozen pretrained body only to compute features, then train a small new head on them. Replacing fc with nn.Identity() (a layer that returns its input unchanged) makes the network output the 512 features. Because the body never changes, we compute every photo's features once and save them.

import torch, torch.nn as nn, torchvision.models as models
d = torch.load("/data/beans.pt")
mean = torch.tensor([0.485, 0.456, 0.406]).view(1, 3, 1, 1)
std = torch.tensor([0.229, 0.224, 0.225]).view(1, 3, 1, 1)
net = models.resnet18(weights=None)
net.load_state_dict(torch.load("/data/resnet18_imagenet.pt"))
net.fc = nn.Identity()               # chop off the ImageNet head
net.eval(); feats = {}
with torch.no_grad():                # no learning here: just read the features
    for s in ("train", "val", "test"):
        X = (d[s + "_x"].float() / 255 - mean) / std
        F = torch.cat([net(X[i:i + 64]) for i in range(0, len(X), 64)])
        feats[s + "_f"], feats[s + "_y"] = F, d[s + "_y"]
        print(f"{s:5} photos {tuple(X.shape)} -> features {tuple(F.shape)}")
torch.save(feats, "/data/beans_feats.pt")
print("first photo, first 5 features:", F[0, :5].numpy().round(2))

Output:

train photos (1034, 3, 160, 160) -> features (1034, 512)
val   photos (133, 3, 160, 160) -> features (133, 512)
test  photos (128, 3, 160, 160) -> features (128, 512)
first photo, first 5 features: [0.47 0.08 0.16 1.62 0.16]

Now train only the new head, nn.Linear(512, 3), on the saved features (200 full-batch Adam steps):

import torch, torch.nn as nn
torch.manual_seed(67)
f = torch.load("/data/beans_feats.pt")       # 512 numbers per photo, saved before
head = nn.Linear(512, 3)                     # the ONLY part we train
opt = torch.optim.Adam(head.parameters(), lr=1e-2)
def acc(s):
    with torch.no_grad():
        return f"{(head(f[s + '_f']).argmax(1) == f[s + '_y']).float().mean().item():.1%}"
for step in range(1, 201):
    loss = nn.functional.cross_entropy(head(f["train_f"]), f["train_y"])
    opt.zero_grad(); loss.backward(); opt.step()
    if step in (1, 10, 50, 100, 200):
        print(f"step {step:3}: loss {loss.item():.3f}  train {acc('train')}"
              f"  validation {acc('val')}")
print("weights trained:", sum(p.numel() for p in head.parameters()), "| test", acc("test"))

Output:

step   1: loss 1.191  train 34.7%  validation 36.8%
step  10: loss 0.759  train 69.8%  validation 71.4%
step  50: loss 0.328  train 88.9%  validation 84.2%
step 100: loss 0.218  train 93.4%  validation 85.7%
step 200: loss 0.137  train 97.2%  validation 88.0%
weights trained: 1539 | test 82.8%

1,539 trained weights (512 × 3 + 3), a few seconds of training, 88.0% validation (82.8% test): far better than 11.7 million weights trained from scratch for about four minutes.

Freezing layers and replacing the head

When we want one model trained end to end, with photos going in, we stop the body from changing by freezing it. Every parameter has a flag requires_grad; with requires_grad = False no gradient is computed for it and the optimiser cannot change it. Replacing the classifier head means assigning a new layer to net.fc; a brand-new layer starts with requires_grad = True. Unfreezing sets the flag back to True.

import torch, torch.nn as nn, torchvision.models as models
net = models.resnet18(weights=None)
net.load_state_dict(torch.load("/data/resnet18_imagenet.pt"))
count = lambda: f"{sum(p.numel() for p in net.parameters() if p.requires_grad):,}"
print("trainable at the start:   ", count())
for p in net.parameters():
    p.requires_grad = False          # freeze: no gradient, no change
print("after freezing everything:", count())
net.fc = nn.Linear(512, 3)           # replace the head (new layers start unfrozen)
print("frozen body + new head:   ", count())
for p in net.layer4.parameters():
    p.requires_grad = True           # unfreeze the last block
print("+ layer4 unfrozen:        ", count())
print("output for one photo:", tuple(net(torch.randn(1, 3, 160, 160)).shape))

Output:

trainable at the start:    11,689,512
after freezing everything: 0
frozen body + new head:    1,539
+ layer4 unfrozen:         8,395,267
output for one photo: (1, 3)

Most of ResNet-18's weights (8.4 of 11.7 million) sit in its last block, layer4.

Fine-tuning with discriminative learning rates

Fine-tuning: keep training the pretrained weights themselves, gently, on the new data. The body already knows a lot, the new head knows nothing, so we give them different learning rates: small for the body, bigger for the head. These are discriminative learning rates. Howard and Ruder (2018) give each layer its own rate and set each lower layer's rate to the one above divided by 2.6 (η^(l-1) = η^l / 2.6). They also use gradual unfreezing: unfreeze the model one layer at a time, "starting from the last layer as this contains the least general knowledge".

import torch, torch.nn as nn, torchvision.models as models, time
torch.manual_seed(67); d = torch.load("/data/beans.pt")
mean = torch.tensor([0.485, 0.456, 0.406]).view(1, 3, 1, 1)
std = torch.tensor([0.229, 0.224, 0.225]).view(1, 3, 1, 1)
prep = lambda x: (x.float() / 255 - mean) / std
X, y, Xv, yv = prep(d["train_x"]), d["train_y"], prep(d["val_x"]), d["val_y"]
Xt, yt = prep(d["test_x"]), d["test_y"]
def acc(net, X, y):
    net.eval()
    with torch.no_grad():
        p = torch.cat([net(X[i:i + 64]).argmax(1) for i in range(0, len(X), 64)])
    return f"{(p == y).float().mean().item():.1%}"
net = models.resnet18(weights=None)
net.load_state_dict(torch.load("/data/resnet18_imagenet.pt"))   # start from ImageNet
net.fc = nn.Linear(512, 3)                                      # new head
body = [p for n, p in net.named_parameters() if not n.startswith("fc.")]
opt = torch.optim.Adam([{"params": body, "lr": 1e-4},           # discriminative
                        {"params": net.fc.parameters(), "lr": 1e-3}])  # learning rates
t = time.time()
for epoch in range(3):
    net.train(); perm = torch.randperm(len(X))
    for i in range(0, len(X), 32):
        b = perm[i:i + 32]; xb = X[b]
        if torch.rand(1) < 0.5: xb = xb.flip(3)                 # augmentation: mirror
        loss = nn.functional.cross_entropy(net(xb), y[b])
        opt.zero_grad(); loss.backward(); opt.step()
    print(f"epoch {epoch + 1}: train {acc(net, X, y)}  validation {acc(net, Xv, yv)}"
          f"  ({time.time() - t:.0f} s)")
print("test accuracy:", acc(net, Xt, yt))

Output:

epoch 1: train 97.1%  validation 93.2%  (39 s)
epoch 2: train 99.2%  validation 94.7%  (76 s)
epoch 3: train 99.4%  validation 95.5%  (117 s)
test accuracy: 93.8%

Each half of the batches is mirrored left to right (xb.flip(3)): data augmentation from Episode 63. Starting from ImageNet, three epochs give 95.5% validation and 93.8% test: same network, same photos as the from-scratch run; only the starting weights changed.

The mistake: one big learning rate for everything

The same program with both rates set to 1e-2:

opt = torch.optim.Adam([{"params": body, "lr": 1e-2},           # ONE big rate
                        {"params": net.fc.parameters(), "lr": 1e-2}])  # for everything

Output:

epoch 1: train 38.4%  validation 44.4%  (41 s)
epoch 2: train 40.8%  validation 39.1%  (82 s)
epoch 3: train 56.8%  validation 56.4%  (120 s)
test accuracy: 57.0%

The big steps push the pretrained weights far from what they knew: catastrophic forgetting (new training wiping out old knowledge). Gentle rates for the body protect it.

Domain shift

A domain is the kind of data a model sees (photos of what, taken how, where). Domain shift means new data looks different from the training data. ImageNet (dogs, cars, cups) → bean-leaf close-ups is already a shift. In the field there is also evening light, a blurry phone, a black-and-white camera. We simulate these on the 128 test photos:

import torch, torch.nn as nn, torchvision.models as models
torch.manual_seed(67); d, f = torch.load("/data/beans.pt"), torch.load("/data/beans_feats.pt")
head = nn.Linear(512, 3); opt = torch.optim.Adam(head.parameters(), lr=1e-2)
for step in range(200):                       # the same head as before
    loss = nn.functional.cross_entropy(head(f["train_f"]), f["train_y"])
    opt.zero_grad(); loss.backward(); opt.step()
body = models.resnet18(weights=None)
body.load_state_dict(torch.load("/data/resnet18_imagenet.pt"))
body.fc = nn.Identity(); body.eval()
mean = torch.tensor([0.485, 0.456, 0.406]).view(1, 3, 1, 1)
std = torch.tensor([0.229, 0.224, 0.225]).view(1, 3, 1, 1)
x = d["test_x"].float() / 255                 # the 128 test photos
shifted = {"same kind of photos": x,
           "evening light": x * 0.35,
           "blurry phone camera": nn.functional.avg_pool2d(x, 9, 1, 4),
           "black and white": x.mean(1, keepdim=True).expand(-1, 3, -1, -1)}
with torch.no_grad():
    for name, xs in shifted.items():
        pred = head(body((xs - mean) / std)).argmax(1)
        print(f"{name:20}: test accuracy {(pred == d['test_y']).float().mean().item():.1%}")

Output:

same kind of photos : test accuracy 82.8%
evening light       : test accuracy 75.0%
blurry phone camera : test accuracy 58.6%
black and white     : test accuracy 63.3%

The leaves did not change, only the camera. Fixes: collect training photos the way users will really take them, augment with those changes (see Practice 3), and keep checking the model on fresh real-world photos.

Three ways, one table

Approach Weights trained Time here (CPU) Validation Test
From scratch (random ResNet-18) 11,178,051 (all) about 4 minutes, 5 epochs 69.9% (jumpy) not measured
Feature extraction (frozen body + new head) 1,539 about 15 s for features + 2 s training 88.0% 82.8%
Fine-tuning, 2 learning rates 11,178,051 (all) about 2 minutes, 3 epochs 95.5% 93.8%
Fine-tuning, one big rate (0.01) 11,178,051 (all) about 2 minutes, 3 epochs 56.4% 57.0%

Which one when? Little data and photos similar to the source task: start with feature extraction (fast, hard to overfit). More data, or a very different domain: fine-tune, unfreeze more layers, use discriminative rates. Always start from pretrained weights when a suitable model exists.

Common mistakes

import torch, torch.nn as nn, torchvision.models as models
net = models.resnet18(weights=None)
net.fc = nn.Linear(512, 3)                                      # new head FIRST...
net.load_state_dict(torch.load("/data/resnet18_imagenet.pt"))   # ...then load ImageNet
RuntimeError: Error(s) in loading state_dict for ResNet:
    size mismatch for fc.weight: copying a param with shape torch.Size([1000, 512]) from checkpoint, the shape in current model is torch.Size([3, 512]).
    size mismatch for fc.bias: copying a param with shape torch.Size([1000]) from checkpoint, the shape in current model is torch.Size([3]).
import torch, torchvision.models as models
net = models.resnet18(weights=None); net.eval()
leaf = torch.rand(1, 1, 160, 160)       # a black-and-white photo: ONE channel
print(net(leaf).shape)
RuntimeError: Given groups=1, weight of size [64, 3, 7, 7], expected input[1, 1, 160, 160] to have 3 channels, but got 1 channels instead
Mistake Exact message / symptom Why Fix
Loading weights after replacing fc size mismatch for fc.weight: copying a param with shape torch.Size([1000, 512]) from checkpoint, the shape in current model is torch.Size([3, 512]). the file's head has 1,000 outputs load first, then net.fc = nn.Linear(512, 3) (or load_state_dict(..., strict=False) after removing the fc. keys)
A one-channel (grey) photo RuntimeError: Given groups=1, weight of size [64, 3, 7, 7], expected input[1, 1, 160, 160] to have 3 channels, but got 1 channels instead ImageNet models take RGB x.expand(-1, 3, -1, -1)
One big learning rate for everything validation 44.4% → 39.1% → 56.4%, test 57.0% catastrophic forgetting small rate for the body, bigger for the head
Skipping ImageNet normalisation lower accuracy the body expects the colour statistics it trained with same mean and std as in training
Forgetting net.eval() predictions change with the batch batch norm uses batch statistics in training mode net.eval() and torch.no_grad() for testing
Freezing but passing frozen params with a big rate nothing learns, or wasted time frozen parameters get no gradient give the optimiser only the parameters that train

Mini project: a leaf-doctor function

Wrap the feature-extraction model in predict(photo) that returns a label and a confidence:

import torch, torch.nn as nn, torchvision.models as models
torch.manual_seed(67); d, f = torch.load("/data/beans.pt"), torch.load("/data/beans_feats.pt")
head = nn.Linear(512, 3); opt = torch.optim.Adam(head.parameters(), lr=1e-2)
for step in range(200):
    loss = nn.functional.cross_entropy(head(f["train_f"]), f["train_y"])
    opt.zero_grad(); loss.backward(); opt.step()
body = models.resnet18(weights=None); body.load_state_dict(torch.load("/data/resnet18_imagenet.pt"))
body.fc = nn.Identity(); body.eval()
mean = torch.tensor([0.485, 0.456, 0.406]).view(1, 3, 1, 1)
std = torch.tensor([0.229, 0.224, 0.225]).view(1, 3, 1, 1)
def predict(photo):                      # photo: uint8 tensor (3, 160, 160)
    x = (photo.float().unsqueeze(0) / 255 - mean) / std
    with torch.no_grad():
        p = head(body(x)).softmax(1)[0]
    return d["classes"][p.argmax()], round(p.max().item(), 3)
for i in (0, 60, 127):
    print("true:", d["classes"][d["test_y"][i]], "| predicted:", predict(d["test_x"][i]))

Output:

true: angular_leaf_spot | predicted: ('bean_rust', 0.548)
true: bean_rust | predicted: ('bean_rust', 0.972)
true: healthy | predicted: ('healthy', 0.878)

Practice

1. Fine-tune with layer4 + head only (everything else frozen).

Solution
import torch, torch.nn as nn, torchvision.models as models, time
torch.manual_seed(67); d = torch.load("/data/beans.pt")
mean = torch.tensor([0.485, 0.456, 0.406]).view(1, 3, 1, 1)
std = torch.tensor([0.229, 0.224, 0.225]).view(1, 3, 1, 1)
prep = lambda x: (x.float() / 255 - mean) / std
X, y, Xv, yv = prep(d["train_x"]), d["train_y"], prep(d["val_x"]), d["val_y"]
def acc(net, X, y):
    net.eval()
    with torch.no_grad():
        p = torch.cat([net(X[i:i + 64]).argmax(1) for i in range(0, len(X), 64)])
    return f"{(p == y).float().mean().item():.1%}"
net = models.resnet18(weights=None)
net.load_state_dict(torch.load("/data/resnet18_imagenet.pt"))
for p in net.parameters(): p.requires_grad = False       # freeze everything...
for p in net.layer4.parameters(): p.requires_grad = True # ...except layer4
net.fc = nn.Linear(512, 3)                               # and the new head
opt = torch.optim.Adam([{"params": net.layer4.parameters(), "lr": 1e-4},
                        {"params": net.fc.parameters(), "lr": 1e-3}])
t = time.time()
for epoch in range(3):
    net.train(); perm = torch.randperm(len(X))
    for i in range(0, len(X), 32):
        b = perm[i:i + 32]
        loss = nn.functional.cross_entropy(net(X[b]), y[b])
        opt.zero_grad(); loss.backward(); opt.step()
    print(f"epoch {epoch + 1}: validation {acc(net, Xv, yv)}  ({time.time() - t:.0f} s)")

Output:

epoch 1: validation 92.5%  (85 s)
epoch 2: validation 94.7%  (167 s)
epoch 3: validation 94.7%  (223 s)

Training only the last block and the head (8.4 of 11.2 million weights) reaches 94.7%, close to full fine-tuning (95.5%) and far above feature extraction (88.0%).

2. Use three learning rates: layer3, layer4 and the head (Howard and Ruder's ÷ 2.6 rule).

Solution
import torch, torch.nn as nn, torchvision.models as models, time
torch.manual_seed(67); d = torch.load("/data/beans.pt")
mean = torch.tensor([0.485, 0.456, 0.406]).view(1, 3, 1, 1)
std = torch.tensor([0.229, 0.224, 0.225]).view(1, 3, 1, 1)
prep = lambda x: (x.float() / 255 - mean) / std
X, y, Xv, yv = prep(d["train_x"]), d["train_y"], prep(d["val_x"]), d["val_y"]
def acc(net, X, y):
    net.eval()
    with torch.no_grad():
        p = torch.cat([net(X[i:i + 64]).argmax(1) for i in range(0, len(X), 64)])
    return f"{(p == y).float().mean().item():.1%}"
net = models.resnet18(weights=None)
net.load_state_dict(torch.load("/data/resnet18_imagenet.pt"))
for p in net.parameters(): p.requires_grad = False
net.fc = nn.Linear(512, 3)
for block in (net.layer3, net.layer4):
    for p in block.parameters(): p.requires_grad = True
lr = 1e-3                                                  # Howard and Ruder: divide by 2.6 per layer down
opt = torch.optim.Adam([{"params": net.fc.parameters(), "lr": lr},
                        {"params": net.layer4.parameters(), "lr": lr / 2.6},
                        {"params": net.layer3.parameters(), "lr": lr / 2.6 ** 2}])
t = time.time()
for epoch in range(3):
    net.train(); perm = torch.randperm(len(X))
    for i in range(0, len(X), 32):
        b = perm[i:i + 32]
        loss = nn.functional.cross_entropy(net(X[b]), y[b])
        opt.zero_grad(); loss.backward(); opt.step()
    print(f"epoch {epoch + 1}: validation {acc(net, Xv, yv)}  ({time.time() - t:.0f} s)")

Output:

epoch 1: validation 90.2%  (101 s)
epoch 2: validation 94.7%  (144 s)
epoch 3: validation 94.7%  (162 s)

3. Add blur augmentation (train the head on sharp AND blurred photos) and retest the blurry test photos.

Solution
import torch, torch.nn as nn, torchvision.models as models
torch.manual_seed(67); d = torch.load("/data/beans.pt")
mean = torch.tensor([0.485, 0.456, 0.406]).view(1, 3, 1, 1)
std = torch.tensor([0.229, 0.224, 0.225]).view(1, 3, 1, 1)
body = models.resnet18(weights=None); body.load_state_dict(torch.load("/data/resnet18_imagenet.pt"))
body.fc = nn.Identity(); body.eval()
blur = lambda x: nn.functional.avg_pool2d(x, 9, 1, 4)
def feats(x):
    with torch.no_grad():
        return torch.cat([body((x[i:i + 64] - mean) / std) for i in range(0, len(x), 64)])
x, xt = d["train_x"].float() / 255, d["test_x"].float() / 255
F_train = torch.cat([feats(x), feats(blur(x))])          # every photo twice: sharp and blurred
y_train = torch.cat([d["train_y"], d["train_y"]])
head = nn.Linear(512, 3); opt = torch.optim.Adam(head.parameters(), lr=1e-2)
for step in range(200):
    loss = nn.functional.cross_entropy(head(F_train), y_train)
    opt.zero_grad(); loss.backward(); opt.step()
with torch.no_grad():
    for name, xs in (("sharp test photos", xt), ("blurry test photos", blur(xt))):
        print(f"{name}: {(head(feats(xs)).argmax(1) == d['test_y']).float().mean().item():.1%}")

Output:

sharp test photos: 82.0%
blurry test photos: 79.7%

Before, the blurry test photos scored 58.6%. Training on blurred copies makes the model robust to that particular shift.

4. Use ResNet-50 features (2,048 per photo) instead of ResNet-18's 512.

Solution
import torch, torch.nn as nn, torchvision.models as models
torch.manual_seed(67); d = torch.load("/data/beans.pt")
mean = torch.tensor([0.485, 0.456, 0.406]).view(1, 3, 1, 1)
std = torch.tensor([0.229, 0.224, 0.225]).view(1, 3, 1, 1)
body = models.resnet50(weights=None)
body.load_state_dict(torch.load("/data/resnet50_imagenet.pt"))   # ImageNet weights for ResNet-50
body.fc = nn.Identity(); body.eval()                              # 2,048 features per photo
def feats(s):
    x = (d[s + "_x"].float() / 255 - mean) / std
    with torch.no_grad():
        return torch.cat([body(x[i:i + 32]) for i in range(0, len(x), 32)])
F, Fv, Ft = feats("train"), feats("val"), feats("test")
head = nn.Linear(F.shape[1], 3); opt = torch.optim.Adam(head.parameters(), lr=1e-2)
for step in range(200):
    loss = nn.functional.cross_entropy(head(F), d["train_y"])
    opt.zero_grad(); loss.backward(); opt.step()
acc = lambda F, y: f"{(head(F).argmax(1) == y).float().mean().item():.1%}"
with torch.no_grad():
    print("features per photo:", F.shape[1], "| validation", acc(Fv, d["val_y"]), "| test", acc(Ft, d["test_y"]))

Output:

features per photo: 2048 | validation 90.2% | test 88.3%

Quick quiz

We freeze every parameter of ResNet-18, then net.fc = nn.Linear(512, 10). How many parameters are trainable? (a) 0 (b) 5,130 (c) 11,689,512 (d) 512

Answer

(b) 5,130: 512 × 10 weights + 10 biases. The frozen body adds nothing, and the new layer starts unfrozen.

  1. What does nn.Identity() do as net.fc?
    AnswerReturns its input unchanged, so the network outputs the 512 body features.
  2. Why can feature extraction save features to a file?
    AnswerThe body is frozen, so the same photo always gives the same features: compute them once.
  3. What does requires_grad = False do?
    AnswerNo gradient is computed for that parameter, so the optimiser never changes it (frozen).
  4. Why a smaller learning rate for the body than for the new head?
    AnswerThe body already holds useful knowledge; big steps would destroy it (catastrophic forgetting), while the random new head needs bigger steps.
  5. What did ImageNet's ResNet-18 head call an angular-leaf-spot photo?
    Answer"hippopotamus" (61%): its 1,000 classes contain no bean diseases.

Interview questions

  1. What is transfer learning and why does it work? Reusing a model trained on a source task for a target task. Early layers learn general features (edges, colours, textures) that are useful for almost any images, so only task-specific parts need to be learned from the small target dataset.
  2. Feature extraction vs fine-tuning? Feature extraction freezes the pretrained body and trains only a new head (fast, little data, low overfitting risk). Fine-tuning also updates the body with small learning rates (higher accuracy with enough data or when the domain differs).
  3. How do you freeze layers in PyTorch? Set p.requires_grad = False for their parameters (and pass only trainable parameters to the optimiser); keep BN layers in eval mode if you want their statistics frozen too.
  4. What are discriminative learning rates and gradual unfreezing? Different learning rates per layer group (smaller for lower, more general layers; Howard and Ruder divide by 2.6 per layer), and unfreezing from the top layer down one stage at a time.
  5. What is catastrophic forgetting? When training on a new task overwrites what a network learned before, for example fine-tuning a pretrained model with a large learning rate.
  6. What is domain shift and how do you handle it? Test or production data distributed differently from training data (lighting, camera, region). Handle it with representative data collection, augmentation, fine-tuning on target-domain data and monitoring on fresh data.
  7. Why must you use the same normalisation as the pretrained model? Its weights were learned on inputs with those statistics; different statistics shift every feature.

Glossary

Term Meaning (plain words + example)
Transfer learning Reusing what a model learned on one task for another: ImageNet ResNet-18 → bean leaves
Pretrained model A network already trained on another (usually bigger) task, e.g. ResNet-18 on ImageNet
ImageNet A large labelled photo collection; its contest had 1,000 classes, about 1.2 million training photos
torchvision models Ready-made architectures (models.resnet18, models.resnet50, ...) with optional pretrained weights
Weights file / state_dict The saved numbers of a network: a dictionary from layer names to tensors
weights=None Build the architecture with random weights (no training, no download)
ResNet18_Weights.IMAGENET1K_V1 torchvision's ImageNet weights for ResNet-18 (69.758% top-1)
Body (backbone) The layers that turn an input into features (everything before fc in ResNet)
Head (classifier head) The last layer(s) turning features into class scores; ResNet's fc
Feature A number describing something about the input; ResNet-18's body gives 512
Feature extraction Using a frozen pretrained body only to compute features and training a new head on them
nn.Identity() A layer that returns its input unchanged; replaces a head to expose the features
Freezing Stopping parameters from changing: requires_grad = False
requires_grad A tensor's flag: should PyTorch compute a gradient for it?
Unfreezing Letting frozen parameters train again: requires_grad = True
Replacing the classifier head Assigning a new output layer, e.g. net.fc = nn.Linear(512, 3)
Fine-tuning Continuing to train pretrained weights, gently, on the new task
Discriminative learning rates Different learning rates for different layers: small for the body, bigger for the head
Gradual unfreezing Unfreezing layers from the top down, one at a time, during fine-tuning
Catastrophic forgetting New training wiping out old knowledge, e.g. one big learning rate: 93.8% → 57.0% test
Domain The kind of data a model sees: what, how and where the photos were taken
Domain shift New data looking different from training data: blurry photos dropped 82.8% → 58.6%
Normalisation (ImageNet) Subtract mean [0.485, 0.456, 0.406], divide by std [0.229, 0.224, 0.225] per channel
Data augmentation Training on randomly changed copies (mirrored, blurred) of the photos

Summary

  • From scratch on 1,034 photos, ResNet-18 reached only about 70% validation, jumping around.
  • A pretrained ResNet-18 (ImageNet, 69.758% top-1) knows general visual features, but its head knows only ImageNet's classes ("hippopotamus" for a sick leaf).
  • Feature extraction: chop off the head (nn.Identity()), save 512 features per photo, train nn.Linear(512, 3): 1,539 weights, 88.0% validation in seconds.
  • Freezing with requires_grad = False, replacing the head, unfreezing layer4 (8.4 million weights).
  • Fine-tuning with discriminative learning rates (1e-4 body, 1e-3 head): 95.5% validation, 93.8% test. One big rate (1e-2) caused catastrophic forgetting (57.0% test).
  • Domain shift: the same leaves photographed blurry or in black and white dropped to 58.6% and 63.3%.

Next episode: Computer vision tasks: classification vs object detection vs segmentation, bounding boxes, IoU, NMS, anchor boxes, YOLO, mAP and U-Net.