Machine Learning Full Course · Episode 67. Every code sample below was run with Python 3.13, PyTorch 2.15.0 and torchvision 0.29.2 on a CPU. The output shown is the real output; training times depend on the machine. The Beans photos are loaded from /data/beans.pt and the ImageNet weights from /data/resnet18_imagenet.pt (see "Getting the data and weights" below; on your own machine models.resnet18(weights="IMAGENET1K_V1") downloads the same weights).
The big idea: a network trained on a huge dataset (here ResNet-18 on ImageNet) has already learned general visual features (edges, colours, textures, parts). Instead of training from scratch on a small dataset, reuse that pretrained body and train only what is new. Feature extraction keeps the body frozen and trains a new head on its features; fine-tuning also trains the body, gently, with discriminative learning rates. On 1,034 bean-leaf photos: from scratch about 70% validation accuracy, feature extraction 88.0%, fine-tuning 95.5% (93.8% on the test set).
Sources: torchvision's ResNet18_Weights.IMAGENET1K_V1 metadata (printed in the code below: 69.758% top-1, 1,000 classes, 11,689,512 parameters). Jason Yosinski, Jeff Clune, Yoshua Bengio and Hod Lipson, "How transferable are features in deep neural networks?", 2014 (arXiv 1411.1792). Jeremy Howard and Sebastian Ruder, "Universal Language Model Fine-tuning for Text Classification", 2018 (arXiv 1801.06146). The Beans dataset, Makerere AI Lab (Hugging Face). The weights file is torchvision's ResNet-18 ImageNet weights as mirrored at timm/resnet18.tv_in1k.
Words you need first
- Pretrained model: a network that has already been trained, usually on a much bigger task. Example: ResNet-18 trained on ImageNet's 1.2 million photos.
- Weights file /
state_dict: the saved numbers of a trained network. In PyTorch,net.state_dict()is a dictionary from layer names ("layer1.0.conv1.weight","fc.bias", ...) to tensors;torch.savewrites it andnet.load_state_dict(...)puts it back. - Body (backbone): the layers that turn a photo into features. In ResNet-18, everything before
fc. - Feature: a number that describes something about the input. ResNet-18's body gives 512 features per photo.
- Head (classifier head): the last layer(s) that turn features into class scores. In ResNet-18 it is
net.fc = nn.Linear(512, 1000). - Transfer learning: reusing what a model learned on one task (the source) for a new task (the target).
- ImageNet: a huge labelled photo collection; its classification contest used 1,000 classes and about 1.2 million training photos (Episode 66).
- Normalisation: standardising each colour channel with a mean and standard deviation, here ImageNet's: mean
[0.485, 0.456, 0.406], std[0.229, 0.224, 0.225]. - Validation set / test set (Episode 15): photos held out to choose settings (validation) and to give the final, honest score (test).
The story: a farmer's bean leaves
A farmer grows beans. Some leaves get small brown spots (angular leaf spot), some get rusty patches (bean rust), and catching it early saves the crop. There is no plant expert nearby, but there is a phone: take a photo of a leaf and get an answer. The Beans dataset from the Makerere AI Lab has exactly such photos: 1,034 training, 133 validation and 128 test images in 3 classes (angular_leaf_spot, bean_rust, healthy), 500 × 500 pixels (we resize them to 160 × 160).
Real-life example for transfer learning: an experienced cook joins a new restaurant. Ten years in a big hotel kitchen taught them cutting, frying and tasting (the pretrained body). The new restaurant has a new menu (a new head: 3 bean classes instead of 1,000 ImageNet classes). Nobody trains them for ten years again; a few days with the menu is enough (training a little on small data).
Getting the data and weights
tools/get_beans.py in the course repo downloads the Beans parquet files from Hugging Face, resizes the photos to 160 × 160 and saves /data/beans.pt (train_x, val_x, test_x as uint8 tensors of shape (N, 3, 160, 160), plus labels and class names). It also saves torchvision's ResNet-18 (and ResNet-50) ImageNet weights as state_dict files. On a normal machine you can simply write:
import torchvision.models as models
net = models.resnet18(weights=models.ResNet18_Weights.IMAGENET1K_V1) # downloads once, then cached
The obvious attempt: train from scratch
ResNet-18 (Episode 66) with random weights and a 3-class head, Adam with learning rate 0.001, batches of 32, 5 epochs.
import torch, torch.nn as nn, torchvision.models as models, time
torch.manual_seed(67); d = torch.load("/data/beans.pt")
mean = torch.tensor([0.485, 0.456, 0.406]).view(1, 3, 1, 1)
std = torch.tensor([0.229, 0.224, 0.225]).view(1, 3, 1, 1)
prep = lambda x: (x.float() / 255 - mean) / std
X, y, Xv, yv = prep(d["train_x"]), d["train_y"], prep(d["val_x"]), d["val_y"]
def acc(net, X, y):
net.eval()
with torch.no_grad():
p = torch.cat([net(X[i:i + 64]).argmax(1) for i in range(0, len(X), 64)])
return f"{(p == y).float().mean().item():.1%}"
net = models.resnet18(weights=None) # same design, RANDOM weights
net.fc = nn.Linear(512, 3) # 3 outputs: our 3 leaf classes
opt = torch.optim.Adam(net.parameters(), lr=1e-3); t = time.time()
for epoch in range(5):
net.train(); perm = torch.randperm(len(X))
for i in range(0, len(X), 32):
b = perm[i:i + 32]
loss = nn.functional.cross_entropy(net(X[b]), y[b])
opt.zero_grad(); loss.backward(); opt.step()
print(f"epoch {epoch + 1}: train {acc(net, X, y)} validation {acc(net, Xv, yv)}"
f" ({time.time() - t:.0f} s)")
Output:
epoch 1: train 69.7% validation 66.9% (74 s)
epoch 2: train 64.9% validation 55.6% (139 s)
epoch 3: train 69.1% validation 69.2% (178 s)
epoch 4: train 78.3% validation 75.2% (219 s)
epoch 5: train 73.5% validation 69.9% (257 s)
Validation accuracy jumps around (66.9, 55.6, 69.2, 75.2, 69.9%) and ends near 70%. Why: 11.7 million weights must be learned from 1,034 photos, starting from nothing: the network must first discover edges, colours and textures before it can tell rust from spots. Too few examples make learning noisy and overfitting easy.
Pretrained models: what does ImageNet's ResNet-18 think a leaf is?
import torch, torchvision.models as models
w = models.ResNet18_Weights.IMAGENET1K_V1 # facts that ship with the weights
print("ImageNet top-1:", w.meta["_metrics"]["ImageNet-1K"]["acc@1"], "%",
"| classes:", len(w.meta["categories"]), "| parameters:", f"{w.meta['num_params']:,}")
net = models.resnet18(weights=None)
net.load_state_dict(torch.load("/data/resnet18_imagenet.pt")) # the trained weights
net.eval()
d = torch.load("/data/beans.pt") # bean-leaf photos, 160 x 160
mean = torch.tensor([0.485, 0.456, 0.406]).view(1, 3, 1, 1)
std = torch.tensor([0.229, 0.224, 0.225]).view(1, 3, 1, 1)
for i in (4, 50, 120):
x = (d["val_x"][i:i + 1].float() / 255 - mean) / std
with torch.no_grad():
p = net(x).softmax(1)[0]
best = p.argmax().item()
print(f"{d['classes'][d['val_y'][i]]:17} -> {w.meta['categories'][best]} ({p[best]:.0%})")
Output:
ImageNet top-1: 69.758 % | classes: 1000 | parameters: 11,689,512
angular_leaf_spot -> hippopotamus (61%)
bean_rust -> head cabbage (58%)
healthy -> leaf beetle (19%)
The ImageNet head only knows ImageNet's 1,000 classes, and none of them is a bean disease, so it calls our leaves "hippopotamus", "head cabbage" and "leaf beetle". The head is useless for us, but the body may not be.
What the layers learned
Yosinski, Clune, Bengio and Lipson (2014) measured how transferable each layer's features are. Their findings: on the first layers, networks learn features "similar to Gabor filters and color blobs" (edge and colour detectors): general, useful for any photo. The last layers are specific to the original task. Transferring features and then fine-tuning even improved generalisation compared with training from random weights, and transferability drops as the source and target tasks become less similar.
| Layers | What they detect | Transfer? |
|---|---|---|
| first (conv1, layer1) | edges, colour blobs | very general: keep |
| middle (layer2, layer3) | textures, simple parts | mostly general |
| last block (layer4) | object parts tied to ImageNet | partly specific: maybe retrain |
head (fc, 1,000 outputs) |
ImageNet's classes | task-specific: replace |
Feature extraction
Feature extraction: use the frozen pretrained body only to compute features, then train a small new head on them. Replacing fc with nn.Identity() (a layer that returns its input unchanged) makes the network output the 512 features. Because the body never changes, we compute every photo's features once and save them.
import torch, torch.nn as nn, torchvision.models as models
d = torch.load("/data/beans.pt")
mean = torch.tensor([0.485, 0.456, 0.406]).view(1, 3, 1, 1)
std = torch.tensor([0.229, 0.224, 0.225]).view(1, 3, 1, 1)
net = models.resnet18(weights=None)
net.load_state_dict(torch.load("/data/resnet18_imagenet.pt"))
net.fc = nn.Identity() # chop off the ImageNet head
net.eval(); feats = {}
with torch.no_grad(): # no learning here: just read the features
for s in ("train", "val", "test"):
X = (d[s + "_x"].float() / 255 - mean) / std
F = torch.cat([net(X[i:i + 64]) for i in range(0, len(X), 64)])
feats[s + "_f"], feats[s + "_y"] = F, d[s + "_y"]
print(f"{s:5} photos {tuple(X.shape)} -> features {tuple(F.shape)}")
torch.save(feats, "/data/beans_feats.pt")
print("first photo, first 5 features:", F[0, :5].numpy().round(2))
Output:
train photos (1034, 3, 160, 160) -> features (1034, 512)
val photos (133, 3, 160, 160) -> features (133, 512)
test photos (128, 3, 160, 160) -> features (128, 512)
first photo, first 5 features: [0.47 0.08 0.16 1.62 0.16]
Now train only the new head, nn.Linear(512, 3), on the saved features (200 full-batch Adam steps):
import torch, torch.nn as nn
torch.manual_seed(67)
f = torch.load("/data/beans_feats.pt") # 512 numbers per photo, saved before
head = nn.Linear(512, 3) # the ONLY part we train
opt = torch.optim.Adam(head.parameters(), lr=1e-2)
def acc(s):
with torch.no_grad():
return f"{(head(f[s + '_f']).argmax(1) == f[s + '_y']).float().mean().item():.1%}"
for step in range(1, 201):
loss = nn.functional.cross_entropy(head(f["train_f"]), f["train_y"])
opt.zero_grad(); loss.backward(); opt.step()
if step in (1, 10, 50, 100, 200):
print(f"step {step:3}: loss {loss.item():.3f} train {acc('train')}"
f" validation {acc('val')}")
print("weights trained:", sum(p.numel() for p in head.parameters()), "| test", acc("test"))
Output:
step 1: loss 1.191 train 34.7% validation 36.8%
step 10: loss 0.759 train 69.8% validation 71.4%
step 50: loss 0.328 train 88.9% validation 84.2%
step 100: loss 0.218 train 93.4% validation 85.7%
step 200: loss 0.137 train 97.2% validation 88.0%
weights trained: 1539 | test 82.8%
1,539 trained weights (512 × 3 + 3), a few seconds of training, 88.0% validation (82.8% test): far better than 11.7 million weights trained from scratch for about four minutes.
Freezing layers and replacing the head
When we want one model trained end to end, with photos going in, we stop the body from changing by freezing it. Every parameter has a flag requires_grad; with requires_grad = False no gradient is computed for it and the optimiser cannot change it. Replacing the classifier head means assigning a new layer to net.fc; a brand-new layer starts with requires_grad = True. Unfreezing sets the flag back to True.
import torch, torch.nn as nn, torchvision.models as models
net = models.resnet18(weights=None)
net.load_state_dict(torch.load("/data/resnet18_imagenet.pt"))
count = lambda: f"{sum(p.numel() for p in net.parameters() if p.requires_grad):,}"
print("trainable at the start: ", count())
for p in net.parameters():
p.requires_grad = False # freeze: no gradient, no change
print("after freezing everything:", count())
net.fc = nn.Linear(512, 3) # replace the head (new layers start unfrozen)
print("frozen body + new head: ", count())
for p in net.layer4.parameters():
p.requires_grad = True # unfreeze the last block
print("+ layer4 unfrozen: ", count())
print("output for one photo:", tuple(net(torch.randn(1, 3, 160, 160)).shape))
Output:
trainable at the start: 11,689,512
after freezing everything: 0
frozen body + new head: 1,539
+ layer4 unfrozen: 8,395,267
output for one photo: (1, 3)
Most of ResNet-18's weights (8.4 of 11.7 million) sit in its last block, layer4.
Fine-tuning with discriminative learning rates
Fine-tuning: keep training the pretrained weights themselves, gently, on the new data. The body already knows a lot, the new head knows nothing, so we give them different learning rates: small for the body, bigger for the head. These are discriminative learning rates. Howard and Ruder (2018) give each layer its own rate and set each lower layer's rate to the one above divided by 2.6 (η^(l-1) = η^l / 2.6). They also use gradual unfreezing: unfreeze the model one layer at a time, "starting from the last layer as this contains the least general knowledge".
import torch, torch.nn as nn, torchvision.models as models, time
torch.manual_seed(67); d = torch.load("/data/beans.pt")
mean = torch.tensor([0.485, 0.456, 0.406]).view(1, 3, 1, 1)
std = torch.tensor([0.229, 0.224, 0.225]).view(1, 3, 1, 1)
prep = lambda x: (x.float() / 255 - mean) / std
X, y, Xv, yv = prep(d["train_x"]), d["train_y"], prep(d["val_x"]), d["val_y"]
Xt, yt = prep(d["test_x"]), d["test_y"]
def acc(net, X, y):
net.eval()
with torch.no_grad():
p = torch.cat([net(X[i:i + 64]).argmax(1) for i in range(0, len(X), 64)])
return f"{(p == y).float().mean().item():.1%}"
net = models.resnet18(weights=None)
net.load_state_dict(torch.load("/data/resnet18_imagenet.pt")) # start from ImageNet
net.fc = nn.Linear(512, 3) # new head
body = [p for n, p in net.named_parameters() if not n.startswith("fc.")]
opt = torch.optim.Adam([{"params": body, "lr": 1e-4}, # discriminative
{"params": net.fc.parameters(), "lr": 1e-3}]) # learning rates
t = time.time()
for epoch in range(3):
net.train(); perm = torch.randperm(len(X))
for i in range(0, len(X), 32):
b = perm[i:i + 32]; xb = X[b]
if torch.rand(1) < 0.5: xb = xb.flip(3) # augmentation: mirror
loss = nn.functional.cross_entropy(net(xb), y[b])
opt.zero_grad(); loss.backward(); opt.step()
print(f"epoch {epoch + 1}: train {acc(net, X, y)} validation {acc(net, Xv, yv)}"
f" ({time.time() - t:.0f} s)")
print("test accuracy:", acc(net, Xt, yt))
Output:
epoch 1: train 97.1% validation 93.2% (39 s)
epoch 2: train 99.2% validation 94.7% (76 s)
epoch 3: train 99.4% validation 95.5% (117 s)
test accuracy: 93.8%
Each half of the batches is mirrored left to right (xb.flip(3)): data augmentation from Episode 63. Starting from ImageNet, three epochs give 95.5% validation and 93.8% test: same network, same photos as the from-scratch run; only the starting weights changed.
The mistake: one big learning rate for everything
The same program with both rates set to 1e-2:
opt = torch.optim.Adam([{"params": body, "lr": 1e-2}, # ONE big rate
{"params": net.fc.parameters(), "lr": 1e-2}]) # for everything
Output:
epoch 1: train 38.4% validation 44.4% (41 s)
epoch 2: train 40.8% validation 39.1% (82 s)
epoch 3: train 56.8% validation 56.4% (120 s)
test accuracy: 57.0%
The big steps push the pretrained weights far from what they knew: catastrophic forgetting (new training wiping out old knowledge). Gentle rates for the body protect it.
Domain shift
A domain is the kind of data a model sees (photos of what, taken how, where). Domain shift means new data looks different from the training data. ImageNet (dogs, cars, cups) → bean-leaf close-ups is already a shift. In the field there is also evening light, a blurry phone, a black-and-white camera. We simulate these on the 128 test photos:
import torch, torch.nn as nn, torchvision.models as models
torch.manual_seed(67); d, f = torch.load("/data/beans.pt"), torch.load("/data/beans_feats.pt")
head = nn.Linear(512, 3); opt = torch.optim.Adam(head.parameters(), lr=1e-2)
for step in range(200): # the same head as before
loss = nn.functional.cross_entropy(head(f["train_f"]), f["train_y"])
opt.zero_grad(); loss.backward(); opt.step()
body = models.resnet18(weights=None)
body.load_state_dict(torch.load("/data/resnet18_imagenet.pt"))
body.fc = nn.Identity(); body.eval()
mean = torch.tensor([0.485, 0.456, 0.406]).view(1, 3, 1, 1)
std = torch.tensor([0.229, 0.224, 0.225]).view(1, 3, 1, 1)
x = d["test_x"].float() / 255 # the 128 test photos
shifted = {"same kind of photos": x,
"evening light": x * 0.35,
"blurry phone camera": nn.functional.avg_pool2d(x, 9, 1, 4),
"black and white": x.mean(1, keepdim=True).expand(-1, 3, -1, -1)}
with torch.no_grad():
for name, xs in shifted.items():
pred = head(body((xs - mean) / std)).argmax(1)
print(f"{name:20}: test accuracy {(pred == d['test_y']).float().mean().item():.1%}")
Output:
same kind of photos : test accuracy 82.8%
evening light : test accuracy 75.0%
blurry phone camera : test accuracy 58.6%
black and white : test accuracy 63.3%
The leaves did not change, only the camera. Fixes: collect training photos the way users will really take them, augment with those changes (see Practice 3), and keep checking the model on fresh real-world photos.
Three ways, one table
| Approach | Weights trained | Time here (CPU) | Validation | Test |
|---|---|---|---|---|
| From scratch (random ResNet-18) | 11,178,051 (all) | about 4 minutes, 5 epochs | 69.9% (jumpy) | not measured |
| Feature extraction (frozen body + new head) | 1,539 | about 15 s for features + 2 s training | 88.0% | 82.8% |
| Fine-tuning, 2 learning rates | 11,178,051 (all) | about 2 minutes, 3 epochs | 95.5% | 93.8% |
| Fine-tuning, one big rate (0.01) | 11,178,051 (all) | about 2 minutes, 3 epochs | 56.4% | 57.0% |
Which one when? Little data and photos similar to the source task: start with feature extraction (fast, hard to overfit). More data, or a very different domain: fine-tune, unfreeze more layers, use discriminative rates. Always start from pretrained weights when a suitable model exists.
Common mistakes
import torch, torch.nn as nn, torchvision.models as models
net = models.resnet18(weights=None)
net.fc = nn.Linear(512, 3) # new head FIRST...
net.load_state_dict(torch.load("/data/resnet18_imagenet.pt")) # ...then load ImageNet
RuntimeError: Error(s) in loading state_dict for ResNet:
size mismatch for fc.weight: copying a param with shape torch.Size([1000, 512]) from checkpoint, the shape in current model is torch.Size([3, 512]).
size mismatch for fc.bias: copying a param with shape torch.Size([1000]) from checkpoint, the shape in current model is torch.Size([3]).
import torch, torchvision.models as models
net = models.resnet18(weights=None); net.eval()
leaf = torch.rand(1, 1, 160, 160) # a black-and-white photo: ONE channel
print(net(leaf).shape)
RuntimeError: Given groups=1, weight of size [64, 3, 7, 7], expected input[1, 1, 160, 160] to have 3 channels, but got 1 channels instead
| Mistake | Exact message / symptom | Why | Fix |
|---|---|---|---|
Loading weights after replacing fc |
size mismatch for fc.weight: copying a param with shape torch.Size([1000, 512]) from checkpoint, the shape in current model is torch.Size([3, 512]). |
the file's head has 1,000 outputs | load first, then net.fc = nn.Linear(512, 3) (or load_state_dict(..., strict=False) after removing the fc. keys) |
| A one-channel (grey) photo | RuntimeError: Given groups=1, weight of size [64, 3, 7, 7], expected input[1, 1, 160, 160] to have 3 channels, but got 1 channels instead |
ImageNet models take RGB | x.expand(-1, 3, -1, -1) |
| One big learning rate for everything | validation 44.4% → 39.1% → 56.4%, test 57.0% | catastrophic forgetting | small rate for the body, bigger for the head |
| Skipping ImageNet normalisation | lower accuracy | the body expects the colour statistics it trained with | same mean and std as in training |
Forgetting net.eval() |
predictions change with the batch | batch norm uses batch statistics in training mode | net.eval() and torch.no_grad() for testing |
| Freezing but passing frozen params with a big rate | nothing learns, or wasted time | frozen parameters get no gradient | give the optimiser only the parameters that train |
Mini project: a leaf-doctor function
Wrap the feature-extraction model in predict(photo) that returns a label and a confidence:
import torch, torch.nn as nn, torchvision.models as models
torch.manual_seed(67); d, f = torch.load("/data/beans.pt"), torch.load("/data/beans_feats.pt")
head = nn.Linear(512, 3); opt = torch.optim.Adam(head.parameters(), lr=1e-2)
for step in range(200):
loss = nn.functional.cross_entropy(head(f["train_f"]), f["train_y"])
opt.zero_grad(); loss.backward(); opt.step()
body = models.resnet18(weights=None); body.load_state_dict(torch.load("/data/resnet18_imagenet.pt"))
body.fc = nn.Identity(); body.eval()
mean = torch.tensor([0.485, 0.456, 0.406]).view(1, 3, 1, 1)
std = torch.tensor([0.229, 0.224, 0.225]).view(1, 3, 1, 1)
def predict(photo): # photo: uint8 tensor (3, 160, 160)
x = (photo.float().unsqueeze(0) / 255 - mean) / std
with torch.no_grad():
p = head(body(x)).softmax(1)[0]
return d["classes"][p.argmax()], round(p.max().item(), 3)
for i in (0, 60, 127):
print("true:", d["classes"][d["test_y"][i]], "| predicted:", predict(d["test_x"][i]))
Output:
true: angular_leaf_spot | predicted: ('bean_rust', 0.548)
true: bean_rust | predicted: ('bean_rust', 0.972)
true: healthy | predicted: ('healthy', 0.878)
Practice
1. Fine-tune with layer4 + head only (everything else frozen).
Solution
import torch, torch.nn as nn, torchvision.models as models, time
torch.manual_seed(67); d = torch.load("/data/beans.pt")
mean = torch.tensor([0.485, 0.456, 0.406]).view(1, 3, 1, 1)
std = torch.tensor([0.229, 0.224, 0.225]).view(1, 3, 1, 1)
prep = lambda x: (x.float() / 255 - mean) / std
X, y, Xv, yv = prep(d["train_x"]), d["train_y"], prep(d["val_x"]), d["val_y"]
def acc(net, X, y):
net.eval()
with torch.no_grad():
p = torch.cat([net(X[i:i + 64]).argmax(1) for i in range(0, len(X), 64)])
return f"{(p == y).float().mean().item():.1%}"
net = models.resnet18(weights=None)
net.load_state_dict(torch.load("/data/resnet18_imagenet.pt"))
for p in net.parameters(): p.requires_grad = False # freeze everything...
for p in net.layer4.parameters(): p.requires_grad = True # ...except layer4
net.fc = nn.Linear(512, 3) # and the new head
opt = torch.optim.Adam([{"params": net.layer4.parameters(), "lr": 1e-4},
{"params": net.fc.parameters(), "lr": 1e-3}])
t = time.time()
for epoch in range(3):
net.train(); perm = torch.randperm(len(X))
for i in range(0, len(X), 32):
b = perm[i:i + 32]
loss = nn.functional.cross_entropy(net(X[b]), y[b])
opt.zero_grad(); loss.backward(); opt.step()
print(f"epoch {epoch + 1}: validation {acc(net, Xv, yv)} ({time.time() - t:.0f} s)")
Output:
epoch 1: validation 92.5% (85 s)
epoch 2: validation 94.7% (167 s)
epoch 3: validation 94.7% (223 s)
Training only the last block and the head (8.4 of 11.2 million weights) reaches 94.7%, close to full fine-tuning (95.5%) and far above feature extraction (88.0%).
2. Use three learning rates: layer3, layer4 and the head (Howard and Ruder's ÷ 2.6 rule).
Solution
import torch, torch.nn as nn, torchvision.models as models, time
torch.manual_seed(67); d = torch.load("/data/beans.pt")
mean = torch.tensor([0.485, 0.456, 0.406]).view(1, 3, 1, 1)
std = torch.tensor([0.229, 0.224, 0.225]).view(1, 3, 1, 1)
prep = lambda x: (x.float() / 255 - mean) / std
X, y, Xv, yv = prep(d["train_x"]), d["train_y"], prep(d["val_x"]), d["val_y"]
def acc(net, X, y):
net.eval()
with torch.no_grad():
p = torch.cat([net(X[i:i + 64]).argmax(1) for i in range(0, len(X), 64)])
return f"{(p == y).float().mean().item():.1%}"
net = models.resnet18(weights=None)
net.load_state_dict(torch.load("/data/resnet18_imagenet.pt"))
for p in net.parameters(): p.requires_grad = False
net.fc = nn.Linear(512, 3)
for block in (net.layer3, net.layer4):
for p in block.parameters(): p.requires_grad = True
lr = 1e-3 # Howard and Ruder: divide by 2.6 per layer down
opt = torch.optim.Adam([{"params": net.fc.parameters(), "lr": lr},
{"params": net.layer4.parameters(), "lr": lr / 2.6},
{"params": net.layer3.parameters(), "lr": lr / 2.6 ** 2}])
t = time.time()
for epoch in range(3):
net.train(); perm = torch.randperm(len(X))
for i in range(0, len(X), 32):
b = perm[i:i + 32]
loss = nn.functional.cross_entropy(net(X[b]), y[b])
opt.zero_grad(); loss.backward(); opt.step()
print(f"epoch {epoch + 1}: validation {acc(net, Xv, yv)} ({time.time() - t:.0f} s)")
Output:
epoch 1: validation 90.2% (101 s)
epoch 2: validation 94.7% (144 s)
epoch 3: validation 94.7% (162 s)
3. Add blur augmentation (train the head on sharp AND blurred photos) and retest the blurry test photos.
Solution
import torch, torch.nn as nn, torchvision.models as models
torch.manual_seed(67); d = torch.load("/data/beans.pt")
mean = torch.tensor([0.485, 0.456, 0.406]).view(1, 3, 1, 1)
std = torch.tensor([0.229, 0.224, 0.225]).view(1, 3, 1, 1)
body = models.resnet18(weights=None); body.load_state_dict(torch.load("/data/resnet18_imagenet.pt"))
body.fc = nn.Identity(); body.eval()
blur = lambda x: nn.functional.avg_pool2d(x, 9, 1, 4)
def feats(x):
with torch.no_grad():
return torch.cat([body((x[i:i + 64] - mean) / std) for i in range(0, len(x), 64)])
x, xt = d["train_x"].float() / 255, d["test_x"].float() / 255
F_train = torch.cat([feats(x), feats(blur(x))]) # every photo twice: sharp and blurred
y_train = torch.cat([d["train_y"], d["train_y"]])
head = nn.Linear(512, 3); opt = torch.optim.Adam(head.parameters(), lr=1e-2)
for step in range(200):
loss = nn.functional.cross_entropy(head(F_train), y_train)
opt.zero_grad(); loss.backward(); opt.step()
with torch.no_grad():
for name, xs in (("sharp test photos", xt), ("blurry test photos", blur(xt))):
print(f"{name}: {(head(feats(xs)).argmax(1) == d['test_y']).float().mean().item():.1%}")
Output:
sharp test photos: 82.0%
blurry test photos: 79.7%
Before, the blurry test photos scored 58.6%. Training on blurred copies makes the model robust to that particular shift.
4. Use ResNet-50 features (2,048 per photo) instead of ResNet-18's 512.
Solution
import torch, torch.nn as nn, torchvision.models as models
torch.manual_seed(67); d = torch.load("/data/beans.pt")
mean = torch.tensor([0.485, 0.456, 0.406]).view(1, 3, 1, 1)
std = torch.tensor([0.229, 0.224, 0.225]).view(1, 3, 1, 1)
body = models.resnet50(weights=None)
body.load_state_dict(torch.load("/data/resnet50_imagenet.pt")) # ImageNet weights for ResNet-50
body.fc = nn.Identity(); body.eval() # 2,048 features per photo
def feats(s):
x = (d[s + "_x"].float() / 255 - mean) / std
with torch.no_grad():
return torch.cat([body(x[i:i + 32]) for i in range(0, len(x), 32)])
F, Fv, Ft = feats("train"), feats("val"), feats("test")
head = nn.Linear(F.shape[1], 3); opt = torch.optim.Adam(head.parameters(), lr=1e-2)
for step in range(200):
loss = nn.functional.cross_entropy(head(F), d["train_y"])
opt.zero_grad(); loss.backward(); opt.step()
acc = lambda F, y: f"{(head(F).argmax(1) == y).float().mean().item():.1%}"
with torch.no_grad():
print("features per photo:", F.shape[1], "| validation", acc(Fv, d["val_y"]), "| test", acc(Ft, d["test_y"]))
Output:
features per photo: 2048 | validation 90.2% | test 88.3%
Quick quiz
We freeze every parameter of ResNet-18, then net.fc = nn.Linear(512, 10). How many parameters are trainable? (a) 0 (b) 5,130 (c) 11,689,512 (d) 512
Answer
(b) 5,130: 512 × 10 weights + 10 biases. The frozen body adds nothing, and the new layer starts unfrozen.
- What does
nn.Identity()do asnet.fc?Answer
Returns its input unchanged, so the network outputs the 512 body features. - Why can feature extraction save features to a file?
Answer
The body is frozen, so the same photo always gives the same features: compute them once. - What does
requires_grad = Falsedo?Answer
No gradient is computed for that parameter, so the optimiser never changes it (frozen). - Why a smaller learning rate for the body than for the new head?
Answer
The body already holds useful knowledge; big steps would destroy it (catastrophic forgetting), while the random new head needs bigger steps. - What did ImageNet's ResNet-18 head call an angular-leaf-spot photo?
Answer
"hippopotamus" (61%): its 1,000 classes contain no bean diseases.
Interview questions
- What is transfer learning and why does it work? Reusing a model trained on a source task for a target task. Early layers learn general features (edges, colours, textures) that are useful for almost any images, so only task-specific parts need to be learned from the small target dataset.
- Feature extraction vs fine-tuning? Feature extraction freezes the pretrained body and trains only a new head (fast, little data, low overfitting risk). Fine-tuning also updates the body with small learning rates (higher accuracy with enough data or when the domain differs).
- How do you freeze layers in PyTorch? Set
p.requires_grad = Falsefor their parameters (and pass only trainable parameters to the optimiser); keep BN layers in eval mode if you want their statistics frozen too. - What are discriminative learning rates and gradual unfreezing? Different learning rates per layer group (smaller for lower, more general layers; Howard and Ruder divide by 2.6 per layer), and unfreezing from the top layer down one stage at a time.
- What is catastrophic forgetting? When training on a new task overwrites what a network learned before, for example fine-tuning a pretrained model with a large learning rate.
- What is domain shift and how do you handle it? Test or production data distributed differently from training data (lighting, camera, region). Handle it with representative data collection, augmentation, fine-tuning on target-domain data and monitoring on fresh data.
- Why must you use the same normalisation as the pretrained model? Its weights were learned on inputs with those statistics; different statistics shift every feature.
Glossary
| Term | Meaning (plain words + example) |
|---|---|
| Transfer learning | Reusing what a model learned on one task for another: ImageNet ResNet-18 → bean leaves |
| Pretrained model | A network already trained on another (usually bigger) task, e.g. ResNet-18 on ImageNet |
| ImageNet | A large labelled photo collection; its contest had 1,000 classes, about 1.2 million training photos |
| torchvision models | Ready-made architectures (models.resnet18, models.resnet50, ...) with optional pretrained weights |
Weights file / state_dict |
The saved numbers of a network: a dictionary from layer names to tensors |
weights=None |
Build the architecture with random weights (no training, no download) |
ResNet18_Weights.IMAGENET1K_V1 |
torchvision's ImageNet weights for ResNet-18 (69.758% top-1) |
| Body (backbone) | The layers that turn an input into features (everything before fc in ResNet) |
| Head (classifier head) | The last layer(s) turning features into class scores; ResNet's fc |
| Feature | A number describing something about the input; ResNet-18's body gives 512 |
| Feature extraction | Using a frozen pretrained body only to compute features and training a new head on them |
nn.Identity() |
A layer that returns its input unchanged; replaces a head to expose the features |
| Freezing | Stopping parameters from changing: requires_grad = False |
requires_grad |
A tensor's flag: should PyTorch compute a gradient for it? |
| Unfreezing | Letting frozen parameters train again: requires_grad = True |
| Replacing the classifier head | Assigning a new output layer, e.g. net.fc = nn.Linear(512, 3) |
| Fine-tuning | Continuing to train pretrained weights, gently, on the new task |
| Discriminative learning rates | Different learning rates for different layers: small for the body, bigger for the head |
| Gradual unfreezing | Unfreezing layers from the top down, one at a time, during fine-tuning |
| Catastrophic forgetting | New training wiping out old knowledge, e.g. one big learning rate: 93.8% → 57.0% test |
| Domain | The kind of data a model sees: what, how and where the photos were taken |
| Domain shift | New data looking different from training data: blurry photos dropped 82.8% → 58.6% |
| Normalisation (ImageNet) | Subtract mean [0.485, 0.456, 0.406], divide by std [0.229, 0.224, 0.225] per channel |
| Data augmentation | Training on randomly changed copies (mirrored, blurred) of the photos |
Summary
- From scratch on 1,034 photos, ResNet-18 reached only about 70% validation, jumping around.
- A pretrained ResNet-18 (ImageNet, 69.758% top-1) knows general visual features, but its head knows only ImageNet's classes ("hippopotamus" for a sick leaf).
- Feature extraction: chop off the head (
nn.Identity()), save 512 features per photo, trainnn.Linear(512, 3): 1,539 weights, 88.0% validation in seconds. - Freezing with
requires_grad = False, replacing the head, unfreezinglayer4(8.4 million weights). - Fine-tuning with discriminative learning rates (1e-4 body, 1e-3 head): 95.5% validation, 93.8% test. One big rate (1e-2) caused catastrophic forgetting (57.0% test).
- Domain shift: the same leaves photographed blurry or in black and white dropped to 58.6% and 63.3%.
Next episode: Computer vision tasks: classification vs object detection vs segmentation, bounding boxes, IoU, NMS, anchor boxes, YOLO, mAP and U-Net.