SyntaxPilot
Courses/Machine Learning/Lesson 68
Video coming soonThis lesson’s video is being made

Start with the notes below; the video appears here as soon as it is on YouTube.

Machine Learning · Lesson 68 of 90

Computer vision tasks

image classification vs object detection vs segmentationbounding boxintersection over union (IoU)anchor boxesYOLOnon-maximum suppression (NMS)mean average precision (mAP)semantic vs instance segmentationU-Net

Machine Learning Full Course · Episode 68. Every code sample below was run with Python 3.13, PyTorch 2.15.0 and torchvision 0.29.2 on a CPU, and the output shown is the real output. Most pieces are written from scratch in plain Python first, then checked against torchvision.ops.

The big idea: one label per image is not enough when a photo holds several objects. Object detection gives each object a bounding box, a label and a confidence score; segmentation labels every pixel. To build a detector we need a way to score a box (IoU), a way to remove duplicate boxes (non-maximum suppression), ready-made box shapes to correct (anchor boxes), a fast single-network design (YOLO), and a way to grade the whole detector (AP and mAP). For segmentation, U-Net turns an image into a same-size mask using a contracting path, an expanding path and skip connections.

Sources: Joseph Redmon, Santosh Divvala, Ross Girshick and Ali Farhadi, "You Only Look Once: Unified, Real-Time Object Detection", 2015 (arXiv 1506.02640, full text). Shaoqing Ren, Kaiming He, Ross Girshick and Jian Sun, "Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks", 2015 (full text). Olaf Ronneberger, Philipp Fischer and Thomas Brox, "U-Net: Convolutional Networks for Biomedical Image Segmentation", 2015 (arXiv 1505.04597). COCO detection metrics as listed in the hf-vision detection_metrics README.

Words you need first

  • Computer vision: getting computers to understand images and video.
  • Image classification: one label for the whole image ("bean rust"). Episodes 65-67.
  • Object detection: for every object, a box + a label + a confidence score.
  • Segmentation: a label for every pixel of the image.
  • Pixel coordinates: x goes right, y goes down; (0, 0) is the top-left pixel.
  • Confidence score: a number from 0 to 1 saying how sure the detector is that a box holds an object (of that class).
  • Precision and recall (Episode 24): precision = correct predictions ÷ all predictions; recall = objects found ÷ all real objects.
  • 1 × 1 convolution and ResNet body (Episodes 66-67).

The story: three leaves in one photo

The farmer from last episode now sends a photo with three leaves: two sick, one healthy. Our classifier gives one label, "bean rust", for the whole photo. Which leaf? Where? How many are sick? A single label cannot answer; we need positions.

Task Output Answers Example
Image classification one label (+ probability) what is in the image? "bean rust, 94%"
Object detection a box + label + score per object what and where? 2 boxes "bean rust", 1 box "healthy"
Semantic segmentation a class for every pixel what is each pixel? every leaf pixel = 1, background = 0
Instance segmentation class + object id for every pixel which object is each pixel? leaf 1, leaf 2, leaf 3

Bounding boxes

A bounding box is the smallest upright rectangle around an object. We store it as (x1, y1, x2, y2): top-left corner and bottom-right corner. Other common formats are (x, y, width, height) (COCO) and (centre x, centre y, width, height) (YOLO); torchvision.ops.box_convert converts between them. Always check which one a dataset or library uses.

IoU: intersection over union

Real-life picture: two sheets of paper on a table. The intersection is where they overlap; the union is the whole area covered by either.

IoU = intersection ÷ union, where union = area(A) + area(B) − intersection. IoU is 1 for a perfect match and 0 for no overlap. The overlap's width is min(right edges) − max(left edges), and the same for the height.

The obvious attempt:

def area(b):                        # box = (x1, y1, x2, y2): top-left and bottom-right corners
    return (b[2] - b[0]) * (b[3] - b[1])
def iou_wrong(a, b):                # the obvious attempt
    w = min(a[2], b[2]) - max(a[0], b[0])
    h = min(a[3], b[3]) - max(a[1], b[1])
    inter = w * h
    return inter / (area(a) + area(b) - inter)
truth = (50, 50, 150, 150)          # where the leaf really is
for name, guess in [("perfect", (50, 50, 150, 150)), ("shifted", (70, 60, 170, 160)),
                    ("far away", (300, 300, 400, 400))]:
    print(f"{name:9} IoU = {iou_wrong(truth, guess):.3f}")

Output:

perfect   IoU = 1.000
shifted   IoU = 0.562
far away  IoU = -9.000

IoU = −9.000 is impossible. Why: when the boxes do not touch, the "overlap" width and height are both negative (−150 × −150 = +22,500), a fake overlap bigger than either box. Fix: clamp both at zero.

def area(b):
    return (b[2] - b[0]) * (b[3] - b[1])
def iou(a, b):                      # intersection over union
    w = max(0, min(a[2], b[2]) - max(a[0], b[0]))     # no overlap -> 0, never negative
    h = max(0, min(a[3], b[3]) - max(a[1], b[1]))
    inter = w * h
    return inter / (area(a) + area(b) - inter)
truth = (50, 50, 150, 150)
for name, guess in [("perfect", (50, 50, 150, 150)), ("shifted", (70, 60, 170, 160)),
                    ("too big", (25, 25, 175, 175)), ("half off", (100, 50, 200, 150)),
                    ("far away", (300, 300, 400, 400))]:
    print(f"{name:9} IoU = {iou(truth, guess):.3f}")
import torch, torchvision.ops as ops          # the library version agrees
print("torchvision:", ops.box_iou(torch.tensor([truth]).float(),
                                   torch.tensor([(70, 60, 170, 160)]).float()).item())

Output:

perfect   IoU = 1.000
shifted   IoU = 0.562
too big   IoU = 0.444
half off  IoU = 0.333
far away  IoU = 0.000
torchvision: 0.5625

torchvision.ops.box_iou gives the same 0.5625. A detection usually counts as correct when its IoU with a true box is at least a threshold such as 0.5 (PASCAL VOC style).

Non-maximum suppression (NMS)

A detector makes many guesses, each with a confidence score, and neighbouring guesses often find the same object. Counting every box above 0.5 confidence counts each leaf three times. Non-maximum suppression fixes it: sort by confidence, keep the best box, delete every remaining box whose IoU with it is above a threshold (its copies), and repeat.

def area(b): return (b[2] - b[0]) * (b[3] - b[1])
def iou(a, b):
    w = max(0, min(a[2], b[2]) - max(a[0], b[0])); h = max(0, min(a[3], b[3]) - max(a[1], b[1]))
    return w * h / (area(a) + area(b) - w * h)
# a detector's raw output for a photo with TWO sick leaves: (box, confidence score)
raw = [((48, 52, 152, 148), 0.92), ((55, 45, 160, 155), 0.85), ((40, 60, 145, 150), 0.78),
       ((260, 80, 380, 200), 0.88), ((270, 90, 385, 210), 0.70), ((250, 75, 370, 190), 0.65),
       ((150, 300, 180, 330), 0.12)]
print("boxes over 0.5 confidence:", sum(s > 0.5 for _, s in raw), "-> 6 leaves?!")
def nms(dets, iou_threshold=0.5):
    dets = sorted(dets, key=lambda d: d[1], reverse=True)   # most confident first
    keep = []
    while dets:
        best = dets.pop(0); keep.append(best)               # keep the best box...
        dets = [d for d in dets if iou(best[0], d[0]) < iou_threshold]   # ...drop its copies
    return keep
kept = nms([d for d in raw if d[1] > 0.5])
for box, score in kept:
    print("kept", box, "score", score)
import torch, torchvision.ops as ops
B = torch.tensor([b for b, s in raw if s > 0.5]).float(); S = torch.tensor([s for b, s in raw if s > 0.5])
print("torchvision.ops.nms keeps indices:", ops.nms(B, S, 0.5).tolist())

Output:

boxes over 0.5 confidence: 6 -> 6 leaves?!
kept (48, 52, 152, 148) score 0.92
kept (260, 80, 380, 200) score 0.88
torchvision.ops.nms keeps indices: [0, 3]

The NMS IoU threshold is a setting: Faster R-CNN's region proposals use 0.7 ("We fix the IoU threshold for NMS at 0.7"); 0.5 is a common choice for final detections. Too low merges neighbouring objects; too high keeps duplicates (see Practice 2).

Anchor boxes

Real-life example: a parking lot. Spaces are painted at fixed spots (grid positions) in several shapes (bikes, cars, buses: anchor boxes), and a car parks a little off the lines (the offset the network learns).

An anchor box is a fixed reference box of a set size and shape at a grid position. The network predicts, for each anchor, whether it holds an object and how to shift and stretch it. Faster R-CNN (Ren, He, Girshick, Sun, 2015): "By default we use 3 scales and 3 aspect ratios, yielding k=9 anchors at each sliding position" (areas 128², 256², 512²; ratios 1:1, 1:2, 2:1). Aspect ratio = height ÷ width. During training, anchors with a high IoU to a true box learn to fit it.

def area(b): return (b[2] - b[0]) * (b[3] - b[1])
def iou(a, b):
    w = max(0, min(a[2], b[2]) - max(a[0], b[0])); h = max(0, min(a[3], b[3]) - max(a[1], b[1]))
    return w * h / (area(a) + area(b) - w * h)
def anchors_at(cx, cy, scales=(32, 64, 128), ratios=(0.5, 1.0, 2.0)):
    out = []
    for s in scales:                 # same area s * s ...
        for r in ratios:             # ... different shapes: height / width = r
            w, h = s / r ** 0.5, s * r ** 0.5
            out.append((cx - w / 2, cy - h / 2, cx + w / 2, cy + h / 2))
    return out
stride = 32                          # one anchor position every 32 pixels
grid = [(x * stride + 16, y * stride + 16) for y in range(7) for x in range(7)]
anchors = [a for cx, cy in grid for a in anchors_at(cx, cy)]
print("positions:", len(grid), "| anchors per position:", len(anchors_at(0, 0)),
      "| total anchors:", len(anchors))
leaf = (100, 24, 188, 200)           # a tall, thin leaf: 88 wide, 176 high
best = max(anchors, key=lambda a: iou(a, leaf))
w, h = best[2] - best[0], best[3] - best[1]
print(f"best anchor: {w:.0f} wide x {h:.0f} high, IoU {iou(best, leaf):.2f}")
print("anchors with IoU > 0.5:", sum(iou(a, leaf) > 0.5 for a in anchors))
mid = lambda b: ((b[0] + b[2]) / 2, (b[1] + b[3]) / 2)
(ax, ay), (lx, ly) = mid(best), mid(leaf)
print(f"what the network learns: move {lx - ax:+.0f}, {ly - ay:+.0f} px,"
      f" width x{(leaf[2] - leaf[0]) / w:.2f}, height x{(leaf[3] - leaf[1]) / h:.2f}")

Output:

positions: 49 | anchors per position: 9 | total anchors: 441
best anchor: 91 wide x 181 high, IoU 0.95
anchors with IoU > 0.5: 4
what the network learns: move +0, +0 px, width x0.97, height x0.97

The tall anchor (91 × 181) already overlaps the tall leaf with IoU 0.95, so only a tiny correction is needed.

YOLO: You Only Look Once

Earlier detectors first proposed regions and then classified each one. YOLO (Redmon, Divvala, Girshick, Farhadi, 2015): "A single neural network predicts bounding boxes and class probabilities directly from full images in one evaluation." It divides the image into an S × S grid; each cell predicts B boxes (x, y, w, h and a confidence = Pr(Object) × IOU) and C class probabilities, encoded as an S × S × (B·5 + C) tensor. For PASCAL VOC: S = 7, B = 2, C = 20, a 7 × 7 × 30 tensor. The base model ran at 45 frames per second (Fast YOLO: 155) and reached 63.4% mAP on VOC 2007. Later YOLO versions added anchor boxes and many other improvements.

import torch, torch.nn as nn, torchvision.models as models
torch.manual_seed(68)
S, B, C = 7, 2, 20                    # YOLO (2015) on PASCAL VOC: 7 x 7 grid, 2 boxes, 20 classes
print("numbers per grid cell: B * 5 + C =", B * 5 + C)
body = models.resnet18(weights=None)
body = nn.Sequential(*list(body.children())[:-2])     # keep the conv layers, drop pooling + fc
head = nn.Conv2d(512, B * 5 + C, kernel_size=1)       # a 1 x 1 conv: 30 numbers per cell
photo = torch.randn(1, 3, 224, 224)
features = body(photo)
out = head(features)
print("photo", tuple(photo.shape), "-> features", tuple(features.shape), "-> output", tuple(out.shape))
cell = out[0, :, 3, 4]                                # the cell in row 3, column 4
x, y, w, h, conf = cell[:5].sigmoid().tolist()        # box 1: centre, size, confidence
print(f"cell (3, 4), box 1: x {x:.2f} y {y:.2f} w {w:.2f} h {h:.2f} confidence {conf:.2f}")
print("class scores per cell:", cell[10:].shape[0], "| boxes per photo:", S * S * B)

Output:

numbers per grid cell: B * 5 + C = 30
photo (1, 3, 224, 224) -> features (1, 512, 7, 7) -> output (1, 30, 7, 7)
cell (3, 4), box 1: x 0.71 y 0.42 w 0.18 h 0.74 confidence 0.50
class scores per cell: 20 | boxes per photo: 98

A ResNet-18 body on a 224 × 224 photo already gives a 7 × 7 grid of 512 features; a 1 × 1 convolution turns it into 30 numbers per cell. The network is untrained, so these numbers mean nothing yet. 98 boxes per photo is why NMS always follows.

Scoring a detector: AP and mAP

  1. A prediction is a true positive if its IoU with a true box is at least the threshold and that true box was not already matched; otherwise it is a false positive.
  2. Go through predictions from most to least confident, recording recall and precision after each: the precision-recall curve.
  3. Average precision (AP) = area under that curve (we make precision non-increasing first, the usual "interpolated" AP).
  4. mAP (mean average precision) = mean of AP over all classes.
  5. COCO AP averages AP over IoU thresholds 0.50, 0.55, ..., 0.95 ("AP at IoU=.50:.05:.95"); AP50 is "similar to mAP PASCAL VOC metric" and AP75 is the "strict metric".
def area(b): return (b[2] - b[0]) * (b[3] - b[1])
def iou(a, b):
    w = max(0, min(a[2], b[2]) - max(a[0], b[0])); h = max(0, min(a[3], b[3]) - max(a[1], b[1]))
    return w * h / (area(a) + area(b) - w * h)
truth = [(10, 10, 60, 60), (100, 20, 150, 80), (200, 200, 260, 250), (300, 40, 350, 90)]
dets = [((12, 8, 61, 58), 0.95), ((105, 25, 152, 84), 0.90), ((400, 400, 450, 450), 0.80),
        ((196, 205, 255, 252), 0.70), ((12, 12, 58, 63), 0.60), ((320, 60, 370, 110), 0.40)]
def average_precision(dets, truth, thr):
    dets = sorted(dets, key=lambda d: d[1], reverse=True)  # most confident first
    used, tp, points = set(), 0, []
    for k, (box, score) in enumerate(dets, 1):
        best = max(range(len(truth)), key=lambda i: iou(box, truth[i]))
        if iou(box, truth[best]) >= thr and best not in used:
            used.add(best); tp += 1                         # a true positive
        points.append((tp / len(truth), tp / k))            # (recall, precision)
    ap, last_r = 0.0, 0.0
    for i, (r, p) in enumerate(points):                     # area under the P-R curve
        best_p = max(p2 for r2, p2 in points[i:])           # (precision made non-increasing)
        ap += (r - last_r) * best_p; last_r = r
    return ap, points
ap, points = average_precision(dets, truth, 0.5)
for r, p in points:
    print(f"recall {r:.2f}  precision {p:.2f}")
print(f"AP at IoU 0.5: {ap:.3f}")
for thr in (0.75, 0.9):
    print(f"AP at IoU {thr}: {average_precision(dets, truth, thr)[0]:.3f}")
coco = [average_precision(dets, truth, t / 100)[0] for t in range(50, 100, 5)]
print(f"COCO-style AP (IoU 0.50 to 0.95, 10 steps): {sum(coco) / len(coco):.3f}")

Output:

recall 0.25  precision 1.00
recall 0.50  precision 1.00
recall 0.50  precision 0.67
recall 0.75  precision 0.75
recall 0.75  precision 0.60
recall 0.75  precision 0.50
AP at IoU 0.5: 0.688
AP at IoU 0.75: 0.688
AP at IoU 0.9: 0.000
COCO-style AP (IoU 0.50 to 0.95, 10 steps): 0.463

At IoU 0.9 none of our boxes is precise enough, so AP falls to 0; averaging over many thresholds rewards tight boxes.

Segmentation: semantic vs instance

A mask is an image-sized grid of labels, one per pixel. Semantic segmentation gives each pixel's class; instance segmentation also says which object it belongs to. Exact shapes let us measure, for example, how much of a leaf is rusty.

photo = ["................",           # a tiny 8 x 16 "photo": # = leaf pixel
         "..####......###.",
         ".######....#####",
         ".######....#####",
         "..####......###.",
         "................",
         "......##........",
         "......##........"]
semantic = [[1 if c == "#" else 0 for c in row] for row in photo]   # class per pixel: leaf or not
instance = [[0] * 16 for _ in photo]; count = 0
for y in range(8):
    for x in range(16):
        if semantic[y][x] and not instance[y][x]:
            count += 1; todo = [(y, x)]                     # a new leaf: flood-fill it
            while todo:
                j, i = todo.pop()
                if 0 <= j < 8 and 0 <= i < 16 and semantic[j][i] and not instance[j][i]:
                    instance[j][i] = count
                    todo += [(j + 1, i), (j - 1, i), (j, i + 1), (j, i - 1)]
print("semantic (class per pixel)    instance (which leaf)")
for s, n in zip(semantic, instance):
    print("".join(map(str, s)), "             ", "".join(str(v) if v else "." for v in n))
print("classes in the semantic mask:", sorted({v for r in semantic for v in r}),
      "| leaves found:", count)

Output:

semantic (class per pixel)    instance (which leaf)
0000000000000000               ................
0011110000001110               ..1111......222.
0111111000011111               .111111....22222
0111111000011111               .111111....22222
0011110000001110               ..1111......222.
0000000000000000               ................
0000001100000000               ......33........
0000001100000000               ......33........
classes in the semantic mask: [0, 1] | leaves found: 3

The flood fill starts at an unlabelled leaf pixel and spreads to all touching leaf pixels, giving each connected leaf its own number.

U-Net

U-Net (Ronneberger, Fischer, Brox, 2015, for biomedical images) has "a contracting path to capture context and a symmetric expanding path that enables precise localization". Going down, convolutions and pooling make maps smaller with more channels (what); going up, transposed convolutions (nn.ConvTranspose2d, a learned upsampling) bring them back to full size (where). Skip connections copy each down-level's maps across and concatenate them (torch.cat) onto the way up, restoring fine detail lost by pooling. The final 1 × 1 convolution gives a score per class per pixel; the paper segments a 512 × 512 image in "less than a second on a recent GPU".

import torch, torch.nn as nn
torch.manual_seed(68)
def block(cin, cout):                     # two 3 x 3 convs, like the U-Net paper
    return nn.Sequential(nn.Conv2d(cin, cout, 3, padding=1), nn.ReLU(),
                         nn.Conv2d(cout, cout, 3, padding=1), nn.ReLU())
class TinyUNet(nn.Module):
    def __init__(self, classes=3):
        super().__init__()
        self.d1, self.d2, self.mid = block(3, 16), block(16, 32), block(32, 64)
        self.pool = nn.MaxPool2d(2)
        self.up2, self.u2 = nn.ConvTranspose2d(64, 32, 2, stride=2), block(64, 32)
        self.up1, self.u1 = nn.ConvTranspose2d(32, 16, 2, stride=2), block(32, 16)
        self.out = nn.Conv2d(16, classes, 1)          # a score for every class, at every pixel
    def forward(self, x):
        a = self.d1(x); b = self.d2(self.pool(a)); m = self.mid(self.pool(b))
        print("down:", tuple(a.shape), tuple(b.shape), "bottom:", tuple(m.shape))
        u = self.u2(torch.cat([self.up2(m), b], 1))    # up, then glue on the skip
        u = self.u1(torch.cat([self.up1(u), a], 1))
        print("up:  ", tuple(u.shape))
        return self.out(u)
net = TinyUNet()
scores = net(torch.randn(1, 3, 64, 64))
print("output:", tuple(scores.shape), "-> mask:", tuple(scores.argmax(1).shape))
print("parameters:", f"{sum(p.numel() for p in net.parameters()):,}")

Output:

down: (1, 16, 64, 64) (1, 32, 32, 32) bottom: (1, 64, 16, 16)
up:   (1, 16, 64, 64)
output: (1, 3, 64, 64) -> mask: (1, 64, 64)
parameters: 117,075

Common mistakes

Mistake Exact message / symptom Why Fix
Overlap without clamping far away IoU = -9.000 negative width × negative height max(0, ...) for width and height
Counting raw detections 6 boxes over 0.5 for 2 leaves neighbouring guesses find the same object non-maximum suppression
Mixing box formats boxes in the wrong place, IoU near 0 (x, y, w, h) read as (x1, y1, x2, y2) torchvision.ops.box_convert
Reporting one loose AP threshold AP 0.688 at 0.5 but 0.000 at 0.9 IoU 0.5 hides sloppy boxes also report COCO AP over 0.50-0.95
Flipping the photo but not the labels boxes/masks on the wrong side augmentation must move labels too flip boxes and masks the same way
NMS across different classes a "rust" box deletes an overlapping "healthy" box suppression should compare same-class boxes per-class NMS (torchvision.ops.batched_nms)

Mini project: count the sick leaves

Combine the pieces: filter by confidence, run NMS, then count the boxes per class. With the raw detections from the NMS section, the 6 confident boxes become 2 kept boxes: 2 sick leaves. Add a class to each detection and run NMS per class (torchvision.ops.batched_nms(boxes, scores, classes, 0.5)) to count each disease separately.

Practice

1. Write iou() for (x, y, width, height) boxes.

Solution
def iou_xywh(a, b):                  # boxes as (x, y, width, height), (x, y) = top-left corner
    a = (a[0], a[1], a[0] + a[2], a[1] + a[3])   # convert to (x1, y1, x2, y2)
    b = (b[0], b[1], b[0] + b[2], b[1] + b[3])
    w = max(0, min(a[2], b[2]) - max(a[0], b[0])); h = max(0, min(a[3], b[3]) - max(a[1], b[1]))
    area = lambda r: (r[2] - r[0]) * (r[3] - r[1])
    return w * h / (area(a) + area(b) - w * h)
print(iou_xywh((50, 50, 100, 100), (70, 60, 100, 100)))   # same boxes as the video: 0.5625
print(iou_xywh((0, 0, 10, 10), (5, 0, 10, 10)))           # the quiz: 0.333...
import torch, torchvision.ops as ops
print(ops.box_convert(torch.tensor([[50., 50., 100., 100.]]), "xywh", "xyxy"))

Output:

0.5625
0.3333333333333333
tensor([[ 50.,  50., 150., 150.]])

2. Run NMS with thresholds 0.3 and 0.8.

Solution
def area(b): return (b[2] - b[0]) * (b[3] - b[1])
def iou(a, b):
    w = max(0, min(a[2], b[2]) - max(a[0], b[0])); h = max(0, min(a[3], b[3]) - max(a[1], b[1]))
    return w * h / (area(a) + area(b) - w * h)
raw = [((48, 52, 152, 148), 0.92), ((55, 45, 160, 155), 0.85), ((40, 60, 145, 150), 0.78),
       ((260, 80, 380, 200), 0.88), ((270, 90, 385, 210), 0.70), ((250, 75, 370, 190), 0.65)]
def nms(dets, iou_threshold):
    dets = sorted(dets, key=lambda d: d[1], reverse=True); keep = []
    while dets:
        best = dets.pop(0); keep.append(best)
        dets = [d for d in dets if iou(best[0], d[0]) < iou_threshold]
    return keep
print("IoUs with the best box:", [round(iou(raw[0][0], b), 2) for b, _ in raw[1:3]])
for t in (0.3, 0.5, 0.8, 0.9):
    print(f"threshold {t}: {len(nms(raw, t))} boxes kept")

Output:

IoUs with the best box: [0.76, 0.78]
threshold 0.3: 2 boxes kept
threshold 0.5: 2 boxes kept
threshold 0.8: 6 boxes kept
threshold 0.9: 6 boxes kept

The copies overlap the best box with IoU 0.76-0.78, so a threshold above that (0.8) keeps every duplicate.

3. Add a second class to the AP code and compute mAP.

Solution
def area(b): return (b[2] - b[0]) * (b[3] - b[1])
def iou(a, b):
    w = max(0, min(a[2], b[2]) - max(a[0], b[0])); h = max(0, min(a[3], b[3]) - max(a[1], b[1]))
    return w * h / (area(a) + area(b) - w * h)
def average_precision(dets, truth, thr):
    dets = sorted(dets, key=lambda d: d[1], reverse=True)
    used, tp, points = set(), 0, []
    for k, (box, score) in enumerate(dets, 1):
        best = max(range(len(truth)), key=lambda i: iou(box, truth[i]))
        if iou(box, truth[best]) >= thr and best not in used:
            used.add(best); tp += 1
        points.append((tp / len(truth), tp / k))
    ap, last_r = 0.0, 0.0
    for i, (r, p) in enumerate(points):
        ap += (r - last_r) * max(p2 for _, p2 in points[i:]); last_r = r
    return ap
data = {   # class: (true boxes, detections)
  "bean_rust": ([(10, 10, 60, 60), (100, 20, 150, 80), (200, 200, 260, 250), (300, 40, 350, 90)],
                [((12, 8, 61, 58), 0.95), ((105, 25, 152, 84), 0.90), ((400, 400, 450, 450), 0.80),
                 ((196, 205, 255, 252), 0.70), ((12, 12, 58, 63), 0.60), ((320, 60, 370, 110), 0.40)]),
  "leaf_spot": ([(30, 120, 90, 170), (150, 150, 200, 210)],
                [((32, 118, 88, 172), 0.85), ((10, 300, 40, 330), 0.75), ((152, 148, 205, 205), 0.50)])}
aps = {c: average_precision(d, t, 0.5) for c, (t, d) in data.items()}
for c, ap in aps.items(): print(f"AP@0.5 {c}: {ap:.3f}")
print(f"mAP@0.5 = mean over classes: {sum(aps.values()) / len(aps):.3f}")

Output:

AP@0.5 bean_rust: 0.688
AP@0.5 leaf_spot: 0.833
mAP@0.5 = mean over classes: 0.760

4. Add a third level to the tiny U-Net.

Solution
import torch, torch.nn as nn
def block(cin, cout):
    return nn.Sequential(nn.Conv2d(cin, cout, 3, padding=1), nn.ReLU(), nn.Conv2d(cout, cout, 3, padding=1), nn.ReLU())
class UNet3(nn.Module):                                   # three levels down, three up
    def __init__(self, classes=3):
        super().__init__()
        self.d1, self.d2, self.d3, self.mid = block(3, 16), block(16, 32), block(32, 64), block(64, 128)
        self.pool = nn.MaxPool2d(2)
        self.up3, self.u3 = nn.ConvTranspose2d(128, 64, 2, stride=2), block(128, 64)
        self.up2, self.u2 = nn.ConvTranspose2d(64, 32, 2, stride=2), block(64, 32)
        self.up1, self.u1 = nn.ConvTranspose2d(32, 16, 2, stride=2), block(32, 16)
        self.out = nn.Conv2d(16, classes, 1)
    def forward(self, x):
        a = self.d1(x); b = self.d2(self.pool(a)); c = self.d3(self.pool(b)); m = self.mid(self.pool(c))
        u = self.u3(torch.cat([self.up3(m), c], 1))
        u = self.u2(torch.cat([self.up2(u), b], 1))
        u = self.u1(torch.cat([self.up1(u), a], 1))
        return self.out(u)
net = UNet3()
print("bottom:", tuple(net.mid(net.pool(net.d3(net.pool(net.d2(net.pool(net.d1(torch.randn(1, 3, 64, 64)))))))).shape))
print("output:", tuple(net(torch.randn(1, 3, 64, 64)).shape))
print("parameters:", f"{sum(p.numel() for p in net.parameters()):,}")

Output:

bottom: (1, 128, 8, 8)
output: (1, 3, 64, 64)
parameters: 482,067

One more level makes the bottom 8 × 8 with 128 channels and grows the network from 117,075 to 482,067 parameters.

Quick quiz

truth = (0, 0, 10, 10), guess = (5, 0, 15, 10), overlap 50. IoU? (a) 0.25 (b) 0.333 (c) 0.5 (d) 1.0

Answer

(b) 0.333: union = 100 + 100 − 50 = 150, and 50 ÷ 150 = 0.333.

  1. Detection vs segmentation?
    AnswerDetection gives a box per object; segmentation gives a label per pixel.
  2. What does NMS remove?
    AnswerLower-scored boxes that overlap a kept box above the IoU threshold (duplicates).
  3. What is YOLO's output size on PASCAL VOC?
    Answer7 × 7 × 30: S = 7, B = 2 boxes × 5 numbers + C = 20 classes.
  4. How many anchors per position in Faster R-CNN?
    Answer9: 3 scales × 3 aspect ratios.
  5. What do U-Net's skip connections carry?
    AnswerEach down-level's feature maps, concatenated onto the up path, to restore fine detail.

Interview questions

  1. Explain IoU and where it is used. Intersection area ÷ union area of two boxes (or masks). Used to match predictions to ground truth, in NMS, in anchor assignment, and in mAP thresholds.
  2. How does non-maximum suppression work? Sort by score; repeatedly keep the top box and discard remaining boxes with IoU above a threshold with it; usually per class.
  3. What are anchor boxes and why use them? Predefined boxes of several sizes and aspect ratios at each grid position; the network predicts offsets and objectness relative to them, which makes learning box shapes easier.
  4. One-stage vs two-stage detectors? Two-stage (Faster R-CNN) proposes regions then classifies them; one-stage (YOLO) predicts boxes and classes in one pass, usually faster.
  5. What is mAP@0.5 vs COCO mAP? mAP@0.5 counts a match at IoU ≥ 0.5; COCO's main AP averages over IoU 0.50-0.95 in steps of 0.05, rewarding precise localisation.
  6. Semantic vs instance segmentation? Semantic labels pixel classes only; instance also separates individual objects of the same class.
  7. Why does U-Net need skip connections? Pooling loses spatial detail; copying encoder features to the decoder gives precise boundaries.

Glossary

Term Meaning (plain words + example)
Computer vision Getting computers to understand images: classify, detect, segment
Image classification One label for the whole image: "bean rust"
Object detection A box + label + confidence for every object in an image
Segmentation A label for every pixel
Semantic segmentation Each pixel's class: leaf (1) or background (0)
Instance segmentation Each pixel's class and which object: leaf 1, leaf 2, leaf 3
Mask An image-sized grid of per-pixel labels
Bounding box Smallest upright rectangle around an object, e.g. (50, 50, 150, 150)
(x1, y1, x2, y2) / (x, y, w, h) Corner format / corner + size format of a box
Confidence score How sure the detector is that a box holds an object (0 to 1)
Intersection The overlapping area of two boxes
Union The area covered by either box: area A + area B − intersection
IoU (intersection over union) Intersection ÷ union: 1 = perfect, 0 = no overlap; shifted box = 0.5625
Non-maximum suppression (NMS) Keep the best box, drop boxes overlapping it above a threshold, repeat
Anchor box A fixed reference box (size + shape) at a grid position that the network adjusts
Aspect ratio Height ÷ width of a box: 1:2, 1:1, 2:1
Offset The small shift and stretch from an anchor to the real box
Faster R-CNN Ren, He, Girshick, Sun (2015): two-stage detector with 9 anchors per position
YOLO "You Only Look Once" (Redmon et al., 2015): one network, one pass, S × S grid, 45 fps
Grid cell (YOLO) One of the S × S squares; predicts boxes for objects centred in it
PASCAL VOC A detection benchmark with 20 object classes
COCO A large detection benchmark whose main metric averages AP over IoU 0.50-0.95
True / false positive (detection) A prediction that matches an unmatched true box at the IoU threshold / one that does not
Precision-recall curve Precision vs recall as we accept more and more predictions
Average precision (AP) Area under the precision-recall curve for one class
mAP (mean average precision) Mean of AP over classes (and, for COCO, over IoU thresholds)
Flood fill Spreading a label to all touching pixels of the same kind
U-Net Ronneberger, Fischer, Brox (2015): contracting path + expanding path + skip connections
Contracting / expanding path The down half (smaller maps, more channels) / the up half (back to full size)
Transposed convolution A learned upsampling layer, nn.ConvTranspose2d(..., 2, stride=2) doubles height and width
Skip connection (U-Net) Copying a down-level's maps across and concatenating them on the way up

Summary

  • Classification: one label; detection: boxes + labels + scores; segmentation: a label per pixel (semantic or instance).
  • IoU = intersection ÷ union; clamp the overlap at 0 or far-apart boxes give −9.000.
  • NMS keeps the best box and removes its copies: 6 boxes → 2 leaves.
  • Anchor boxes: 9 shapes per position (Faster R-CNN), the network learns small offsets.
  • YOLO: one network, one look, a 7 × 7 × 30 output on PASCAL VOC, 45 fps.
  • AP = area under the precision-recall curve (0.688 at IoU 0.5, 0.463 COCO-style); mAP = mean over classes.
  • U-Net: down, up, skip connections: a 64 × 64 photo → a 64 × 64 mask.

Next episode: RNNs and sequences: sequence data, recurrent neural networks, the hidden state, unrolling through time, backpropagation through time and vanishing gradients.