SyntaxPilot
Courses/Machine Learning/Lesson 60
Video coming soonThis lesson’s video is being made

Start with the notes below; the video appears here as soon as it is on YouTube.

Machine Learning · Lesson 60 of 90

PyTorch 3: data loading

Dataset class__len__ and __getitem__DataLoaderbatch_sizeshufflingdrop_lastcollate_fnnum_workerstransformstrain and validation loaders

Machine Learning Full Course · Episode 60. Every code sample below was run with Python 3.13 and PyTorch 2.15.0 (CPU, 2 cores). The output shown is the real output; timings vary from machine to machine and run to run.

The big idea: last episode every batch was a slice of one tensor already in memory. Real data breaks that: 50,000 photos would need 30.1 GB as one float32 tensor, hand-cut batches leave a short last batch (40 instead of 64), and sentences of different lengths can't even become one tensor. PyTorch splits the job in two. A Dataset answers two questions: how many examples (__len__) and "give me example number i" (__getitem__), loading each example only when asked. A DataLoader turns a Dataset into batches: batch_size, shuffle (a new order every epoch), drop_last, a collate_fn that joins examples into a batch (we pad uneven sentences), and num_workers helper processes that load in parallel (2.61 s → 1.36 s per epoch here). Transforms fix or randomly vary (augment) each example inside __getitem__. Training and validation each get their own loader.

Source: torchvision, PyTorch's computer-vision library, provides ready-made transforms in torchvision.transforms.v2, including Compose, Normalize and RandomHorizontalFlip (torchvision docs).

Words you need first

  • Tensor, shape, dtype, padding a shape (Episode 58).
  • Training loop, batch, iteration, epoch, model.train() / model.eval() (Episode 59).
  • Train / validation / test split (Episode 15): validation data is kept aside to measure the model while you build it.
  • Normalisation (Episode 16): rescaling numbers to a standard range, e.g. −1 … 1.
  • CSV file: a text file of rows with values separated by commas.
  • Class, method: a Python blueprint and the functions that belong to it.
  • Process: a running program; several processes can work at the same time.

The problem: loading everything and batching by hand

import torch
n, h, w = 50_000, 224, 224                        # 50,000 colour photos, 224 x 224 pixels
bytes_needed = n * 3 * h * w * 4                  # 3 colours, float32 = 4 bytes each
print(f"all photos as one tensor: {bytes_needed / 1e9:.1f} GB of memory")
X = torch.arange(1000.0).unsqueeze(1)             # 1,000 examples that DO fit
batches = [X[i:i + 64] for i in range(0, 1000, 64)]   # hand-made batches of 64
print("batches:", len(batches), "| last 3 sizes:", [len(b) for b in batches[-3:]])
sentences = [[4, 9, 2], [7, 1, 8, 3, 5]]          # two sentences as word ids: lengths 3 and 5
try:
    print(torch.tensor(sentences))
except ValueError as e:
    print("ValueError:", e)

Output:

all photos as one tensor: 30.1 GB of memory
batches: 16 | last 3 sizes: [64, 64, 40]
ValueError: expected sequence of length 3 at dim 1 (got 5)
  • 50,000 × 3 × 224 × 224 × 4 bytes = 30.1 GB: more than most laptops have.
  • 1,000 is not a multiple of 64, so the last batch has 40 examples.
  • A tensor must be a perfect rectangle; sentences of 3 and 5 word ids are not. (Word id: a number standing for a word in a vocabulary.)

Why: we loaded all the data before training and wrote the batching ourselves. What we want: load one example only when needed, and a helper that groups, shuffles and batches them.

The Dataset class

A library: the catalogue says how many books there are (__len__); the librarian fetches book No. 25 only when asked (__getitem__); a helper with a trolley brings 4 random books per trip (the DataLoader).

  • Dataset (from torch.utils.data import Dataset): a class that answers two questions.
  • __len__(self): how many examples are there? len(ds) calls it.
  • __getitem__(self, i): return example number i, usually as tensors (x, y). ds[i] calls it.
  • Dunder methods: names with double underscores on both sides; Python calls them for built-in syntax.
  • __init__ keeps only cheap things (file names, rows); the heavy loading (opening a photo) happens in __getitem__. This is lazy loading.
import torch, csv
from torch.utils.data import Dataset
with open("marks.csv", "w") as f:                  # a small file: study hours, sleep hours, passed?
    f.write("hours,sleep,passed\n" + "".join(f"{h},{s},{int(h + s / 2 > 7)}\n"
                                              for h in range(10) for s in range(4, 10)))
class MarksDataset(Dataset):
    def __init__(self, path):
        with open(path) as f: self.rows = list(csv.DictReader(f))   # keep the rows, not tensors
    def __len__(self): return len(self.rows)       # how many examples?
    def __getitem__(self, i):                      # give me example number i, as tensors
        r = self.rows[i]
        x = torch.tensor([float(r["hours"]), float(r["sleep"])])
        return x, torch.tensor(float(r["passed"]))
ds = MarksDataset("marks.csv")
print("len(ds):", len(ds))
print("ds[0]: ", ds[0], "  ds[59]:", ds[59])
x, y = ds[25]; print("ds[25] -> x =", x.tolist(), " y =", y.item())

Output:

len(ds): 60
ds[0]:  (tensor([0., 4.]), tensor(0.))   ds[59]: (tensor([9., 9.]), tensor(1.))
ds[25] -> x = [4.0, 5.0]  y = 0.0

Forgetting __len__:

import torch
from torch.utils.data import Dataset, DataLoader
class NoLen(Dataset):                              # forgot __len__
    def __getitem__(self, i): return torch.tensor([float(i)])
try:
    for xb in DataLoader(NoLen(), batch_size=4, shuffle=True): pass
except Exception as e:
    print(type(e).__name__ + ":", e)

Output:

TypeError: object of type 'NoLen' has no len()

To shuffle, the loader must know how many example numbers exist.

The DataLoader

import torch
from torch.utils.data import TensorDataset, DataLoader
torch.manual_seed(60)
X = torch.arange(10).unsqueeze(1); y = X.squeeze(1) * 2     # 10 tiny examples: x -> 2x
ds = TensorDataset(X, y)                                    # a ready-made Dataset for tensors
loader = DataLoader(ds, batch_size=4)                       # no shuffle
print("batches per epoch:", len(loader))
for xb, yb in loader: print("  x:", xb.squeeze(1).tolist(), " shape:", list(xb.shape))
for epoch in (1, 2):
    order = [xb.squeeze(1).tolist() for xb, _ in DataLoader(ds, batch_size=4, shuffle=True)]
    print(f"shuffle, epoch {epoch}:", order)
print("drop_last=True:", [len(xb) for xb, _ in DataLoader(ds, batch_size=4, drop_last=True)])

Output:

batches per epoch: 3
  x: [0, 1, 2, 3]  shape: [4, 1]
  x: [4, 5, 6, 7]  shape: [4, 1]
  x: [8, 9]  shape: [2, 1]
shuffle, epoch 1: [[0, 2, 5, 3], [6, 4, 7, 9], [8, 1]]
shuffle, epoch 2: [[3, 0, 7, 1], [5, 4, 2, 8], [6, 9]]
drop_last=True: [4, 4]
Option Meaning Typical value
batch_size examples per batch 32, 64, 128
shuffle a new random order every epoch True for training, False for validation
drop_last skip the short final batch True when every batch must be full
collate_fn function that joins a list of examples into one batch default stacks tensors
num_workers helper processes loading in parallel 0 (main process) … number of CPU cores

len(loader) is the number of batches = iterations per epoch: ⌈10 / 4⌉ = 3, or ⌊10 / 4⌋ = 2 with drop_last. TensorDataset(X, y) is a ready-made Dataset whose example i is (X[i], y[i]).

collate_fn: building a batch

The loader picks example numbers, calls __getitem__ for each, then gives the list to a collate function, which joins them into one batch. The default stacks tensors, so every example must have the same shape.

import torch
from torch.utils.data import DataLoader
sentences = [torch.tensor([4, 9, 2]), torch.tensor([7, 1, 8, 3, 5]), torch.tensor([6, 2])]   # word ids
try:
    next(iter(DataLoader(sentences, batch_size=3)))       # default: stack them into one tensor
except RuntimeError as e:
    print("RuntimeError:", e)
def pad_collate(batch):                                  # our own way to build a batch
    longest = max(len(s) for s in batch)
    padded = torch.zeros(len(batch), longest, dtype=torch.long)    # 0 = padding
    for i, s in enumerate(batch): padded[i, :len(s)] = s
    lengths = torch.tensor([len(s) for s in batch])
    return padded, lengths
xb, lengths = next(iter(DataLoader(sentences, batch_size=3, collate_fn=pad_collate)))
print(xb); print("lengths:", lengths.tolist())

Output:

RuntimeError: stack expects each tensor to be equal size, but got [3] at entry 0 and [5] at entry 1
tensor([[4, 9, 2, 0, 0],
        [7, 1, 8, 3, 5],
        [6, 2, 0, 0, 0]])
lengths: [3, 5, 2]

Padding: filling short examples with a placeholder (0) up to the longest length. The lengths tell the model which positions are real. (A plain Python list works as a Dataset too: it has len() and [i]. next(iter(loader)) takes just the first batch.)

Transforms and data augmentation

  • Transform: a function applied to each example inside __getitem__, as it is loaded.
  • Preprocessing: fixed changes such as scaling pixels 0 … 255 to 0 … 1 and normalising to −1 … 1.
  • Data augmentation: random small changes (noise, flips, crops) so the model sees a slightly different copy every epoch: more variety without more data.
import torch; from torch.utils.data import Dataset
class Photos(Dataset):                                   # pretend photos: 4 pixels each, values 0..255
    def __init__(self, transform=None):
        g = torch.Generator().manual_seed(60)            # the same 6 "photos" every time
        self.data = torch.randint(0, 256, (6, 4), generator=g).float(); self.transform = transform
    def __len__(self): return len(self.data)
    def __getitem__(self, i):                            # the transform runs on every example
        return self.transform(self.data[i]) if self.transform else self.data[i]
def preprocess(x):                                       # fixed: scale to 0..1, then normalise to -1..1
    return (x / 255 - 0.5) / 0.5
def augment(x):                                          # random: preprocess + a little noise
    return preprocess(x) + 0.05 * torch.randn_like(x)
r = lambda t: [round(v, 3) for v in t.tolist()]
print("raw:        ", Photos()[0].tolist())
print("transformed:", r(Photos(preprocess)[0]))
aug = Photos(augment)
print("augmented 1:", r(aug[0]))
print("augmented 2:", r(aug[0]), "<- same photo, new noise")

Output (the noise differs each run):

raw:         [205.0, 65.0, 70.0, 138.0]
transformed: [0.608, -0.49, -0.451, 0.082]
augmented 1: [0.651, -0.319, -0.462, 0.111]
augmented 2: [0.515, -0.446, -0.54, 0.103] <- same photo, new noise

205 / 255 = 0.804; (0.804 − 0.5) / 0.5 = 0.608. For real images, torchvision.transforms.v2 offers Compose, Normalize, RandomHorizontalFlip and many more. Augment only the training data; validation data gets the fixed preprocessing only.

num_workers: loading in parallel

With num_workers=0 the main program loads every example itself and the model waits. With num_workers=N, N helper processes (workers) prepare batches in parallel.

import torch, time
from torch.utils.data import Dataset, DataLoader
class SlowDisk(Dataset):                                 # each example takes 10 ms to read, like a file
    def __len__(self): return 256
    def __getitem__(self, i):
        time.sleep(0.01)
        return torch.tensor([float(i)])
for workers in (0, 2):
    t = time.time()
    for xb in DataLoader(SlowDisk(), batch_size=32, num_workers=workers): pass
    print(f"num_workers={workers}: one epoch in {time.time() - t:.2f} s")

Output (this machine has 2 CPU cores):

num_workers=0: one epoch in 2.61 s
num_workers=2: one epoch in 1.36 s

Start near the number of CPU cores; PyTorch prints a UserWarning ("Our suggested max number of worker in current system is 2...") if you ask for more than it suggests. Workers cost memory and start-up time, so 0 is fine for small in-memory data. On Windows and macOS, put the training code under if __name__ == "__main__": when using workers.

Train and validation loaders

import torch, torch.nn as nn
from torch.utils.data import TensorDataset, DataLoader, random_split
torch.manual_seed(60)
X = torch.rand(1200, 2) * 4 - 2; y = (X.pow(2).sum(1) < 1).float().unsqueeze(1)   # episode 59's circle
train_ds, val_ds = random_split(TensorDataset(X, y), [1000, 200])        # 1,000 train, 200 validation
train_loader = DataLoader(train_ds, batch_size=100, shuffle=True)         # shuffle the training data
val_loader = DataLoader(val_ds, batch_size=100)                           # never shuffle validation
model = nn.Sequential(nn.Linear(2, 16), nn.ReLU(), nn.Linear(16, 16), nn.ReLU(), nn.Linear(16, 1))
loss_fn, opt = nn.BCEWithLogitsLoss(), torch.optim.SGD(model.parameters(), lr=0.5)
for epoch in range(1, 41):
    model.train()
    for xb, yb in train_loader:                                           # the loader hands us batches
        loss = loss_fn(model(xb), yb); opt.zero_grad(); loss.backward(); opt.step()
    model.eval(); correct = 0
    with torch.no_grad():
        for xb, yb in val_loader: correct += ((model(xb) > 0).float() == yb).sum().item()
    if epoch in (1, 10, 40): print(f"epoch {epoch:2}: validation accuracy {correct / len(val_ds):.1%}")
print("train batches:", len(train_loader), "| validation batches:", len(val_loader))

Output:

epoch  1: validation accuracy 80.5%
epoch 10: validation accuracy 96.5%
epoch 40: validation accuracy 97.0%
train batches: 10 | validation batches: 2
  • random_split(ds, [1000, 200]) returns two Subset datasets that pick random, non-overlapping example numbers.
  • Training loader: shuffle=True. Validation loader: no shuffling needed, only measuring.
  • Never train on validation data, or its score stops meaning anything (leakage, episode 15).

Common mistakes

Mistake Exact message / symptom Why Fix
Loading every photo into one tensor all photos as one tensor: 30.1 GB of memory memory load in __getitem__
Hand-made batches last 3 sizes: [64, 64, 40] leftovers DataLoader (+ drop_last)
Uneven lists to torch.tensor ValueError: expected sequence of length 3 at dim 1 (got 5) not a rectangle pad
No __len__ TypeError: object of type 'NoLen' has no len() shuffle needs the size add __len__
Uneven examples with default collate RuntimeError: stack expects each tensor to be equal size, but got [3] at entry 0 and [5] at entry 1 default stacks collate_fn with padding
Heavy loading in __init__ slow start, huge memory not lazy keep file names; open in __getitem__
Too many workers UserWarning: This DataLoader will create 4 worker processes in total... more workers than cores about the number of cores
Augmenting / training on validation data a validation score that means nothing leakage separate loaders, fixed preprocessing only

Mini project: train episode 59's model on MarksDataset

import torch, torch.nn as nn, csv
from torch.utils.data import Dataset, DataLoader, random_split
torch.manual_seed(60)
with open("marks.csv", "w") as f:
    f.write("hours,sleep,passed\n" + "".join(f"{h},{s},{int(h + s / 2 > 7)}\n" for h in range(10) for s in range(4, 10)))
class MarksDataset(Dataset):
    def __init__(self, path):
        with open(path) as f: self.rows = list(csv.DictReader(f))
    def __len__(self): return len(self.rows)
    def __getitem__(self, i):
        r = self.rows[i]
        x = torch.tensor([float(r["hours"]) / 9, float(r["sleep"]) / 9])   # scale to about 0..1
        return x, torch.tensor([float(r["passed"])])
train_ds, val_ds = random_split(MarksDataset("marks.csv"), [48, 12])
train_loader, val_loader = DataLoader(train_ds, batch_size=8, shuffle=True), DataLoader(val_ds, batch_size=12)
model = nn.Sequential(nn.Linear(2, 8), nn.ReLU(), nn.Linear(8, 1))
loss_fn, opt = nn.BCEWithLogitsLoss(), torch.optim.SGD(model.parameters(), lr=0.5)
for epoch in range(1, 301):
    model.train()
    for xb, yb in train_loader:
        loss = loss_fn(model(xb), yb); opt.zero_grad(); loss.backward(); opt.step()
    if epoch in (1, 100, 300):
        model.eval()
        with torch.no_grad():
            xb, yb = next(iter(val_loader)); acc = ((model(xb) > 0).float() == yb).float().mean()
        print(f"epoch {epoch:3}: validation accuracy {acc:.1%}")

Output:

epoch   1: validation accuracy 75.0%
epoch 100: validation accuracy 100.0%
epoch 300: validation accuracy 100.0%

The label is returned as [1.] (shape [1]) so a batch has shape [8, 1], matching the model's output.

Practice

1. Write a Dataset that reads one .txt file per example.

Solution
import torch, os
from torch.utils.data import Dataset, DataLoader
os.makedirs("reviews", exist_ok=True)                       # one small text file per example
for i, (text, label) in enumerate([("great chai", 1), ("cold and bland", 0), ("loved it", 1), ("too sweet", 0)]):
    with open(f"reviews/{i}_{label}.txt", "w") as f: f.write(text)
class ReviewFiles(Dataset):
    def __init__(self, folder):
        self.paths = sorted(os.path.join(folder, n) for n in os.listdir(folder))   # only the file names
    def __len__(self): return len(self.paths)
    def __getitem__(self, i):
        with open(self.paths[i]) as f: text = f.read()                          # open the file only now
        label = int(self.paths[i].rsplit("_", 1)[1][0])
        return text, torch.tensor(label)
ds = ReviewFiles("reviews")
print(len(ds), ds[1])
for texts, labels in DataLoader(ds, batch_size=2): print(texts, labels.tolist())

Output:

4 ('cold and bland', tensor(0))
('great chai', 'cold and bland') [1, 0]
('loved it', 'too sweet') [1, 0]

The default collate keeps strings as a tuple and stacks the label tensors.

2. Add a random flip transform (x.flip(0) with probability 0.5).

Solution
import torch
from torch.utils.data import Dataset
class Photos(Dataset):
    def __init__(self, transform=None):
        g = torch.Generator().manual_seed(60)
        self.data = torch.randint(0, 256, (6, 4), generator=g).float(); self.transform = transform
    def __len__(self): return len(self.data)
    def __getitem__(self, i):
        x = self.data[i]
        return self.transform(x) if self.transform else x
def random_flip(x, p=0.5):                                 # flip the pixel order with probability p
    return x.flip(0) if torch.rand(()) < p else x
torch.manual_seed(0)
ds = Photos(random_flip)
print("original:", Photos()[0].tolist())
for k in range(4): print(f"call {k + 1}:  ", ds[0].tolist())

Output:

original: [205.0, 65.0, 70.0, 138.0]
call 1:   [138.0, 70.0, 65.0, 205.0]
call 2:   [205.0, 65.0, 70.0, 138.0]
call 3:   [138.0, 70.0, 65.0, 205.0]
call 4:   [138.0, 70.0, 65.0, 205.0]

3. Time num_workers = 0, 1, 2 on your machine.

Solution

Use the SlowDisk program above with for workers in (0, 1, 2). Output on the 2-core recording machine:

num_workers=0: one epoch in 2.61 s
num_workers=1: one epoch in 2.69 s
num_workers=2: one epoch in 1.35 s

One worker just moves the loading to one helper process (no faster, plus a little start-up cost); two workers load two batches at once and roughly halve the time.

Quick quiz

len(ds) = 1,050, DataLoader(ds, batch_size=100, drop_last=True). What is len(loader)?

Answer

10. Ten full batches of 100; the leftover 50 are dropped. Without drop_last it would be 11.

More questions:

  1. Which two methods must a map-style Dataset define?
    Answer__len__ and __getitem__.
  2. Where should a photo be opened: __init__ or __getitem__?
    Answer__getitem__, one example at a time; __init__ keeps only the file names.
  3. What does collate_fn receive?
    AnswerA list of examples (whatever __getitem__ returned) for one batch.
  4. Should the validation loader shuffle?
    AnswerNot needed: it only measures; the order does not change the score.
  5. What is data augmentation?
    AnswerRandom, label-preserving changes to training examples (flips, noise, crops) so the model sees more variety.

Interview questions

  1. Dataset vs DataLoader? A Dataset gives single examples by index and knows its length; a DataLoader samples indices, batches with a collate function, shuffles and loads in parallel.
  2. Why lazy loading? Data larger than memory; only the current batch is in memory, and loading overlaps with training when workers are used.
  3. How do you batch variable-length sequences? A custom collate_fn that pads to the longest in the batch and returns lengths (or an attention mask).
  4. What does num_workers do, and trade-offs? Spawns worker processes that prepare batches in parallel; faster for I/O- or transform-heavy data but costs memory and start-up time.
  5. What is drop_last for? Dropping the final incomplete batch, useful when layers or losses assume a fixed batch size.

Glossary

Term Meaning (plain words + example)
Dataset A class giving example number i and the number of examples: MarksDataset
__len__ Method answering "how many examples?": len(ds) → 60
__getitem__ Method returning example i: ds[25] → (x, y)
Dunder method A method named with double underscores that Python calls for built-in syntax
Lazy loading Loading each example only when it is requested
TensorDataset Ready-made Dataset over existing tensors: TensorDataset(X, y)
DataLoader Turns a Dataset into batches: shuffles, collates, uses workers
batch_size Number of examples per batch: 4, 100
Shuffling A new random order of examples every epoch (shuffle=True)
drop_last Skip the final short batch: [4, 4, 2] → [4, 4]
Collate function (collate_fn) Joins a list of examples into one batch; the default stacks tensors
Padding Filling short sequences with a placeholder (0) to a common length
Word id A number standing for a word in a vocabulary
Transform A function applied to each example as it is loaded: scale, normalise
Compose Chaining several transforms into one
Data augmentation Random small changes to training examples: noise, flips
num_workers Number of helper processes loading batches in parallel
Worker A helper process that runs __getitem__ and collate for the loader
random_split Splits a Dataset into random non-overlapping parts: [1000, 200]
Subset A Dataset that is a selection of another Dataset's examples
Training loader Shuffled loader the model learns from
Validation loader Loader used only to measure the model during development
torchvision PyTorch's computer-vision library, with datasets and transforms

Summary

  • Don't load everything and slice by hand: memory runs out, leftovers and uneven examples break things.
  • A Dataset answers __len__ and __getitem__, loading lazily.
  • A DataLoader gives batches: batch_size, shuffle, drop_last, a collate_fn (padding for uneven data) and num_workers.
  • Transforms preprocess and augment each example on the way in.
  • Train with a shuffled training loader; measure with a separate validation loader.

Next episode: Optimisers: momentum, RMSProp, Adam and AdamW, compared on loss curves.