Machine Learning Full Course · Episode 58. Every code sample below was run with Python 3.13, NumPy 2.5.3 and PyTorch 2.15.0 (the CPU build; the recording machine has no NVIDIA GPU). The output shown is the real output. Timings change from machine to machine; the numbers computed do not.
The big idea: hand-written backpropagation breaks silently. Swap sigmoid for tanh, forget to change the derivative, and the gradient comes out 0 where the truth is −4, with no error. PyTorch fixes this with two things. A tensor is a grid of numbers with a shape, a dtype and a device (CPU or GPU). Autograd records every operation done to a tensor with requires_grad=True; calling backward() on the loss walks that record backwards (backpropagation) and puts each gradient in .grad. You only write the forward pass. Three traps come with it: gradients accumulate (−2, −4, −6), the graph is freed after backward(), and updating a weight in place while recording is refused. The fixes: update inside torch.no_grad(), empty gradients with .grad.zero_(), and detach() before converting to NumPy.
Sources: PyTorch was launched by Facebook (now Meta) in 2016, and in September 2022 it moved to the PyTorch Foundation, part of the Linux Foundation (Engadget, 12 Sep 2022). NVIDIA introduced CUDA, an architecture for computing on its GPUs, on 16 November 2006 (Computer Graphics World).
Words you need first
- Array, NumPy, shape, broadcasting (Episode 2): NumPy stores numbers in grids called arrays; broadcasting stretches smaller arrays to match bigger ones.
- Matrix product
@(Episode 4): rows of the left matrix times columns of the right. - Derivative, chain rule, numerical derivative (Episode 6).
- Computational graph (Episode 7): a diagram of operations; each node is one step of the calculation.
- Loss, gradient descent, learning rate (Episode 8).
- Backpropagation, local gradient, weight update rule (Episode 57): multiply local gradients from the loss backwards; w ← w − lr × dL/dw.
- tanh (Episode 56): an activation function squashing any number into −1 … 1; its slope is 1 − a².
The problem: hand-written derivatives break silently
import math # last episode: hand-written backprop for one neuron
x, y, w, b = 2.0, 1.0, 0.5, -1.0
act = lambda z: math.tanh(z) # we switched sigmoid -> tanh...
dact = lambda a: a * (1 - a) # ...but forgot to change its derivative
def loss(w, b): return (act(w * x + b) - y) ** 2
a = act(w * x + b)
grad_w = 2 * (a - y) * dact(a) * x # chain rule, by hand
num_w = (loss(w + 1e-5, b) - loss(w - 1e-5, b)) / 2e-5
print(f"hand-written dL/dw = {grad_w + 0:.6f}")
print(f"numerical dL/dw = {num_w:.6f}")
print("match!" if abs(grad_w - num_w) < 1e-4 else "WRONG gradient, and no error message")
Output:
hand-written dL/dw = 0.000000
numerical dL/dw = -4.000000
WRONG gradient, and no error message
z = 0.5 × 2 − 1 = 0, so a = tanh(0) = 0 and the wrong derivative a(1 − a) is 0. The true slope of tanh at 0 is 1 − a² = 1, giving 2 × (0 − 1) × 1 × 2 = −4. Training with the wrong value would never move w.
Why: the derivative was typed by a person, separately from the model. Change one, forget the other, and they drift apart without an error. Also, NumPy arrays only run on the CPU.
What we want: arrays that remember how they were made (so the computer can differentiate them itself) and that can run on a graphics card. That is PyTorch.
PyTorch and the tensor
- PyTorch: a free, open-source deep learning library for Python. Install it with
pip install torch; import it astorch. - Tensor: a grid of numbers with any number of dimensions. NumPy's array, plus a device and gradient tracking.
| Real life | Tensor | Shape |
|---|---|---|
| The price of one chai: 15 | 0-D tensor, a scalar | [] |
| A bill: 15, 20, 40 | 1-D tensor, a vector | [3] |
| A train timetable | 2-D tensor, a matrix | [rows, cols] |
| A stack of 32 colour photos | 4-D tensor | [32, 3, 224, 224] (photo, colour, height, width) |
import torch
t = torch.tensor([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]]) # a tensor: a grid of numbers
print(t)
print("shape:", t.shape, " ndim:", t.ndim, " dtype:", t.dtype)
i = torch.tensor([1, 2, 3]) # whole numbers -> int64
print("i.dtype:", i.dtype, " t.int():", t.int().dtype, " half:", t.half().dtype)
print("zeros:", torch.zeros(2, 2).tolist(), " arange:", torch.arange(5).tolist())
print("random:", torch.rand(3).shape, torch.randn(2, 3).shape)
print(torch.__version__)
Output:
tensor([[1., 2., 3.],
[4., 5., 6.]])
shape: torch.Size([2, 3]) ndim: 2 dtype: torch.float32
i.dtype: torch.int64 t.int(): torch.int32 half: torch.float16
zeros: [[0.0, 0.0], [0.0, 0.0]] arange: [0, 1, 2, 3, 4]
random: torch.Size([3]) torch.Size([2, 3])
2.15.0+cu132
(+cu132 is part of the version name of this PyTorch build; it does not mean a GPU is present.)
Shape and dtype
- shape: the size along each dimension, e.g.
torch.Size([2, 3]). ndim: the number of dimensions. - dtype (data type): the type of every number inside; one dtype per tensor.
| dtype | Bytes per number | Default for | Example |
|---|---|---|---|
torch.float32 |
4 | decimals | weights, inputs |
torch.float64 |
8 | NumPy decimals | torch.from_numpy(np.array([1.0])) |
torch.float16 (.half()) |
2 | mixed precision (episode 64) | |
torch.int64 ("long") |
8 | whole numbers | class labels |
torch.bool |
1 | comparisons | t > 2 |
float32 is precise enough for neural networks and fast on GPUs. Mixing dtypes in + promotes the result, but a matrix product needs the same dtype on both sides:
import torch
a = torch.tensor([1, 2, 3]) # int64
b = torch.tensor([0.5, 0.5, 0.5]) # float32
print(a + b, (a + b).dtype) # mixing is fine: promoted to float
W = torch.randn(3, 3)
print(a @ W) # but a matrix product needs the same dtype
Output:
tensor([1.5000, 2.5000, 3.5000]) torch.float32
Traceback (most recent call last):
File "main.py", line 6, in <module>
print(a @ W) # but a matrix product needs the same dtype
~~^~~
RuntimeError: expected m1 and m2 to have the same dtype, but got: long int != float
Fix: a.float() @ W. "long int" is PyTorch's internal name for int64.
Reshape, view and indexing
Twelve eggs can sit in a line, in a 3 × 4 tray or a 4 × 3 tray: the same eggs, arranged differently.
- reshape: the same numbers in a new shape. -1 means "work this size out for me".
- view: a new shape on the same memory, no copy. Change the view and the original changes.
- Indexing: picking parts of a tensor:
m[1](row 1),m[:, 2](column 2),m[m > 8](a boolean mask: keep the elements where the condition is True).
import torch
t = torch.arange(12) # 0, 1, ..., 11: shape [12]
m = t.reshape(3, 4) # same 12 numbers, as 3 rows x 4 columns
print(m)
v = t.view(4, 3); v[0, 0] = 100 # view: a new shape on the SAME memory
print("t[0] after changing v:", t[0].item())
print("m[1]:", m[1].tolist(), " m[:, 2]:", m[:, 2].tolist(), " m[m > 8]:", m[m > 8].tolist())
print("reshape(2, -1):", t.reshape(2, -1).shape)
print(m.T.is_contiguous()); m.T.view(12) # transposed: not one tidy block of memory
Output:
tensor([[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11]])
t[0] after changing v: 100
m[1]: [4, 5, 6, 7] m[:, 2]: [2, 6, 10] m[m > 8]: [100, 9, 10, 11]
reshape(2, -1): torch.Size([2, 6])
False
Traceback (most recent call last):
File "main.py", line 9, in <module>
print(m.T.is_contiguous()); m.T.view(12) # transposed: not one tidy block of memory
~~~~~~~~^^^^
RuntimeError: view size is not compatible with input tensor's size and stride (at least one dimension spans across two contiguous subspaces). Use .reshape(...) instead.
m was a reshape of a contiguous tensor, so it is also a view of the same memory: that is why 100 appears in m[m > 8].
Contiguous means the numbers are stored in memory in reading order (row after row). .T (transpose) swaps rows and columns by changing how memory is read, without moving numbers, so the result is not contiguous. view only works on contiguous memory; reshape returns a view when it can and a copy when it must. .contiguous() makes a contiguous copy.
Broadcasting
Two chai stalls (rows) sell chai, samosa and biscuits (columns). Tax is per item (one number per column); a discount is per shop (one number per row).
import torch
prices = torch.tensor([[40.0, 60.0, 25.0], # 2 shops x 3 items (chai, samosa, biscuit)
[45.0, 55.0, 30.0]])
gst = torch.tensor([1.05, 1.05, 1.12]) # one tax rate per item: shape [3]
print(prices * gst) # [2, 3] * [3] -> stretched over every row
discount = torch.tensor([0.9, 1.0]) # one discount per shop: shape [2]
print(prices * discount)
Output:
tensor([[42.0000, 63.0000, 28.0000],
[47.2500, 57.7500, 33.6000]])
Traceback (most recent call last):
File "main.py", line 7, in <module>
print(prices * discount)
~~~~~~~^~~~~~~~~~
RuntimeError: The size of tensor a (3) must match the size of tensor b (2) at non-singleton dimension 1
The fix is a column, shape [2, 1]:
discount = torch.tensor([[0.9], [1.0]]) # one discount per shop: shape [2, 1]
print(prices * discount) # [2, 3] * [2, 1] -> stretched over every column
Output:
tensor([[36.0000, 54.0000, 22.5000],
[45.0000, 55.0000, 30.0000]])
Broadcasting rules: line the shapes up from the right; each pair of sizes must be equal or one of them must be 1; a size of 1 (or a missing dimension) is stretched. [2, 3] × [3] ✓, [2, 3] × [2, 1] ✓, [2, 3] × [2] ✗ (3 vs 2). The stretch is virtual: no copies are made.
Devices: CPU, GPU and CUDA
| Wedding kitchen | Computer |
|---|---|
| A master chef: few hands, any recipe | CPU (central processing unit): a few powerful cores |
| 1,000 helpers chopping onions at once | GPU (graphics processing unit): thousands of simple cores, ideal for the millions of multiply-adds in a network |
| The way the helpers receive orders | CUDA: NVIDIA's platform for general computing on its GPUs (introduced 2006); in PyTorch the GPU device is called "cuda" |
import torch
print("GPU available?", torch.cuda.is_available())
device = "cuda" if torch.cuda.is_available() else "cpu" # the standard one-liner
x = torch.randn(1000, 1000, device=device)
print("x lives on:", x.device)
y = torch.randn(1000, 1000)
print((x @ y).shape, "fine: both on", device)
try:
g = torch.randn(3, device="cuda") # ask for a GPU that isn't there
except RuntimeError as e:
print("RuntimeError:", str(e).split(".")[0] + ".")
Output (this machine has no NVIDIA GPU):
GPU available? False
x lives on: cpu
torch.Size([1000, 1000]) fine: both on cpu
RuntimeError: Found no NVIDIA driver on your system.
Rules:
- Pick the device once with the one-liner; never hard-code
"cuda". - Move a tensor with
x = x.to(device)(it returns a new tensor). - Both sides of an operation must be on the same device; otherwise PyTorch raises "Expected all tensors to be on the same device".
- No NVIDIA card? Free notebook services such as Google Colab and Kaggle offer GPUs for practice.
NumPy interop
import numpy as np, torch
arr = np.array([1.0, 2.0, 3.0])
t = torch.from_numpy(arr) # shares memory with the NumPy array
c = torch.tensor(arr) # a copy
arr[0] = 99
print("from_numpy:", t.tolist(), " copy:", c.tolist(), " dtype:", t.dtype)
back = (t * 2).numpy() # tensor -> NumPy
print(type(back).__name__, back)
w = torch.tensor([1.0, 2.0], requires_grad=True)
print(w.numpy())
Output:
from_numpy: [99.0, 2.0, 3.0] copy: [1.0, 2.0, 3.0] dtype: torch.float64
ndarray [198. 4. 6.]
Traceback (most recent call last):
File "main.py", line 10, in <module>
print(w.numpy())
~~~~~~~^^
RuntimeError: Can't call numpy() on Tensor that requires grad. Use tensor.detach().numpy() instead.
torch.from_numpy(arr)shares memory (like a view);torch.tensor(arr)copies.- NumPy's default float is float64; convert with
.float()before mixing with float32 model weights. .numpy()needs a CPU tensor that does not require grad:x.detach().cpu().numpy().
Autograd
- requires_grad=True: "record every operation done to me".
- grad_fn: each result remembers the function that made it (
PowBackward0,SigmoidBackward0). Together these records are the computational graph. - backward(): called on the loss, walks the graph backwards multiplying local gradients (backpropagation).
- .grad: where each gradient lands.
- Leaf tensor: a tensor you created yourself (like
w), not the result of an operation; gradients are stored on leaves. - Autograd = automatic differentiation.
import torch # the same neuron as episode 57, now with autograd
x, y = torch.tensor(2.0), torch.tensor(1.0)
w = torch.tensor(0.5, requires_grad=True) # "please track every operation on me"
b = torch.tensor(-1.0, requires_grad=True)
z = w * x + b
a = torch.sigmoid(z)
L = (a - y) ** 2
print("L =", L, " made by:", type(a.grad_fn).__name__)
L.backward() # backpropagation, done for us
print("dL/dw =", w.grad.item(), " dL/db =", b.grad.item())
a = torch.tanh(w * x + b) # switch to tanh: no derivative to rewrite
w.grad = None; ((a - y) ** 2).backward()
print("tanh: dL/dw =", w.grad.item())
Output:
L = tensor(0.2500, grad_fn=<PowBackward0>) made by: SigmoidBackward0
dL/dw = -0.5 dL/db = -0.25
tanh: dL/dw = -4.0
−0.5 and −0.25 are exactly episode 57's hand-written results; −4 matches the numerical check above.
The dynamic computation graph
PyTorch builds the graph while the code runs (define-by-run). Each run records only the operations that really happened, so ordinary Python if, for and while work inside a model, and you can print anything mid-way.
import torch
w = torch.tensor(3.0, requires_grad=True)
def f(w, steps): # ordinary Python: a loop and an if
out = w
for _ in range(steps):
out = out * w if out < 50 else out + 1
return out
for steps in (2, 5):
w.grad = None; y = f(w, steps); y.backward()
print(f"steps={steps}: y = {y.item():.0f}, dy/dw = {w.grad.item():.0f}")
Output:
steps=2: y = 27, dy/dw = 27
steps=5: y = 83, dy/dw = 108
Two steps: y = w³, dy/dw = 3w² = 27. Five steps: 3 → 9 → 27 → 81, then the if switches to +1 twice: y = w⁴ + 2, dy/dw = 4w³ = 108.
Autograd at scale
import numpy as np, torch, time; rng = np.random.default_rng(57)
X = rng.random((64, 784)) # the same network as episode 57
W1, W2 = rng.normal(0, .03, (784, 1000)), rng.normal(0, .03, (1000, 10))
Z = X @ W1; H = np.maximum(0, Z); O = H @ W2 # episode 57: backward pass by hand
dO = 2 * (O - O.mean()) / O.size; dW2 = H.T @ dO; dW1 = X.T @ (dO @ W2.T * (Z > 0))
Xt = torch.from_numpy(X) # now PyTorch: forward pass only
T1 = torch.from_numpy(W1).requires_grad_(); T2 = torch.from_numpy(W2).requires_grad_()
t = time.time(); L = (torch.relu(Xt @ T1) @ T2).var(correction=0); L.backward()
print(f"autograd: {T1.numel() + T2.numel():,} gradients in {(time.time() - t) * 1000:.0f} ms")
print("same as hand backprop?", np.allclose(T1.grad.numpy(), dW1), np.allclose(T2.grad.numpy(), dW2))
Output (one run):
autograd: 794,000 gradients in 9 ms
same as hand backprop? True True
requires_grad_() (with an underscore) switches tracking on in place. var(correction=0) is the plain average of squared deviations, matching NumPy's .var().
The training-loop traps
Trap 1 and 2: gradients accumulate; the graph is freed
import torch
w = torch.tensor(0.5, requires_grad=True)
for i in range(3):
L = (w * 2.0 - 1.5) ** 2 # the same loss every time
L.backward()
print(f"call {i + 1}: w.grad = {w.grad.item()}")
try:
L.backward() # once more, on a graph already used
except RuntimeError as e:
print("RuntimeError:", str(e).split(" (")[0])
Output:
call 1: w.grad = -2.0
call 2: w.grad = -4.0
call 3: w.grad = -6.0
RuntimeError: Trying to backward through the graph a second time
The true gradient is 2 × (0.5 × 2 − 1.5) × 2 = −2 each time, but .grad adds new gradients to what is already there. After backward(), PyTorch frees the values the graph saved, to save memory; running the forward pass again builds a new graph. (retain_graph=True keeps it, rarely needed.)
Trap 3: updating a leaf while recording
import torch
w = torch.tensor(0.5, requires_grad=True)
for step in range(3):
L = (w * 2.0 - 1.5) ** 2
L.backward()
w -= 0.1 * w.grad # the update rule from episode 57
Output:
Traceback (most recent call last):
File "main.py", line 6, in <module>
w -= 0.1 * w.grad # the update rule from episode 57
^^^^^^^^^^^^^^^^^
RuntimeError: a leaf Variable that requires grad is being used in an in-place operation.
An in-place operation changes a tensor's own memory (-=, zero_(), anything ending in _). Doing it on a recorded leaf would make the update part of the graph, so PyTorch refuses.
The fixes
import torch # gradient descent on one weight, the PyTorch way
w = torch.tensor(0.5, requires_grad=True)
for step in range(1, 31):
L = (w * 2.0 - 1.5) ** 2 # we want w * 2 = 1.5, so w = 0.75
L.backward()
with torch.no_grad(): # "this is an update, not part of the model"
w -= 0.1 * w.grad
w.grad.zero_() # empty the gradient for the next step
if step in (1, 2, 10, 30): print(f"step {step:2}: w = {w.item():.4f}, loss = {L.item():.6f}")
with torch.no_grad(): p = w * 2.0
print("inside no_grad, requires_grad =", p.requires_grad)
d = (w * 2.0).detach()
print("detach: requires_grad =", d.requires_grad, " numpy:", d.numpy())
Output:
step 1: w = 0.7000, loss = 0.250000
step 2: w = 0.7400, loss = 0.010000
step 10: w = 0.7500, loss = 0.000000
step 30: w = 0.7500, loss = 0.000000
inside no_grad, requires_grad = False
detach: requires_grad = False numpy: 1.5
with torch.no_grad():switches recording off inside the block: for updates and for predictions (saves memory and time)..grad.zero_()empties a gradient in place (next episode,optimizer.zero_grad()does it for every parameter).- Run the forward pass again every step for a fresh graph.
detach()returns the same numbers cut off from the graph.- Accumulation is a feature: it lets you sum gradients from several small batches (gradient accumulation, episode 64).
Common mistakes
| Mistake | Exact message / symptom | Why | Fix |
|---|---|---|---|
| Hand derivative not updated | hand-written dL/dw = 0.000000 vs numerical -4.000000 |
model and maths drift apart | autograd |
int64 @ float32 |
RuntimeError: expected m1 and m2 to have the same dtype, but got: long int != float |
one dtype per operation | a.float() @ W |
m.T.view(12) |
RuntimeError: view size is not compatible with input tensor's size and stride ... Use .reshape(...) instead. |
not contiguous | reshape or .contiguous().view |
[2, 3] * [2] |
RuntimeError: The size of tensor a (3) must match the size of tensor b (2) at non-singleton dimension 1 |
shapes align from the right | shape [2, 1] |
Hard-coded device="cuda" |
RuntimeError: Found no NVIDIA driver on your system. ... |
no GPU | torch.cuda.is_available() |
.numpy() on a tracked tensor |
RuntimeError: Can't call numpy() on Tensor that requires grad. Use tensor.detach().numpy() instead. |
still in the graph | detach() |
Calling backward() twice |
RuntimeError: Trying to backward through the graph a second time ... |
graph freed | rerun the forward pass |
w -= lr * w.grad while recording |
RuntimeError: a leaf Variable that requires grad is being used in an in-place operation. |
the update would be recorded | with torch.no_grad(): |
Never zeroing .grad |
-2.0, -4.0, -6.0 |
gradients accumulate | w.grad.zero_() |
Mini project: XOR with tensors and autograd
Episode 57's XOR network, with no hand-written backward pass:
import torch; torch.manual_seed(58) # mini project: XOR with tensors + autograd (episode 57, no hand backprop)
X = torch.tensor([[0., 0.], [0., 1.], [1., 0.], [1., 1.]]); y = torch.tensor([[0.], [1.], [1.], [0.]])
W1 = torch.empty(2, 3).uniform_(-1, 1).requires_grad_(); b1 = torch.zeros(3, requires_grad=True)
W2 = torch.empty(3, 1).uniform_(-1, 1).requires_grad_(); b2 = torch.zeros(1, requires_grad=True)
params = [W1, b1, W2, b2]
for epoch in range(1, 5001):
out = torch.sigmoid(torch.sigmoid(X @ W1 + b1) @ W2 + b2)
loss = ((out - y) ** 2).mean()
loss.backward()
with torch.no_grad():
for p in params: p -= 2.0 * p.grad; p.grad.zero_()
if epoch in (1, 1000, 5000): print(f"epoch {epoch}: loss {loss.item():.4f}")
print("outputs:", [round(v, 3) for v in out.detach().squeeze(1).tolist()])
Output:
epoch 1: loss 0.2820
epoch 1000: loss 0.0030
epoch 5000: loss 0.0003
outputs: [0.009, 0.984, 0.981, 0.021]
Zero, one, one, zero: XOR learned, and we never wrote a derivative.
Practice
1. Make a random tensor of shape [32, 3, 28, 28] and reshape it to [32, -1].
Solution
import torch
x = torch.rand(32, 3, 28, 28) # 32 colour images, 28 x 28 pixels
flat = x.reshape(32, -1)
print(x.shape, "->", flat.shape) # 3 * 28 * 28 = 2352
Output:
torch.Size([32, 3, 28, 28]) -> torch.Size([32, 2352])
2. Use autograd to find dy/dx of y = x³ + 2x at x = 2.
Solution
import torch
x = torch.tensor(2.0, requires_grad=True)
y = x ** 3 + 2 * x
y.backward()
print("y =", y.item(), " dy/dx =", x.grad.item()) # 3x^2 + 2 = 14
Output:
y = 12.0 dy/dx = 14.0
3. Fit w and b so that w·x + b matches y = 3x + 1.
Solution
import torch; torch.manual_seed(58)
x = torch.linspace(-1, 1, 50); y = 3 * x + 1 # data from the line y = 3x + 1
w = torch.zeros((), requires_grad=True); b = torch.zeros((), requires_grad=True)
for step in range(1, 301):
loss = ((w * x + b - y) ** 2).mean()
loss.backward()
with torch.no_grad():
w -= 0.1 * w.grad; b -= 0.1 * b.grad
w.grad.zero_(); b.grad.zero_()
if step in (1, 100, 300): print(f"step {step}: w = {w.item():.4f}, b = {b.item():.4f}, loss = {loss.item():.6f}")
Output:
step 1: w = 0.2082, b = 0.2000, loss = 4.122449
step 100: w = 2.9977, b = 1.0000, loss = 0.000002
step 300: w = 3.0000, b = 1.0000, loss = 0.000000
4. Break it on purpose: remove the two zero_() calls and watch.
Solution
import torch
x = torch.linspace(-1, 1, 50); y = 3 * x + 1
w = torch.zeros((), requires_grad=True); b = torch.zeros((), requires_grad=True)
for step in range(1, 301):
loss = ((w * x + b - y) ** 2).mean()
loss.backward()
with torch.no_grad():
w -= 0.1 * w.grad; b -= 0.1 * b.grad
# forgot: w.grad.zero_(); b.grad.zero_()
if step in (1, 10, 50, 300): print(f"step {step}: w = {w.item():.4f}, b = {b.item():.4f}, loss = {loss.item():.4f}")
Output:
step 1: w = 0.2082, b = 0.2000, loss = 4.1224
step 10: w = 5.8241, b = 0.9760, loss = 2.2501
step 50: w = 0.8376, b = 1.7253, loss = 3.3575
step 300: w = 5.0037, b = 1.9260, loss = 3.2670
The accumulated gradients keep pushing long after the answer is passed: w swings around 3 and the loss never settles.
Quick quiz
w = torch.tensor(2.0, requires_grad=True)
for _ in range(2):
L = w * w
L.backward()
print(w.grad)
Answer
tensor(8.) dL/dw = 2w = 4 each time; each loop builds a fresh graph (no error), and .grad accumulates: 4 + 4 = 8.
More questions:
- What is the shape of a batch of 16 grayscale 28 × 28 images?
Answer
[16, 1, 28, 28](one colour channel). - What dtype does
torch.tensor([1, 2])have? Andtorch.tensor([1.0, 2.0])?Answer
int64 and float32. - Can
[4, 1]and[3]broadcast? To what shape?Answer
Yes: from the right, 1 vs 3 (stretch) and 4 vs missing (stretch):[4, 3]. - Why does
torch.from_numpy(arr)change whenarrchanges?Answer
It shares memory with the NumPy array instead of copying. - What does
torch.no_grad()do?Answer
Turns off recording inside its block, so results do not require grad and no graph is built.
Interview questions
- What is a tensor in PyTorch? An n-dimensional array with a shape, a dtype and a device, which can also record operations for automatic differentiation.
- view vs reshape?
viewalways shares memory and needs a contiguous tensor;reshapereturns a view when possible and copies otherwise. - Explain broadcasting. Shapes are aligned from the right; dimensions must be equal or 1; size-1 or missing dimensions are virtually expanded.
- How does autograd work? Operations on tensors with
requires_grad=Truebuild a graph ofgrad_fnnodes during the forward pass;backward()traverses it in reverse applying the chain rule and accumulates gradients into leaf tensors'.grad. - What does dynamic computation graph mean? The graph is built anew on every forward pass as the code runs, so Python control flow is allowed and the graph matches what actually executed.
- Why must you zero gradients?
.gradaccumulates acrossbackward()calls; without zeroing, each step uses the sum of all past gradients. - Difference between
torch.no_grad()anddetach()?no_gradis a context that stops recording for everything inside;detach()returns one tensor cut off from the graph.
Glossary
| Term | Meaning (plain words + example) |
|---|---|
| PyTorch | Open-source Python library for deep learning; import torch |
| Tensor | A grid of numbers with any number of dimensions: torch.tensor([[1., 2.], [3., 4.]]) |
| Scalar / vector / matrix | 0-D, 1-D and 2-D tensors: one chai price, a bill, a timetable |
| Shape | Size along each dimension: torch.Size([2, 3]) |
| ndim | Number of dimensions: 2 for a matrix |
| dtype | Type of every number in a tensor: torch.float32, torch.int64 |
| float32 / float64 / float16 | Decimal numbers using 4 / 8 / 2 bytes each |
| int64 (long) | 8-byte whole numbers, the default for integer tensors and class labels |
| reshape | Same numbers, new shape: arange(12).reshape(3, 4) |
| view | A new shape on the same memory: changing it changes the original |
| -1 in reshape | "Work this size out": reshape(2, -1) → [2, 6] |
| Contiguous | Stored in memory in reading order; .T makes a tensor non-contiguous |
Transpose (.T) |
Swap rows and columns |
| Indexing | Picking parts: m[1], m[:, 2] |
| Boolean mask | A True/False tensor used to select elements: m[m > 8] |
| Broadcasting | Automatic stretching of size-1 or missing dimensions: [2, 3] * [3] |
| Device | Where a tensor lives: "cpu" or "cuda" |
| CPU | Central processing unit: a few powerful cores |
| GPU | Graphics processing unit: thousands of simple cores working in parallel |
| CUDA | NVIDIA's platform for computing on its GPUs; PyTorch's name for the GPU device |
.to(device) |
Returns a copy of a tensor on another device |
| NumPy interop | Moving data between NumPy and PyTorch: torch.from_numpy, .numpy() |
torch.from_numpy |
Wraps a NumPy array without copying (shared memory) |
| Autograd | Automatic differentiation: PyTorch computes gradients from the recorded forward pass |
| requires_grad | Flag telling PyTorch to record operations on a tensor: torch.tensor(0.5, requires_grad=True) |
| grad_fn | The recorded function that produced a tensor: SigmoidBackward0 |
| Computational graph | The record of operations from inputs to the loss |
| Dynamic computation graph | A graph rebuilt on every run as the code executes (define-by-run) |
| backward() | Runs backpropagation from a scalar loss and fills .grad |
| .grad | Where a leaf tensor's gradient is stored |
| Leaf tensor | A tensor created by you (not by an operation), such as a weight |
| Gradient accumulation | .grad adds new gradients to old ones: −2, −4, −6 |
| In-place operation | Changes a tensor's own memory: w -= ..., zero_() |
| torch.no_grad() | A block in which nothing is recorded: updates and predictions |
| zero_() | Sets a tensor to zero in place: w.grad.zero_() |
| detach() | Same numbers, cut off from the graph: x.detach().numpy() |
.item() |
Turns a one-number tensor into a Python number |
Summary
- Hand-written derivatives break silently when the model changes; autograd removes them.
- A tensor has a shape, a dtype and a device;
viewshares memory,reshapeis the safe choice, and broadcasting stretches size-1 dimensions from the right. - Pick the device with
torch.cuda.is_available(); keep both sides of an operation on the same device.from_numpyshares memory. requires_grad=True+ forward pass +loss.backward()→ gradients in.grad, identical to hand backprop (794,000 of them in milliseconds).- Update inside
torch.no_grad(), zero gradients every step, rerun the forward pass, anddetach()before NumPy.
Next episode: PyTorch 2: nn.Module and the training loop: layers, loss functions, optimisers and the full training loop.