Machine Learning Full Course · Episode 69. Every code sample below was run with Python 3.13 and PyTorch 2.15.0 on a CPU, and the output shown is the real output; training times depend on the machine.
The big idea: sequences (sentences, daily readings, cricket balls) come in an order and in any length. Averaging the items throws the order away, and a fixed-size input layer fits only one length. A recurrent neural network (RNN) reads one time step at a time and keeps a fixed-size hidden state, a running note, updated by the same weights at every step: h_t = tanh(Wx·x_t + Wh·h_(t-1) + b). Drawn step by step (unrolled), it is a deep network with one layer per step, trained with backpropagation through time, and that is exactly why its gradients vanish over long distances.
Sources: Jeffrey Elman, "Finding Structure in Time", 1990 (summary: Swarthmore notes on Elman's simple recurrent network: "Hidden layer activations are copied to a context layer and fed back into the network on the next time step"). Paul Werbos, "Backpropagation through time: what it does and how to do it", 1990 (Zenodo record, DOI 10.1109/5.58337). Yoshua Bengio, Patrice Simard and Paolo Frasconi, "Learning long-term dependencies with gradient descent is difficult", IEEE Transactions on Neural Networks 5(2):157-166, 1994 (bibtex).
Words you need first
- Sequence: items that come in an order: the words of a review, the rainfall of each day, the balls of an over.
- Time step: one position in a sequence (word 1, word 2, ...), even when it is not really time.
- One-hot vector (Episode 16): a row with a single 1 marking which word it is: "good" =
[0, 0, 0, 1, 0, 0]in our 6-word vocabulary. - Vocabulary: the list of all words the model knows; here
["the", "movie", "was", "good", "bad", "not"]. - tanh (Episode 56): an activation that squashes any number into −1 ... 1.
- Backpropagation and the chain rule (Episode 57): gradients flow backwards, multiplied layer by layer.
- Vanishing gradient (Episode 66): the learning signal shrinking towards zero on its way back through many layers.
The story: reading a review
"The movie was good, not bad" is a happy review; "the movie was bad, not good" is an unhappy one. Same words, different order. And reviews have any length. Real-life example for the fix: reading with a running note. You read one word at a time; you keep a short note in your head ("so far good... now a not...") and update it after each word; and you use the same reading habit for every word, so any length works.
Two obvious attempts that fail
import torch, torch.nn as nn
torch.manual_seed(69)
vocab = ["the", "movie", "was", "good", "bad", "not"]
onehot = lambda s: torch.eye(6)[[vocab.index(w) for w in s.split()]] # one row per word
model = nn.Linear(6, 1) # obvious attempt 1: average the words, then score
score = lambda s: model(onehot(s).mean(0)).item()
a = "the movie was good not bad" # a positive review
b = "the movie was bad not good" # a negative review
print(f"{a!r}: {score(a):+.4f}")
print(f"{b!r}: {score(b):+.4f}")
print("same score?", score(a) == score(b))
fixed = nn.Linear(6 * 6, 1) # obvious attempt 2: glue exactly 6 words together
print(f"6 words: {fixed(onehot(a).flatten()).item():+.4f}")
print("4 words:", fixed(onehot("the movie was good").flatten()).item())
Output:
'the movie was good not bad': -0.2856
'the movie was bad not good': -0.2856
same score? True
6 words: +0.4613
RuntimeError: addmm: mat1 and mat2 shapes cannot be multiplied (1x24 and 36x1)
- Averaging gives exactly the same score (−0.2856) for both reviews: the average of the same rows is the same, whatever the order. No training can fix that.
- Gluing a fixed number of words (6 × 6 = 36 inputs) works for 6 words, but a 4-word review gives 24 inputs and crashes:
mat1 and mat2 shapes cannot be multiplied (1x24 and 36x1).
The RNN cell and the hidden state
h_t = tanh( Wx · x_t + Wh · h_(t-1) + b )
x_t: this step's input (a one-hot word, a temperature, ...).h_(t-1): the hidden state so far, a vector of fixed size (the running note);h_0is all zeros.h_t: the new hidden state. Recurrent means it is fed back in at the next step.Wx,Wh,b: the weights, the same at every step (shared weights), so any length works.
Elman's 1990 simple recurrent network did exactly this: the hidden activations are copied and fed back as extra input at the next time step.
Unrolling through time
Drawing the loop as one copy of the cell per step is unrolling through time:
h0 = 0 → [cell] → h1 → [cell] → h2 → [cell] → h3 → answer
↑ x1 ↑ x2 ↑ x3
Every copy shares the same weights; unrolled, an RNN is a deep network with one layer per time step.
import torch, torch.nn as nn
torch.manual_seed(69)
rnn = nn.RNN(input_size=3, hidden_size=2, batch_first=True) # PyTorch's RNN layer
Wx, Wh = rnn.weight_ih_l0, rnn.weight_hh_l0 # its weights: input, hidden
b = rnn.bias_ih_l0 + rnn.bias_hh_l0
x = torch.randn(1, 4, 3) # 1 sequence, 4 time steps, 3 numbers per step
h = torch.zeros(2) # the memory starts empty
with torch.no_grad():
for t in range(4): # unrolled: the SAME weights at every step
h = torch.tanh(Wx @ x[0, t] + Wh @ h + b)
print(f"step {t + 1}: hidden state = {h.numpy().round(4)}")
out, h_last = rnn(x)
print("nn.RNN, last step: ", h_last[0, 0].detach().numpy().round(4))
print("output shape (batch, steps, hidden):", tuple(out.shape))
print("weights:", sum(p.numel() for p in rnn.parameters()), "for ANY sequence length")
Output:
step 1: hidden state = [-0.4114 -0.9595]
step 2: hidden state = [ 0.9746 -0.9355]
step 3: hidden state = [-0.0376 -0.8954]
step 4: hidden state = [ 0.8791 -0.8479]
nn.RNN, last step: [ 0.8791 -0.8479]
output shape (batch, steps, hidden): (1, 4, 2)
weights: 14 for ANY sequence length
Our hand-written loop gives exactly nn.RNN's hidden state. nn.RNN stores weight_ih_l0 (Wx), weight_hh_l0 (Wh) and two biases: 6 + 4 + 2 + 2 = 14 numbers, for a sequence of any length.
Many-to-one, many-to-many, one-to-many
| Pattern | Uses | Example |
|---|---|---|
| Many-to-one | the last hidden state only | a review → happy or not |
| Many-to-many | every step's hidden state | each word → its part of speech (noun, verb, ...) |
| One-to-many | one input, then each output fed back as the next input | a photo → a caption; a starting note → a melody |
import torch, torch.nn as nn
torch.manual_seed(69)
rnn = nn.RNN(input_size=6, hidden_size=16, batch_first=True)
x = torch.randn(2, 5, 6) # 2 sequences, 5 steps, 6 numbers per step
out, h = rnn(x) # out: every step's hidden state, h: the last one
print("out:", tuple(out.shape), "| h:", tuple(h.shape))
one = nn.Linear(16, 1) # many-to-one: one answer per sequence
print("many-to-one (review -> score): ", tuple(one(h[0]).shape))
tag = nn.Linear(16, 3) # many-to-many: one answer per step
print("many-to-many (word -> tag): ", tuple(tag(out).shape))
cell, back = nn.RNNCell(6, 16), nn.Linear(16, 6) # one-to-many: one input, many outputs
inp, hs, made = torch.randn(1, 6), torch.zeros(1, 16), []
for t in range(5):
hs = cell(inp, hs); inp = back(hs) # each output becomes the next input
made.append(inp)
print("one-to-many (1 input -> 5 steps):", tuple(torch.stack(made, 1).shape))
Output:
out: (2, 5, 16) | h: (1, 2, 16)
many-to-one (review -> score): (2, 1)
many-to-many (word -> tag): (2, 5, 3)
one-to-many (1 input -> 5 steps): (1, 5, 6)
With batch_first=True, out is (batch, steps, hidden) = every step's hidden state, and h is (layers, batch, hidden) = the last step only.
Training an RNN: does order help?
3,000 random reviews of 3-8 words; the true label counts +1 per "good", −1 per "bad", and "not" flips the next one, so order matters. Short reviews get empty (all-zero) steps at the start so they fit in one batch of 8 steps: padding (more on padding and packing in Episode 70). A bag of words model (word counts, order lost) against a many-to-one RNN, both trained with 400 full-batch Adam steps:
import torch, torch.nn as nn, random
random.seed(69); torch.manual_seed(69)
vocab = ["the", "movie", "was", "good", "bad", "not"]
def review(): # a random review and its true label
total = 0
while total == 0:
words = random.choices(vocab, k=random.randint(3, 8)); total, flip = 0, 1
for w in words:
if w == "not": flip = -1
elif w in ("good", "bad"): total += flip * (1 if w == "good" else -1); flip = 1
return words, int(total > 0)
def encode(words): # one-hot rows; short reviews get empty
x = torch.zeros(8, 6) # (all-zero) steps at the START
for t, w in enumerate(words): x[8 - len(words) + t, vocab.index(w)] = 1
return x
data = [review() for _ in range(3000)]
X = torch.stack([encode(w) for w, _ in data]); y = torch.tensor([float(l) for _, l in data])
Xtr, ytr, Xte, yte = X[:2500], y[:2500], X[2500:], y[2500:]
class RNNClassifier(nn.Module): # many-to-one: read every word, answer once
def __init__(self):
super().__init__()
self.rnn, self.out = nn.RNN(6, 32, batch_first=True), nn.Linear(32, 1)
def forward(self, x):
_, h = self.rnn(x) # h = the hidden state after the LAST step
return self.out(h[0]).squeeze(1)
bag = nn.Linear(6, 1)
models = {"bag of words": (bag, lambda x: bag(x.sum(1)).squeeze(1)), "RNN": (RNNClassifier(),) * 2}
for name, (model, f) in models.items():
opt = torch.optim.Adam(model.parameters(), lr=0.01)
for step in range(400):
loss = nn.functional.binary_cross_entropy_with_logits(f(Xtr), ytr)
opt.zero_grad(); loss.backward(); opt.step()
with torch.no_grad():
print(f"{name:12}: test accuracy {((f(Xte) > 0).float() == yte).float().mean():.1%}")
rnn = models["RNN"][0]
with torch.no_grad():
for s in ("the movie was good not bad", "the movie was bad not good"):
p = torch.sigmoid(rnn(encode(s.split()).unsqueeze(0))).item()
print(f"RNN: {s!r} -> {p:.2f} chance positive")
Output:
bag of words: test accuracy 66.8%
RNN : test accuracy 99.8%
RNN: 'the movie was good not bad' -> 1.00 chance positive
RNN: 'the movie was bad not good' -> 0.00 chance positive
The bag of words cannot see where the "not" is (66.8%); the RNN gets 99.8% and scores our two reviews correctly.
Backpropagation through time (BPTT)
Unroll the RNN and run ordinary backpropagation on the unrolled network: the error flows from the last step back to the first. Because the weights are shared, each weight's gradient is the sum of its gradients at every time step. Werbos (1990) described this as backpropagation through time. Each step back multiplies the gradient by Wh and by tanh's slope (at most 1), so over many steps it can shrink (vanish) or grow (explode).
import torch, torch.nn as nn
torch.manual_seed(69)
rnn = nn.RNN(input_size=1, hidden_size=16, batch_first=True)
for steps in (5, 10, 20, 50, 100):
x = torch.randn(1, steps, 1, requires_grad=True)
out, h = rnn(x)
h.sum().backward() # a "loss" at the very last step
g = x.grad[0, :, 0].abs()
print(f"{steps:3} steps: gradient at the last step {g[-1]:.1e}, at step 1 {g[0]:.1e}")
Output:
5 steps: gradient at the last step 8.9e-02, at step 1 5.5e-02
10 steps: gradient at the last step 8.6e-02, at step 1 1.0e-03
20 steps: gradient at the last step 5.8e-02, at step 1 9.4e-06
50 steps: gradient at the last step 6.6e-02, at step 1 5.2e-12
100 steps: gradient at the last step 7.5e-02, at step 1 2.1e-23
At the last step the gradient stays around 0.06-0.09; at step 1 it falls from 5.5e-02 (5 steps) to 2.1e-23 (100 steps).
Vanishing gradients in RNNs
The learning signal for early steps shrinks towards zero, so a plain RNN struggles to connect something at the start of a sequence with an answer at the end: a long-term dependency. Bengio, Simard and Frasconi (1994) showed that learning long-term dependencies with gradient descent is difficult. The opposite problem, exploding gradients, is usually handled with gradient clipping: rescale the gradient whenever its size (norm) is above a limit (Practice 3).
The memory test: the label is the sign of the first number; every later number is just noise. Three fresh RNNs per length, 200 full-batch Adam steps each:
import torch, torch.nn as nn
torch.manual_seed(69)
def task(steps, n=1000): # remember the FIRST number's sign; then only noise
x = torch.randn(n, steps, 1)
y = (x[:, 0, 0] > 0).float()
return x, y
class Net(nn.Module):
def __init__(self):
super().__init__(); self.rnn, self.out = nn.RNN(1, 16, batch_first=True), nn.Linear(16, 1)
def forward(self, x): return self.out(self.rnn(x)[1][0]).squeeze(1)
for steps in (10, 30, 60, 100):
accs = []
for run in range(3): # 3 fresh networks per length, then the average
X, y = task(steps); Xt, yt = task(steps, 500)
net = Net(); opt = torch.optim.Adam(net.parameters(), lr=0.01)
for i in range(200):
loss = nn.functional.binary_cross_entropy_with_logits(net(X), y)
opt.zero_grad(); loss.backward(); opt.step()
with torch.no_grad(): accs.append(((net(Xt) > 0).float() == yt).float().mean().item())
print(f"remember the first of {steps:3} numbers: test accuracy",
" ".join(f"{a:.0%}" for a in accs), f"| average {sum(accs) / 3:.1%}")
Output:
remember the first of 10 numbers: test accuracy 91% 99% 99% | average 96.4%
remember the first of 30 numbers: test accuracy 98% 50% 48% | average 65.1%
remember the first of 60 numbers: test accuracy 96% 49% 96% | average 80.3%
remember the first of 100 numbers: test accuracy 52% 51% 50% | average 51.0%
At 10 steps all three networks learn the task (91-99%). At 30 and 60 steps some runs learn it and others stay stuck at about 50%, a coin flip; at 100 steps all three are stuck (52%, 51%, 50%). The signal from step 1 is too weak for gradient descent to find reliably. Episode 70's LSTM and GRU add gates to fix exactly this.
Common mistakes
import torch, torch.nn as nn
rnn = nn.RNN(input_size=6, hidden_size=16, batch_first=True)
x = torch.randn(2, 5, 4) # but each step has only 4 numbers
out, h = rnn(x)
RuntimeError: input.size(-1) must be equal to input_size. Expected 6, got 4
| Mistake | Exact message / symptom | Why | Fix |
|---|---|---|---|
| Averaging the words | the same score (−0.2856) for "good not bad" and "bad not good" | order is thrown away | read the sequence in order (RNN) |
| A fixed-size input layer | RuntimeError: addmm: mat1 and mat2 shapes cannot be multiplied (1x24 and 36x1) |
only one length fits | one cell reused at every step |
| Wrong number of features per step | RuntimeError: input.size(-1) must be equal to input_size. Expected 6, got 4 |
the last dimension of x ≠ input_size |
input_size = x.shape[-1] |
Forgetting batch_first=True |
batch and time silently swapped | the default layout is (steps, batch, features) |
batch_first=True or x.transpose(0, 1) |
Using out when you meant h (or the reverse) |
wrong shapes: (batch, steps, hidden) vs (1, batch, hidden) |
out = every step, h = last step |
many-to-one: h[-1]; many-to-many: out |
| Padding at the end for a many-to-one model | the answer is read after many empty steps | the last step is padding | pad at the start, or pack sequences (Episode 70) |
| Expecting long memory from a plain RNN | runs stuck at 50% when remembering far back | vanishing gradients | LSTM / GRU (Episode 70) |
Mini project: a word-by-word sentiment tracker (many-to-many)
Label every step with the sign of the running total so far (negative / neutral / positive) and train a many-to-many tagger that uses every hidden state:
import torch, torch.nn as nn, random
random.seed(69); torch.manual_seed(69)
vocab = ["the", "movie", "was", "good", "bad", "not"]
def review(): # label EVERY step: the running total's sign so far (+1 / -1 / 0)
words = random.choices(vocab, k=8); total, flip, tags = 0, 1, []
for w in words:
if w == "not": flip = -1
elif w in ("good", "bad"): total += flip * (1 if w == "good" else -1); flip = 1
tags.append(0 if total < 0 else 1 if total == 0 else 2) # 3 classes: negative, neutral, positive
return words, tags
data = [review() for _ in range(3000)]
X = torch.stack([torch.eye(6)[[vocab.index(w) for w in ws]] for ws, _ in data])
Y = torch.tensor([t for _, t in data]) # (3000, 8): one label per step
class Tagger(nn.Module): # many-to-many
def __init__(self):
super().__init__(); self.rnn, self.out = nn.RNN(6, 32, batch_first=True), nn.Linear(32, 3)
def forward(self, x): return self.out(self.rnn(x)[0]) # use EVERY step's hidden state
net = Tagger(); opt = torch.optim.Adam(net.parameters(), lr=0.01)
for step in range(300):
loss = nn.functional.cross_entropy(net(X[:2500]).reshape(-1, 3), Y[:2500].reshape(-1))
opt.zero_grad(); loss.backward(); opt.step()
with torch.no_grad():
pred = net(X[2500:]).argmax(2)
print("per-step test accuracy:", f"{(pred == Y[2500:]).float().mean():.1%}")
print("words:", data[2500][0]); print("true :", data[2500][1]); print("pred :", pred[0].tolist())
Output:
per-step test accuracy: 99.2%
words: ['was', 'movie', 'the', 'bad', 'not', 'good', 'good', 'the']
true : [1, 1, 1, 0, 0, 0, 0, 0]
pred : [1, 1, 1, 0, 0, 0, 0, 0]
Practice
1. Run the cell loop for 10 steps by hand and compare with nn.RNN.
Solution
import torch, torch.nn as nn
torch.manual_seed(69)
rnn = nn.RNN(input_size=3, hidden_size=2, batch_first=True)
Wx, Wh, b = rnn.weight_ih_l0, rnn.weight_hh_l0, rnn.bias_ih_l0 + rnn.bias_hh_l0
x = torch.randn(1, 10, 3) # 10 time steps this time
h = torch.zeros(2)
with torch.no_grad():
for t in range(10):
h = torch.tanh(Wx @ x[0, t] + Wh @ h + b)
out, h_last = rnn(x)
print("our loop, step 10:", h.numpy().round(4))
print("nn.RNN, step 10:", h_last[0, 0].numpy().round(4))
print("same?", torch.allclose(h, h_last[0, 0], atol=1e-6))
Output:
our loop, step 10: [ 0.5263 -0.8005]
nn.RNN, step 10: [ 0.5263 -0.8005]
same? True
2. Make the review task many-to-many: label every step. See the mini project above.
3. Clip gradients with torch.nn.utils.clip_grad_norm_.
Solution
import torch, torch.nn as nn
torch.manual_seed(69)
rnn = nn.RNN(1, 16, batch_first=True)
with torch.no_grad(): rnn.weight_hh_l0.mul_(4) # big recurrent weights: gradients can explode
x = torch.randn(32, 50, 1)
out, h = rnn(x)
(out ** 2).sum().backward()
before = torch.nn.utils.clip_grad_norm_(rnn.parameters(), max_norm=1.0) # returns the norm BEFORE clipping
after = torch.sqrt(sum((p.grad ** 2).sum() for p in rnn.parameters()))
print(f"gradient norm before clipping: {before:.1f}")
print(f"gradient norm after clipping: {after:.3f}")
Output:
gradient norm before clipping: 96325.1
gradient norm after clipping: 1.000
clip_grad_norm_ returns the total norm before clipping and rescales every gradient so the total norm is at most max_norm. Call it between loss.backward() and opt.step().
4. Try hidden_size 4 and 64 on the review task.
Solution
import torch, torch.nn as nn, random
random.seed(69); torch.manual_seed(69)
vocab = ["the", "movie", "was", "good", "bad", "not"]
def review(): # a random review and its true label
total = 0
while total == 0:
words = random.choices(vocab, k=random.randint(3, 8)); total, flip = 0, 1
for w in words:
if w == "not": flip = -1
elif w in ("good", "bad"): total += flip * (1 if w == "good" else -1); flip = 1
return words, int(total > 0)
def encode(words): # one-hot rows; short reviews get empty
x = torch.zeros(8, 6) # (all-zero) steps at the START
for t, w in enumerate(words): x[8 - len(words) + t, vocab.index(w)] = 1
return x
data = [review() for _ in range(3000)]
X = torch.stack([encode(w) for w, _ in data]); y = torch.tensor([float(l) for _, l in data])
Xtr, ytr, Xte, yte = X[:2500], y[:2500], X[2500:], y[2500:]
class RNNClassifier(nn.Module):
def __init__(self, hidden):
super().__init__()
self.rnn, self.out = nn.RNN(6, hidden, batch_first=True), nn.Linear(hidden, 1)
def forward(self, x):
return self.out(self.rnn(x)[1][0]).squeeze(1)
for hidden in (4, 64):
torch.manual_seed(69); model = RNNClassifier(hidden)
opt = torch.optim.Adam(model.parameters(), lr=0.01)
for step in range(400):
loss = nn.functional.binary_cross_entropy_with_logits(model(Xtr), ytr)
opt.zero_grad(); loss.backward(); opt.step()
with torch.no_grad():
acc = ((model(Xte) > 0).float() == yte).float().mean()
n = sum(p.numel() for p in model.parameters())
print(f"hidden_size {hidden:2}: {n:5} weights, test accuracy {acc:.1%}")
Output:
hidden_size 4: 53 weights, test accuracy 84.0%
hidden_size 64: 4673 weights, test accuracy 98.2%
Quick quiz
rnn = nn.RNN(input_size=10, hidden_size=32, batch_first=True), x = torch.randn(8, 20, 10), out, h = rnn(x). out.shape? (a) (8, 32) (b) (8, 20, 32) (c) (1, 8, 32) (d) (20, 8, 10)
Answer
(b) (8, 20, 32): every step's hidden state. (c) is h, the last step only.
- Why can't averaging word vectors tell "good not bad" from "bad not good"?
Answer
The average of the same rows is the same in any order. - What is the hidden state?
Answer
A fixed-size vector that summarises everything read so far; updated at every step and fed into the next. - How many weights does
nn.RNN(3, 2)have?Answer
14: 2×3 + 2×2 + 2 + 2. - What is backpropagation through time?
Answer
Ordinary backpropagation on the unrolled RNN; shared weights sum their gradients over all steps. - Why do gradients vanish in an RNN?
Answer
Each step back multiplies byWhand tanh's slope (≤ 1); many such factors shrink the gradient towards 0.
Interview questions
- What makes RNNs different from feed-forward networks? They process a sequence step by step with a hidden state carried forward and weights shared across time, so they handle any length and depend on order.
- Write the vanilla RNN update.
h_t = tanh(W_xh x_t + W_hh h_(t-1) + b); outputsy_t = W_hy h_t + cwhen needed. - Explain BPTT and truncated BPTT. BPTT backpropagates through the unrolled sequence and sums gradients for shared weights; truncated BPTT only goes back a fixed number of steps to save memory and time.
- Why are long-term dependencies hard for vanilla RNNs? Gradients through many steps are products of many Jacobians; they vanish (or explode), so the effect of early inputs is hard to learn (Bengio, Simard, Frasconi, 1994).
- How do you handle exploding gradients? Gradient clipping by norm (
clip_grad_norm_), smaller learning rates, careful initialisation. - Give examples of many-to-one, many-to-many and one-to-many. Sentiment classification; part-of-speech tagging; image captioning or music generation.
Glossary
| Term | Meaning (plain words + example) |
|---|---|
| Sequence data | Items in an order: words of a review, daily rainfall, cricket balls |
| Time step | One position in a sequence: word 1, word 2, ... |
| Vocabulary | The list of words a model knows: our 6 words |
| One-hot vector | A row with one 1 marking the item: "good" = [0, 0, 0, 1, 0, 0] |
| Bag of words | Representing text by word counts only; order is lost |
| Recurrent neural network (RNN) | A network that reads one step at a time and feeds its hidden state back in |
| Hidden state | The fixed-size running note h_t carried from step to step |
| RNN cell | The one-step update h_t = tanh(Wx·x_t + Wh·h_(t-1) + b) |
| Shared weights | The same Wx, Wh, b used at every time step |
| Simple recurrent network (Elman network) | Elman's 1990 RNN: hidden activations copied back as input at the next step |
| Unrolling through time | Drawing the RNN as one copy of the cell per step: a deep network |
nn.RNN |
PyTorch's RNN layer; returns out (every step) and h (last step) |
batch_first=True |
Input laid out as (batch, steps, features) instead of (steps, batch, features) |
| Many-to-one | Read a sequence, answer once: review → sentiment |
| Many-to-many | An answer at every step: word → tag |
| One-to-many | One input, a sequence out, each output fed back in: photo → caption |
| Padding | Filling short sequences with empty (all-zero) steps so a batch has one length |
| Backpropagation through time (BPTT) | Backpropagation on the unrolled RNN; shared weights sum their gradients over steps |
| Vanishing gradient (in RNNs) | The gradient reaching early steps shrinks towards 0: 2.1e-23 at 100 steps back |
| Long-term dependency | An answer that depends on something many steps earlier |
| Exploding gradient | The gradient growing huge through many steps |
| Gradient clipping | Rescaling the gradient when its norm is above a limit: clip_grad_norm_ |
Summary
- Sequences have order and any length: averaging loses the order (same score −0.2856), fixed inputs crash on other lengths.
- An RNN keeps a hidden state and applies the same cell at every step; our loop matched
nn.RNNexactly with 14 weights. - Unrolled, it is a deep network; wire it many-to-one, many-to-many or one-to-many.
- On order-dependent reviews: bag of words 66.8%, RNN 99.8%.
- BPTT trains it; the gradient at step 1 falls to 2.1e-23 at 100 steps: vanishing gradients, so long memory is unreliable.
Next episode: LSTMs and GRUs: the cell state and forget, input and output gates, GRUs, bidirectional and stacked RNNs, padding and packing.