SyntaxPilot
Courses/Machine Learning/Lesson 70
Video coming soonThis lesson’s video is being made

Start with the notes below; the video appears here as soon as it is on YouTube.

Machine Learning · Lesson 70 of 90

LSTMs and GRUs

long short-term memory (LSTM)cell stateforget gateinput gateoutput gategated recurrent unit (GRU)bidirectional RNNstacked RNNpadding and packing sequencesLSTM for time series

Machine Learning Full Course · Episode 70. Every code sample below was run with Python 3.13 and PyTorch 2.15.0 on a CPU, and the output shown is the real output; training times depend on the machine.

The big idea: a plain RNN rewrites its whole hidden state at every step (h = tanh(W·x + U·h + b)), so information from far back is squashed again and again, and its gradient vanishes. Making the hidden state bigger does not help. The LSTM adds a separate long-term memory, the cell state c, that is changed only by multiplying by a forget gate and adding new information through an input gate; an output gate decides what to show as h. Because the cell-state path has no squashing and no weight matrix, the gradient can survive hundreds of steps when the forget gate is open. The GRU does the same job with two gates and no separate cell state.

Sources: Sepp Hochreiter and Jürgen Schmidhuber, "Long Short-Term Memory", Neural Computation 9(8):1735-1780, 1997 (record: "constant error carousels", multiplicative gate units, "minimal time lags in excess of 1000 discrete time steps"). Felix A. Gers, Jürgen Schmidhuber and Fred A. Cummins, "Learning to Forget: Continual Prediction with LSTM", Neural Computation 12:2451-2471, 2000 (record: the adaptive forget gate). Kyunghyun Cho et al., "Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation", 2014 (arXiv 1406.1078, section 2.3: a new hidden unit with reset and update gates, "motivated by the LSTM unit but is much simpler to compute and implement"). Mike Schuster and Kuldip K. Paliwal, "Bidirectional Recurrent Neural Networks", IEEE Transactions on Signal Processing 45(11), 1997 (summary).

Words you need first

  • Sequence, time step, hidden state (Episode 69): items in order; one position; the RNN's running note h.
  • RNN cell (Episode 69): h_t = tanh(Wx·x_t + Wh·h_(t-1) + b), the same weights at every step.
  • Sigmoid (Episode 22): squashes any number into 0 ... 1; sigmoid(3) = 0.953, sigmoid(0) = 0.5.
  • tanh (Episode 56): squashes any number into −1 ... 1.
  • Vanishing gradient (Episodes 62, 69): the learning signal shrinking towards 0 as it is multiplied backwards through many steps.
  • Element-wise product *: multiply two vectors number by number: [1, 2] * [3, 4] = [3, 8].
  • Gradient clipping (Episode 62): cap the size of the gradient before each update.

The story: a bigger note still forgets

The memory test from Episode 69: a sequence of random numbers, and the answer is only whether the first number was positive; everything after it is noise. A plain RNN with 16 hidden numbers was at about 50% after 100 numbers. The obvious fix: a bigger note.

import torch, torch.nn as nn
torch.manual_seed(70)
def task(steps, n=1000):                      # remember the FIRST number's sign; the rest is noise
    x = torch.randn(n, steps, 1)
    return x, (x[:, 0, 0] > 0).float()
class Net(nn.Module):
    def __init__(self, kind, hidden):
        super().__init__(); self.rnn, self.out = kind(1, hidden, batch_first=True), nn.Linear(hidden, 1)
    def forward(self, x):
        out, _ = self.rnn(x)
        return self.out(out[:, -1]).squeeze(1)    # many-to-one: the last step's hidden state
def accuracy(kind, hidden, steps=100):
    X, y = task(steps); Xt, yt = task(steps, 500)
    net = Net(kind, hidden); opt = torch.optim.Adam(net.parameters(), lr=0.01)
    for i in range(200):
        loss = nn.functional.binary_cross_entropy_with_logits(net(X), y)
        opt.zero_grad(); loss.backward(); opt.step()
    with torch.no_grad(): return ((net(Xt) > 0).float() == yt).float().mean().item()
for kind, hidden in [(nn.RNN, 16), (nn.RNN, 128)]:     # attempt: a BIGGER note
    accs = [accuracy(kind, hidden) for run in range(3)]
    print(f"{kind.__name__:4} hidden {hidden:3}, first of 100 numbers:", " ".join(f"{a:.0%}" for a in accs),
          f"| average {sum(accs) / 3:.1%}")

Output (about 6 minutes on a laptop CPU):

RNN  hidden  16, first of 100 numbers: 51% 46% 50% | average 49.2%
RNN  hidden 128, first of 100 numbers: 48% 50% 62% | average 53.3%

Eight times the memory, and it is still a coin flip. Why: every step rewrites the whole note through the weights and tanh, so nothing is ever kept as it is; and backwards, the gradient is multiplied by Wh and tanh's slope at every step. More numbers just fade the same way. We need a memory that stays the same by default and changes only on purpose.

Real-life example: a shopkeeper's khata

Kirana shop's khata (ledger) LSTM
The khata book: balances kept for months Cell state c: long-term memory vector
Rubbing out a debt that was cleared Forget gate f: how much of each number in c to keep (0 to 1)
Writing today's purchase Input gate i: how much of the new candidate to add
Reading out one customer's total Output gate o: how much of c to show as h

The LSTM cell

A gate is a sigmoid of a weighted sum of this step's input and the previous hidden state, so each of its numbers is between 0 (closed) and 1 (open). It is multiplied element-wise with something to let a fraction of it through.

f = sigmoid(Wf·x_t + Uf·h_(t-1) + bf)     forget gate
i = sigmoid(Wi·x_t + Ui·h_(t-1) + bi)     input gate
o = sigmoid(Wo·x_t + Uo·h_(t-1) + bo)     output gate
g = tanh   (Wg·x_t + Ug·h_(t-1) + bg)     candidate: a possible new entry (−1 ... 1)

c_t = f * c_(t-1) + i * g                 keep some, add some
h_t = o * tanh(c_t)                       show part of it
  • Cell state c: the long-term memory ("the notebook").
  • Hidden state h: what the LSTM shows to the next step and the next layer (also its output).
  • Candidate g: the new information this step might write.
  • PyTorch stores the four weight blocks stacked in the order input, forget, candidate (cell), output in weight_ih_l0, weight_hh_l0, bias_ih_l0, bias_hh_l0.

Why it works: the + path

Follow c from step to step: c_t = f * c_(t-1) + .... There is no tanh and no weight matrix on this path. Backwards, the gradient of c_t with respect to c_(t-1) is just f. If the forget gate stays near 1, the signal passes through almost untouched for hundreds of steps. Hochreiter and Schmidhuber's 1997 paper calls this constant error flow ("constant error carousels") and showed time lags of more than 1000 steps; the forget gate itself was added by Gers, Schmidhuber and Cummins in 2000 so that the cell could also learn to reset itself.

An LSTM by hand matches nn.LSTM

import torch, torch.nn as nn
torch.manual_seed(70)
lstm = nn.LSTM(input_size=3, hidden_size=2, batch_first=True)   # PyTorch's LSTM layer
W, U = lstm.weight_ih_l0, lstm.weight_hh_l0           # 4 blocks stacked: i, f, g, o
b = lstm.bias_ih_l0 + lstm.bias_hh_l0
x = torch.randn(1, 4, 3)                     # 1 sequence, 4 steps, 3 numbers per step
h, c = torch.zeros(2), torch.zeros(2)        # short-term note h, long-term notebook c
with torch.no_grad():
    for t in range(4):
        i, f, g, o = (W @ x[0, t] + U @ h + b).chunk(4)
        i, f, o, g = torch.sigmoid(i), torch.sigmoid(f), torch.sigmoid(o), torch.tanh(g)
        c = f * c + i * g                    # forget some, then add some
        h = o * torch.tanh(c)                # show part of the notebook
        print(f"step {t + 1}: forget {f.numpy().round(2)}  cell {c.numpy().round(4)}")
    out, (h_n, c_n) = lstm(x)
print("our h:", h.numpy().round(4), "| nn.LSTM h:", h_n[0, 0].numpy().round(4))
print("our c:", c.numpy().round(4), "| nn.LSTM c:", c_n[0, 0].numpy().round(4))

Output:

step 1: forget [0.23 0.55]  cell [-0.293  -0.4175]
step 2: forget [0.4  0.75]  cell [-0.3946 -0.8965]
step 3: forget [0.36 0.66]  cell [ 0.1088 -1.1657]
step 4: forget [0.12 0.33]  cell [ 0.4811 -0.691 ]
our h: [ 0.2755 -0.3744] | nn.LSTM h: [ 0.2755 -0.3744]
our c: [ 0.4811 -0.691 ] | nn.LSTM c: [ 0.4811 -0.691 ]

Our four lines of maths give exactly the same h and c as nn.LSTM.

Does the gradient survive?

import torch, torch.nn as nn
torch.manual_seed(70)
rnn, lstm = nn.RNN(1, 16, batch_first=True), nn.LSTM(1, 16, batch_first=True)
lstm_open = nn.LSTM(1, 16, batch_first=True)
with torch.no_grad():                    # forget gate = bias block 2 of 4 (rows 16..31)
    lstm_open.bias_ih_l0[16:32] = 3.0    # sigmoid(3) = 0.95: keep 95% of the notebook
    lstm_open.bias_hh_l0[16:32] = 0.0
for name, layer in [("RNN", rnn), ("LSTM, random", lstm), ("LSTM, forget open", lstm_open)]:
    x = torch.randn(1, 100, 1, requires_grad=True)
    out, _ = layer(x)
    out[0, -1].sum().backward()          # a "loss" at step 100
    print(f"{name:17}: step-1 gradient {x.grad[0, 0, 0].abs():.1e}")

Output:

RNN              : step-1 gradient 3.3e-25
LSTM, random     : step-1 gradient 3.9e-20
LSTM, forget open: step-1 gradient 1.4e-01
  • The plain RNN's gradient is gone (3.3e-25).
  • An LSTM with random gates also vanishes (3.9e-20): its forget gates start near sigmoid(0) = 0.5, and 0.5 multiplied a hundred times is tiny. The design makes long memory possible; the network must learn to keep the forget gate open.
  • With the forget-gate bias set to 3 (sigmoid(3) = 0.95), a real signal reaches step 1 (1.4e-01). Starting the forget gate open is a common initialisation trick.

The GRU (gated recurrent unit)

r = sigmoid(Wr·x_t + Ur·h_(t-1) + br)     reset gate: how much of the old h to use
z = sigmoid(Wz·x_t + Uz·h_(t-1) + bz)     update gate: how much of the old h to keep
n = tanh(Wn·x_t + bn_x + r * (Un·h_(t-1) + bn_h))   the new candidate note
h_t = (1 − z) * n + z * h_(t-1)
  • No separate cell state, only h, and two gates instead of three.
  • When z is near 1, the old note is carried forward unchanged (the same + trick).
  • PyTorch's blocks are stacked reset, update, new. (Some papers write the blend the other way round, with z and 1 − z swapped; PyTorch keeps z of the old state.)
import torch, torch.nn as nn
torch.manual_seed(70)
gru = nn.GRU(input_size=3, hidden_size=2, batch_first=True)
W, U = gru.weight_ih_l0, gru.weight_hh_l0     # 3 blocks stacked: reset, update, new
bW, bU = gru.bias_ih_l0, gru.bias_hh_l0
x, h = torch.randn(1, 4, 3), torch.zeros(2)
with torch.no_grad():
    for t in range(4):
        rW, zW, nW = (W @ x[0, t] + bW).chunk(3)
        rU, zU, nU = (U @ h + bU).chunk(3)
        r, z = torch.sigmoid(rW + rU), torch.sigmoid(zW + zU)   # reset, update gates
        n = torch.tanh(nW + r * nU)                # the new candidate note
        h = (1 - z) * n + z * h                    # keep z of the old note
        print(f"step {t + 1}: update gate {z.numpy().round(2)}  h {h.numpy().round(4)}")
    out, h_n = gru(x)
print("our h:", h.numpy().round(4), "| nn.GRU h:", h_n[0, 0].numpy().round(4))
for layer in (nn.RNN(10, 32), nn.LSTM(10, 32), nn.GRU(10, 32)):
    n = sum(p.numel() for p in layer.parameters())
    print(f"{type(layer).__name__:4} (10 in, 32 hidden): {n:5} weights")

Output:

step 1: update gate [0.36 0.47]  h [-0.5204 -0.1441]
step 2: update gate [0.4  0.59]  h [ 0.2348 -0.2101]
step 3: update gate [0.26 0.29]  h [ 0.1524 -0.0221]
step 4: update gate [0.37 0.34]  h [0.3092 0.0209]
our h: [0.3092 0.0209] | nn.GRU h: [0.3092 0.0209]
RNN  (10 in, 32 hidden):  1408 weights
LSTM (10 in, 32 hidden):  5632 weights
GRU  (10 in, 32 hidden):  4224 weights

An LSTM has 4 blocks of weights and a GRU 3, so they have 4× and 3× the weights of a plain RNN of the same size.

The memory test: RNN vs LSTM vs GRU

import torch, torch.nn as nn
torch.manual_seed(70)
def task(steps, n=1000):                     # remember the FIRST number's sign; the rest is noise
    x = torch.randn(n, steps, 1)
    return x, (x[:, 0, 0] > 0).float()
def keep_open(layer, block):                 # start the "keep" gate open: bias +3 -> sigmoid = 0.95
    H = layer.hidden_size
    with torch.no_grad():
        layer.bias_ih_l0[block * H:(block + 1) * H] = 3.0; layer.bias_hh_l0[block * H:(block + 1) * H] = 0.0
class Net(nn.Module):
    def __init__(self, kind):
        super().__init__(); self.rnn, self.out = kind(1, 16, batch_first=True), nn.Linear(16, 1)
        if kind is nn.LSTM: keep_open(self.rnn, 1)    # LSTM blocks: input, FORGET, cell, output
        if kind is nn.GRU: keep_open(self.rnn, 1)     # GRU blocks: reset, UPDATE, new
    def forward(self, x):
        out, _ = self.rnn(x)
        return self.out(out[:, -1]).squeeze(1)       # many-to-one: answer from the last step
def accuracy(kind, steps=200):
    X, y = task(steps); Xt, yt = task(steps, 500)
    net = Net(kind); opt = torch.optim.Adam(net.parameters(), lr=0.003)
    for i in range(400):
        loss = nn.functional.binary_cross_entropy_with_logits(net(X), y)
        opt.zero_grad(); loss.backward()
        nn.utils.clip_grad_norm_(net.parameters(), 1.0); opt.step()
    with torch.no_grad(): return ((net(Xt) > 0).float() == yt).float().mean().item()
for kind in (nn.RNN, nn.LSTM, nn.GRU):
    accs = [accuracy(kind) for run in range(3)]       # 3 fresh networks each
    print(f"{kind.__name__:4}: remember the first of 200 numbers:", " ".join(f"{a:.0%}" for a in accs),
          f"| average {sum(accs) / 3:.1%}")

Output (about 4.5 minutes on a laptop CPU):

RNN : remember the first of 200 numbers: 91% 50% 44% | average 61.5%
LSTM: remember the first of 200 numbers: 95% 97% 90% | average 93.9%
GRU : remember the first of 200 numbers: 93% 97% 92% | average 93.9%

With the same training (Adam, lr=0.003, gradient clipping at 1.0, 400 full-batch steps), one plain RNN out of three got lucky; all six gated networks remembered a number from 200 steps back.

Bidirectional and stacked RNNs

  • Bidirectional RNN: one reader goes left → right, another right → left, and their hidden states are joined (concatenated) at every step. Useful when later words change the meaning ("I went to the bank to catch fish" vs "... take out money"). It needs the whole sequence, so it cannot be used for live, step-by-step forecasting.
  • Stacked RNN (num_layers=2): layer 2 reads layer 1's outputs step by step, a deeper reading.
import torch, torch.nn as nn
torch.manual_seed(70)
x = torch.randn(8, 20, 10)               # 8 sequences, 20 steps, 10 numbers per step
for name, kw in [("1 layer, one way", {}), ("2 layers (stacked)", {"num_layers": 2}),
                 ("bidirectional", {"bidirectional": True}),
                 ("2 layers, bidirectional", {"num_layers": 2, "bidirectional": True})]:
    out, (h, c) = nn.LSTM(10, 32, batch_first=True, **kw)(x)
    print(f"{name:24} out {tuple(out.shape)}  h {tuple(h.shape)}")
bi = nn.LSTM(10, 32, batch_first=True, bidirectional=True)
out, (h, c) = bi(x)
print("forward reader ends at the LAST step: ", torch.allclose(out[:, -1, :32], h[0]))
print("backward reader ends at the FIRST step:", torch.allclose(out[:, 0, 32:], h[1]))

Output:

1 layer, one way         out (8, 20, 32)  h (1, 8, 32)
2 layers (stacked)       out (8, 20, 32)  h (2, 8, 32)
bidirectional            out (8, 20, 64)  h (2, 8, 32)
2 layers, bidirectional  out (8, 20, 64)  h (4, 8, 32)
forward reader ends at the LAST step:  True
backward reader ends at the FIRST step: True
Layer out h (and c)
1 layer, one way (batch, steps, H) (1, batch, H)
num_layers=L (batch, steps, H) (top layer) (L, batch, H)
bidirectional=True (batch, steps, 2H) (2, batch, H)
both (batch, steps, 2H) (2L, batch, H)

The forward reader finishes at the last step and the backward reader at the first step, so for a many-to-one classifier use h[-2] and h[-1] (the top layer's two directions), not out[:, -1].

Padding and packing sequences

  • A batch is one tensor, so every sequence in it must have the same length. Padding fills short sequences with zeros up to the longest (pad_sequence).
  • But the LSTM reads the zeros as real steps, and the short sequence's final state changes.
  • Packing (pack_padded_sequence(padded, lengths, ...)) tells PyTorch the true lengths so it skips the pads; pad_packed_sequence turns the output back into a padded tensor.
import torch, torch.nn as nn
from torch.nn.utils.rnn import pad_sequence, pack_padded_sequence, pad_packed_sequence
torch.manual_seed(70)
lstm = nn.LSTM(2, 3, batch_first=True)
seqs = [torch.randn(5, 2), torch.randn(2, 2)]       # a 5-step and a 2-step sequence
lengths = torch.tensor([5, 2])
padded = pad_sequence(seqs, batch_first=True)       # the short one gets zeros at the END
print("padded:", tuple(padded.shape), "| zero steps added to the short one:", 5 - 2)
with torch.no_grad():
    _, (h_alone, _) = lstm(seqs[1].unsqueeze(0))     # the short sequence on its own
    _, (h_pad, _) = lstm(padded)                     # padded, no packing
    packed = pack_padded_sequence(padded, lengths, batch_first=True, enforce_sorted=False)
    out_p, (h_pack, _) = lstm(packed)                # packed: the pads are skipped
print("short alone :", h_alone[0, 0].numpy().round(4))
print("padded only :", h_pad[0, 1].numpy().round(4), "<- read the zeros")
print("packed      :", h_pack[0, 1].numpy().round(4))
out, lens = pad_packed_sequence(out_p, batch_first=True)
print("unpacked out:", tuple(out.shape), "lengths", lens.tolist())

Output:

padded: (2, 5, 2) | zero steps added to the short one: 3
short alone : [0.1643 0.0223 0.0145]
padded only : [ 0.1859 -0.1529 -0.0255] <- read the zeros
packed      : [0.1643 0.0223 0.0145]
unpacked out: (2, 5, 3) lengths [5, 2]

An LSTM for time series

A time series is one number per time step in order (Episode 50). To forecast, slide a window: the last 30 days in, the next day out (many-to-one). Split by time (train on the past, test on the future), scale with training statistics only, and always compare with the naive baseline "tomorrow = today". The data below is made up so that we know the truth: a yearly season plus hot and cold spells that build up and fade (an AR(2) process).

import torch, torch.nn as nn, math
torch.manual_seed(70)
days, weather = torch.arange(4 * 365.0), torch.zeros(4 * 365)   # 4 years, made-up data
for t in range(2, len(days)):                     # hot and cold spells that build and fade
    weather[t] = 1.5 * weather[t - 1] - 0.8 * weather[t - 2] + 0.5 * torch.randn(1).item()
temp = 26 + 7 * torch.sin(2 * math.pi * days / 365) + weather   # season + spells, °C
W, split = 30, 3 * 365 - 30                        # last 30 days in; 3 years train, 1 test
X = torch.stack([temp[i:i + W] for i in range(len(temp) - W)]).unsqueeze(-1); y = temp[W:]
mean, std = temp[:3 * 365].mean(), temp[:3 * 365].std()     # TRAIN statistics only
lstm, head = nn.LSTM(1, 32, batch_first=True), nn.Linear(32, 1)
f = lambda x: head(lstm((x - mean) / std)[0][:, -1]).squeeze(1) * std + mean
opt = torch.optim.Adam([*lstm.parameters(), *head.parameters()], lr=0.005)
for epoch in range(300):
    loss = nn.functional.mse_loss((f(X[:split]) - mean) / std, (y[:split] - mean) / std)
    opt.zero_grad(); loss.backward(); opt.step()
with torch.no_grad(): pred, naive = f(X[split:]), X[split:, -1, 0]   # naive: tomorrow = today
mae = lambda p: (p - y[split:]).abs().mean().item()
print(f"test days: {len(naive)} | naive MAE {mae(naive):.3f} °C | LSTM MAE {mae(pred):.3f} °C")
print("next 3 days, true:", y[split:split + 3].numpy().round(1), "LSTM:", pred[:3].numpy().round(1))

Output:

test days: 365 | naive MAE 0.686 °C | LSTM MAE 0.440 °C
next 3 days, true: [26.9 26.3 26.5] LSTM: [28.3 26.1 25.8]

The best possible MAE for this data is about 0.40 °C (the daily random shock has standard deviation 0.5); the LSTM reached 0.440 against the naive 0.686.

Common mistakes

import torch, torch.nn as nn
lstm = nn.LSTM(input_size=4, hidden_size=8, batch_first=True)
x = torch.randn(2, 5, 4)
out, h = lstm(x)                      # the habit from nn.RNN
print("out:", tuple(out.shape))
print("h:", h.shape)
out: (2, 5, 8)
Traceback (most recent call last):
  File "main.py", line 6, in <module>
    print("h:", h.shape)
                ^^^^^^^
AttributeError: 'tuple' object has no attribute 'shape'
Mistake Exact message / symptom Why Fix
out, h = lstm(x) then h.shape AttributeError: 'tuple' object has no attribute 'shape' an LSTM returns out, (h, c) out, (h, c) = lstm(x)
Padding without packing the short sequence's last state changes silently (0.1643 → 0.1859 above) the LSTM reads the zero steps pack_padded_sequence with the true lengths
out[:, -1] of a bidirectional LSTM the backward half has seen only one step the backward reader ends at step 1 torch.cat([h[-2], h[-1]], 1)
A bigger RNN for long memory about 50% at 100 steps back the whole note is rewritten every step LSTM or GRU
Bidirectional for live forecasting great offline scores that cannot happen live the backward reader reads the future a one-way LSTM for forecasts
Forget gate starting near 0.5 gradient 3.9e-20 at 100 steps 0.5¹⁰⁰ is tiny train longer, or start the forget bias at 1-3

Mini project: a 2-layer LSTM forecaster

Change one line of the forecaster: lstm = nn.LSTM(1, 32, num_layers=2, batch_first=True).

test days: 365 | naive MAE 0.686 °C | LSTM MAE 0.431 °C
next 3 days, true: [26.9 26.3 26.5] LSTM: [28.  25.5 25.5]

The second layer improved the test MAE from 0.440 to 0.431 °C; with a theoretical best of about 0.40 there is little left to gain, which is itself a lesson: compare against the best possible error when you know it.

Practice

1. Try forget-gate biases 0, 1, 2, 3 and 5. How big is the gradient at step 1 of 100?

Solution
import torch, torch.nn as nn
torch.manual_seed(70)
for bias in (0.0, 1.0, 2.0, 3.0, 5.0):
    lstm = nn.LSTM(1, 16, batch_first=True)
    with torch.no_grad():
        lstm.bias_ih_l0[16:32] = bias; lstm.bias_hh_l0[16:32] = 0.0   # forget-gate bias
    x = torch.randn(1, 100, 1, requires_grad=True)
    out, _ = lstm(x)
    out[0, -1].sum().backward()
    print(f"forget bias {bias}: sigmoid = {torch.sigmoid(torch.tensor(bias)):.3f}, "
          f"gradient at step 1 of 100 = {x.grad[0, 0, 0].abs():.1e}")
forget bias 0.0: sigmoid = 0.500, gradient at step 1 of 100 = 6.0e-25
forget bias 1.0: sigmoid = 0.731, gradient at step 1 of 100 = 1.5e-09
forget bias 2.0: sigmoid = 0.881, gradient at step 1 of 100 = 4.1e-05
forget bias 3.0: sigmoid = 0.953, gradient at step 1 of 100 = 3.7e-02
forget bias 5.0: sigmoid = 0.993, gradient at step 1 of 100 = 1.1e-01

The more open the forget gate starts, the more signal survives: sigmoid(b) multiplied about 100 times.

2. Write a GRU for a whole batch by hand and compare with nn.GRU.

Solution
import torch, torch.nn as nn
torch.manual_seed(70)
gru = nn.GRU(input_size=3, hidden_size=4, batch_first=True)
W, U, bW, bU = gru.weight_ih_l0, gru.weight_hh_l0, gru.bias_ih_l0, gru.bias_hh_l0
x = torch.randn(5, 7, 3)                      # a batch: 5 sequences, 7 steps
h = torch.zeros(5, 4)
with torch.no_grad():
    for t in range(7):
        rW, zW, nW = (x[:, t] @ W.T + bW).chunk(3, dim=1)   # rows = sequences
        rU, zU, nU = (h @ U.T + bU).chunk(3, dim=1)
        r, z = torch.sigmoid(rW + rU), torch.sigmoid(zW + zU)
        n = torch.tanh(nW + r * nU)
        h = (1 - z) * n + z * h
    _, h_n = gru(x)
print("batch h shape:", tuple(h.shape), "| matches nn.GRU:", torch.allclose(h, h_n[0], atol=1e-6))
batch h shape: (5, 4) | matches nn.GRU: True

3. Write a formula for the number of weights in an RNN, GRU and LSTM with I inputs and H hidden numbers.

Solution

Each block has H×I input weights, H×H hidden weights and two bias vectors of H (PyTorch keeps bias_ih and bias_hh). RNN: 1 block, GRU: 3, LSTM: 4.

import torch.nn as nn
I, H = 10, 32
for name, layer, blocks in [("RNN", nn.RNN(I, H), 1), ("GRU", nn.GRU(I, H), 3), ("LSTM", nn.LSTM(I, H), 4)]:
    formula = blocks * (H * I + H * H + 2 * H)            # weights for x, for h, two biases
    real = sum(p.numel() for p in layer.parameters())
    print(f"{name:4}: formula {formula}, PyTorch {real}")
RNN : formula 1408, PyTorch 1408
GRU : formula 4224, PyTorch 4224
LSTM: formula 5632, PyTorch 5632

4. Build a many-to-one classifier on a 2-layer bidirectional LSTM.

Solution
import torch, torch.nn as nn
torch.manual_seed(70)
class BiClassifier(nn.Module):               # many-to-one with a bidirectional LSTM
    def __init__(self):
        super().__init__()
        self.lstm = nn.LSTM(6, 16, num_layers=2, batch_first=True, bidirectional=True)
        self.out = nn.Linear(2 * 16, 1)
    def forward(self, x):
        _, (h, c) = self.lstm(x)             # h: (layers * 2, batch, 16)
        both = torch.cat([h[-2], h[-1]], dim=1)   # top layer: forward and backward
        return self.out(both).squeeze(1)
x = torch.randn(4, 9, 6)
print("scores:", tuple(BiClassifier()(x).shape))
scores: (4,)

Quick quiz

lstm = nn.LSTM(10, 32, num_layers=2, bidirectional=True, batch_first=True)
x = torch.randn(8, 20, 10)
out, (h, c) = lstm(x)
# h.shape = ?
Answer

(4, 8, 32): one row per layer per direction (2 × 2 = 4), batch 8, hidden 32. out is (8, 20, 64).

Interview questions

  1. Why does a plain RNN struggle with long-term dependencies? Its hidden state is rewritten through a weight matrix and tanh at every step, so the gradient is multiplied by factors usually smaller than 1 at each step back and vanishes (or explodes if they are bigger than 1).
  2. What does each LSTM gate do? Forget: how much of the old cell state to keep; input: how much of the new candidate to write; output: how much of the cell state to expose as the hidden state.
  3. Why does the cell state help gradients? c_t = f * c_(t-1) + i * g: the derivative along the cell path is f, with no weight matrix or squashing, so with f ≈ 1 it passes through many steps.
  4. LSTM vs GRU? A GRU has two gates and no separate cell state, so about 3/4 of the weights and less compute; they often perform similarly, so try both.
  5. When should you not use a bidirectional RNN? When the output must be produced live from the past only (forecasting, streaming speech): the backward direction needs future inputs.
  6. Why pack padded sequences? So the RNN does not read padding as data; the final state of each sequence is taken at its true length.
  7. Why initialise the forget-gate bias to a positive value? So the gate starts mostly open and gradients reach early steps from the start of training.

Glossary

Term Meaning, with an example
LSTM (long short-term memory) A recurrent cell with a separate cell state and three gates; nn.LSTM(10, 32)
Cell state c The LSTM's long-term memory vector, updated by c = f*c + i*g; like a khata book
Hidden state h What the cell shows at each step: its output and short-term memory; h = o*tanh(c)
Gate A sigmoid layer giving numbers 0-1 that scale another vector element-wise; 0 blocks, 1 lets through
Forget gate f How much of each cell-state number to keep; f = 0.95 keeps 95%
Input gate i How much of the candidate to add to the cell state
Output gate o How much of tanh(c) to expose as h
Candidate g The new information a step might write, from tanh (−1 ... 1)
Constant error carousel Hochreiter and Schmidhuber's name for the cell's self-loop that keeps the error signal constant
GRU (gated recurrent unit) A simpler gated cell with reset and update gates and no separate cell state (Cho et al., 2014)
Reset gate r In a GRU, how much of the old h is used to build the new candidate
Update gate z In a GRU, how much of the old h is kept: h = (1−z)*n + z*h_old
Bidirectional RNN Two RNNs reading the sequence in opposite directions, their states concatenated (Schuster and Paliwal, 1997)
Stacked RNN Several RNN layers on top of each other; layer 2 reads layer 1's outputs (num_layers=2)
Padding Filling short sequences with zeros so a batch has one length (pad_sequence)
Packing Giving the RNN the true lengths so it skips the padding (pack_padded_sequence)
pad_packed_sequence Turns a packed output back into a padded tensor plus lengths
Window The fixed number of past steps used as input for a forecast (30 days above)
Naive forecast The baseline "the next value equals the last value"
MAE (mean absolute error) The average size of the errors; 0.44 °C above (Episode 19)
Gradient clipping Capping the gradient's size before an update (clip_grad_norm_)
Forget-gate bias initialisation Starting the forget gate's bias positive (e.g. 1-3) so the gate starts open

Summary

  • A plain RNN rewrites its whole note every step; even a 128-number note stayed at a coin flip on a 100-step memory task.
  • The LSTM keeps a separate cell state, changed by c = f*c + i*g, with forget, input and output gates; our hand-written version matched nn.LSTM exactly.
  • The + path lets gradients survive when the forget gate is open: 1.4e-01 at 100 steps instead of 3.3e-25.
  • The GRU uses reset and update gates and no separate cell; LSTM and GRU both reached 93.9% remembering the first of 200 numbers, against 61.5% for the RNN.
  • Bidirectional = two readers joined; stacked = layers on top; h has layers × directions rows.
  • Pack padded batches so pads are not read; an LSTM forecaster beat the naive baseline (0.440 vs 0.686 °C).

Next: Episode 71, NLP basics: cleaning text, tokenisation, stop words, stemming, lemmatisation, n-grams and a TF-IDF sentiment classifier.