Machine Learning Full Course · Episode 73. Every code sample below was run with Python 3.13 and PyTorch 2.15.0 on a 2-core CPU, and the output shown is the real output.
The big idea: many problems take one sequence in and give a different sequence out (translation, summarising, answering). Word-by-word lookup fails because word order and length change. An encoder-decoder (seq2seq) model reads the whole input with one RNN (the encoder) and writes the output one token at a time with another (the decoder), trained with teacher forcing. But squeezing the whole input into one vector is an information bottleneck: our model reversed 5 letters with 89.8% of words exactly right and 25 letters with 0.0%. Attention lets the decoder look at every encoder state at every step: score each state, turn the scores into weights with softmax, and mix the states into a context vector. With attention the same model got 93.2% of 25-letter words exactly right.
Sources: Ilya Sutskever, Oriol Vinyals and Quoc V. Le (Google), "Sequence to Sequence Learning with Neural Networks", 2014 (arXiv 1409.3215: a multilayer LSTM maps the input to "a vector of a fixed dimensionality" and a second LSTM decodes it; four-layer LSTMs; WMT'14 English to French, BLEU 34.8 against 33.3 for a phrase-based SMT baseline; reversing the source sentence made optimisation easier). Dzmitry Bahdanau, KyungHyun Cho and Yoshua Bengio, "Neural Machine Translation by Jointly Learning to Align and Translate" (arXiv 1409.0473: "the use of a fixed-length vector is a bottleneck"; the model learns to "(soft-)search for parts of a source sentence"; the alignment model is a feedforward network; softmax weights; the context vector is a weighted sum of the annotations from a bidirectional RNN). Minh-Thang Luong, Hieu Pham and Christopher D. Manning, "Effective Approaches to Attention-based Neural Machine Translation", 2015 (arXiv 1508.04025, ar5iv: global attention over all source words vs local attention over a subset; score functions dot, general and concat).
Words you need first
- Embedding (Episode 72): a learned list of numbers for each token;
nn.Embeddingis a lookup table. - RNN, hidden state, GRU (Episodes 69-70): a recurrent network reads a sequence one item at a time and carries a hidden state, a vector that summarises what it has read so far. A GRU (gated recurrent unit) is an RNN with gates that decide what to keep.
- Softmax (Episode 23): turns any list of scores into positive numbers that add up to 1.
softmax([1.8, 0.1]) = [0.85, 0.15]. - Cross-entropy loss (Episodes 22-23): small when the model gives high probability to the right answer.
- Dot product (Episode 3): multiply matching numbers and add:
[1, 2] · [3, 4] = 11. Bigger when two vectors point the same way. - Token: one unit of a sequence: a word, a piece of a word, or a single character.
The story: word-by-word translation fails
A sequence is a list of items whose order matters. Translation is a sequence-to-sequence problem: English sentence in, French sentence out. The obvious first attempt is a dictionary:
fr = {"the": "le", "black": "noir", "cat": "chat", "i": "je", "love": "aime",
"you": "te", "drinks": "boit", "tea": "thé", "my": "mon", "friend": "ami"}
def word_by_word(sentence):
return " ".join(fr[w] for w in sentence.lower().split())
tests = [("the black cat", "le chat noir"),
("I love you", "je t'aime"),
("my friend drinks tea", "mon ami boit du thé")]
for en, correct in tests:
guess = word_by_word(en)
print(f"{en:21} -> {guess:17}", "OK" if guess == correct else "WRONG")
print(f"{'':21} correct: {correct}")
print("words in vs words out:", [(len(e.split()), len(c.split())) for e, c in tests])
Output:
the black cat -> le noir chat WRONG
correct: le chat noir
I love you -> je aime te WRONG
correct: je t'aime
my friend drinks tea -> mon ami boit thé WRONG
correct: mon ami boit du thé
words in vs words out: [(3, 3), (3, 2), (4, 5)]
Two things break. Word order: French puts most colours after the noun (le chat noir), and the object pronoun moves before the verb (je t'aime). Length: 3 English words became 2 French words; 4 became 5 (du, "some", has no English word here). A translator has to read the whole sentence first, then write.
Real-life example: an interpreter at a meeting
| At a meeting | In the model |
|---|---|
| She listens to the whole sentence before speaking | Encoder: reads the whole input |
| She holds what it means in her head | Summary vector: the encoder's last hidden state |
| She speaks in French, word by word, in French order | Decoder: writes the output one token at a time |
The encoder-decoder (seq2seq) architecture
input tokens ──► [ Encoder RNN ] ──► summary vector h ──► [ Decoder RNN ] ──► output tokens, one per step
▲ │
└── previous output token
- Encoder: an RNN (here a GRU) that reads the input tokens one by one. Its final hidden state is a summary of the whole input.
- Decoder: a second RNN that starts from that summary and, at each step, predicts the next output token from the previous one.
- Start token
<s>: a special token that tells the decoder to begin. An end token</s>lets it decide when to stop, so the output length can differ from the input length. - Seq2seq = sequence to sequence = this whole design. Sutskever, Vinyals and Le (2014) used four-layer LSTMs and scored BLEU 34.8 on WMT'14 English to French. BLEU (bilingual evaluation understudy) is a score from 0 to 100 measuring how many word sequences a machine translation shares with human translations.
Teacher forcing, greedy decoding and exposure bias
| In class | In training |
|---|---|
| Write the next letter of "chai" after the letters before it | A decoder step: predict the next token |
| The teacher corrects each letter before you continue | Teacher forcing: feed the TRUE previous token |
| In the exam there is no teacher; one slip and you build on it | Exposure bias: at test time it reads its own guesses |
target = ["c", "h", "a", "i"]
dec_in = ["<s>"] + target[:-1] # teacher forcing: the TRUE previous letter
for step, (x, y) in enumerate(zip(dec_in, target)):
print(f"step {step}: decoder reads {x!r:6} -> must predict {y!r}")
model_guess = ["k", "h", "a", "i"] # suppose step 0 is wrong at test time
print("training input :", dec_in)
print("test-time input:", ["<s>"] + model_guess[:-1])
print(" ('k' was its own wrong guess, and it leaks in)")
step 0: decoder reads '<s>' -> must predict 'c'
step 1: decoder reads 'c' -> must predict 'h'
step 2: decoder reads 'h' -> must predict 'a'
step 3: decoder reads 'a' -> must predict 'i'
training input : ['<s>', 'c', 'h', 'a']
test-time input: ['<s>', 'k', 'h', 'a']
('k' was its own wrong guess, and it leaks in)
- Teacher forcing: during training, the decoder's input at step t is the true token from step t−1 (the target shifted right by one, with
<s>in front). All steps can be trained in parallel because every input is known. - Greedy decoding: at test time there is no true target, so we pick the most likely token, feed it back as the next input, and repeat. (Beam search keeps the few best partial outputs instead of just one.)
- Exposure bias: the model only ever saw correct previous tokens in training, so its own early mistakes at test time put it in situations it never practised.
Seq2seq in PyTorch, and the bottleneck measured
The task: reverse a string of letters. It has the same shape as translation, and it is hard for a summary vector: to write the first output letter, the decoder needs the last input letter. The model (attention is switched off by attn=False):
import torch, torch.nn as nn, torch.nn.functional as F
torch.manual_seed(73); V, H, S = 27, 64, 26 # letters a..z = 0..25, start token = 26
class Seq2Seq(nn.Module):
def __init__(s, attn):
super().__init__(); s.attn = attn; s.emb = nn.Embedding(V, 32)
s.enc = nn.GRU(32, H, batch_first=True); s.dec = nn.GRU(32, H, batch_first=True)
s.out = nn.Linear(2 * H if attn else H, 26)
def forward(s, src, dec_in):
enc_states, h = s.enc(s.emb(src)) # h = the ONE summary vector
dec_states, _ = s.dec(s.emb(dec_in), h)
if not s.attn: return s.out(dec_states), None
scores = dec_states @ enc_states.transpose(1, 2) # Luong dot scores
weights = F.softmax(scores, dim=-1) # attention weights
context = weights @ enc_states # context vectors
return s.out(torch.cat([dec_states, context], -1)), weights
def batch(n, L):
src = torch.randint(26, (n, L)); tgt = src.flip(1) # task: reverse the letters
dec_in = torch.cat([torch.full((n, 1), S), tgt[:, :-1]], 1) # teacher forcing
return src, tgt, dec_in
Training and testing (cross-entropy with teacher forcing for 1,500 steps of 64 strings; testing on 1,000 new strings with greedy decoding):
def train(L, attn, steps=1500):
m = Seq2Seq(attn); opt = torch.optim.Adam(m.parameters(), 3e-3)
for _ in range(steps):
src, tgt, dec_in = batch(64, L)
loss = F.cross_entropy(m(src, dec_in)[0].reshape(-1, 26), tgt.reshape(-1))
opt.zero_grad(); loss.backward(); opt.step()
return m
@torch.no_grad()
def greedy(m, src): # no teacher: feed its own guesses
out = torch.full((len(src), 1), S)
for _ in range(src.shape[1]):
logits, w = m(src, out); out = torch.cat([out, logits[:, -1:].argmax(-1)], 1)
return out[:, 1:], w
for L, attn in [(5, False), (25, False), (25, True)]:
m = train(L, attn); src, tgt, _ = batch(1000, L)
pred, w = greedy(m, src)
letters, words = (pred == tgt).float().mean(), (pred == tgt).all(1).float().mean()
print(f"length {L:2} attention={attn!s:5} letters right {letters:.1%} whole word right {words:.1%}")
word = "learningtoreversewithgrus"; src = torch.tensor([[ord(c) - 97 for c in word]])
pred, w = greedy(m, src)
print("input :", word); print("output:", "".join(chr(97 + i) for i in pred[0]))
print("output letter i looks most at input:", w[0].argmax(-1)[:10].tolist(), "...")
print("weights of output letter 0, last 4 inputs:", [round(x, 2) for x in w[0, 0, -4:].tolist()])
Output:
length 5 attention=False letters right 96.0% whole word right 89.8%
length 25 attention=False letters right 18.7% whole word right 0.0%
length 25 attention=True letters right 97.5% whole word right 93.2%
input : learningtoreversewithgrus
output: surghtiwesreverotgninrael
output letter i looks most at input: [24, 24, 23, 22, 21, 20, 19, 18, 17, 16] ...
weights of output letter 0, last 4 inputs: [0.0, 0.0, 0.0, 1.0]
Without attention: 5 letters → 96.0% of letters and 89.8% of whole words right. 25 letters → 18.7% of letters (random guessing among 26 letters gives about 3.8%) and 0.0% of whole words. Same model, same training.
Why it fails: the information bottleneck
Everything the decoder knows about the input is one vector of 64 numbers (h). Five letters fit; 25 letters in their exact order do not. An information bottleneck is a narrow point that all the information must squeeze through, like a crowd leaving a hall through one small door. Bahdanau, Cho and Bengio: "the use of a fixed-length vector is a bottleneck". Yet the encoder produced a state for every input token (enc_states), and we threw all but the last away.
Real-life example: the interpreter's notes
| At a meeting | In the model |
|---|---|
| A short note for every phrase, not one memory of the whole speech | Encoder states: one vector per input token |
| "Which note matters for my next word?" | Alignment scores: how well each input matches now |
| Read mostly the best note, a little of the others | Attention weights → context vector |
The attention mechanism
At every decoder step t:
- Alignment scores
e_t,j: how well encoder state j matches the decoder's current state. Simplest: the dot product. - Attention weights
α_t,j = softmax(e_t): all ≥ 0 and adding up to 1 over the input positions. - Context vector
c_t = Σ_j α_t,j · enc_state_j: a weighted sum of the encoder states, recomputed at every step. - Predict the next token from
[decoder state ; context vector](;= joined end to end).
Alignment means which input position lines up with which output position. Attention learns it with no alignment labels.
By hand, for "the black cat ." while the decoder is about to write "chat":
import numpy as np
src = ["the", "black", "cat", "."]
enc = np.array([[0.1, 0.0, 0.2], # one encoder state per source word
[0.0, 1.0, 0.1],
[1.0, 0.1, 0.9],
[0.0, 0.0, 0.1]])
dec = np.array([0.9, 0.0, 1.0]) # decoder state while writing "chat"
scores = enc @ dec # 1. alignment scores (dot product)
weights = np.exp(scores) / np.exp(scores).sum() # 2. softmax -> attention weights
context = weights @ enc # 3. context vector = weighted sum
for w, s, a in zip(src, scores, weights):
print(f"{w:6} score {s:4.2f} weight {a:.2f}")
print("weights add up to", weights.sum().round(2))
print("context:", context.round(2))
the score 0.29 weight 0.14
black score 0.10 weight 0.12
cat score 1.80 weight 0.63
. score 0.10 weight 0.12
weights add up to 1.0
context: [0.64 0.18 0.62]
"cat" has the largest score (1.8), so it gets 63% of the weight, and the context vector is mostly cat's state.
Seq2seq with attention
Lines 12-15 of the model are the attention: scores = dec_states @ enc_states.transpose(1, 2) has shape (batch, output steps, input tokens); softmax(..., dim=-1) normalises over the input tokens; weights @ enc_states gives one context vector per output step. The same run printed:
- 25 letters with attention: 97.5% of letters and 93.2% of whole words right (from 0.0%).
learningtoreversewithgrus→surghtiwesreverotgninrael, exactly reversed.- Output letter 0 puts weight 1.0 on input 24 (the last letter); then 24, 23, 22, 21, … a near-perfect reversed diagonal.
Bahdanau and Luong attention
| Bahdanau et al. (2014) | Luong, Pham, Manning (2015) | |
|---|---|---|
| Score | additive: v · tanh(W1·h_s + W2·s) (a one-hidden-layer feedforward network) |
multiplicative: dot h_t · h_s, general h_t · W·h_s (also concat) |
| Encoder | bidirectional RNN (each state knows words before and after) | top layer of a stacked LSTM |
| Variants | global (all source words) vs local (a window) |
Here h_s is an encoder state, s / h_t the decoder state, W, W1, W2 learned matrices and v a learned vector.
import numpy as np
rng = np.random.default_rng(73)
enc = rng.normal(size=(4, 3)); dec = rng.normal(size=3) # 4 source states, 1 decoder
Wa = rng.normal(size=(3, 3)); W1, W2 = rng.normal(size=(2, 5, 3)); v = rng.normal(size=5)
softmax = lambda s: np.exp(s - s.max()) / np.exp(s - s.max()).sum()
scores = {"Luong dot ": enc @ dec, # h_t . h_s
"Luong general ": enc @ (Wa @ dec), # h_t . W h_s
"Bahdanau add ": np.tanh(enc @ W1.T + W2 @ dec) @ v} # v . tanh(W1 h_s + W2 s)
for name, s in scores.items():
print(name, "scores ", s.round(2))
print(" " * 14, "weights", softmax(s).round(2))
Luong dot scores [ 1.66 -1.58 -2.93 -4.68]
weights [0.95 0.04 0.01 0. ]
Luong general scores [0.4 4.03 0.06 0.01]
weights [0.03 0.94 0.02 0.02]
Bahdanau add scores [1.67 0.89 1.83 1.19]
weights [0.31 0.14 0.36 0.19]
Untrained, the three score functions disagree; training makes them point at the useful input. All follow the same recipe: score → softmax → weighted sum.
Common mistakes
import torch
torch.manual_seed(73)
dec = torch.randn(2, 3, 8) # batch 2, 3 output steps, 8 numbers
enc = torch.randn(2, 5, 8) # batch 2, 5 input words
try:
scores = torch.bmm(dec, enc) # forgot to transpose enc
except RuntimeError as e:
print("RuntimeError:", e)
scores = torch.bmm(dec, enc.transpose(1, 2)) # (2, 3, 5)
wrong = torch.softmax(scores, dim=1) # softmax over OUTPUT steps
right = torch.softmax(scores, dim=-1) # softmax over INPUT words
print("wrong: each step's weights sum to", [f"{x:.2f}" for x in wrong[0].sum(-1)])
print("right: each step's weights sum to", [f"{x:.2f}" for x in right[0].sum(-1)])
RuntimeError: Expected size for first two dimensions of batch2 tensor to be: [2, 8] but got: [2, 5].
wrong: each step's weights sum to ['0.99', '2.84', '1.16']
right: each step's weights sum to ['1.00', '1.00', '1.00']
| Mistake | Exact message / symptom | Why | Fix |
|---|---|---|---|
torch.bmm(dec, enc) |
RuntimeError: Expected size for first two dimensions of batch2 tensor to be: [2, 8] but got: [2, 5]. |
(2,3,8) × (2,5,8) doesn't fit | torch.bmm(dec, enc.transpose(1, 2)) |
softmax(scores, dim=1) |
no error; each output step's weights sum to 0.99, 2.84, 1.16 | normalised over output steps | dim=-1 (over input tokens) |
| Word-by-word translation | le noir chat |
order and length change | encoder-decoder |
| One summary vector for long inputs | 0.0% of 25-letter words right | information bottleneck | attention |
| Evaluating with teacher forcing on | great scores, poor real outputs | real use has no true previous token | greedy or beam decoding |
| Forgetting the start token shift | the decoder sees the answer it must predict | dec_in = target instead of target shifted right |
cat([<s>], target[:-1]) |
Mini project: dates to ISO format
Turn "13 november 2026" into "2026-11-13". Input and output have different lengths and different orders: a real seq2seq task (with attention). Inputs are padded with spaces to 18 characters.
import torch, torch.nn as nn, torch.nn.functional as F, random, datetime
torch.manual_seed(73); random.seed(73)
chars = sorted(set("0123456789- abcdefghijklmnopqrstuvwxyz")); S = len(chars) # S = start token
ix = {c: i for i, c in enumerate(chars)}; H = 64
def sample():
d = datetime.date(1950, 1, 1) + datetime.timedelta(random.randrange(40000))
return f"{d.day} {d.strftime('%B').lower()} {d.year}".ljust(18), d.isoformat() # "13 november 2026"
class Seq2Seq(nn.Module):
def __init__(s):
super().__init__(); s.emb = nn.Embedding(S + 1, 32)
s.enc = nn.GRU(32, H, batch_first=True); s.dec = nn.GRU(32, H, batch_first=True)
s.out = nn.Linear(2 * H, S)
def forward(s, src, dec_in):
e, h = s.enc(s.emb(src)); d, _ = s.dec(s.emb(dec_in), h)
w = F.softmax(d @ e.transpose(1, 2), -1)
return s.out(torch.cat([d, w @ e], -1))
def batch(n):
pairs = [sample() for _ in range(n)]
src = torch.tensor([[ix[c] for c in a] for a, _ in pairs]); tgt = torch.tensor([[ix[c] for c in b] for _, b in pairs])
return src, tgt, torch.cat([torch.full((n, 1), S), tgt[:, :-1]], 1)
m = Seq2Seq(); opt = torch.optim.Adam(m.parameters(), 3e-3)
for step in range(1500):
src, tgt, dec_in = batch(64)
loss = F.cross_entropy(m(src, dec_in).reshape(-1, S), tgt.reshape(-1))
opt.zero_grad(); loss.backward(); opt.step()
@torch.no_grad()
def convert(texts):
src = torch.tensor([[ix[c] for c in t.ljust(18)] for t in texts]); out = torch.full((len(texts), 1), S)
for _ in range(10): out = torch.cat([out, m(src, out)[:, -1:].argmax(-1)], 1)
return ["".join(chars[i] for i in row[1:]) for row in out]
src, tgt, _ = batch(1000)
pred = convert(["".join(chars[i] for i in r) for r in src])
print("test dates exactly right:", f"{sum(p == ''.join(chars[i] for i in t) for p, t in zip(pred, tgt)) / 10:.1f}%")
for t, p in zip(["13 november 2026", "2 october 1969", "26 january 1950"], convert(["13 november 2026", "2 october 1969", "26 january 1950"])):
print(f"{t:18} -> {p}")
Output:
test dates exactly right: 100.0%
13 november 2026 -> 2026-11-13
2 october 1969 -> 1969-10-02
26 january 1950 -> 1950-01-26
Practice
1. Try lengths 10 and 40, with and without attention. Then print the attention weights for one 8-letter word.
Solution
import sys; sys.argv = ["x"]
exec(open("s2s.py").read().split("for L, attn in")[0]) # reuse the model, batch, train, greedy
for L, attn in [(10, False), (10, True), (40, False), (40, True)]:
m = train(L, attn); src, tgt, _ = batch(1000, L); pred, w = greedy(m, src)
print(f"length {L:2} attention={attn!s:5} whole word right {(pred == tgt).all(1).float().mean():.1%}")
word = "chaitime"; m = train(8, True); src = torch.tensor([[ord(c) - 97 for c in word]])
pred, w = greedy(m, src)
print("output:", "".join(chr(97 + i) for i in pred[0]))
print("attention weights x10 (rows = output letters, columns = input letters", list(word), "):")
for row in (w[0] * 10).round().int().clamp(max=9).tolist(): print(" ", "".join(str(v) for v in row))
length 10 attention=False whole word right 18.2%
length 10 attention=True whole word right 97.8%
length 40 attention=False whole word right 0.0%
length 40 attention=True whole word right 42.3%
output: emitiahc
attention weights x10 (rows = output letters, columns = input letters ['c', 'h', 'a', 'i', 't', 'i', 'm', 'e'] ):
00000009
00000090
00000900
00009000
00082000
00900000
19000000
90000000
s2s.py is the whole program from the episode (the model plus the training and testing code). Even 10 letters break the plain model (18.2%), while attention gets 97.8%. At 40 letters attention reaches 42.3% in the same 1,500 steps: longer sequences need more training, but without attention the score stays at 0.0%. The weights (×10, rounded, capped at 9) form a reversed diagonal: output letter i looks at input letter 7−i.
2. Replace the dot score with Luong's "general" score.
Solution
Add a matrix W between the two states: scores = dec_states @ s.W(enc_states).transpose(1, 2).
exec(open("s2s.py").read().split("for L, attn in")[0]) # reuse the model, batch, train, greedy
class General(Seq2Seq): # Luong "general": h_t . W h_s
def __init__(s, attn):
super().__init__(attn); s.W = nn.Linear(H, H, bias=False)
def forward(s, src, dec_in):
enc_states, h = s.enc(s.emb(src)); dec_states, _ = s.dec(s.emb(dec_in), h)
weights = F.softmax(dec_states @ s.W(enc_states).transpose(1, 2), dim=-1)
return s.out(torch.cat([dec_states, weights @ enc_states], -1)), weights
Seq2Seq = General
m = train(25, True); src, tgt, _ = batch(1000, 25); pred, _ = greedy(m, src)
print(f"general score, length 25: whole word right {(pred == tgt).all(1).float().mean():.1%}")
general score, length 25: whole word right 90.2%
The extra matrix lets the decoder ask for something different from what the encoder state literally contains.
3. Make the encoder bidirectional.
Solution
Hint: s.enc = nn.GRU(32, H, batch_first=True, bidirectional=True) gives states of 2H numbers, so project them back with s.proj = nn.Linear(2 * H, H) (enc_states = s.proj(enc_states)), and start the decoder from h[0:1] + h[1:2], the sum of the forward and backward final states, shape (1, batch, H). Compare its accuracy at length 25 with and without attention.
4. Why can't teacher forcing be used at test time? Answer
Because at test time the true output is what we are trying to produce; the decoder must use its own previous predictions.
Quick quiz
dec_states: (32, 10, 64) # batch, output steps, numbers
enc_states: (32, 25, 64) # batch, input words, numbers
weights = softmax(dec_states @ enc_states.transpose(1, 2), dim=-1)
# weights.shape = ?
Answer
(32, 10, 25): one weight for every (output step, input word) pair, and each row of 25 adds up to 1.
Interview questions
- What is the information bottleneck in seq2seq? The encoder compresses the whole input into one fixed-size vector, so long inputs lose information; accuracy drops with length.
- How does attention fix it? At each decoder step it computes a fresh context vector as a weighted sum of all encoder states, so the decoder can read any input position directly.
- What is teacher forcing, and what is exposure bias? Training the decoder on the true previous token; the mismatch at test time, when it reads its own (possibly wrong) tokens.
- Bahdanau vs Luong attention? Additive (feedforward) scores with a bidirectional encoder vs multiplicative (dot/general) scores on the top LSTM layer; Luong also proposed local attention.
- Why softmax over the input dimension? Each output step needs a probability distribution over input positions (weights summing to 1).
- Greedy vs beam search? Greedy takes the single best token each step; beam search keeps the k best partial outputs and usually finds better whole sequences.
- What does an attention weight matrix tell you? Which input positions each output step relied on: a soft alignment you can plot.
Glossary
| Term | Meaning, with an example |
|---|---|
| Sequence | Items whose order matters; the letters of "chai" |
| Sequence-to-sequence (seq2seq) | A model that maps one sequence to another, possibly of different length; English → French |
| Encoder-decoder architecture | An encoder network reads the input into states; a decoder network produces the output from them |
| Encoder | The network that reads the input; here a GRU over the input letters |
| Decoder | The network that writes the output one token per step |
| Hidden state | The vector an RNN carries between steps, summarising what it has read (64 numbers here) |
| Summary (context) vector, fixed | The encoder's last hidden state handed to the decoder in plain seq2seq |
Start token <s> / end token </s> |
Special tokens that begin the output / mark where it stops |
| Teacher forcing | Training the decoder with the true previous token as input; <s> c h a → c h a i |
| Greedy decoding | At test time, take the most likely token and feed it back in, step by step |
| Beam search | Keep the k best partial outputs at each step instead of one |
| Exposure bias | Train/test mismatch: true previous tokens in training, own guesses at test time |
| Information bottleneck | A narrow point all information must pass through; one 64-number vector for 25 letters |
| Attention mechanism | At each output step, score all encoder states, softmax the scores, mix the states |
| Alignment | Which input position corresponds to which output position; "chat" ↔ "cat" |
| Alignment score | A number for how well an encoder state matches the decoder's need; cat: 1.8 |
| Attention weights | Softmax of the scores: ≥ 0, sum to 1; cat: 0.63 |
| Context vector | Weighted sum of encoder states with the attention weights; recomputed every step |
| Bahdanau (additive) attention | Score = v · tanh(W1·h_s + W2·s), with a bidirectional encoder (2014) |
| Luong (multiplicative) attention | Score = dot (h_t · h_s) or general (h_t · W·h_s) (2015) |
| Global vs local attention | Attend over all input positions vs a window of them |
| Bidirectional RNN | Two RNNs, left-to-right and right-to-left, joined so each state sees both sides |
| BLEU | A 0-100 score of word-sequence overlap with human translations |
torch.bmm |
Batch matrix multiply: (b, n, m) × (b, m, p) → (b, n, p) |
Summary
- Word-by-word translation fails: order and length change between languages.
- Seq2seq: an encoder reads, a decoder writes, trained with teacher forcing and run with greedy decoding.
- One summary vector is a bottleneck: 25 letters, 0.0% words right.
- Attention: scores → softmax weights → context vector at every step: 93.2%, with a learned reversed diagonal.
- Bahdanau (additive) and Luong (dot, general) differ only in the score function.
Next: Episode 74, self-attention from scratch: queries, keys and values, scaled dot-product attention, masks and multi-head attention.