These are the notes for AI episode 14. Last episode you invented embeddings, but a lookup table gives the word "bank" the same vector by a river and at a loan desk. Here you fix that: you watch a plain sentence average turn every word to mush, then invent attention (a relevance-weighted average with softmax), queries, keys and values, scaling by √d_k, multi-head attention, positional encoding and the causal mask, and stack them into the Transformer from the 2017 paper Attention Is All You Need. Every Python example runs as-is in the SyntaxPilot compiler; outputs are real (Python 3.13).
Meaning needs context: the Hindi word "kal"
| Sentence | What "kal" means | Which neighbour decided |
|---|---|---|
| "kal" on its own | yesterday or tomorrow | none |
| "kal main gaya tha" (I went) | yesterday | the past-tense verb |
| "kal main jaunga" (I will go) | tomorrow | the future-tense verb |
You looked at the neighbours, and not all of them equally: "main" (I) tells you nothing, the verb decides. That is exactly what attention does.
Attempt 1: average the whole sentence
Two dimensions, money and water. Every word's new vector is the plain average of its sentence:
# word vectors from episode 13: money water
emb = {"the": [0.1, 0.1], "river": [0.0, 1.0],
"bank": [0.7, 0.7], "loan": [1.0, 0.0]}
def average(vectors): # every word counts equally
return [round(sum(col) / len(vectors), 2) for col in zip(*vectors)]
for s in ["the river bank", "the bank loan"]:
vs = [emb[w] for w in s.split()]
print(f"{s:15}", {w: average(vs) for w in s.split()})
Output:
the river bank {'the': [0.27, 0.6], 'river': [0.27, 0.6], 'bank': [0.27, 0.6]}
the bank loan {'the': [0.6, 0.27], 'bank': [0.6, 0.27], 'loan': [0.6, 0.27]}
"bank" does move (more water by the river, more money by the loan), but "the", "river" and "bank" all become the same vector. Every neighbour counted equally, even "the", so every word blended into one soup.
Real-life example: your family WhatsApp group
| In a group chat | In attention |
|---|---|
| 200 new messages | the other tokens in the sentence |
| "Is this about me?" for each message | a relevance score per token |
| Read some closely, skim the rest; attention adds up to 100% | attention weights that add up to 1 |
Scores to weights: the dot product and softmax
- Relevance score = the dot product of two vectors (episode 13): multiply matching numbers and add. Vectors pointing the same way give a big score.
- Softmax turns any list of scores into weights: raise e (≈ 2.718) to each score, then divide each by the total. The weights are always positive and add up to 1.
For "bank" in "the river bank": scores 0.14, 0.70, 0.98 → softmax → weights 0.20, 0.35, 0.46.
Attention in Python
import math
emb = {"the": [0.1, 0.1], "river": [0.0, 1.0],
"bank": [0.7, 0.7], "loan": [1.0, 0.0]}
def softmax(scores): # scores -> weights that add up to 1
e = [math.exp(s) for s in scores]
return [x / sum(e) for x in e]
def attend(word, sentence):
me = emb[word]
scores = [sum(a * b for a, b in zip(me, emb[w])) for w in sentence] # dot product
weights = softmax(scores)
new = [sum(wt * emb[w][d] for wt, w in zip(weights, sentence)) for d in range(2)]
return [round(x, 2) for x in weights], [round(x, 2) for x in new]
for s in ["the river bank", "the bank loan"]:
weights, new = attend("bank", s.split())
print(f"{s:15} weights {weights} bank -> {new}")
Output:
the river bank weights [0.2, 0.35, 0.46] bank -> [0.34, 0.69]
the bank loan weights [0.2, 0.46, 0.35] bank -> [0.69, 0.34]
By the river, bank's new vector is mostly water [0.34, 0.69]; at the loan desk it is mostly money [0.69, 0.34]. Attention = replace each token's vector with a weighted average of the tokens around it, weighted by relevance. The result is a contextual embedding: it changes with the sentence.
The next problem: what you look for is not what you are
When every word attends to the words of its own sentence, it is called self-attention. Test it on "raju made chai, drank it": which word is "it"?
import math
# thing drink person pronoun
emb = {"raju": [0, 0, 1, 0], "made": [0, 0, 0, 0], "chai": [1, 1, 0, 0],
"drank": [0, 0, 0, 0], "it": [0, 0, 0, 1]}
sentence = ["raju", "made", "chai", "drank", "it"]
def dot(a, b): return sum(x * y for x, y in zip(a, b))
def softmax(s):
e = [math.exp(x) for x in s]
return [x / sum(e) for x in e]
scores = [dot(emb["it"], emb[w]) for w in sentence]
for w, wt in zip(sentence, softmax(scores)):
print(f"it -> {w:6} {wt:.2f}")
Output:
it -> raju 0.15
it -> made 0.15
it -> chai 0.15
it -> drank 0.15
it -> it 0.40
"it" gives 40% to itself and only 15% to chai. A plain dot product rewards words that are similar to you, but a pronoun is not similar to the thing it refers to.
Query, key and value: a shopping app
| On a shopping app | In attention |
|---|---|
| You type "red cotton kurta" | Query: what this token is looking for |
| Each product's tags: kurta, cotton, red, size M | Key: what each token offers to be matched on |
| The product you get: photos, price, the real thing | Value: what a token hands over when chosen |
import math
# thing drink person pronoun
emb = {"raju": [0, 0, 1, 0], "made": [0, 0, 0, 0], "chai": [1, 1, 0, 0],
"drank": [0, 0, 0, 0], "it": [0, 0, 0, 1]}
sentence = ["raju", "made", "chai", "drank", "it"]
def dot(a, b): return sum(x * y for x, y in zip(a, b))
def softmax(s):
e = [math.exp(x) for x in s]
return [x / sum(e) for x in e]
def query(v): return [3 * v[3], 0] # "I'm a pronoun: I look for a thing"
def key(v): return [v[0], v[2]] # "I'm a thing" / "I'm a person"
def value(v): return v # what I hand over if chosen
q = query(emb["it"])
weights = softmax([dot(q, key(emb[w])) for w in sentence])
for w, wt in zip(sentence, weights):
print(f"it -> {w:6} {wt:.2f}")
new = [round(sum(wt * value(emb[w])[d] for wt, w in zip(weights, sentence)), 2) for d in range(4)]
print("new it:", new)
Output:
it -> raju 0.04
it -> made 0.04
it -> chai 0.83
it -> drank 0.04
it -> it 0.04
new it: [0.83, 0.83, 0.04, 0.04]
Now "it" gives 83% to chai and its new vector is mostly thing and drink ([thing, drink, person, pronoun]). In a real model, the query, key and value are learned weight matrices (Q = x·W_Q, K = x·W_K, V = x·W_V), trained by backpropagation like episode 9.
Scaling: divide by √d_k
import math
def softmax(s):
e = [math.exp(x) for x in s]
return [round(x / sum(e), 2) for x in e]
scores = [1.0, 2.0, 3.0]
print("small vectors: ", softmax(scores))
big = [x * 64 for x in scores] # 4,096 numbers per vector: dot products grow huge
print("big vectors: ", softmax(big))
print("scaled by sqrt: ", softmax([x / math.sqrt(4096) for x in big]))
Output:
small vectors: [0.09, 0.24, 0.67]
big vectors: [0.0, 0.0, 1.0]
scaled by sqrt: [0.09, 0.24, 0.67]
Long vectors give huge dot products, and softmax turns all-or-nothing [0.0, 0.0, 1.0]: one token grabs everything and learning nearly stops. Dividing by the square root of the key size (√d_k) brings the shares back. The full formula from the 2017 paper is scaled dot-product attention:
Attention(Q, K, V) = softmax(Q · Kᵀ / √d_k) · V
The paper's reason: large dot products push softmax "into regions where it has extremely small gradients".
Multi-head attention
One set of queries and keys asks one kind of question. Multi-head attention runs several attentions side by side, each a head with its own learned Q, K and V (like cricket selectors watching batting, bowling and fielding). One head might link pronouns to things, another match a verb's tense. The original Transformer used h = 8 heads, each 64 numbers wide (d_k = 64), joined back into d_model = 512.
Attention is order-blind
import math
emb = {"raju": [1.0, 0.0], "pays": [0.0, 0.0], "neha": [0.0, 1.0]}
def softmax(s):
e = [math.exp(x) for x in s]
return [x / sum(e) for x in e]
def self_attention(words): # every word attends to every word
out = {}
for w in words:
wts = softmax([sum(a * b for a, b in zip(emb[w], emb[o])) for o in words])
out[w] = [round(sum(t * emb[o][d] for t, o in zip(wts, words)), 3) for d in range(2)]
return out
a = self_attention(["raju", "pays", "neha"])
b = self_attention(["neha", "pays", "raju"])
print("Raju pays Neha:", a)
print("Neha pays Raju:", b)
print("Same result? ", a == b)
Output:
Raju pays Neha: {'raju': [0.576, 0.212], 'pays': [0.333, 0.333], 'neha': [0.212, 0.576]}
Neha pays Raju: {'neha': [0.212, 0.576], 'pays': [0.333, 0.333], 'raju': [0.576, 0.212]}
Same result? True
Attention compares every word with every word, like a bag of words (episode 10), so "Raju pays Neha" and "Neha pays Raju" give the same vectors.
Positional encoding: seat numbers on a train
| On a train | In a Transformer |
|---|---|
| The same passengers | the same token embeddings |
| A seat number on each ticket | a position vector for each position |
| Ticket = passenger + seat | input = embedding + position |
import math
emb = {"raju": [1.0, 0.0], "pays": [0.0, 0.0], "neha": [0.0, 1.0]}
def position(p): # a unique wave pattern per position
return [round(math.sin(p), 2), round(math.cos(p), 2)]
def softmax(s):
e = [math.exp(x) for x in s]
return [x / sum(e) for x in e]
def self_attention(words):
vecs = [emb[w] + position(p) for p, w in enumerate(words)] # word + where it is
out = {}
for w, v in zip(words, vecs):
wts = softmax([sum(a * b for a, b in zip(v, o)) for o in vecs])
out[w] = [round(sum(t * o[d] for t, o in zip(wts, vecs)), 2) for d in range(4)]
return out
a = self_attention(["raju", "pays", "neha"])
b = self_attention(["neha", "pays", "raju"])
print("raju, as payer: ", a["raju"])
print("raju, as receiver:", b["raju"])
print("Same result? ", a == b)
Output:
raju, as payer: [0.76, 0.07, 0.21, 0.82]
raju, as receiver: [0.76, 0.07, 0.84, -0.16]
Same result? False
Adding a position vector to each embedding is positional encoding. The 2017 paper used "sine and cosine functions of different frequencies"; many newer models learn their position signals instead.
The causal mask: no peeking ahead
import math
words = ["raju", "made", "chai", "for", "neha"]
def masked_weights(scores, i): # word i may only look at words 0..i
s = [x if j <= i else -math.inf for j, x in enumerate(scores)]
e = [math.exp(x) for x in s]
return [round(x / sum(e), 2) for x in e]
scores = [1.0, 1.0, 1.0, 1.0, 1.0]
for i, w in enumerate(words):
print(f"{w:5} looks at {masked_weights(scores, i)}")
Output:
raju looks at [1.0, 0.0, 0.0, 0.0, 0.0]
made looks at [0.5, 0.5, 0.0, 0.0, 0.0]
chai looks at [0.33, 0.33, 0.33, 0.0, 0.0]
for looks at [0.25, 0.25, 0.25, 0.25, 0.0]
neha looks at [0.2, 0.2, 0.2, 0.2, 0.2]
Models like GPT write one token at a time, so while training to predict the next token a word must not see the words after it. Scores for later words are set to −∞ before softmax (e^−∞ = 0). This is the causal mask (the paper: "prevent leftward information flow in the decoder to preserve the auto-regressive property").
The Transformer block
tokens → embedding + position → [ multi-head attention → add & norm → feed-forward → add & norm ] × N → better vectors
| Part | What it does |
|---|---|
| Multi-head attention | tokens share information with each other |
| Feed-forward network | a small neural network (episode 9) works on each token alone |
| Residual connection | adds each step's change to the original vector instead of replacing it |
| Layer normalization | rescales each vector so the numbers stay in a steady range |
| Stacking | a block outputs vectors of the same size, so blocks stack: 6 in the original Transformer, 96 in GPT-3 |
History: "Attention Is All You Need" (2017)
| Fact | Value |
|---|---|
| Paper | Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, Polosukhin, submitted 12 June 2017 (arXiv 1706.03762) |
| Idea | a model "based solely on attention mechanisms": no RNN, no convolutions |
| Base model | N = 6 layers, d_model = 512, 8 heads, d_ff = 2048 |
| Training | base model 12 hours, big model 3.5 days, on 8 NVIDIA P100 GPUs |
| Results | WMT 2014 English-German 28.4 BLEU, English-French 41.8 BLEU |
Why it won: an RNN (episode 10) reads one word at a time, each step waiting for the last; attention processes every word at once, which suits GPUs (episode 11).
Glossary
| Term | Meaning | Example |
|---|---|---|
| Context | the words around a word | "gaya tha" around "kal" |
| Attention | a relevance-weighted average of the other tokens' vectors | bank → [0.34, 0.69] by the river |
| Relevance score | how much one token matters to another | dot product of two vectors |
| Dot product | multiply matching numbers, add them up | sum(a * b for a, b in zip(x, y)) |
| Softmax | turns scores into positive weights that add up to 1 | [1, 2, 3] → [0.09, 0.24, 0.67] |
| Attention weights | softmax of the scores | 0.20, 0.35, 0.46 |
| Weighted average | each item counts by its weight | sum(w * v) |
| Self-attention | every token attends to the tokens of the same text | "it" looks at "chai" |
| Contextual embedding | a vector that changes with the surrounding words | two vectors for "bank" |
| Query | what a token is looking for | "I look for a thing" |
| Key | what a token offers to be matched on | "I'm a thing" |
| Value | what a token hands over when chosen | chai's vector |
| Weight matrix | learned numbers that turn a vector into Q, K or V | W_Q, W_K, W_V |
| d_k | the size of each query/key vector | 64 in the original Transformer |
| Scaled dot-product attention | softmax(Q·Kᵀ/√d_k)·V | the 2017 formula |
| Head | one attention with its own Q, K, V | 8 heads in 2017 |
| Multi-head attention | several heads side by side, results joined | 8 × 64 = 512 |
| Order-blind | gives the same result for any word order | "Raju pays Neha" = "Neha pays Raju" |
| Positional encoding | a vector per position added to each embedding | sin(p), cos(p) |
| Causal mask | hides later tokens by setting their scores to −∞ | "raju" sees only itself |
| Autoregressive | generating one token at a time, each fed back in | GPT (episode 15) |
| Feed-forward network | a small dense network applied to each token | episode 9 |
| Residual connection | output = input + change | pencil corrections on notes |
| Layer normalization | rescales a vector to a steady range | after each sub-layer |
| Transformer block | attention + feed-forward + residuals + norms | stacked N times |
| Transformer | a model made of stacked Transformer blocks | GPT, BERT |
| BLEU | a translation quality score | 28.4 English-German |
Common mistakes
| Mistake | Code | Error |
|---|---|---|
| A huge score in softmax | math.exp(1000) |
OverflowError: math range error |
| Query and key of different sizes | zip([1, 2, 3], [1, 2], strict=True) |
ValueError: zip() argument 2 is shorter than argument 1 |
| More tokens than positions | pos_table[3] with only 3 rows |
IndexError: list index out of range |
| Forgetting the √d_k scaling | big dot products | no error; softmax goes all-or-nothing [0.0, 0.0, 1.0] |
| Forgetting positions | attention on "dog bites man" | no error; same as "man bites dog" |
Note on plain zip |
zip([1, 2, 3], [1, 2]) in a dot product |
no error; silently drops the extra number |
Fixes: subtract the largest score before math.exp (same weights, no overflow); give every query and key in a head the same size d_k; stay within the context window (episode 16); always scale and always add positions.
Mini project: a pronoun resolver
Extend the Q, K, V demo with a fifth feature female and the sentence "neha met raju and gave him chai". Write query() for "him" so that it looks for a male person and check it gives the most weight to raju.
Practice
- In the attention demo, attend "bank" in
["river", "bank", "loan"]. Where does bank end up? - In the Q, K, V demo, change
query()so "it" looks for a person. Which word does it find? - In the position demo, run self-attention on "neha pays raju pays" and print every word's vector. Do the two "pays" get the same vector?
- Write a softmax that subtracts the largest score first, and test it on
[1000.0, 1001.0, 1002.0].
Solutions
1.
import math
emb = {"the": [0.1, 0.1], "river": [0.0, 1.0],
"bank": [0.7, 0.7], "loan": [1.0, 0.0]}
def softmax(scores):
e = [math.exp(s) for s in scores]
return [x / sum(e) for x in e]
def attend(word, sentence):
me = emb[word]
scores = [sum(a * b for a, b in zip(me, emb[w])) for w in sentence]
weights = softmax(scores)
new = [sum(wt * emb[w][d] for wt, w in zip(weights, sentence)) for d in range(2)]
return [round(x, 2) for x in weights], [round(x, 2) for x in new]
print(attend("bank", ["river", "bank", "loan"]))
([0.3, 0.4, 0.3], [0.58, 0.58])
Bank listens to river and loan equally (0.3 each) and stays in the middle [0.58, 0.58]: the sentence really is ambiguous.
2.
import math
# thing drink person pronoun
emb = {"raju": [0, 0, 1, 0], "made": [0, 0, 0, 0], "chai": [1, 1, 0, 0],
"drank": [0, 0, 0, 0], "it": [0, 0, 0, 1]}
sentence = ["raju", "made", "chai", "drank", "it"]
def dot(a, b): return sum(x * y for x, y in zip(a, b))
def softmax(s):
e = [math.exp(x) for x in s]
return [x / sum(e) for x in e]
def query(v): return [0, 3 * v[3]] # now: "I look for a PERSON"
def key(v): return [v[0], v[2]] # "I'm a thing" / "I'm a person"
def value(v): return v # what I hand over if chosen
q = query(emb["it"])
weights = softmax([dot(q, key(emb[w])) for w in sentence])
for w, wt in zip(sentence, weights):
print(f"it -> {w:6} {wt:.2f}")
new = [round(sum(wt * value(emb[w])[d] for wt, w in zip(weights, sentence)), 2) for d in range(4)]
print("new it:", new)
it -> raju 0.83
it -> made 0.04
it -> chai 0.04
it -> drank 0.04
it -> it 0.04
new it: [0.04, 0.04, 0.83, 0.04]
3.
import math
emb = {"raju": [1.0, 0.0], "pays": [0.0, 0.0], "neha": [0.0, 1.0]}
def position(p): # a unique wave pattern per position
return [round(math.sin(p), 2), round(math.cos(p), 2)]
def softmax(s):
e = [math.exp(x) for x in s]
return [x / sum(e) for x in e]
def self_attention(words):
vecs = [emb[w] + position(p) for p, w in enumerate(words)] # word + where it is
out = []
for w, v in zip(words, vecs):
wts = softmax([sum(a * b for a, b in zip(v, o)) for o in vecs])
out.append((w, [round(sum(t * o[d] for t, o in zip(wts, vecs)), 2) for d in range(4)]))
return out
for w, v in self_attention(["neha", "pays", "raju", "pays"]):
print(f"{w:5}", v)
neha [0.06, 0.73, 0.21, 0.76]
pays [0.25, 0.25, 0.58, 0.27]
raju [0.64, 0.06, 0.73, -0.28]
pays [0.31, 0.07, 0.46, -0.49]
The two "pays" get different vectors, because they sit in different positions.
4.
import math
def softmax(scores):
top = max(scores) # subtract the largest score first
e = [math.exp(s - top) for s in scores]
return [round(x / sum(e), 2) for x in e]
print(softmax([1.0, 2.0, 3.0]))
print(softmax([1000.0, 1001.0, 1002.0])) # no OverflowError
[0.09, 0.24, 0.67]
[0.09, 0.24, 0.67]
Subtracting the same number from every score doesn't change softmax's weights, but keeps math.exp small.
Quick quiz 1
In "The cat sat on the mat because it was tired", "it" gives cat 0.71, mat 0.12, sat 0.05, others 0.12. What do the weights add up to? A: 0.71 · B: 1.0 · C: any number · D: 100.0
Answer
B: 1.0. Softmax weights are always positive and add up to 1.
Quick quiz 2
Without positional encoding, what does self-attention see in "dog bites man" vs "man bites dog"? A: two different sentences · B: the same words, no order · C: an error · D: only the first word
Answer
B. Attention is order-blind; positions must be added.
Interview questions
- What problem does attention solve? Static embeddings give a word one vector everywhere; attention builds a contextual vector from the relevant surrounding tokens.
- Explain query, key and value. Each token projects its vector into a query (what it looks for), a key (what it offers) and a value (what it passes on); weights = softmax(Q·Kᵀ/√d_k), output = weights · V.
- Why divide by √d_k? Dot products grow with vector size; large scores push softmax into a near one-hot region with tiny gradients.
- Why multiple heads? Each head can learn a different relationship (reference, tense, phrase); their outputs are concatenated and projected.
- Why do Transformers need positional encoding? Self-attention is permutation-invariant: without positions, word order is lost.
- What is the causal mask for? In decoder (GPT-style) models it stops a token from seeing future tokens, so next-token training works and all positions train in parallel.
- Why did Transformers replace RNNs? They process all tokens in parallel (fast on GPUs) and connect distant tokens directly instead of through a fading hidden state.
Summary
- Averaging a sentence turns every word to mush.
- Attention = a relevance-weighted average; softmax turns scores into weights that add to 1.
- Query, key, value: what I look for, what I offer, what I hand over ("it" → chai 0.83).
- Scale by √d_k so softmax isn't all-or-nothing; use many heads for many questions.
- Attention is order-blind: add positional encoding. GPT-style models add a causal mask.
- Transformer block = attention + feed-forward + residuals + layer norm, stacked (6 in 2017, 96 in GPT-3).
- Next episode: what an LLM really is.