Machine Learning Full Course · Episode 72. Every code sample below was run with Python 3.13, PyTorch 2.15.0, gensim 4.4.0, Hugging Face tokenizers 0.23.3 and NLTK 3.10.4, and the output shown is the real output. The GloVe file is glove-wiki-gigaword-50.gz from the gensim-data releases (the 50-number GloVe 6B vectors).
The big idea: one-hot vectors make every pair of words equally unrelated (cosine 0) and are as long as the vocabulary. An embedding is a short, dense, learned vector per word in which similar words are close. We can learn embeddings from plain text with no labels because of the distributional hypothesis: words that appear in the same contexts tend to have similar meanings. word2vec learns them by predicting neighbours (skip-gram) or the middle word (CBOW), made cheap by negative sampling; GloVe learns them from global co-occurrence counts. In PyTorch, nn.Embedding is a trainable lookup table. Words never seen before are handled by subword tokenisation (BPE, WordPiece).
Sources: the distributional hypothesis, "words that occur in the same contexts tend to have similar meanings" (Harris, 1954), and Firth (1957), "a word is characterized by the company it keeps" (ACL wiki). Tomas Mikolov, Kai Chen, Greg Corrado and Jeffrey Dean, "Efficient Estimation of Word Representations in Vector Space", ICLR 2013 (Google Research: vectors from a 1.6-billion-word dataset in less than a day; the two architectures are Continuous Bag-of-Words and Continuous Skip-gram, summary). Mikolov, Sutskever, Chen, Corrado and Dean, "Distributed Representations of Words and Phrases and their Compositionality", 2013 (ar5iv: negative sampling; vec("Madrid") − vec("Spain") + vec("France") is closest to vec("Paris")). Jeffrey Pennington, Richard Socher and Christopher D. Manning, GloVe, 2014 (Stanford NLP: global word-word co-occurrence statistics; the 6B release is Wikipedia 2014 + Gigaword 5, 6 billion tokens, 400K words, uncased, 50/100/200/300 dimensions). Rico Sennrich, Barry Haddow and Alexandra Birch, "Neural Machine Translation of Rare Words with Subword Units" (arXiv 1508.07909: rare and unknown words encoded as sequences of subword units using byte pair encoding). Devlin et al., BERT (arXiv 1810.04805: "WordPiece embeddings ... with a 30,000 token vocabulary"; "We denote split word pieces with ##").
Words you need first
- Vector, dot product, cosine similarity (Episode 3): cosine = dot product ÷ (length × length): 1 = same direction, 0 = unrelated, −1 = opposite.
- One-hot vector (Episode 16): all zeros and a single 1 at the word's id.
- TF-IDF (Episodes 39, 71): one column per word, weighted by count and rarity.
- Sigmoid and log loss (Episode 22):
sigmoidsquashes to 0..1; binary cross-entropy rewards high probability for the right answer. - Softmax (Episode 23): turns scores for many classes into probabilities that add to 1.
The story: one-hot vectors know nothing
import numpy as np
vocab = ["good", "great", "awful", "potato"]
onehot = np.eye(len(vocab)) # one-hot: a single 1 per word
cos = lambda a, b: a @ b / (np.linalg.norm(a) * np.linalg.norm(b))
v = lambda w: onehot[vocab.index(w)]
print("good vs great :", cos(v("good"), v("great")))
print("good vs potato:", cos(v("good"), v("potato")))
print("good vs awful :", cos(v("good"), v("awful")))
print("one-hot size for a 400,000-word vocabulary:", 400_000, "numbers per word")
dense = {"good": [0.9, 0.8], "great": [0.95, 0.9], # 2 made-up "meaning" numbers
"awful": [-0.9, 0.7], "potato": [0.0, -0.9]}
d = lambda w: np.array(dense[w])
print("dense good vs great :", round(cos(d("good"), d("great")), 3))
print("dense good vs potato:", round(cos(d("good"), d("potato")), 3))
Output:
good vs great : 0.0
good vs potato: 0.0
good vs awful : 0.0
one-hot size for a 400,000-word vocabulary: 400000 numbers per word
dense good vs great : 0.999
dense good vs potato: -0.664
Good-great, good-potato and good-awful all have cosine 0: two different one-hot vectors never share a 1, so their dot product is 0. And with a 400,000-word vocabulary each vector has 400,000 numbers, almost all zeros. Two well-chosen dense numbers already separate the words, but nobody can hand-write meaning numbers for 400,000 words. The computer must learn them from text.
Real-life example: the lunch table
| At school | In text |
|---|---|
| Who a new student sits with at lunch | Context words: the words around a word |
| Same friends → probably similar interests | Similar contexts → similar vectors (chai and coffee) |
| "I drink ___ every morning": anyone can fill it | The training task: predict neighbours, labels for free |
The distributional hypothesis and word2vec
- Distributional hypothesis: words in the same contexts tend to have similar meanings (Harris 1954; Firth 1957: "a word is characterized by the company it keeps").
- word2vec (Mikolov et al., 2013): learn each word's vector by predicting its neighbours.
- Skip-gram: from a word, predict each of its context words.
- CBOW (continuous bag of words): from the (averaged) context words, predict the middle word.
- Window: how many words on each side count as context. Window 2 in "i drink chai every morning" gives chai the context i, drink, every, morning.
Negative sampling and the contrastive idea
Predicting a neighbour as a 400,000-way softmax means scoring every word for every pair. Negative sampling instead takes one real (word, context) pair and k random "negative" words, and trains
loss = −log σ(w · c) − Σ_k log σ(−w · n_k)
so real pairs get a large dot product and random pairs a small one. Pull real pairs together, push random pairs apart is the idea of contrastive learning, also used to learn image and sentence embeddings.
word2vec from scratch (skip-gram with negative sampling)
import torch, random, torch.nn.functional as F
random.seed(72); torch.manual_seed(72)
slots = {"D": "chai coffee lassi", "Q": "dosa idli paratha", "C": "delhi mumbai pune"}
templates = ["i drink D every morning", "a cup of D please", "the D is too sweet",
"we ate Q for breakfast", "my mother makes soft Q", "hot Q with chutney",
"the train to C is late", "i live in C", "C is a big city"]
corpus = [" ".join(random.choice(slots[w].split()) if w in slots else w
for w in random.choice(templates).split()).split() for _ in range(2000)]
vocab = sorted({w for s in corpus for w in s}); ix = {w: i for i, w in enumerate(vocab)}
pairs = [(ix[s[i]], ix[s[j]]) for s in corpus for i in range(len(s)) # window of 2
for j in range(max(0, i - 2), min(len(s), i + 3)) if i != j]
center, context = torch.tensor(pairs).T
E_in, E_out = torch.nn.Embedding(len(vocab), 10), torch.nn.Embedding(len(vocab), 10)
opt = torch.optim.Adam([*E_in.parameters(), *E_out.parameters()], lr=0.05)
for epoch in range(150): # skip-gram with negative sampling
neg = torch.randint(len(vocab), (len(center), 5)) # 5 random words each
pos = (E_in(center) * E_out(context)).sum(1) # real pairs: up
negs = (E_in(center).unsqueeze(1) * E_out(neg)).sum(2) # random pairs: down
loss = -(F.logsigmoid(pos).mean() + F.logsigmoid(-negs).mean())
opt.zero_grad(); loss.backward(); opt.step()
print(f"{len(corpus)} sentences, {len(vocab)} words, {len(pairs)} pairs, loss {loss:.3f}")
V = F.normalize(E_in.weight.detach(), dim=1)
for w in ["chai", "dosa", "delhi"]:
best = (V @ V[ix[w]]).argsort(descending=True)[1:3]
print(f"closest to {w:6}:", [(vocab[i], round((V[i] @ V[ix[w]]).item(), 2)) for i in best])
Output:
2000 sentences, 39 words, 27072 pairs, loss 0.453
closest to chai : [('coffee', 0.99), ('lassi', 0.94)]
closest to dosa : [('paratha', 0.98), ('idli', 0.98)]
closest to delhi : [('pune', 0.99), ('mumbai', 0.98)]
We never said what chai or Pune is; the only signal was which words appeared near which. Words that share neighbours got almost the same vectors.
GloVe, similarity and analogies
GloVe learns vectors so that their dot products match how often word pairs co-occur (appear near each other) across the whole corpus. The pretrained 6B vectors cover 400,000 lower-case words.
from gensim.models import KeyedVectors
glove = KeyedVectors.load_word2vec_format("glove50.gz") # GloVe 6B, 50 numbers per word
print("words:", len(glove.index_to_key), "| numbers per word:", glove.vector_size)
for w in ("tea", "chai", "cricket"):
print(f"{w:7} ->", [x for x, s in glove.most_similar(w, topn=5)])
print("similarity good/great:", round(glove.similarity("good", "great"), 3),
"| good/potato:", round(glove.similarity("good", "potato"), 3))
for a, b, c in [("king", "man", "woman"), ("paris", "france", "india"), ("walking", "walk", "swim")]:
best = glove.most_similar(positive=[a, c], negative=[b], topn=1)[0]
print(f"{a} - {b} + {c} = {best[0]} ({best[1]:.3f})")
Output (the 66 MB vector file loads in about 15 seconds):
words: 400000 | numbers per word: 50
tea -> ['coffee', 'drinks', 'drink', 'teas', 'fruit']
chai -> ['po', 'tai', 'lai', 'wai', 'hsin']
cricket -> ['indies', 'twenty20', 'rugby', 'cricketers', 'england']
similarity good/great: 0.798 | good/potato: 0.327
king - man + woman = queen (0.852)
paris - france + india = delhi (0.889)
walking - walk + swim = swimming (0.807)
- Embedding similarity: cosine between two embeddings (good/great 0.798 vs good/potato 0.327).
- Analogy by vector arithmetic:
a − b + cand take the nearest word: king − man + woman = queen; paris − france + india = delhi; walking − walk + swim = swimming. - Embeddings learn the corpus, not the dictionary: in English news and Wikipedia "chai" is mostly a Chinese surname, so its neighbours are po, tai, lai.
The embedding layer: nn.Embedding
nn.Embedding(num_embeddings, embedding_dim) is a table with one row per id. emb(ids) replaces every id with its row; this equals one-hot × table, without building the one-hot vectors. The rows are ordinary weights, trained by backpropagation; nn.Embedding.from_pretrained(vectors, freeze=True) starts from vectors such as GloVe and keeps them fixed.
import torch, torch.nn as nn
torch.manual_seed(72)
emb = nn.Embedding(num_embeddings=5, embedding_dim=3) # 5 words, 3 numbers each
print("the table (5 x 3):\n", emb.weight.data.numpy().round(3))
ids = torch.tensor([[1, 4, 4], [0, 2, 3]]) # 2 sentences of 3 word ids
print("lookup shape:", tuple(emb(ids).shape)) # (batch, words, numbers)
onehot = nn.functional.one_hot(torch.tensor(4), 5).float()
print("row 4: ", emb.weight[4].data.numpy().round(3))
print("one-hot @ table:", (onehot @ emb.weight).data.numpy().round(3))
print("weights:", emb.weight.numel(), "| trainable:", emb.weight.requires_grad)
pre = torch.tensor([[0.9, 0.8], [0.95, 0.9]]) # e.g. vectors from GloVe
frozen = nn.Embedding.from_pretrained(pre, freeze=True)
print("pretrained, frozen:", frozen(torch.tensor([1])).numpy(), frozen.weight.requires_grad)
Output:
the table (5 x 3):
[[-1. -0.725 -0.356]
[-0.271 2.15 0.478]
[-2.956 -0.357 -0.377]
[-0.921 1.724 0.413]
[-1.846 -0.003 -1.055]]
lookup shape: (2, 3, 3)
row 4: [-1.846 -0.003 -1.055]
one-hot @ table: [-1.846 -0.003 -1.055]
weights: 15 | trainable: True
pretrained, frozen: [[0.95 0.9 ]] False
Subword tokenisation: BPE and WordPiece
A word-level vocabulary is fixed, so a new word becomes the unknown token [UNK] (an out-of-vocabulary word). Characters never fail but make sequences long. Subword tokenisation keeps common words whole and splits rare ones into frequent pieces.
Byte pair encoding (BPE): start from characters (plus an end-of-word mark _), count every neighbouring pair weighted by word frequency, merge the most frequent pair into a new symbol, repeat. Encoding a new word applies the learned merges in order.
import re; from collections import Counter
vocab = {"l o w _": 5, "l o w e r _": 2, "n e w e s t _": 6, "w i d e s t _": 3} # _ = end
merges = []
glue = lambda a, b, s: re.sub(r"(?<!\S)" + re.escape(a + " " + b) + r"(?!\S)", a + b, s)
for step in range(8): # BPE: merge the most frequent pair
pairs = Counter()
for word, n in vocab.items():
s = word.split()
for a, b in zip(s, s[1:]): pairs[a, b] += n
(a, b), n = pairs.most_common(1)[0]
merges.append((a, b))
vocab = {glue(a, b, w): c for w, c in vocab.items()}
print(f"merge {step + 1}: {a!r} + {b!r} (seen {n} times)")
print("tokens:", list(vocab))
def encode(word): # apply the merges, in order
s = " ".join(word) + " _"
for a, b in merges: s = glue(a, b, s)
return s.split()
print("unseen 'lowest' ->", encode("lowest"), "| 'slow' ->", encode("slow"))
Output:
merge 1: 'e' + 's' (seen 9 times)
merge 2: 'es' + 't' (seen 9 times)
merge 3: 'est' + '_' (seen 9 times)
merge 4: 'l' + 'o' (seen 7 times)
merge 5: 'lo' + 'w' (seen 7 times)
merge 6: 'n' + 'e' (seen 6 times)
merge 7: 'ne' + 'w' (seen 6 times)
merge 8: 'new' + 'est_' (seen 6 times)
tokens: ['low _', 'low e r _', 'newest_', 'w i d est_']
unseen 'lowest' -> ['low', 'est_'] | 'slow' -> ['s', 'low', '_']
WordPiece: a close cousin used by BERT (30,000 pieces); a piece inside a word starts with ##.
from nltk.corpus import movie_reviews as mr
from tokenizers import Tokenizer, models, trainers, pre_tokenizers
tok = Tokenizer(models.WordPiece(unk_token="[UNK]")) # WordPiece, as used by BERT
tok.pre_tokenizer = pre_tokenizers.Whitespace()
trainer = trainers.WordPieceTrainer(vocab_size=8000, special_tokens=["[UNK]"],
show_progress=False)
tok.train_from_iterator((mr.raw(f) for f in mr.fileids()), trainer) # 2,000 reviews
print("vocabulary:", tok.get_vocab_size(), "pieces")
for w in ["movie", "unbelievably", "rewatchable", "biryani", "overacting"]:
print(f"{w:13} -> {tok.encode(w).tokens}")
Output:
vocabulary: 8000 pieces
movie -> ['movie']
unbelievably -> ['unbelie', '##v', '##ably']
rewatchable -> ['re', '##w', '##atch', '##able']
biryani -> ['bir', '##y', '##an', '##i']
overacting -> ['overact', '##ing']
Common mistakes
import torch, torch.nn as nn
emb = nn.Embedding(num_embeddings=5, embedding_dim=3) # ids 0..4 only
try:
print(emb(torch.tensor([1, 5]))) # word id 5 does not exist
except IndexError as e:
print("IndexError:", e)
IndexError: index out of range in self
| Mistake | Exact message / symptom | Why | Fix |
|---|---|---|---|
An id ≥ num_embeddings |
IndexError: index out of range in self (uncaught, it ends a long traceback through torch's internals) |
the table has rows 0 ... n−1 | num_embeddings = vocabulary size + special tokens |
| Comparing words with one-hot vectors | every similarity is 0.0 | one-hot vectors carry no meaning | learned dense embeddings |
| Trusting pretrained vectors blindly | chai → po, tai, lai | vectors reflect their training corpus | inspect neighbours; train or fine-tune on your text |
| Word-level vocabulary for new text | new words become [UNK] |
the table is fixed | subword tokenisation |
| Fine-tuning embeddings on a tiny dataset | vectors drift, overfitting | few examples per word | from_pretrained(..., freeze=True) |
| Forgetting that GloVe 6B is lower case | KeyError: "Key 'Delhi' not present" |
the vocabulary is uncased | lower-case before lookup |
Mini project: average GloVe vectors as review features
Download glove-wiki-gigaword-50.gz from the gensim-data releases on GitHub as glove50.gz (or use gensim.downloader.load("glove-wiki-gigaword-50")). Represent each movie review (Episode 71) by the average of its words' GloVe vectors (50 numbers) and train logistic regression.
import numpy as np, re
from gensim.models import KeyedVectors
from nltk.corpus import movie_reviews as mr
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
glove = KeyedVectors.load_word2vec_format("glove50.gz")
docs = [mr.raw(f) for f in mr.fileids()]; y = [mr.categories(f)[0] for f in mr.fileids()]
def doc_vector(text): # average the GloVe vectors of the known words
words = [w for w in re.findall(r"[a-z']+", text) if w in glove.key_to_index]
return np.mean([glove[w] for w in words], axis=0)
X = np.stack([doc_vector(d) for d in docs])
print("features per review:", X.shape[1])
s = cross_val_score(LogisticRegression(max_iter=2000, C=10), X, y, cv=5)
print(f"average GloVe + logistic regression: 5-fold accuracy {s.mean():.1%} ± {s.std():.1%}")
Output:
features per review: 50
average GloVe + logistic regression: 5-fold accuracy 72.2% ± 2.3%
72.2% against 87.8% for TF-IDF in Episode 71: averaging throws away order and which words matter, and general-purpose 50-number vectors miss sentiment detail. Embeddings shine when a model learns how to combine them (an LSTM, Episode 70, or attention, Episode 73).
Practice
1. Add a fourth slot (cricket words) to our word2vec.
Solution
Add "B": ["bat", "ball", "stumps"] to slots, three templates ("he hit the B hard", "a new B for the match", "the B is in the bag"), and "bat" to the printed words.
2000 sentences, 48 words, 28804 pairs, loss 0.456
closest to chai : [('lassi', 1.0), ('coffee', 0.99)]
closest to dosa : [('paratha', 0.99), ('idli', 0.94)]
closest to delhi : [('pune', 0.94), ('mumbai', 0.81)]
closest to bat : [('stumps', 1.0), ('ball', 0.99)]
2. Turn the skip-gram into CBOW: average the context vectors and predict the middle word.
Solution
import torch, random
import torch.nn.functional as F
random.seed(72); torch.manual_seed(72)
slots = {"D": ["chai", "coffee", "lassi"], "Q": ["dosa", "idli", "paratha"],
"C": ["delhi", "mumbai", "pune"]}
templates = ["i drink D every morning", "a cup of D please", "the D is too sweet",
"we ate Q for breakfast", "my mother makes soft Q", "hot Q with chutney",
"the train to C is late", "i live in C", "C is a big city"]
corpus = []
for _ in range(2000): # 2,000 made-up sentences
s = random.choice(templates)
for k, words in slots.items(): s = s.replace(k, random.choice(words))
corpus.append(s.split())
vocab = sorted({w for s in corpus for w in s}); ix = {w: i for i, w in enumerate(vocab)}
rows = [] # CBOW: (up to 4 context ids, middle id)
for s in corpus:
for i in range(len(s)):
c = [ix[s[j]] for j in range(max(0, i - 2), min(len(s), i + 3)) if j != i]
rows.append((c + [0] * (4 - len(c)), len(c), ix[s[i]]))
ctx = torch.tensor([r[0] for r in rows]); n_ctx = torch.tensor([r[1] for r in rows])
mid = torch.tensor([r[2] for r in rows])
mask = (torch.arange(4) < n_ctx.unsqueeze(1)).float()
E_in, E_out = torch.nn.Embedding(len(vocab), 10), torch.nn.Embedding(len(vocab), 10)
opt = torch.optim.Adam([*E_in.parameters(), *E_out.parameters()], lr=0.05)
for epoch in range(150): # CBOW with negative sampling
neg = torch.randint(len(vocab), (len(mid), 5)) # 5 random words each
ctx_mean = (E_in(ctx) * mask.unsqueeze(2)).sum(1) / mask.sum(1, keepdim=True) # average
pos = (ctx_mean * E_out(mid)).sum(1) # predict the middle word
negs = (ctx_mean.unsqueeze(1) * E_out(neg)).sum(2)
loss = -(F.logsigmoid(pos).mean() + F.logsigmoid(-negs).mean())
opt.zero_grad(); loss.backward(); opt.step()
print(f"{len(corpus)} sentences, {len(rows)} (context, word) rows, loss {loss:.3f}")
V = F.normalize(E_in.weight.detach(), dim=1)
for w in ["chai", "dosa", "delhi"]:
sims = V @ V[ix[w]]; best = sims.argsort(descending=True)[1:3]
print(f"closest to {w:6}:", [(vocab[i], round(sims[i].item(), 2)) for i in best])
2000 sentences, 9808 (context, word) rows, loss 0.170
closest to chai : [('coffee', 0.88), ('lassi', 0.78)]
closest to dosa : [('paratha', 0.99), ('idli', 0.97)]
closest to delhi : [('pune', 0.98), ('mumbai', 0.96)]
3. Find analogies GloVe gets wrong.
Solution
from gensim.models import KeyedVectors
glove = KeyedVectors.load_word2vec_format("glove50.gz")
for a, b, c in [("doctor", "man", "woman"), ("tokyo", "japan", "nepal"), ("bigger", "big", "small"),
("sitar", "india", "spain"), ("better", "good", "bad")]:
top = glove.most_similar(positive=[a, c], negative=[b], topn=3)
print(f"{a} - {b} + {c} ->", [(w, round(s, 2)) for w, s in top])
doctor - man + woman -> [('nurse', 0.84), ('child', 0.77), ('pregnant', 0.76)]
tokyo - japan + nepal -> [('kathmandu', 0.93), ('dhaka', 0.86), ('katmandu', 0.84)]
bigger - big + small -> [('larger', 0.92), ('smaller', 0.88), ('large', 0.88)]
sitar - india + spain -> [('aimar', 0.74), ('boozer', 0.73), ('osvaldo', 0.72)]
better - good + bad -> [('worse', 0.9), ('too', 0.83), ('unfortunately', 0.82)]
Right: Kathmandu, worse; nearly right: smaller is second after larger; wrong: sitar − india + spain gives football names, not guitar. And doctor − man + woman = nurse shows that embeddings also learn the stereotypes in their text (Episode 84 covers fairness).
4. Run 12 BPE merges on your own word list.
Solution
import re; from collections import Counter
vocab = {" ".join(w) + " _": n for w, n in {"playing": 4, "played": 3, "player": 5, "plays": 2, "replay": 2}.items()}
merges = []
glue = lambda a, b, s: re.sub(r"(?<!\S)" + re.escape(a + " " + b) + r"(?!\S)", a + b, s)
for step in range(12): # BPE: merge the most frequent pair
pairs = Counter()
for word, n in vocab.items():
s = word.split()
for a, b in zip(s, s[1:]): pairs[a, b] += n
(a, b), n = pairs.most_common(1)[0]
merges.append((a, b))
vocab = {glue(a, b, w): c for w, c in vocab.items()}
print(f"merge {step + 1}: {a!r} + {b!r} (seen {n} times)")
print("tokens:", list(vocab))
def encode(word): # apply the merges, in order
s = " ".join(word) + " _"
for a, b in merges: s = glue(a, b, s)
return s.split()
print("unseen: replayed ->", encode("replayed"), "| players ->", encode("players"))
merge 1: 'p' + 'l' (seen 16 times)
merge 2: 'pl' + 'a' (seen 16 times)
merge 3: 'pla' + 'y' (seen 16 times)
merge 4: 'play' + 'e' (seen 8 times)
merge 5: 'playe' + 'r' (seen 5 times)
merge 6: 'player' + '_' (seen 5 times)
merge 7: 'play' + 'i' (seen 4 times)
merge 8: 'playi' + 'n' (seen 4 times)
merge 9: 'playin' + 'g' (seen 4 times)
merge 10: 'playing' + '_' (seen 4 times)
merge 11: 'playe' + 'd' (seen 3 times)
merge 12: 'played' + '_' (seen 3 times)
tokens: ['playing_', 'played_', 'player_', 'play s _', 'r e play _']
unseen: replayed -> ['r', 'e', 'played_'] | players -> ['player', 's', '_']
Quick quiz
emb = nn.Embedding(10000, 64)
ids = torch.randint(10000, (32, 20))
out = emb(ids)
# out.shape = ?
Answer
(32, 20, 64): every id is replaced by its 64-number row. The table itself is (10000, 64).
Interview questions
- Why not one-hot vectors? They are huge, sparse and orthogonal: every pair of words has similarity 0, so nothing learned about one word transfers to another.
- Skip-gram vs CBOW? Skip-gram predicts context words from the centre word (often better for rare words); CBOW predicts the centre word from the averaged context (faster).
- What problem does negative sampling solve? It replaces a softmax over the whole vocabulary with a few binary decisions (real pair vs random words), making training cheap.
- word2vec vs GloVe? word2vec learns from local windows by prediction; GloVe fits vectors to global co-occurrence counts. Both give similar-quality static vectors.
- What is a limitation of static embeddings? One vector per word, whatever the context ("bank" of a river and a bank for money get the same vector); contextual models (transformers, Episodes 74-76) fix this.
- Why subword tokenisation? No unknown words, a vocabulary of a fixed size, and shared pieces (play, ##ing) let rare words borrow meaning.
- What does
nn.Embeddingcompute? A row lookup in a weight matrix, equivalent to multiplying a one-hot vector by that matrix.
Glossary
| Term | Meaning, with an example |
|---|---|
| One-hot representation | A vector with a single 1 at the word's id; good = [1, 0, 0, 0] |
| Dense representation | A short vector where every number is used; good = [0.9, 0.8] |
| Embedding | A learned dense vector for a word (or item); 50 numbers per word in GloVe 6B-50d |
| Vocabulary | The list of all tokens a model knows, each with an id |
| Distributional hypothesis | Words in similar contexts have similar meanings (Harris, Firth) |
| Context word / window | The words around a word / how many on each side count |
| word2vec | Mikolov et al.'s 2013 method that learns embeddings by predicting neighbours |
| Skip-gram | word2vec variant: centre word → each context word |
| CBOW | Continuous bag of words: averaged context → centre word |
| Negative sampling | Train on one real pair plus a few random "wrong" words instead of a full softmax |
| Contrastive learning | Learning by pulling matching pairs together and pushing non-matching pairs apart |
| GloVe | Global Vectors (Pennington, Socher, Manning, 2014): embeddings fitted to co-occurrence counts |
| Co-occurrence | How often two words appear near each other in a corpus |
| Cosine similarity | Similarity of two vectors' directions, −1 to 1; good/great = 0.798 |
| Analogy (vector arithmetic) | a − b + c ≈ d; king − man + woman ≈ queen |
nn.Embedding |
PyTorch's trainable lookup table of embeddings |
from_pretrained |
Build an embedding layer from given vectors; freeze=True keeps them fixed |
Out-of-vocabulary (OOV) / [UNK] |
A word not in the vocabulary / the token used for it |
| Subword tokenisation | Splitting rare words into frequent pieces; overacting → overact ##ing |
| BPE (byte pair encoding) | Subword method that repeatedly merges the most frequent pair of symbols |
| WordPiece | BERT's subword method; ## marks a piece inside a word |
Summary
- One-hot vectors give cosine 0 for every pair of words; we need learned dense embeddings.
- The distributional hypothesis lets us learn them from plain text: our skip-gram with negative sampling put chai next to coffee and dosa next to paratha.
- GloVe's pretrained vectors do arithmetic (king − man + woman = queen) but reflect their corpus (chai as a surname).
nn.Embeddingis a trainable lookup table, equal to one-hot × table.- BPE and WordPiece split unseen words into known pieces: lowest → low + est.
Next: Episode 73, seq2seq and attention: encoder-decoder models, teacher forcing, the bottleneck and the attention mechanism.