SyntaxPilot
Courses/AI: Zero to Agents/Lesson 12

AI: Zero to Agents · Lesson 12 of 40 · 12 min

Text to numbers: tokens

queued

These are the notes for AI episode 12. A neural network only eats numbers. Photos already are numbers, but how do you turn a sentence into numbers? You try one number per word (it crashes on new words), then one number per character (text becomes six times longer), and then invent the method real language models use: subword tokens built with byte pair encoding (BPE). Then you look inside GPT-2's real tokenizer to see why "strawberry" is three pieces, why Hindi costs more, and why every AI bill and limit is counted in tokens. Every Python example from the compiler runs as-is in the SyntaxPilot compiler; outputs are real (Python 3.13). The GPT-2 examples were run with the Hugging Face tokenizers library on GPT-2's published tokenizer.json.

Cutting text into pieces: a sabzi

In the kitchen In a language model
Cut vegetables into pieces before cooking Tokenization: cut text into small pieces
One piece (a cube of potato) Token: a word, part of a word, or a symbol
Labelled boxes, one per kind of piece Token ID: each kind of token's number; the list of all tokens is the vocabulary

The big question: how big should the pieces be?

Attempt 1: one number per word

# give every word in Raju's notebook a number
notebook = "Raju pays Neha . Neha pays Raju . Raju sells chai"
vocab = {}
for w in notebook.split():
    if w not in vocab:
        vocab[w] = len(vocab)
print("vocabulary:", vocab)

def encode(text):
    return [vocab[w] for w in text.split()]

print(encode("Neha sells chai"))
print(encode("Priya pays Raju"))

Output:

vocabulary: {'Raju': 0, 'pays': 1, 'Neha': 2, '.': 3, 'sells': 4, 'chai': 5}
[2, 4, 5]
Traceback (most recent call last):
  File "main.py", line 13, in <module>
    print(encode("Priya pays Raju"))
          ~~~~~~^^^^^^^^^^^^^^^^^^^
  File "main.py", line 10, in encode
    return [vocab[w] for w in text.split()]
            ~~~~~^^^
KeyError: 'Priya'

A new name crashes it. Every name, typo or new slang word would. A bigger notebook (all of Wikipedia) makes the vocabulary millions of words long, and there is still always a new word tomorrow. Words outside the vocabulary are called out-of-vocabulary (OOV) words.

The quieter problem: word forms

words = ["play", "plays", "played", "playing", "player", "replay"]
vocab = {w: i for i, w in enumerate(words)}
print(vocab)
print("play vs playing: ids", vocab["play"], "and", vocab["playing"], "- no shared piece")
{'play': 0, 'plays': 1, 'played': 2, 'playing': 3, 'player': 4, 'replay': 5}
play vs playing: ids 0 and 3 - no shared piece

Six unrelated numbers: nothing tells the model they share "play", so it must learn each one separately. Words are too big a piece.

Attempt 2: one number per character

text = "Priya pays Raju"
ids = [ord(ch) for ch in text]             # every character has a number
print(ids)
print("back to text:", "".join(chr(i) for i in ids))

page = "Raju sells chai near the station every morning. " * 40   # about one page
print("words:", len(page.split()), "  characters:", len(page))
[80, 114, 105, 121, 97, 32, 112, 97, 121, 115, 32, 82, 97, 106, 117]
back to text: Priya pays Raju
words: 320   characters: 1920

ord gives a character's number (its Unicode code point) and chr turns it back. Nothing is ever unknown. But a 320-word page becomes 1,920 tokens: six times longer, and from episode 11 the work is paid per token (six times the compute). And a single letter means almost nothing. Characters are too small a piece.

The middle way: typing on your phone

You type letter by letter at first; some chunks come up again and again ("ing", "the", "tion"); the keyboard learns to offer them as one tap; and a brand-new word can still be typed from known chunks plus letters. That's the idea of subword tokens.

The fix: byte pair encoding (BPE)

  1. Start with every word split into characters, plus an end-of-word mark (_).
  2. Count every pair of neighbouring tokens across the whole text.
  3. Merge the most frequent pair into one new token (l + o → lo, then lo + w → low).
  4. Repeat until the vocabulary is the size you want. The number of merges is a hyperparameter (episode 11).

To tokenize new text, split it into characters and apply the learned merges in the same order.

from collections import Counter
words = ["low", "lower", "lowest", "newer", "wider", "new", "low", "newest"]
corpus = [list(w) + ["_"] for w in words]      # characters, "_" = end of word

def merge(w, a, b):                            # join every a,b pair in w
    i = 0
    while i < len(w) - 1:
        if w[i] == a and w[i + 1] == b:
            w[i:i + 2] = [a + b]
        i += 1

merges = []
for step in range(1, 9):
    pairs = Counter()
    for w in corpus:
        for a, b in zip(w, w[1:]):
            pairs[a, b] += 1
    (a, b), count = pairs.most_common(1)[0]    # the most frequent pair...
    merges.append((a, b))
    for w in corpus:
        merge(w, a, b)                          # ...becomes one new token
    print(f"merge {step}: {a} + {b} -> {a + b}   (seen {count} times)")

def tokenize(word):
    w = list(word) + ["_"]
    for a, b in merges:
        merge(w, a, b)
    return w

for word in ["newest", "lowering"]:
    print(word, "->", tokenize(word))

Output:

merge 1: l + o -> lo   (seen 4 times)
merge 2: lo + w -> low   (seen 4 times)
merge 3: e + r -> er   (seen 3 times)
merge 4: er + _ -> er_   (seen 3 times)
merge 5: n + e -> ne   (seen 3 times)
merge 6: ne + w -> new   (seen 3 times)
merge 7: low + _ -> low_   (seen 2 times)
merge 8: e + s -> es   (seen 2 times)
newest -> ['new', 'es', 't', '_']
lowering -> ['low', 'er', 'i', 'n', 'g', '_']

The never-seen word "lowering" becomes low + er + single letters: no crash, and it shares low with "lower" and "lowest". (er and er_ are different tokens: one is in the middle of a word, the other at its end.)

Giving tokens IDs, and a crash

A model needs numbers, not pieces, so every token in the vocabulary gets an ID: every character seen, plus the merged tokens.

words = ["low", "lower", "lowest", "newer", "wider", "new", "low", "newest"]
base = sorted(set("".join(words))) + ["_"]               # every character seen
merged = ["lo", "low", "er", "er_", "ne", "new", "low_", "es"]   # our 8 merges
vocab = base + merged
token_id = {t: i for i, t in enumerate(vocab)}
print("vocabulary size:", len(vocab))
print("newest   ->", [token_id[t] for t in ["new", "es", "t", "_"]])
print("lowering ->", [token_id[t] for t in ["low", "er", "i", "n", "g", "_"]])

Output:

vocabulary size: 19
newest   -> [16, 18, 8, 10]
Traceback (most recent call last):
  File "main.py", line 8, in <module>
    print("lowering ->", [token_id[t] for t in ["low", "er", "i", "n", "g", "_"]])
                          ~~~~~~~~^^^
KeyError: 'g'

The letter g never appeared in our eight words, so it has no ID. BPE can only promise "never unknown" if its starting pieces can spell absolutely anything.

Bytes: the smallest pieces

Every text in a computer is stored as bytes, numbers from 0 to 255. UTF-8 is the standard way to turn any text into bytes.

for text in ["g", "lowering", "चाय", "₹50"]:
    data = text.encode("utf-8")              # any text -> bytes (numbers 0-255)
    print(f"{text!r}: {len(text)} chars -> {len(data)} bytes")
    print("   ", list(data))
print("decoded back:", bytes([224, 164, 154, 224, 164, 190, 224, 164, 175]).decode("utf-8"))

Output:

'g': 1 chars -> 1 bytes
    [103]
'lowering': 8 chars -> 8 bytes
    [108, 111, 119, 101, 114, 105, 110, 103]
'चाय': 3 chars -> 9 bytes
    [224, 164, 154, 224, 164, 190, 224, 164, 175]
'₹50': 3 chars -> 5 bytes
    [226, 130, 185, 53, 48]
decoded back: चाय

Each Devanagari character is 3 bytes in UTF-8, and the rupee sign is 3 too. Start BPE from all 256 possible bytes and nothing can ever be missing: this is byte-level BPE. GPT-2's tokenizer is exactly that: 256 bytes + 50,000 learned merges + 1 special token = 50,257 tokens (counted from its tokenizer.json).

Where BPE came from

  • BPE began as a compression trick: replace the most frequent pair of bytes with a new symbol.
  • In 2015, Rico Sennrich, Barry Haddow and Alexandra Birch (University of Edinburgh) used it for machine translation, so rare and unknown words could be handled as subword units (Neural Machine Translation of Rare Words with Subword Units). Their paper's example uses the same kind of words: low, lowest, newer, wider.
  • Subword tokens: frequent words stay whole, rare words are split, nothing is ever unknown. GPT-3 uses GPT-2's byte-level BPE tokenizer (from the GPT-3 paper): it works on bytes, so any text at all, in any language or emoji, can be encoded.

A real tokenizer: GPT-2

>>> from tokenizers import Tokenizer
>>> tok = Tokenizer.from_file("gpt2.json")     # GPT-2's real tokenizer
>>> tok.get_vocab_size()
50257
>>> for text in ["Hello, how are you?", "I love chai", "strawberry", "Priya pays Raju"]:
...     enc = tok.encode(text); print(len(enc.ids), enc.tokens, enc.ids)
6 ['Hello', ',', 'Ġhow', 'Ġare', 'Ġyou', '?'] [15496, 11, 703, 389, 345, 30]
4 ['I', 'Ġlove', 'Ġch', 'ai'] [40, 1842, 442, 1872]
3 ['st', 'raw', 'berry'] [301, 1831, 8396]
5 ['Pri', 'ya', 'Ġpays', 'ĠRaj', 'u'] [34487, 3972, 13831, 13308, 84]
  • The vocabulary has 50,257 tokens: 256 bytes, 50,000 merges and one special token.
  • Ġ marks a space before the word: " how" is one token.
  • "chai" is rare in GPT-2's training text: " ch" + "ai". "Priya" becomes "Pri" + "ya". No KeyError, just pieces.
  • "strawberry" is st + raw + berry. The model never sees the letters inside each token, which is why language models have famously miscounted the r's in strawberry.

Tokens and languages

>>> for text in ["How are you?", "आप कैसे हैं?", "<|endoftext|>"]:
...     ids = tok.encode(text).ids; print(repr(text), len(text), "chars ->", len(ids), "tokens")
'How are you?' 12 chars -> 4 tokens
'आप कैसे हैं?' 12 chars -> 19 tokens
'<|endoftext|>' 13 chars -> 1 tokens        # one special token, id 50256

Same question, same length: 4 tokens in English, 19 in Hindi, because GPT-2 learned its merges mostly from English and Hindi falls back to byte pieces. Newer tokenizers have bigger vocabularies and do much better, but OpenAI's own help pages note that languages other than English usually have a higher token-to-character ratio, which makes them more expensive.

<|endoftext|> is a special token (id 50256): a signal that one document has ended.

Why tokens matter

Rule of thumb (OpenAI help pages, English) 1 token ≈ 4 characters ≈ ¾ of a word; 100 tokens ≈ 75 words
Cost AI APIs bill per token, for input and output
Context window how much a model can read at once is counted in tokens (GPT-3: 2,048)
Speed answers are generated one token at a time

Glossary

Term Meaning Example
Tokenization cutting text into tokens "I love chai" → I, love, ch, ai
Token one piece of text a model reads "Ġhow", "berry", ","
Token ID the number for a token "Hello" = 15496 in GPT-2
Vocabulary the list of all tokens a model knows GPT-2: 50,257
Tokenizer the program that turns text into token IDs and back GPT-2's tokenizer.json
Encode / decode text → IDs / IDs → text encode("Neha sells chai") = [2, 4, 5]
Word-level tokens one token per whole word crashes on "Priya"
Out-of-vocabulary (OOV) a word not in the vocabulary KeyError: 'Priya'
Character-level tokens one token per character 320 words → 1,920 tokens
Unicode code point a character's number ord("P") = 80
Subword token a frequent piece of a word "low", "er", "berry"
Byte pair encoding (BPE) build tokens by merging the most frequent pair, repeatedly l + o → lo → low
Merge gluing two neighbouring tokens into one new token e + r -> er
End-of-word mark a marker so word endings are distinct _ in our demo
Byte a number from 0 to 255; all text is stored as bytes "g" = 103
UTF-8 the standard way to turn any text into bytes चाय = 9 bytes
Byte-level BPE BPE starting from all 256 bytes, so any text can be encoded GPT-2, GPT-3
Special token a reserved token used as a signal <|endoftext|> = 50256
Context window how many tokens a model can read at once GPT-3: 2,048
Hyperparameter a setting chosen before training (episode 11) the number of merges

Common mistakes

Mistake Code Error
A word-level vocabulary meets a new word encode("Priya pays Raju") KeyError: 'Priya'
A token missing from the base vocabulary token_id["g"] KeyError: 'g'
Passing a whole word to ord ord("chai") TypeError: ord() expected a character, but string of length 4 found
Assuming 1 token = 1 word 300 words → "300 tokens" no error; it's closer to 400
Asking a model to count letters inside words "How many r's in strawberry?" wrong answers: it sees tokens, not letters

Fixes: use subword tokens; loop over characters ([ord(ch) for ch in "chai"]); count tokens with the model's own tokenizer; for letter-level tasks, spell the word out or use code.

Mini project: your own tokenizer

Package BPE into two functions, learn_bpe(text, n_merges) and tokenize(text, merges), then give every token an ID.

from collections import Counter

def learn_bpe(text, n_merges):
    corpus = [list(w) + ["_"] for w in text.split()]
    merges = []
    for _ in range(n_merges):
        pairs = Counter((a, b) for w in corpus for a, b in zip(w, w[1:]))
        if not pairs:
            break
        (a, b), _ = pairs.most_common(1)[0]
        merges.append((a, b))
        for w in corpus:
            i = 0
            while i < len(w) - 1:
                if w[i] == a and w[i + 1] == b:
                    w[i:i + 2] = [a + b]
                i += 1
    return merges

def tokenize(text, merges):
    out = []
    for word in text.split():
        w = list(word) + ["_"]
        for a, b in merges:
            i = 0
            while i < len(w) - 1:
                if w[i] == a and w[i + 1] == b:
                    w[i:i + 2] = [a + b]
                i += 1
        out += w
    return out

text = "chai chaiwala chai chai tea teacup chai chaiwala tea"
merges = learn_bpe(text, 6)
print("merges:", merges)
tokens = tokenize("chaiwala sells tea", merges)
vocab = {t: i for i, t in enumerate(sorted(set(tokens)))}
print(tokens)
print([vocab[t] for t in tokens])
merges: [('c', 'h'), ('ch', 'a'), ('cha', 'i'), ('chai', '_'), ('a', '_'), ('t', 'e')]
['chai', 'w', 'a', 'l', 'a_', 's', 'e', 'l', 'l', 's', '_', 'te', 'a_']
[3, 8, 1, 5, 2, 6, 4, 5, 5, 6, 0, 7, 2]

"sells" never appeared, so it falls back to letters. Try more merges and more text.

Practice

  1. Run the BPE demo with 12 merges. Which new tokens appear, and how is "slowest" tokenized?
  2. Add "slower" and a second "newest" to the word list and re-run with 8 merges. What changes?
  3. Estimate the tokens in a 1,500-word English essay.
Solutions

1. Change range(1, 9) to range(1, 13) and add "slowest" to the final loop:

merge 1: l + o -> lo   (seen 4 times)
merge 2: lo + w -> low   (seen 4 times)
merge 3: e + r -> er   (seen 3 times)
merge 4: er + _ -> er_   (seen 3 times)
merge 5: n + e -> ne   (seen 3 times)
merge 6: ne + w -> new   (seen 3 times)
merge 7: low + _ -> low_   (seen 2 times)
merge 8: e + s -> es   (seen 2 times)
merge 9: es + t -> est   (seen 2 times)
merge 10: est + _ -> est_   (seen 2 times)
merge 11: low + er_ -> lower_   (seen 1 times)
merge 12: low + est_ -> lowest_   (seen 1 times)
newest -> ['new', 'est_']
lowering -> ['low', 'er', 'i', 'n', 'g', '_']
slowest -> ['s', 'lowest_']

est_ (the ending "-est") becomes a token, and whole words start appearing as single tokens.

2. With words = ["low", "lower", "lowest", "newer", "wider", "new", "low", "newest", "slower", "newest"]:

merge 1: w + e -> we   (seen 6 times)
merge 2: l + o -> lo   (seen 5 times)
merge 3: r + _ -> r_   (seen 4 times)
merge 4: n + e -> ne   (seen 4 times)
merge 5: w + _ -> w_   (seen 3 times)
merge 6: lo + we -> lowe   (seen 3 times)
merge 7: s + t -> st   (seen 3 times)
merge 8: st + _ -> st_   (seen 3 times)
newest -> ['ne', 'we', 'st_']
lowering -> ['lowe', 'r', 'i', 'n', 'g', '_']

Different text, different pair counts, different merges: a tokenizer reflects the text it was trained on (which is why GPT-2 handles English better than Hindi).

3. 1,500 ÷ 75 × 100 = about 2,000 tokens.

Quick quiz 1

An English email of 600 words. Using 100 tokens ≈ 75 words, about how many tokens? A: 450 · B: 600 · C: 800 · D: 2,400

Answer

C: about 800. 600 ÷ 75 × 100 = 800.

Quick quiz 2

Why did older models answer "2" to "How many r's are in strawberry?" A: they can't do maths · B: they see tokens, not letters · C: the word is too long · D: they were never trained on fruit

Answer

B. GPT-2's tokenizer gives st + raw + berry: the letters inside each token aren't directly visible to the model.

Interview questions

  1. What is a token? A piece of text (a word, part of a word, a symbol, or a byte) that the model reads as one unit, identified by a token ID.
  2. Why not use whole words? The vocabulary never ends (out-of-vocabulary words crash or become an "unknown" token) and related forms (play, playing) share nothing.
  3. Why not use characters? Sequences get several times longer (more compute, smaller effective context) and single characters carry little meaning.
  4. How does BPE work? Start from characters (or bytes), repeatedly merge the most frequent adjacent pair into a new token until the vocabulary reaches the chosen size; apply the merges in order to new text.
  5. Why does tokenization matter in practice? Pricing, context limits and speed are all per token; non-English text often needs more tokens; letter-level tasks (spelling, counting letters) are hard because letters are hidden inside tokens.
  6. What is a special token? A reserved token used as a signal, such as GPT-2's <|endoftext|> marking the end of a document.

Summary

  • Tokenization cuts text into tokens; each has a token ID from the vocabulary.
  • Words are too big: KeyError: 'Priya', and play/playing share nothing.
  • Characters are too small: never unknown, but 6× longer.
  • BPE merges the most frequent pair, again and again: subword tokens. Starting from all 256 bytes (byte-level BPE) means nothing is ever unknown.
  • GPT-2: 50,257 tokens; strawberry = st + raw + berry; Hindi costs more.
  • Everything is counted in tokens: cost, context window, speed. 100 tokens ≈ 75 English words.
  • Next episode: embeddings, numbers that carry meaning.