SyntaxPilot
Courses/AI: Zero to Agents/Lesson 18

AI: Zero to Agents · Lesson 18 of 40 · 11 min

Multimodal AI

queued

These are the notes for AI episode 18. So far our models only read text: tokens in, tokens out. Here you invent how an AI turns pictures and sound into something a transformer can read: pixels, image patches, the Vision Transformer (ViT), why fixed-label classifiers break, CLIP and contrastive learning, zero-shot classification, vision-language models (VLMs), and audio as samples, frames and spectrograms. Every Python example runs as-is in the SyntaxPilot compiler; outputs are real (Python 3.13).

A different kind of input (text, images, audio, video) is called a modality. A model that handles more than one is multimodal.

Real-life example: describing a photo over the phone

On the phone In a multimodal model
Your friend can't see the photo; you only have words a transformer only reads tokens
Describing it dot by dot: hopeless one token per pixel: far too many
Describing it square by square image patches
"That's a cat on a sofa": connecting it to words CLIP: matching images with text

A photo is just numbers

A digital photo is a grid of tiny dots called pixels; each pixel is three numbers (how much red, green and blue, from 0 to 255). This tiny 3 × 3 photo uses the colours of the Indian flag:

# a tiny 3 x 3 photo: each pixel is (red, green, blue), from 0 to 255
photo = [[(255, 153, 51), (255, 153, 51), (255, 153, 51)],   # saffron
         [(255, 255, 255), (0, 0, 128), (255, 255, 255)],    # white, navy
         [(19, 136, 8), (19, 136, 8), (19, 136, 8)]]         # green

pixels = sum(len(row) for row in photo)
print("pixels:", pixels)
print("numbers:", pixels * 3)
print("centre pixel:", photo[1][1])

Output:

pixels: 9
numbers: 27
centre pixel: (0, 0, 128)

An image is just numbers, like text after tokenizing. A real phone photo has millions of pixels.

Attempt: every pixel is a token

# a small photo: 224 x 224 pixels. Make every pixel one token?
width, height = 224, 224
tokens = width * height
print("tokens:", f"{tokens:,}")
print("attention scores per head:", f"{tokens * tokens:,}")
print("fits in GPT-3's 2,048-token window?", tokens <= 2048)

Output:

tokens: 50,176
attention scores per head: 2,517,630,976
fits in GPT-3's 2,048-token window? False

50,176 tokens and 2.5 billion attention scores for each attention head (attention compares every token with every token, episode 14). It doesn't even fit in GPT-3's 2,048-token window. A single pixel means almost nothing on its own, and there are far too many.

Patches

Cut the image into squares, like a chessboard, and flatten each square into one list. Each square is a patch, and each patch becomes one token.

image = [[1, 1, 0, 0],                # a tiny 4 x 4 black-and-white image
         [1, 1, 0, 0],
         [0, 0, 1, 1],
         [0, 0, 1, 1]]
P = 2                                 # patch size: 2 x 2 pixels
patches = []
for r in range(0, len(image), P):     # cut into squares, like a chessboard
    for c in range(0, len(image[0]), P):
        patch = [image[r + i][c + j] for i in range(P) for j in range(P)]
        patches.append(patch)         # flatten: one patch -> one list
for n, p in enumerate(patches):
    print("patch", n, p)
side = 224 // 16                      # a real photo, 16 x 16 patches
print("224 x 224 photo ->", side * side, "patches")
print("attention scores:", f"{(side * side) ** 2:,}")

Output:

patch 0 [1, 1, 1, 1]
patch 1 [0, 0, 0, 0]
patch 2 [0, 0, 0, 0]
patch 3 [1, 1, 1, 1]
224 x 224 photo -> 196 patches
attention scores: 38,416
Way to read a 224 × 224 photo Tokens Attention scores per head
one token per pixel 50,176 2,517,630,976
16 × 16 patches 196 38,416

The Vision Transformer (ViT)

Photo 224 x 224 ──► 196 patches (16 x 16) ──► patch embeddings ──► Transformer
                                              + position embeddings
  • Each flattened patch is turned into a vector by a learned linear projection: a patch embedding (like a word embedding, episode 13).
  • Position embeddings are added so the model knows where each patch was.
  • The vectors go into the same kind of transformer used for text.
  • Paper: Dosovitskiy et al. (Google Research), "An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale" (arXiv 2010.11929, 2020): "a pure transformer applied directly to sequences of image patches can perform very well on image classification tasks."
  • The part of a model that turns an image into vectors is the image encoder.

Attempt: a fixed list of labels

The classic way to teach "these patches mean cat": train on photos labelled by hand, from a fixed list.

# a classifier trained on a FIXED list of labels
labels = ["cat", "dog", "car"]

def classify(scores):                 # always picks one of its labels
    return labels[scores.index(max(scores))]

print(classify([0.9, 0.05, 0.05]))    # a photo of a cat
print(classify([0.4, 0.1, 0.5]))      # a photo of an auto-rickshaw
print("can it say 'auto-rickshaw'?", "auto-rickshaw" in labels)

Output:

cat
car
can it say 'auto-rickshaw'? False

A model trained on a fixed list of labels can never name anything outside it, and labelling millions of kinds of things by hand is impossible.

CLIP: learning from photos and their captions

A family photo album In CLIP
A photo with a caption under it ("Diwali at nani's house") an image-text pair (400 million collected from the internet)
Mix up the captions and match each one back contrastive learning: pull true pairs together, push wrong pairs apart
Now you can caption new photos with any words zero-shot: any sentence can be a label

An image encoder and a text encoder each output vectors of the same size, in one shared embedding space. Cosine similarity (episode 13) measures how closely two vectors point the same way (1 = same direction).

import math
def cos(a, b):                        # how similar two vectors point
    dot = sum(x * y for x, y in zip(a, b))
    size = math.sqrt(sum(x * x for x in a)) * math.sqrt(sum(y * y for y in b))
    return dot / size

# pretend encoders already turned each photo and caption into a vector
images = {"photo 1": [0.9, 0.1, 0.0], "photo 2": [0.1, 0.9, 0.1],
          "photo 3": [0.0, 0.2, 0.9]}
texts = {"a cat on a sofa": [0.8, 0.2, 0.1], "a plate of samosas": [0.1, 0.8, 0.0],
         "an auto-rickshaw": [0.1, 0.1, 0.9]}

print(" " * 9 + "".join(f"{t[:12]:>14}" for t in texts))
for name, v in images.items():        # every photo vs every caption: N x N
    print(f"{name:9}" + "".join(f"{cos(v, w):14.2f}" for w in texts.values()))

Output:

           a cat on a s  a plate of s  an auto-rick
photo 1            0.98          0.23          0.12
photo 2            0.36          0.99          0.23
photo 3            0.17          0.22          0.99

The diagonal is high: each photo matches its own caption best. Training makes this happen: in each batch of N pairs it raises the N true matches and lowers the N² − N wrong ones. CLIP (Radford et al., OpenAI, arXiv 2103.00020, 2021) was trained on "400 million (image, text) pairs collected from the internet". (The vectors in our demo are made up by hand to show the idea; real ones have hundreds of numbers and are learned.)

Zero-shot classification

Labels are just sentences, so you can write any label you like, the way the CLIP paper did with "A photo of a {label}."

import math
def cos(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    size = math.sqrt(sum(x * x for x in a)) * math.sqrt(sum(y * y for y in b))
    return dot / size
# the text encoder turns ANY sentence into a vector (pretend values)
text_vec = {"a photo of a cat": [0.8, 0.2, 0.1],
            "a photo of a samosa": [0.1, 0.8, 0.0],
            "a photo of an auto-rickshaw": [0.1, 0.1, 0.9],
            "a photo of a scooter": [0.3, 0.1, 0.7]}
new_photo = [0.05, 0.15, 0.95]        # never seen in training
scores = {t: round(cos(new_photo, v), 2) for t, v in text_vec.items()}
for t, s in scores.items():
    print(f"{s:5}  {t}")
print("best:", max(scores, key=scores.get))

Output:

 0.21  a photo of a cat
 0.16  a photo of a samosa
  1.0  a photo of an auto-rickshaw
 0.94  a photo of a scooter
best: a photo of an auto-rickshaw

Doing a task with no training examples for it is zero-shot. The scooter is close behind the auto-rickshaw, which makes sense: they look similar.

Vision-language models (VLMs)

Take the patch vectors from the image encoder, adjust them to the size of the language model's token vectors, and put them in front of the text.

# a vision-language model reads image AND text tokens together
patches = 196                         # from the image encoder
image_tokens = [f"<img{i}>" for i in range(patches)]
question = "what is written on this auto-rickshaw ?".split()

sequence = image_tokens + question    # one sequence, into the transformer
print("image tokens:", len(image_tokens))
print("text tokens: ", len(question))
print("sequence:", sequence[:2], "...", sequence[-2:])

Output:

image tokens: 196
text tokens:  7
sequence: ['<img0>', '<img1>'] ... ['auto-rickshaw', '?']

Attention lets every word of the question look at every patch. A language model that reads image tokens too is a vision-language model (VLM): that's what happens when you upload a photo to a chatbot. Images cost tokens, so a big photo uses up context window.

Sound

A microphone measures air pressure many times a second: a wave. How many measurements per second is the sample rate (16,000 is common for speech). Like patches, we cut the wave into short overlapping windows called frames.

import math
rate = 16000                          # microphone: 16,000 numbers per second
wave = [math.sin(2 * math.pi * 440 * t / rate) for t in range(rate)]
print("1 second of sound:", len(wave), "numbers")
print("first five:", [round(x, 2) for x in wave[:5]])

frame, hop = 400, 160                 # 25 ms windows, every 10 ms
frames = [wave[s:s + frame] for s in range(0, len(wave) - frame + 1, hop)]
print("frames per second:", len(frames))
print("30 seconds of speech:", len(frames) * 30, "frames")
print("instead of", f"{rate * 30:,}", "numbers")

Output:

1 second of sound: 16000 numbers
first five: [0.0, 0.17, 0.34, 0.5, 0.64]
frames per second: 98
30 seconds of speech: 2940 frames
instead of 480,000 numbers
Wave (16,000 numbers / s) ──► frames (25 ms, every 10 ms) ──► spectrogram ──► Transformer ──► text
  • For each frame, measure which pitches are loud; lined up over time this is a picture of sound: a spectrogram (time × pitch).
  • The transformer reads it in chunks, just like patches.
  • Whisper (Radford et al., OpenAI, arXiv 2212.04356, December 2022) is a speech-to-text model trained to predict transcripts for 680,000 hours of multilingual and multitask audio from the internet.
  • The pattern for every modality: cut into chunks → turn chunks into vectors → feed a transformer.

Where you already meet multimodal AI

What you do What happens inside
Photo of a restaurant bill: "what's the total?" a vision-language model reads the image tokens
Talking to a voice assistant speech-to-text first, then the language model
Searching your gallery for "beach sunset" words and photos compared in one shared space (CLIP-style)
A phone describing a photo for someone who can't see it image captioning: image in, words out

Glossary

Term Meaning Example from this episode
Modality a kind of input or output: text, image, audio, video a photo vs a sentence
Multimodal handles more than one modality a chatbot that reads photos
Pixel one dot of a digital image, stored as numbers a 224 × 224 photo has 50,176
Patch a small square of an image, flattened into one list 16 × 16 pixels → one token
Patch embedding the vector a patch is turned into like a word embedding
Position embedding a vector that tells the model where a token or patch was top-left vs bottom-right
Vision Transformer (ViT) a transformer that reads image patches "An Image is Worth 16x16 Words"
Image encoder the part that turns an image into vectors ViT inside CLIP
Fixed-label classifier picks one label from a fixed list auto-rickshaw → "car"
Image-text pair a photo plus its caption "Diwali at nani's house"
Contrastive learning learning by telling true pairs apart from wrong ones the N × N matrix
Shared embedding space image and text vectors of the same size, comparable photo 1 ≈ "a cat on a sofa"
Cosine similarity how closely two vectors point the same way (1 = same) 0.98 on the diagonal
CLIP OpenAI's model trained on 400 million image-text pairs 2021
Zero-shot doing a task with no training examples for it "a photo of an auto-rickshaw"
Vision-language model (VLM) a language model that reads image tokens too 196 image + 7 text tokens
Sample rate measurements of sound per second 16,000
Frame a short window of audio 25 ms, every 10 ms
Spectrogram a picture of sound: which pitches are loud over time what Whisper reads
Speech-to-text turning spoken audio into written text Whisper

Common mistakes

Code Exact error Why Fix
P = 3 on a 4 × 4 image IndexError: list index out of range the patch size doesn't divide the image size pick P so that width % P == 0, or pad the image
cos([0, 0, 0], [1, 2, 3]) ZeroDivisionError: float division by zero an all-zero vector has length 0 check for zero length before dividing
zip([1, 2, 3], [1, 2], strict=True) ValueError: zip() argument 2 is shorter than argument 1 image and text vectors of different sizes (plain zip silently cuts the longer one: wrong scores, no error) both encoders must output the same size; use strict=True to catch it

Mini project: a zero-shot photo sorter

Invent 3-number vectors for 5 photos (food, vehicles, animals) and 6 label sentences ("a photo of a dosa", "a photo of a bus", ...). Use cos to print the best label for each photo, and add a "not sure" answer when the best score is below 0.8.

Practice

  1. Make the demo image 6 × 6 and use P = 3. How many patches, and how long is each?
  2. Add "a photo of a bus" with vector [0.2, 0.3, 0.8] to the zero-shot demo. What changes?
  3. How many tokens is a 448 × 448 photo with 16 × 16 patches? How many attention scores?
  4. Change the audio sample rate to 8,000 (keep frame, hop = 400, 160). How many frames per second?
Solutions

1.

image = [[1, 1, 1, 0, 0, 0],            # a 6 x 6 image
         [1, 1, 1, 0, 0, 0],
         [1, 1, 1, 0, 0, 0],
         [0, 0, 0, 1, 1, 1],
         [0, 0, 0, 1, 1, 1],
         [0, 0, 0, 1, 1, 1]]
P = 3                                 # patch size: 3 x 3 pixels

patches = []
for r in range(0, len(image), P):
    for c in range(0, len(image[0]), P):
        patch = [image[r + i][c + j] for i in range(P) for j in range(P)]
        patches.append(patch)

for n, p in enumerate(patches):
    print("patch", n, p)
patch 0 [1, 1, 1, 1, 1, 1, 1, 1, 1]
patch 1 [0, 0, 0, 0, 0, 0, 0, 0, 0]
patch 2 [0, 0, 0, 0, 0, 0, 0, 0, 0]
patch 3 [1, 1, 1, 1, 1, 1, 1, 1, 1]

4 patches of 9 numbers each (6 / 3 = 2 per side).

2.

 0.21  a photo of a cat
 0.16  a photo of a samosa
  1.0  a photo of an auto-rickshaw
 0.94  a photo of a scooter
 0.96  a photo of a bus
best: a photo of an auto-rickshaw

The bus scores 0.96, close but still second. Adding a label needs no retraining.

3.

side = 448 // 16
print(side * side, 'tokens')
print(f'{(side * side) ** 2:,}', 'attention scores')
784 tokens
614,656 attention scores

Double the width and height and you get 4 times the tokens and 16 times the attention scores, which is why chatbots often shrink big photos.

4.

1 second of sound: 8000 numbers
first five: [0.0, 0.34, 0.64, 0.86, 0.98]
frames per second: 48
30 seconds of speech: 1440 frames
instead of 240,000 numbers

With the same frame size in samples, each frame now covers 50 ms instead of 25 ms, so there are about half as many.

Quick quiz 1

A 64 × 64 image with 16 × 16 patches: how many image tokens? A: 4 · B: 16 · C: 64 · D: 4,096

Answer

B: 16. 64 / 16 = 4 patches per side, 4 × 4 = 16. 4,096 would be one token per pixel.

Quick quiz 2

You want to find "a photo of a cricket bat", but nobody labelled cricket bats. Use? A: a fixed-label classifier · B: a CLIP-style text prompt · C: more pixels per token · D: a bigger sample rate

Answer

B: a CLIP-style text prompt. A new label is just a new sentence: zero-shot.

Interview questions

  1. Why not feed raw pixels to a transformer? Too many tokens: attention cost grows with the square of the length, and single pixels carry little meaning.
  2. How does a Vision Transformer tokenize an image? It splits it into fixed-size patches (e.g. 16 × 16), flattens and linearly projects each into a patch embedding, and adds position embeddings.
  3. What is contrastive learning in CLIP? In a batch of N image-text pairs, it maximizes the similarity of the N true pairs and minimizes it for the N² − N wrong pairs.
  4. How does zero-shot classification work with CLIP? Encode the image and a text prompt per label ("a photo of a {label}"), and pick the label with the highest cosine similarity.
  5. How does a vision-language model combine images and text? Image-encoder outputs are projected to the LLM's token size and placed in the same sequence as the text tokens.
  6. How is audio turned into model input? Sample the wave, cut it into short overlapping frames, compute a spectrogram, and feed it to a transformer (as in Whisper).
  7. Why do images cost context-window space? Each patch becomes a token, so bigger images mean more tokens.

Summary

  • Pixels as tokens are far too many; cut images into patches (ViT: 16 × 16, 196 tokens for 224 × 224).
  • A fixed-label classifier can't name new things.
  • CLIP learns from 400 million image-text pairs with contrastive learning into a shared space, giving zero-shot labels.
  • VLMs put image tokens and text tokens in one sequence.
  • Sound: sample rate → frames → spectrogram → transformer (Whisper, 680,000 hours).
  • Next episode: the limits of LLMs: hallucination, knowledge cutoff and more.