SyntaxPilot
Courses/AI: Zero to Agents/Lesson 19

AI: Zero to Agents · Lesson 19 of 40 · 10 min

Limits of LLMs

queued

These are the notes for AI episode 19. You know how an LLM is built, trained into an assistant and taught to see and hear. Here is the other side: where it fails, and why. You'll meet hallucination, why tests reward guessing over "I don't know" (abstaining), the knowledge cutoff, why models struggle to count letters, why tools are the fix for exact work, bias learned from training data, and a checklist for using an LLM safely. Every Python example runs as-is in the SyntaxPilot compiler; outputs are real (Python 3.13).

Real-life example: the uncle who knows everything

The uncle An LLM
Has an answer for every question always produces a fluent reply
"Which bus goes to the airport?" "Number 47, definitely!" sounds sure
There is no bus 47; it just sounded right hallucination: plausible, confident, false
Never says "I don't know" rarely abstains unless it is trained and asked to

Hallucination: likely words, not facts

The word-after-word model from episode 15, trained on four sentences that are all true:

import random
from collections import defaultdict
random.seed(5)
facts = """the capital of india is delhi . the largest planet is jupiter .
the capital of japan is tokyo . the fastest animal is the cheetah ."""

nxt, w = defaultdict(list), facts.split()   # every sentence here is TRUE
for a, b in zip(w, w[1:]):
    nxt[a].append(b)

for _ in range(4):
    out = ["the", "capital", "of", "india", "is"]
    while out[-1] != ".":
        out.append(random.choice(nxt[out[-1]]))
    print(" ".join(out))

Output:

the capital of india is tokyo .
the capital of india is delhi .
the capital of india is jupiter .
the capital of india is jupiter .

After the word "is", the model has seen "delhi", "jupiter", "tokyo" and "the". It picks a likely next word and never checks whether the sentence is true. A fluent, confident answer that is false is a hallucination.

A real case: Mata v. Avianca (New York, 2023)

  • Lawyers used ChatGPT to help write a filing in a US federal court.
  • It cited numerous court cases that did not exist, with invented quotations; ChatGPT even assured them the cases were real.
  • Judge P. Kevin Castel sanctioned the lawyers with a $5,000 fine (decision of 22 June 2023).

Why guessing pays

A 2025 paper by Adam Tauman Kalai, Ofir Nachum, Santosh Vempala and Edwin Zhang (OpenAI and Georgia Tech), "Why Language Models Hallucinate" (arXiv 2509.04664), argues that most tests give no credit for "I don't know", so guessing beats honesty. Simulate an exam:

import random
random.seed(2)
# 100 exam questions. The model truly knows 60 answers.
def exam(strategy, wrong_penalty):
    score = 0
    for q in range(100):
        if q < 60:
            score += 1                          # knows it: correct
        elif strategy == "guess":               # doesn't know: guess
            if random.random() < 0.25: score += 1
            else: score -= wrong_penalty
        # "I don't know" scores 0
    return score
for penalty in [0, 1]:
    print(f"wrong answer costs {penalty}:",
          "guesser", exam("guess", penalty), "| honest", exam("idk", penalty))

Output:

wrong answer costs 0: guesser 69 | honest 60
wrong answer costs 1: guesser 28 | honest 60
Exam rule Guesser Honest ("I don't know")
a wrong answer costs 0 69 60
a wrong answer costs 1 28 60

When a wrong answer costs nothing, bluffing wins, like a student on an exam. Saying "I don't know" is called abstaining, and scoring should reward it.

Which facts get made up?

  • Facts seen thousands of times (the capital of India) are usually right.
  • Rare facts with no pattern to learn (a person's birthday, a thesis title) get guessed: the same paper asked a model for one author's birthday three times and got three different wrong dates.
  • Rule of thumb: the rarer the fact, the more you should check it.

The knowledge cutoff

Training data is collected up to some date and then frozen.

# the model's knowledge: frozen on the day training data was collected
learned = {"school topper 2023": "aman", "school topper 2024": "neha"}

def model(question):
    if question in learned:
        return learned[question]
    # never seen it: return the most similar thing it DID see
    best = max(learned, key=lambda q: sum(a == b for a, b in zip(q, question)))
    return learned[best] + "   (sounds sure!)"

print("2024 topper:", model("school topper 2024"))
print("2026 topper:", model("school topper 2026"))

Output:

2024 topper: neha
2026 topper: aman   (sounds sure!)

The 2026 question gets the 2023 answer, stated just as confidently. The date where a model's training data stops is its knowledge cutoff. The fix is to give the model fresh information: web search, or your own documents (RAG, episode 24).

Counting letters: tokens hide letters

The model never sees letters; it sees token ids (tokens, episode 12).

# the model never sees letters: it sees token ids
vocab = {"DE": 101, "EP": 102, "SE": 103, "EK": 104}

def tokenize(word):                   # split into 2-letter pieces
    return [word[i:i + 2] for i in range(0, len(word), 2)]

pieces = tokenize("DEEPSEEK")
print("pieces:", pieces)
print("what the model sees:", [vocab[p] for p in pieces])
print("how many D? (python):", "DEEPSEEK".count("D"))

Output:

pieces: ['DE', 'EP', 'SE', 'EK']
what the model sees: [101, 102, 103, 104]
how many D? (python): 1

The letter D isn't visible in [101, 102, 103, 104]; the model has to remember how each token is spelled. The hallucination paper reports models answering anywhere from 2 to 7 when asked how many D's are in DEEPSEEK. One line of Python gets 1, every time. The fix for exact work is a tool: let the model run code or use a calculator (function calling, episode 28).

Exact maths: predicting digits is not calculating

# a "model" that learned maths only from text it has seen
seen = {"2 + 2": "4", "12 + 30": "42", "25 * 4": "100", "123 * 456": "56088"}

def model(question):
    if question in seen:
        return seen[question]
    # never seen: copy the answer of the most similar-looking question
    best = max(seen, key=lambda q: sum(a == b for a, b in zip(q, question)))
    return seen[best]

for q in ["25 * 4", "123 * 457", "12 + 31"]:   # eval: trusted input only
    print(f"{q:10} model: {model(q):6}  python: {eval(q)}")

Output:

25 * 4     model: 100     python: 100
123 * 457  model: 56088   python: 56211
12 + 31    model: 42      python: 43

The model's answers look plausible and are wrong: it copies the answer of the most similar-looking question it has seen. Real models are far better than this toy, but they still produce digits as predicted text. For exact maths, let the model call a calculator or run code. (eval runs any Python expression: use it only on input you wrote yourself, never on text from users.)

Bias: models copy their training text

from collections import Counter
# training text, copied from the web, with its habits
text = """the doctor said he was busy . the doctor said he was late .
the doctor said she was busy . the nurse said she was kind .
the nurse said she was late . the engineer said he was busy ."""

after = {"doctor": Counter(), "nurse": Counter(), "engineer": Counter()}
w = text.split()
for i in range(len(w) - 2):
    if w[i] in after:
        after[w[i]][w[i + 2]] += 1    # "the doctor said ___"

for job, c in after.items():
    print(f"the {job} said ->", c.most_common(1)[0][0], dict(c))

Output:

the doctor said -> he {'he': 2, 'she': 1}
the nurse said -> she {'she': 2}
the engineer said -> he {'he': 1}

Nothing in the code is unfair; the training text was. When a model repeats unfair patterns from its data, that's bias. RLHF (episode 17) helps but doesn't erase it; more in responsible AI (episode 38).

Three more limits you have already met

Limit Why Episode
A different answer each time sampling and temperature 16
Forgets long conversations the context window 16
Shaky with exact maths and long logic it predicts text, it doesn't calculate reasoning models, 22

Grounding: answer only from a source

# grounding: answer ONLY from a document you were given
notice = """the annual day is on 14 december.
the bus leaves the school gate at 8 am. parents may attend."""

def answer(question, keyword):
    for line in notice.split("."):
        if keyword in line:
            return line.strip() + "   [notice]"
    return "I don't know: the notice doesn't say."

print(answer("when is annual day?", "annual day"))
print(answer("when does the bus leave?", "bus"))
print(answer("what is the chief guest's name?", "chief guest"))

Output:

the annual day is on 14 december   [notice]
the bus leaves the school gate at 8 am   [notice]
I don't know: the notice doesn't say.

The function answers only from the notice and says "I don't know" when the notice doesn't cover the question. Answering only from a source you supply is grounding. Real apps ground answers with web search, or by retrieving the right paragraphs from your documents (RAG, episode 24).

How to use an LLM safely

  1. Verify facts, names, numbers and especially citations.
  2. Give it the source: paste the document or turn on search, so it reads instead of remembering.
  3. Use tools for exact work: code, a calculator.
  4. Allow "I don't know": tell it that's a fine answer (bluffing becomes less likely, not impossible).
  5. A human checks before anything important happens.

Glossary

Term Meaning Example from this episode
Hallucination a fluent, confident answer that is false "the capital of india is tokyo ."
Plausible sounds likely or reasonable, whether or not it's true bus number 47
Abstain decline to answer: "I don't know" the honest exam taker
Binary grading 1 for right, 0 for anything else the exam with penalty 0
Knowledge cutoff the date where a model's training data stops 2026 topper → 2023's answer
Token id the number the model sees for a token "DE" → 101
Tokenizer splits text into tokens 2-letter pieces
Tool outside code the model can call for exact work "DEEPSEEK".count("D")
Bias unfair patterns a model copies from its data doctor → "he", nurse → "she"
Grounding giving the model sources to read instead of relying on memory pasting the document
Human in the loop a person checks before an important action checking citations before filing

Common mistakes

Code Exact error Why Fix
learned["school topper 2026"] KeyError: 'school topper 2026' asking for something the data never had check with in, or .get(key, "I don't know")
[vocab[p] for p in ["DE", "XY"]] KeyError: 'XY' a piece missing from the vocabulary real tokenizers fall back to smaller pieces, down to single bytes
max({}, key=len) ValueError: max() iterable argument is empty nothing learned, nothing to compare handle the empty case: say "I don't know"

Mini project: an honest quiz bot

Build a dictionary of 10 facts. Write answer(question) that returns the fact if it's known, and "I don't know" otherwise. Then score it with the exam rules (right = 1, wrong = −1, "I don't know" = 0) against a bot that always guesses a random fact from the dictionary.

Practice

  1. Change the exam penalty to 0.5. Who wins?
  2. Make model() return "I don't know" for unseen questions.
  3. Count the r's in "strawberry" with Python.
  4. Ask a chatbot for a rare fact (for example, the birthday of a teacher at your school who isn't famous), then try to verify it.
Solutions

1. Replace the loop with for penalty in [0.5]:

wrong answer costs 0.5: guesser 53.5 | honest 60

Even a half-mark penalty makes honesty win here.

2.

learned = {"school topper 2023": "aman", "school topper 2024": "neha"}

def model(question):
    return learned.get(question, "I don't know")

print("2024 topper:", model("school topper 2024"))
print("2026 topper:", model("school topper 2026"))
2024 topper: neha
2026 topper: I don't know

3.

print("strawberry".count("r"))
3

4. Expect a confident answer for a fact that isn't public. If it can't be verified, treat it as a guess.

Quick quiz 1

A test gives 1 mark for right, 0 for wrong, and 0 for "I don't know". A model learns to? A: always say I don't know · B: guess when unsure · C: refuse every question · D: answer more slowly

Answer

B: guess when unsure. Guessing can only gain marks; "I don't know" never does.

Quick quiz 2

"How many r's in strawberry?" The most reliable way for an LLM app is? A: a higher temperature · B: a longer prompt · C: let it run "strawberry".count("r") · D: ask it twice

Answer

C: let it run code. The model sees tokens, not letters; code counts exactly: 3.

Interview questions

  1. What is a hallucination, and why does it happen? A fluent but false output; the model generates likely text and doesn't check truth, and rare facts with no learnable pattern get guessed.
  2. How do evaluations encourage hallucination? Binary scoring gives no credit for abstaining, so guessing scores higher than "I don't know".
  3. What is a knowledge cutoff, and how do apps work around it? The end date of the training data; apps add search or retrieval (RAG) to supply fresh information.
  4. Why do LLMs struggle to count letters? They operate on tokens (subword pieces as ids), not individual characters.
  5. How do you reduce hallucinations in an application? Ground answers in retrieved sources, use tools for exact work, allow and reward abstaining, and add human review for high-stakes outputs.
  6. Where does model bias come from? Patterns in the training data, which the model learns to reproduce.

Summary

  • Hallucination: the model predicts likely words, not facts; it happens even with all-true training text.
  • Tests that give 0 for "I don't know" reward guessing; rare facts suffer most.
  • The knowledge cutoff means old news or guesses after a date.
  • Models see tokens, not letters: use tools for exact work.
  • Models copy the bias in their data.
  • Use LLMs safely: verify, give sources, use tools, allow "I don't know", keep a human in the loop.
  • Next episode: prompting basics.