These are the notes for AI episode 25. You changed prompts, thresholds and chunk sizes in the last episodes. How do you know a change made your AI app better and didn't quietly break something? You run an eval: many test questions with golden answers, a grader that scores each reply, and a score you can compare between versions. You'll see why exact match marks correct answers wrong, how a key-fact check catches a regression, how to write success criteria, how LLM-as-a-judge works with a rubric, the judge's biases, and how to check a judge against people. Every Python example runs as-is in the SyntaxPilot compiler; outputs are real (Python 3.13). The bots and the judge are toys.
Real-life example: Ramesh changes his recipe
| What happened | In AI |
|---|---|
| One customer says "more ginger!", Ramesh changes the recipe | you change a prompt or a model |
| He tastes one cup: perfect, ships it | you try one question and ship |
| Next day the regulars complain | users hit the cases you didn't test |
| Next time 20 regulars taste it first | an eval set: many cases, run before release |
Try it once
def make_bot(r, fb): # toy bot: first matching rule wins
return lambda q: next((a for k, a in r.items() if k in q.lower()), fb)
v1 = make_bot({"chai": "Chai is 15 rupees.", "close": "We close at 10 pm.",
"deliver": "Sorry, we don't deliver."}, "Sorry, I don't know.")
v2 = make_bot({"samosa": "Samosa is 20 rupees!", "chai": "Chai is 15!"},
"We're open 7 am to 10 pm, come anytime!") # "be helpful"
print("v1:", v1("How much is a samosa?"))
print("v2:", v2("How much is a samosa?")) # try it once... looks great!
Output:
v1: Sorry, I don't know.
v2: Samosa is 20 rupees!
Version 2 looks better, so it ships. The next morning:
def make_bot(r, fb): # toy bot: first matching rule wins
return lambda q: next((a for k, a in r.items() if k in q.lower()), fb)
v1 = make_bot({"chai": "Chai is 15 rupees.", "close": "We close at 10 pm.",
"deliver": "Sorry, we don't deliver."}, "Sorry, I don't know.")
v2 = make_bot({"samosa": "Samosa is 20 rupees!", "chai": "Chai is 15!"},
"We're open 7 am to 10 pm, come anytime!") # "be helpful"
for q in ["Do you deliver?", "Do you sell pizza?", "Is there parking?"]:
print(f"{q:20} -> v2: {v2(q)}")
Output:
Do you deliver? -> v2: We're open 7 am to 10 pm, come anytime!
Do you sell pizza? -> v2: We're open 7 am to 10 pm, come anytime!
Is there parking? -> v2: We're open 7 am to 10 pm, come anytime!
Why we missed it: one good example proves almost nothing. AI answers change from question to question, so you test many questions every time something changes.
Real-life example: a school exam
| A school exam | An AI eval |
|---|---|
| The question paper covers the syllabus, plus tricky questions | eval set (test cases), including edge cases |
| The answer key | golden answers (what a correct reply must contain) |
| The examiner marks every paper the same way | grader (code or a model) |
| The marks: 72/100 | score: compare versions by number |
Grader 1: exact match
def make_bot(r, fb): # toy bot: first matching rule wins
return lambda q: next((a for k, a in r.items() if k in q.lower()), fb)
v1 = make_bot({"chai": "Chai is 15 rupees.", "close": "We close at 10 pm.",
"deliver": "Sorry, we don't deliver."}, "Sorry, I don't know.")
v2 = make_bot({"samosa": "Samosa is 20 rupees!", "chai": "Chai is 15!"},
"We're open 7 am to 10 pm, come anytime!") # "be helpful"
TESTS = [("How much is chai?", "Chai costs Rs 15."), # golden answers
("When do you close?", "We close at 10 pm."),
("Do you deliver?", "No, we don't deliver.")]
def exact_match(output, gold): # from Anthropic's eval guide
return output.strip().lower() == gold.lower()
for q, gold in TESTS:
out = v1(q)
print(exact_match(out, gold), "|", out)
Output:
False | Chai is 15 rupees.
True | We close at 10 pm.
False | Sorry, we don't deliver.
exact_match is the grader from Anthropic's evaluation guide (it normalises whitespace and case). "Chai is 15 rupees." and "Chai costs Rs 15." state the same fact but don't match. Exact match is right when there is one correct form (a category label, a number, yes/no), not for free-form sentences.
Grader 2: check the key fact
def make_bot(r, fb): # toy bot: first matching rule wins
return lambda q: next((a for k, a in r.items() if k in q.lower()), fb)
v1 = make_bot({"chai": "Chai is 15 rupees.", "close": "We close at 10 pm.",
"deliver": "Sorry, we don't deliver."}, "Sorry, I don't know.")
v2 = make_bot({"samosa": "Samosa is 20 rupees!", "chai": "Chai is 15!"},
"We're open 7 am to 10 pm, come anytime!") # "be helpful"
TESTS = [("How much is chai?", "15"), ("How much is a samosa?", "20"),
("When do you close?", "10 pm"), ("Do you deliver?", "don't"),
("Do you sell pizza?", "don't know")] # edge case: not on menu
print(f"{'question':24}{'v1':6}v2")
for q, fact in TESTS:
marks = ["pass" if fact in b(q).lower() else "FAIL" for b in (v1, v2)]
print(f"{q:24}{marks[0]:6}{marks[1]}")
for name, bot in [("v1", v1), ("v2", v2)]:
score = sum(fact in bot(q).lower() for q, fact in TESTS)
print(f"{name} score: {score}/{len(TESTS)}")
Output:
question v1 v2
How much is chai? pass pass
How much is a samosa? FAIL pass
When do you close? pass pass
Do you deliver? pass FAIL
Do you sell pizza? pass FAIL
v1 score: 4/5
v2 score: 3/5
- Each test stores the key fact a correct answer must contain.
- "Do you sell pizza?" is an edge case on purpose: the right answer is "I don't know".
- v2 fixed the samosa question but broke delivery and pizza: v1 4/5, v2 3/5. A change that breaks something that used to work is a regression.
Fix and re-run
def make_bot(r, fb): # toy bot: first matching rule wins
return lambda q: next((a for k, a in r.items() if k in q.lower()), fb)
v1 = make_bot({"chai": "Chai is 15 rupees.", "close": "We close at 10 pm.",
"deliver": "Sorry, we don't deliver."}, "Sorry, I don't know.")
v2 = make_bot({"samosa": "Samosa is 20 rupees!", "chai": "Chai is 15!"},
"We're open 7 am to 10 pm, come anytime!") # "be helpful"
v3 = make_bot({"samosa": "Samosa is 20 rupees.", "chai": "Chai is 15 rupees.",
"close": "We close at 10 pm.", "deliver": "Sorry, we don't deliver."},
"Sorry, I don't know.") # v1's honesty + v2's samosa answer
TESTS = [("How much is chai?", "15"), ("How much is a samosa?", "20"),
("When do you close?", "10 pm"), ("Do you deliver?", "don't"),
("Do you sell pizza?", "don't know")]
for name, bot in [("v1", v1), ("v2", v2), ("v3", v3)]:
score = sum(fact in bot(q).lower() for q, fact in TESTS)
print(f"{name}: {score}/{len(TESTS)}", "ship it" if score == 5 else "")
Output:
v1: 4/5
v2: 3/5
v3: 5/5 ship it
Re-run the whole eval after every change (prompt, model, chunk size, threshold), like a programmer's test suite.
Success criteria
| Weak | Strong |
|---|---|
| "The bot should be good" | "Answers 95% of menu questions correctly" |
| "Good performance" | "Accurate sentiment classification" (Anthropic's example) |
From Anthropic's guide Define success criteria and build evaluations:
- Criteria should be Specific, Measurable, Achievable and Relevant.
- Most apps need several metrics at once, e.g. accuracy, latency (how long users wait) and cost.
- Be task-specific: mirror your real questions, and "Don't forget to factor in edge cases!"
- Automate when possible: multiple-choice, string match, code-graded, LLM-graded.
- Prioritise volume over quality: many automatically graded tests beat a few hand-graded ones.
LLM-as-a-judge
Some qualities have no single key fact: politeness, clarity, whether a summary is faithful. For those, another model grades the answer: LLM-as-a-judge. Judges have biases:
def judge(first, second): # toy LLM judge: longer wins, a tie goes first
return "first" if len(first) >= len(second) else "second"
good = "Sorry, we don't deliver. Pick up only."
ramble = ("Thank you so much for asking! We truly value every customer, and "
"our team works very hard every single day. Do visit us soon!")
print("good vs ramble :", judge(good, ramble)) # ramble never answers!
a = "We close at 10 pm."
b = "We close by 10 pm."
print("a vs b :", judge(a, b))
print("b vs a :", judge(b, a)) # swap the order
Output:
good vs ramble : second
a vs b : first
b vs a : first
- Verbosity bias: the judge prefers longer answers, even one that never answers.
- Position bias: it prefers whichever answer it saw first (
a vs bandb vs aboth give "first"). - Fix for position bias: judge both orders and trust the verdict only when they agree.
A rubric
RUBRIC = { # specific, checkable criteria
"answers": lambda q, a: "deliver" in a.lower(),
"correct": lambda q, a: "don't" in a.lower(),
"polite": lambda q, a: any(w in a.lower() for w in ["sorry", "please"]),
"short": lambda q, a: len(a.split()) < 20,
}
def grade(q, answer):
failed = [name for name, check in RUBRIC.items() if not check(q, answer)]
return f"{len(RUBRIC) - len(failed)}/{len(RUBRIC)} failed: {failed}"
q = "Do you deliver?"
print("good :", grade(q, "Sorry, we don't deliver. Pick up only."))
print("ramble:", grade(q, "Thank you so much for asking! We truly value "
"every customer. Do visit us soon!"))
Output:
good : 4/4 failed: []
ramble: 1/4 failed: ['answers', 'correct', 'polite']
A rubric is a list of specific checks. Instead of "is this good?", the judge answers "does it answer the question? is the fact correct? is it polite? is it short?", and you learn what failed. Anthropic's guide: "Generally best practice to evaluate with a different model from the one used to generate the evaluated output."
Judge the judge
human = [1, 0, 1, 1, 0, 1, 0, 1, 1, 0] # 1 = good answer, said by people
judge = [1, 0, 1, 0, 0, 1, 1, 1, 1, 0] # the LLM judge, same answers
same = sum(h == j for h, j in zip(human, judge))
print(f"judge agrees with people: {same}/{len(human)} = {same/len(human):.0%}")
for i, (h, j) in enumerate(zip(human, judge)):
if h != j:
print(f" answer {i}: human said {h}, judge said {j} <- read these")
Output:
judge agrees with people: 8/10 = 80%
answer 3: human said 1, judge said 0 <- read these
answer 6: human said 0, judge said 1 <- read these
Check the judge against labels from people and read the disagreements to improve the rubric.
Zheng and colleagues, "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" (arXiv, 9 June 2023): strong judges such as GPT-4 reached over 80% agreement with human preferences, the same level at which humans agree with each other. The paper also studied position, verbosity and self-enhancement bias (favouring answers like its own) and limited reasoning ability.
Choosing a grader
| Code-based | LLM judge | Human |
|---|---|---|
| exact match for labels, key facts, valid JSON | tone, clarity, faithfulness, with a rubric | the most flexible |
| thousands of tests in seconds, same result every time | fast, but has biases: check it against people | slow and expensive |
| use it whenever you can | for what code can't check | for small, important samples |
Offline and online evaluation
| Offline | Online |
|---|---|
| your eval set, before release | real users, after release |
| catches regressions before users see them | user feedback (thumbs up/down), A/B tests |
An A/B test sends half the users to one version and half to the other, and compares the results. Every failure a real user finds becomes a new test case.
Glossary
| Term | Meaning | Example from this episode |
|---|---|---|
| Eval (evaluation) | measuring an AI app on many test cases | v1 4/5, v2 3/5 |
| Eval set / test cases | the questions you test with | 5 menu questions |
| Golden answer | the known correct answer for a test | "Chai costs Rs 15." |
| Key fact | what a correct answer must contain | "15" |
| Edge case | an unusual or tricky input | "Do you sell pizza?" |
| Grader | code or a model that scores a reply | exact_match, the rubric |
| Exact match | output equals the golden answer (after normalising) | False for a correct answer |
| Normalising | making text comparable (spaces, case) | .strip().lower() |
| Score / accuracy | fraction of tests passed | 4/5 = 80% |
| Regression | a change breaks something that worked | v2 broke delivery |
| Success criteria | specific, measurable goals | "95% of menu questions" |
| Latency | how long a user waits for an answer | seconds per reply |
| LLM-as-a-judge | a model grading another model's answers | the toy judge |
| Rubric | a list of specific grading checks | answers, correct, polite, short |
| Verbosity bias | a judge preferring longer answers | the ramble won |
| Position bias | a judge preferring the first answer shown | "first" both times |
| Self-enhancement bias | a judge favouring answers like its own | from Zheng et al. 2023 |
| Agreement | how often the judge matches people | 8/10 = 80% |
| Offline eval | testing before release on your eval set | the 5 tests |
| Online eval | measuring with real users after release | thumbs up/down |
| A/B test | two versions to two groups, compare results | half v1, half v3 |
Common mistakes
| Code / situation | Exact error or symptom | Why | Fix |
|---|---|---|---|
| tested 1 question, shipped | v2 broke delivery and pizza answers | one example proves almost nothing | an eval set with edge cases, every change |
exact_match("Chai is 15 rupees.", "Chai costs Rs 15.") |
False (a correct answer marked wrong) |
same fact, different words | check the key fact, or use a rubric |
score / len(TESTS) with TESTS = [] |
ZeroDivisionError: division by zero |
an empty eval set | never report a score from zero tests |
| judging in one order only | the first answer always wins | position bias | judge both orders |
| "Is this answer good?" as the judge prompt | vague, biased scores | no specific criteria | a rubric of specific checks |
Mini project: an eval harness
Write evaluate(bot, tests) that returns the score and the list of failed questions, and compare(old, new, tests) that prints every test where the new version got worse ("REGRESSION: Do you deliver?"). Add a check that refuses to release (prints "BLOCK") if the new score is lower than the old one or any regression appears. Run it on v1 → v2 and v2 → v3.
Practice
- Add three tests: parking, Sunday hours, card payment. What do v1-v3 score now?
- Write v4 that passes them all.
- Make the judge fair about position: judge both orders, trust only if they agree.
- Write three rubric checks for a polite refusal ("Sorry, we don't deliver.").
Solutions
1 and 2. Add the tests and a v4:
v4 = make_bot({"samosa": "Samosa is 20 rupees.", "chai": "Chai is 15 rupees.",
"close": "We close at 10 pm.", "deliver": "Sorry, we don't deliver.",
"park": "Free parking behind the shop.", "sunday": "Yes, open on Sunday too.",
"card": "Yes, card and UPI are fine."}, "Sorry, I don't know.")
TESTS = [("How much is chai?", "15"), ("How much is a samosa?", "20"),
("When do you close?", "10 pm"), ("Do you deliver?", "don't"),
("Do you sell pizza?", "don't know"), ("Is there parking?", "parking"),
("Are you open on Sunday?", "sunday"), ("Can I pay by card?", "card")]
v1: 4/8
v2: 3/8
v3: 5/8
v4: 8/8 ship it
(The parking, Sunday and card answers are made up for Ramesh's toy shop.)
3. Add to the judge example:
def fair_judge(x, y): # judge both orders, trust only if they agree
one, two = judge(x, y), judge(y, x)
if one == "first" and two == "second": return "x"
if one == "second" and two == "first": return "y"
return "tie / unsure"
print("fair a vs b :", fair_judge(a, b))
print("fair good vs ramble:", fair_judge(good, ramble))
fair a vs b : tie / unsure
fair good vs ramble: y
Swapping fixes position bias (the two equal answers are now a tie), but not verbosity bias: the ramble still wins. That needs a rubric.
4. For example: says it doesn't deliver ("don't deliver" in a.lower()), apologises ("sorry" in a.lower()), offers an alternative ("pick up" in a.lower()).
Quick quiz 1
Version A scores 9/10 on your eval. Version B scores 7/10, but its demo answer looked amazing. Which ships? A: version B · B: version A · C: neither, evals are useless · D: whichever is newer
Answer
B (version A). A great demo is one example; the eval covers many questions, including edge cases.
Quick quiz 2
judge(x, y) returns "first" and judge(y, x) also returns "first". Which bias? A: verbosity bias · B: position bias · C: a regression · D: exact match
Answer
B. It prefers whatever it sees first. Judge both orders.
Interview questions
- How would you evaluate an LLM app before launch? Define specific, measurable success criteria; build an eval set that mirrors real usage with edge cases; grade automatically where possible (exact match, key facts, format checks), with an LLM judge and rubric for subjective qualities; track accuracy, latency and cost; run it on every change.
- Why is exact match a poor grader for chat answers? Correct answers can be phrased many ways; use it for labels or fixed formats, and key-fact checks or rubrics for free text.
- What is a regression in an AI app and how do you catch it? A change that makes previously correct cases fail; catch it by re-running the full eval and comparing per-test results between versions.
- What are the risks of LLM-as-a-judge? Position, verbosity and self-enhancement biases, inconsistent scoring; mitigate with rubrics, order swapping, a different model than the generator, and checking agreement with human labels.
- Offline vs online evaluation? Offline: fixed test set before release. Online: real users after release (feedback, A/B tests, monitoring); failures found online become new offline tests.
- How do you evaluate a RAG system? Separately measure retrieval (is the right chunk in the top-k?) and generation (is the answer correct and grounded in the retrieved text?), plus "I don't know" behaviour on out-of-scope questions.
Summary
- One good example proves nothing: test many questions, every change.
- An eval = an eval set (with edge cases) + golden answers + a grader → a score.
- Exact match only for one correct form; key-fact checks and rubrics for sentences.
- Comparing versions per test catches regressions.
- Write specific, measurable success criteria, and track accuracy, latency and cost.
- LLM-as-a-judge with a rubric, both orders, a different model, checked against people.
- Next episode: what is an AI agent?