These are the notes for AI episode 36. Guardrails (episode 35) limit the damage when an agent is wrong. This episode answers the next questions: how do you find out what an agent did, step by step, and how do you know whether it's getting better or worse? Each idea arrives as the fix for a failure on screen: a final answer that hides a bug, a slow step you can't find, a single passing run that proves nothing, a right answer reached by a wrong path, and a prompt change that keeps the same score while breaking something else. Every Python example runs as-is in the SyntaxPilot compiler; outputs are real (Python 3.13; timings rounded to 0.1 s). The agents are toys (if-rules and a seeded random number) standing in for real model calls.
Real-life example: a delivery app and a plane's black box
| In real life | For an agent |
|---|---|
| Live order tracking: packed, picked up, on the way | trace: every step of one request |
| "Stuck at the restaurant for 25 minutes" | spans with timings: which step is slow or costly |
| A plane's black box, read after a problem | logs you can replay: why did it do that? |
| Many test drives before a car is sold | evals: many runs, many tasks, before shipping |
Problem 1: a silent failure
TRAINS = {"08-11": [12127, 11007]} # the API wants day-month numbers
def search_trains(date):
return TRAINS.get(date, [])
def agent(question): # toy LLM agent
date = "8 Nov" # what the model wrote
trains = search_trains(date)
if not trains:
return "Sorry, no trains to Goa on 8 Nov."
return f"Trains: {trains}"
print(agent("Any trains to Goa on 8 Nov?"))
Output:
Sorry, no trains to Goa on 8 Nov.
There are two trains on that day (line 1). When only the final answer is visible, every mistake in the middle is invisible.
The fix: record every step (a trace)
TRAINS = {"08-11": [12127, 11007]}
trace = [] # the agent's flight recorder
def search_trains(date):
result = TRAINS.get(date, [])
trace.append({"step": "search_trains", "input": date, "output": result})
return result
def agent(question):
trace.append({"step": "llm", "input": question, "output": "date = 8 Nov"})
trains = search_trains("8 Nov")
answer = f"Trains: {trains}" if trains else "Sorry, no trains."
trace.append({"step": "answer", "input": "", "output": answer})
return answer
agent("Any trains to Goa on 8 Nov?")
for i, span in enumerate(trace, 1):
print(i, span["step"], "| in:", span["input"], "| out:", span["output"])
Output:
1 llm | in: Any trains to Goa on 8 Nov? | out: date = 8 Nov
2 search_trains | in: 8 Nov | out: []
3 answer | in: | out: Sorry, no trains.
Step 2 shows the bug: search_trains received "8 Nov", but the API stores dates as "08-11". We only found it because the trace showed the middle step.
Fix it, prove it
TRAINS = {"08-11": [12127, 11007]}
MONTHS = {"Nov": "11", "Dec": "12"}
trace = []
def to_api_date(text): # "8 Nov" -> "08-11"
day, month = text.split()
return f"{int(day):02d}-{MONTHS[month]}"
def search_trains(date):
result = TRAINS.get(to_api_date(date), [])
trace.append({"step": "search_trains", "input": date, "output": result})
return result
trains = search_trains("8 Nov")
print("answer:", f"Trains: {trains}" if trains else "Sorry, no trains.")
for i, span in enumerate(trace, 1):
print(i, span["step"], "| in:", span["input"], "| out:", span["output"])
Output:
answer: Trains: [12127, 11007]
1 search_trains | in: 8 Nov | out: [12127, 11007]
The same trace now proves the fix: in 8 Nov, out both trains.
Traces and spans
OpenTelemetry, a widely used open standard for monitoring software, defines:
- Trace: "the path of a request through your application".
- Span: one "unit of work or operation", such as one model call or one tool call.
Spans form a tree: the whole agent run is the root span (its parent ID is empty) and each step is a child span pointing to its parent.
agent: trains to Goa (root span)
├── llm: plan (child)
├── tool: search_trains (child)
└── llm: answer (child)
| Span field | Meaning | Example |
|---|---|---|
| Name | what this step is | tool: search_trains |
| Parent span ID | which span it belongs to (empty for the root) | agent: trains to Goa |
| Start and end time | when it began and finished | 1.5 s long |
| Attributes | extra key-value details | input, output, tokens, model name |
| Events | things that happened at a moment inside it | "retrying" |
| Status | Unset, Error or Ok | Error if the tool crashed |
Because it's a shared standard, many monitoring tools can read the same traces.
What goes into a span
| Record | Never record | Mask instead |
|---|---|---|
| model name, input, output, tokens, time, cost | passwords | card ****4242 |
| tool name, arguments, result, status | API keys | phone ******3210 |
| parent span, retries | card numbers | email a***@example.com |
A trace is read by many people (developers, support, monitoring tools), so it must be useful without leaking secrets.
Finding the slow and the expensive step
import time
trace = []
def span(name, parent=None): # one timed step of the agent
s = {"name": name, "parent": parent, "start": time.perf_counter()}
trace.append(s)
return s
def end(s, tokens=0):
s["secs"] = time.perf_counter() - s["start"]
s["tokens"] = tokens
root = span("agent: trains to Goa")
s = span("llm: plan", root["name"]); time.sleep(0.2); end(s, 900)
s = span("tool: search_trains", root["name"]); time.sleep(1.5); end(s)
s = span("llm: answer", root["name"]); time.sleep(0.3); end(s, 1400)
end(root, 2300)
for s in trace:
pad = " " if s["parent"] else ""
print(f"{pad}{s['name']:24} {s['secs']:.1f}s {s['tokens']} tokens")
Output:
agent: trains to Goa 2.0s 2300 tokens
llm: plan 0.2s 900 tokens
tool: search_trains 1.5s 0 tokens
llm: answer 0.3s 1400 tokens
The search tool took 1.5 s of 2.0 s, and the answer step used the most tokens. Fix those (cache the tool, send less history) instead of guessing. Measure before you optimise.
Problem 2: "it passed once, ship it"
import random
random.seed(42) # same "luck" every time we run
def agent_run(): # toy agent: right about 70% of the time
return random.random() < 0.7
print("one run:", "PASS" if agent_run() else "FAIL", " -> ship it?")
results = [agent_run() for _ in range(8)]
print("8 runs :", " ".join("P" if r else "F" for r in results))
print("passed :", sum(results), "of 8")
Output:
one run: PASS -> ship it?
8 runs : P P P F P F P P
passed : 6 of 8
A model is not a normal function: the same input can give a different result. One pass is luck, not proof. (random.seed(42) makes the demo repeat the same "luck" every run.)
τ-bench and pass^k
τ-bench (tau-bench; Yao, Shinn, Razavi, Narasimhan, June 2024) tests agents in conversations: a language model plays the user, and the agent must help using domain API tools and policy rules (for example, a retail shop). It introduced pass^k: the chance that k tries of the same task all succeed, a measure of reliability. The paper reports that even strong function-calling agents such as gpt-4o succeeded on under 50% of tasks, with pass^8 under 25% in the retail domain.
The estimate for one task with c passes out of n runs is C(c, k) / C(n, k), averaged over tasks:
from math import comb
runs = { # 8 runs of each task: P = pass
"book 8 Nov train": "PPPPPPPP",
"cancel a ticket": "PPPFPPPF",
"change the date": "PFPPFPPP",
"refund a ticket": "PPFPPFPF",
}
def pass_hat_k(record, k): # chance that k runs in a row ALL pass
n, c = len(record), record.count("P")
return comb(c, k) / comb(n, k)
for k in (1, 2, 4, 8):
score = sum(pass_hat_k(r, k) for r in runs.values()) / len(runs)
print(f"pass^{k} = {score:.0%}")
Output:
pass^1 = 78%
pass^2 = 61%
pass^4 = 38%
pass^8 = 25%
| k | pass^k | Meaning |
|---|---|---|
| 1 | 78% | a typical single try works |
| 2 | 61% | two in a row both work |
| 4 | 38% | four in a row |
| 8 | 25% | only 1 in 4 chance of 8 clean runs |
Same agent, same runs: reliability is the hard part of agents.
Problem 3: right answer, wrong path
def agent(task): # toy agent; returns answer + its trace
trace = ["search_trains(8 Nov)", "book_ticket(12127)",
"book_ticket(12127)", # booked again "to be sure"!
"send_email(asha)"]
return "Booked 12127 on 8 Nov", trace
answer, trace = agent("book a Goa train on 8 Nov")
print("answer test:", "PASS" if "8 Nov" in answer else "FAIL")
Output:
answer test: PASS
The answer test passed, but the trace shows book_ticket was called twice: two tickets, paid twice. We graded the destination, not the journey.
The fix: grade the trajectory
def agent(task): # the same toy agent
trace = ["search_trains(8 Nov)", "book_ticket(12127)",
"book_ticket(12127)",
"send_email(asha)"]
return "Booked 12127 on 8 Nov", trace
def check_trajectory(trace, max_steps=5): # grade the PATH too
problems = []
books = [t for t in trace if t.startswith("book_ticket")]
if len(books) != 1:
problems.append(f"booked {len(books)} times")
if len(trace) > max_steps:
problems.append(f"{len(trace)} steps > {max_steps}")
return problems
answer, trace = agent("book a Goa train on 8 Nov")
print("answer test:", "PASS" if "8 Nov" in answer else "FAIL")
print("path test :", check_trajectory(trace) or "PASS")
Output:
answer test: PASS
path test : ['booked 2 times']
Trajectory (path) checks are cheap and powerful: the right tools, in the right order, no forbidden actions, not too many steps, each irreversible action exactly once. You can only write them because you kept a trace.
Problem 4: same score, new bug
SUITE = [ # test tasks + what must come out
("trains on 8 Nov", "12127"),
("cancel PNR 4521", "cancelled 4521"),
("refund PNR 4521", "refund Rs 450"),
("trains on 08-11", "12127"),
]
def agent(task, prompt): # toy agent: v2 adds a date rule
if "08-11" in task:
return "12127" if "dates" in prompt else "no trains"
if "refund" in task and "dates" in prompt:
return "refund Rs 4500" # the new rule confused it here!
return {"trains on 8 Nov": "12127", "cancel PNR 4521": "cancelled 4521",
"refund PNR 4521": "refund Rs 450"}[task]
for name, prompt in [("v1", "You book trains."),
("v2", "You book trains. Read all dates carefully.")]:
marks = ["P" if agent(task, prompt) == want else "F" for task, want in SUITE]
print(name, " ".join(marks), f"{marks.count('P')}/{len(SUITE)}")
Output:
v1 P P P F 3/4
v2 P P F P 3/4
Both prompts score 3/4, but different tasks fail: v2 fixed the 08-11 date task and broke the refund (Rs 4500 instead of Rs 450). A total score hid an expensive regression.
The fix: an eval suite, compared task by task
change (prompt / model / tool)
│
▼
run the suite: every task, k times
│
▼
compare with the last version TASK BY TASK
│
▼
ship only if no task got worse
- Regression testing: re-running old tests after a change, to catch things that used to work and now don't.
- Every bug you find becomes a new task in the suite, so it can't come back unnoticed.
- Check answers and paths in code where you can; for open-ended answers, a judge model (LLM-as-judge, episode 25) can grade.
- Evals tell you before you ship; traces tell you what happens after (in production).
Glossary
| Term | Meaning | Example from this episode |
|---|---|---|
| Observability | being able to see what a system did and why | traces, spans, logs |
| Trace | the path of one request through the app | the 3-step list |
| Span | one unit of work inside a trace | tool: search_trains |
| Root span | the top span, with no parent | agent: trains to Goa |
| Attribute | a key-value detail on a span | tokens: 1400 |
| OpenTelemetry | an open standard for traces, metrics and logs | |
time.perf_counter() |
Python's precise stopwatch | span timings |
| Latency | how long something takes | 2.0 s for the agent |
| Token | a piece of text the model reads or writes (episode 12) | 1400 tokens |
| Eval | a test of an AI system's behaviour | the suite |
| Non-deterministic | the same input can give different outputs | 6 of 8 runs |
random.seed |
fixes the random numbers so a run repeats | random.seed(42) |
| τ-bench (tau-bench) | an agent benchmark with simulated users, tools and policies | June 2024 |
| pass^k | chance that k tries in a row all pass | 78% → 25% |
math.comb(n, k) |
number of ways to choose k items from n | comb(8, 4) = 70 |
| Trajectory | the sequence of steps an agent took | the trace list |
| Eval suite | a fixed set of test tasks with expected results | SUITE |
| Regression | something that used to work and now doesn't | the refund in v2 |
| LLM-as-judge | a model that grades another model's answers | episode 25 |
Common mistakes
| Mistake | What you see | Why | Fix |
|---|---|---|---|
| Printing only the final answer | Sorry, no trains to Goa on 8 Nov. |
middle steps invisible | trace every step |
| Guessing which step is slow | optimised the wrong thing | no timings | spans with time and tokens |
| Shipping after one pass | one run: PASS, then 6 of 8 |
models vary run to run | run k times, use pass^k |
| Testing only the answer | answer test: PASS, booked twice |
path not graded | trajectory checks |
| Comparing only total scores | 3/4 → 3/4, refund broke |
totals hide regressions | compare task by task |
| Fixing a bug without a test | the bug comes back later | nothing guards it | add it to the suite |
Mini project: a tiny agent test bench
Write bench.py with: a trace list and span()/end() helpers; a toy agent with three tools; a suite of 6 tasks with expected answers and path rules (each booking once, at most 5 steps); run every task 8 times with different seeds, print pass^1 and pass^4 per task, and a table comparing two prompt versions task by task.
Practice
1. Add a "Dec" trip: search trains on "25 Dec" with the fixed agent (no trains are stored for that day) and check the trace.
Solution
TRAINS = {"08-11": [12127, 11007]}
MONTHS = {"Nov": "11", "Dec": "12"}
trace = []
def to_api_date(text):
day, month = text.split()
return f"{int(day):02d}-{MONTHS[month]}"
def search_trains(date):
result = TRAINS.get(to_api_date(date), [])
trace.append({"step": "search_trains", "input": date, "api": to_api_date(date), "output": result})
return result
search_trains("25 Dec")
for span in trace:
print(span)
Output:
{'step': 'search_trains', 'input': '25 Dec', 'api': '25-12', 'output': []}
2. Add a status field to each span: "ERROR" if the step raised an exception, else "OK".
Solution
trace = []
def traced(name, func, *args):
span = {"name": name, "status": "OK"}
try:
span["output"] = func(*args)
except Exception as e:
span["status"] = "ERROR"
span["output"] = f"{type(e).__name__}: {e}"
trace.append(span)
return span["output"]
traced("tool: fare", lambda no: {12127: 450}[no], 12127)
traced("tool: fare", lambda no: {12127: 450}[no], 99999)
for span in trace:
print(f"{span['name']:12} {span['status']:6} {span['output']}")
Output:
tool: fare OK 450
tool: fare ERROR KeyError: 99999
3. Compute pass^3 for the quiz agents A (PPPPPPPF) and B (PPPPPFFF).
Solution
from math import comb
for name, record in [("A", "PPPPPPPF"), ("B", "PPPPPFFF")]:
n, c = len(record), record.count("P")
print(name, f"pass^3 = {comb(c, 3) / comb(n, 3):.0%}")
Output:
A pass^3 = 62%
B pass^3 = 18%
4. Add the 08-11 bug as a suite task and print which tasks changed between versions.
Solution
v1 = {"trains on 8 Nov": "P", "cancel PNR 4521": "P", "refund PNR 4521": "P", "trains on 08-11": "F"}
v2 = {"trains on 8 Nov": "P", "cancel PNR 4521": "P", "refund PNR 4521": "F", "trains on 08-11": "P"}
for task in v1:
if v1[task] != v2[task]:
change = "FIXED" if v2[task] == "P" else "BROKE"
print(f"{change}: {task}")
Output:
BROKE: refund PNR 4521
FIXED: trains on 08-11
Quick quiz 1
Eight runs of the same task: agent A PPPPPPPF, agent B PPPPPFFF. Which is more reliable?
Answer
A. It passed 7 of 8; pass^4 = C(7,4)/C(8,4) = 35/70 = 50%. Agent B: C(5,4)/C(8,4) = 5/70 ≈ 7%.
Quick quiz 2
A user says: "the agent said my refund failed, but I got the money twice!" What do you look at first?
Answer
B: the trace of that run. It shows both refund calls with their inputs and outputs, and the step that reported a failure (probably a timeout followed by a retry without an idempotency key, episode 35). Then add the case to your eval suite.
Interview questions
- What's the difference between a trace and a span? A trace is the whole path of one request; a span is one unit of work inside it, with a parent, timings, attributes and a status.
- Why are agents harder to test than normal functions? They're non-deterministic, take many steps, call tools with side effects, and can reach a right answer by a wrong path.
- What is pass^k? From τ-bench: the probability that k independent tries of a task all succeed; it measures reliability rather than a lucky single success.
- What is trajectory evaluation? Checking the steps an agent took (tools, order, counts, forbidden actions), not just its final answer.
- How do you catch regressions after changing a prompt? Run a fixed eval suite several times per task on both versions and compare task by task, not just total scores.
- What would you log for each LLM call? Model name, input, output, tokens, latency, cost, status and the parent span, without secrets or unnecessary personal data.
- Evals vs monitoring? Evals run before shipping on known tasks; monitoring (traces, alerts) watches real traffic after shipping, and its failures become new eval tasks.
Summary
- Printing only the answer hides bugs: trace every step (input and output).
- Spans (OpenTelemetry) form a tree with timings, attributes and status: find the slow and expensive steps.
- Models vary: run each task k times; pass^k (τ-bench) measures reliability: 78% → 25% here.
- Grade the path (trajectory), not just the answer.
- Keep an eval suite; compare task by task on every change; every bug becomes a test.
Next: episode 37, Agent design patterns and frameworks.
Sources: OpenTelemetry documentation, "Traces" (opentelemetry.io/docs/concepts/signals/traces/); Yao, Shinn, Razavi, Narasimhan, "τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains", arXiv 2406.12045 (17 June 2024). Fetched 5 Nov 2026.