These are the notes for AI episode 38. An AI system can work perfectly in a demo and still hurt people: it can treat equal people unequally, leak their personal data, state made-up facts with confidence, or make decisions nobody can explain or appeal. In this episode you watch each of these happen in code and invent the fix. Every Python example runs as-is in the SyntaxPilot compiler; outputs and errors are real (Python 3.13). The models are toys (word weights and if-rules) standing in for real ML models and LLM calls.
Real-life example: why you trust a good doctor
| A good doctor | Responsible AI |
|---|---|
| Treats you by your symptoms, not your surname or gender | Fairness: same skills, same score |
| Keeps your file locked | Privacy: personal data protected |
| Says "let's run a test" when unsure | Honesty: no made-up facts, "I don't know" is allowed |
| Signs the prescription and explains why | Accountability: reasons, a responsible person, a record |
Responsible AI means building AI systems that are fair, protect privacy, are honest about what they know, and have people who answer for their decisions.
Problem 1: a CV screener learns bias
A company trains a screener on ten years of past hiring. The toy "training" gives every word in a hired CV +1 and every word in a rejected CV −1.
past_hires = [ # 10 years of past CVs (mostly men hired)
("python sql cricket club", 1), ("java sql chess club", 1),
("python aws chess club", 1), ("java python football", 1),
("python sql women's chess club", 0), ("java aws women's coding club", 0),
]
weights = {} # toy "training": words seen in hires go up
for cv, hired in past_hires:
for word in cv.split():
weights[word] = weights.get(word, 0) + (1 if hired else -1)
def score(cv):
return sum(weights.get(word, 0) for word in cv.split())
print("Kabir:", score("python sql chess club captain"))
print("Priya:", score("python sql women's chess club captain"))
Output:
Kabir: 5
Priya: 3
Kabir and Priya have the same skills. Priya scores lower only because her CV says "women's chess club". Nobody wrote a rule against women: in the past data most hires were men, so "women's" appeared mostly in rejected CVs and got a negative weight (−2).
- Bias (in AI): a model treating people or groups unfairly in a systematic way.
- Training data bias: the data reflects old unfair decisions, so the model learns them.
This really happened
Reuters reported on 10 October 2018 that Amazon had been building a CV screener since 2014. It gave candidates one to five stars and was trained on patterns in CVs submitted to the company over a 10-year period. It penalised CVs containing the word "women's" (as in "women's chess club captain"). Amazon edited the programs to be neutral to those terms; the team was later disbanded.
Fix 1a: the swap test (counterfactual test)
Take two inputs that differ only in something that should not matter, and check the output does not change.
weights = {"python": 2, "sql": 1, "chess": 1, "club": 1,
"women's": -2} # learned from the past
def score(cv):
return sum(weights.get(word, 0) for word in cv.split())
def swap_test(cv_a, cv_b): # two CVs, ONE word different
a, b = score(cv_a), score(cv_b)
assert a == b, f"score changed {a} -> {b}: '{cv_b}'"
print("PASS:", cv_a, "=", cv_b)
swap_test("he led python sql chess club", "she led python sql chess club")
swap_test("python sql chess club", "python sql women's chess club")
Output:
PASS: he led python sql chess club = she led python sql chess club
Traceback (most recent call last):
File "main.py", line 13, in <module>
swap_test("python sql chess club", "python sql women's chess club")
~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "main.py", line 9, in swap_test
assert a == b, f"score changed {a} -> {b}: '{cv_b}'"
^^^^^^
AssertionError: score changed 5 -> 3: 'python sql women's chess club'
The AssertionError is good news: the bug is caught in a test, not in a real person's life.
- Counterfactual test ("swap test"): change one attribute that should be irrelevant and check the decision stays the same.
assert condition, message: raisesAssertionError: messagewhen the condition is false.
Fix 1b: compare selection rates per group
results = [("men", True), ("men", True), ("men", False), ("men", True),
("men", True), ("women", False), ("women", True), ("women", False),
("women", False), ("women", False)] # who the screener shortlisted
for group in ["men", "women"]:
picks = [ok for g, ok in results if g == group]
rate = sum(picks) / len(picks)
print(f"{group:6} shortlisted {sum(picks)} of {len(picks)} = {rate:.0%}")
Output:
men shortlisted 4 of 5 = 80%
women shortlisted 1 of 5 = 20%
- Selection rate: the share of a group that gets the positive outcome (shortlisted, approved).
- A big gap (80% vs 20%) does not prove unfairness on its own, but it is an alarm: someone must find out why before the system goes live. Overall accuracy can look fine while one group is treated badly.
Fix 1c: score only what the job needs
SKILLS = {"python": 2, "sql": 1, "java": 2, "aws": 1} # job skills ONLY
def score(cv):
return sum(SKILLS.get(word, 0) for word in cv.split())
def swap_test(cv_a, cv_b):
a, b = score(cv_a), score(cv_b)
assert a == b, f"score changed {a} -> {b}: '{cv_b}'"
print("PASS:", a, "=", b)
swap_test("python sql chess club", "python sql women's chess club")
swap_test("he knows java aws", "she knows java aws")
print("Kabir:", score("python sql chess club captain"))
print("Priya:", score("python sql women's chess club captain"))
Output:
PASS: 3 = 3
PASS: 3 = 3
Kabir: 3
Priya: 3
Warning: deleting one word is not enough in real models. Other features can stand in for a group (the name of a girls' college, a sport, a PIN code). These are called proxies. Keep the swap tests and group checks in your test suite and run them every time the model or data changes.
Problem 2: a log that leaks personal data
log = [] # every prompt is saved for "debugging"
def ask_bot(message):
log.append(message) # sent to the model AND stored
return "Sure, I'll check your refund."
ask_bot("Refund please. My phone is 9876543210, "
"Aadhaar 1234 5678 9012, card 4111 1111 1111 1111")
print("log file:", log[0])
Output:
log file: Refund please. My phone is 9876543210, Aadhaar 1234 5678 9012, card 4111 1111 1111 1111
Phone, Aadhaar and full card number are sent to the model and stored in plain text, where many people can read them. The refund only needed the booking.
- Personal data / PII (personally identifiable information): data that identifies a person: phone, Aadhaar, card number, email, address.
Fix 2: redact before data leaves your code
import re
RULES = [(r"\b\d{4} \d{4} \d{4} \d{4}\b", "[CARD]"), # longest first
(r"\b\d{4} \d{4} \d{4}\b", "[AADHAAR]"),
(r"\b[6-9]\d{9}\b", "[PHONE]")]
def redact(text): # hide personal data BEFORE it leaves
for pattern, label in RULES:
text = re.sub(pattern, label, text)
return text
msg = ("Refund please. My phone is 9876543210, "
"Aadhaar 1234 5678 9012, card 4111 1111 1111 1111")
print("sent :", redact(msg))
Output:
sent : Refund please. My phone is [PHONE], Aadhaar [AADHAAR], card [CARD]
- Redaction: replacing sensitive parts of text with labels before sending, storing or showing it.
- Regular expression (regex): a pattern for matching text.
\d{4}= four digits,\b= word boundary,[6-9]\d{9}= ten digits starting with 6-9 (an Indian mobile number).
The rule-order bug
import re
RULES = [(r"\b\d{4} \d{4} \d{4}\b", "[AADHAAR]"), # Aadhaar rule first...
(r"\b\d{4} \d{4} \d{4} \d{4}\b", "[CARD]"),
(r"\b[6-9]\d{9}\b", "[PHONE]")]
def redact(text):
for pattern, label in RULES:
text = re.sub(pattern, label, text)
return text
print(redact("card 4111 1111 1111 1111"))
Output:
card [AADHAAR] 1111
The 12-digit Aadhaar rule matched the first 12 digits of the 16-digit card first, so the card rule never matched and the last four digits leaked with the wrong label. Put the longest patterns first and test the redactor with samples of every kind.
Privacy habits
- Collect less (data minimisation): only what the task needs.
- Consent: ask, and say clearly what the data is for.
- Redact before sending to a model or writing to a log.
- Delete it when the job is done (retention limits).
India's law: the Digital Personal Data Protection (DPDP) Act, 2023. Its rules were notified on 13 November 2025 and apply in phases: the Data Protection Board at once; consent managers after one year (12 November 2026); most duties (notice and consent, children's data, people's rights, breach notification, security safeguards, erasure) after 18 months (12 May 2027).
Problem 3: confident and wrong
def llm(question): # toy LLM: always sounds confident
return {"answer": "Airlines must pay for late bags.",
"cites": ["Mehta v. SkyAir (2019)", "Rao v. JetFast (2021)"]}
reply = llm("Find court cases about lost airline luggage")
print("answer:", reply["answer"])
for case in reply["cites"]:
print(" source:", case)
Output:
answer: Airlines must pay for late bags.
source: Mehta v. SkyAir (2019)
source: Rao v. JetFast (2021)
One of these cases does not exist. A language model predicts likely words (episode 19), and a realistic case name is very likely words. This is a hallucination: fluent, confident, false output.
This happened in a real court
In Mata v. Avianca (New York federal court), lawyers filed a brief citing court opinions that did not exist, with fake quotes and citations, produced with ChatGPT. On 22 June 2023, Judge P. Kevin Castel sanctioned the two lawyers and their firm $5,000. The people who used the tool without checking were held responsible.
Fix 3: check every source in code
LIBRARY = {"Rao v. JetFast (2021)": "court record #5531"} # sources we can check
def llm(question):
return {"answer": "Airlines must pay for late bags.",
"cites": ["Mehta v. SkyAir (2019)", "Rao v. JetFast (2021)"]}
def check(reply): # never trust a citation you can't find
real = [c for c in reply["cites"] if c in LIBRARY]
fake = [c for c in reply["cites"] if c not in LIBRARY]
for c in fake:
print(" NOT FOUND:", c)
if not real:
return "I couldn't find a real source for this."
return reply["answer"] + " (source: " + ", ".join(real) + ")"
print(check(llm("Find court cases about lost airline luggage")))
Output:
NOT FOUND: Mehta v. SkyAir (2019)
Airlines must pay for late bags. (source: Rao v. JetFast (2021))
A real system also checks that the source actually says what the answer claims (episode 39 checks quotes word for word). If nothing checks out, "I couldn't find a real source" is the honest answer.
Honesty habits
- Show sources and verify them in code; ground answers in retrieved documents (RAG, episode 24).
- Allow "I don't know" in the prompt and in the user interface.
- Tell people when they are talking to an AI, not a human.
- Label AI-generated images, voices and videos.
Problem 4: a black box
def model(applicant): # a black box: just yes or no
return applicant["income"] * 3 > applicant["loan"] + 400000
meera = {"name": "Meera", "income": 300000, "loan": 600000, "pin": "110092"}
print(meera["name"], "->", "APPROVED" if model(meera) else "REJECTED")
print("why?", meera.get("reason"))
Output:
Meera -> REJECTED
why? None
Meera can't fix anything, appeal, or know whether it was a mistake. And the model was given her PIN code, which it doesn't need and which can act as a proxy for a group.
- Black box: a system whose decisions come without reasons people can understand.
Fix 4: reasons, a human in the loop, an audit log
import json
audit_log = []
def decide(applicant): # reasons a person can read and check
reasons = []
if applicant["income"] * 3 < applicant["loan"]:
reasons.append("loan is more than 3x yearly income")
if applicant["missed_emis"] > 2:
reasons.append(f"{applicant['missed_emis']} missed EMIs this year")
decision = "REVIEW BY OFFICER" if reasons else "APPROVED" # human in the loop
audit_log.append({"who": applicant["name"], "decision": decision,
"reasons": reasons, "model": "loan-rules-v2"})
return decision, reasons
print(decide({"name": "Meera", "income": 300000, "loan": 600000, "missed_emis": 3}))
print(decide({"name": "Arjun", "income": 500000, "loan": 900000, "missed_emis": 0}))
print(json.dumps(audit_log[0]))
Output:
('REVIEW BY OFFICER', ['3 missed EMIs this year'])
('APPROVED', [])
{"who": "Meera", "decision": "REVIEW BY OFFICER", "reasons": ["3 missed EMIs this year"], "model": "loan-rules-v2"}
- Explainability: decisions come with reasons a person can read and check.
- Human in the loop: a person reviews or approves important decisions (also used for risky agent actions in episode 35).
- Audit log: a record of who was affected, what was decided, why, and by which model version.
- Accountability: a named person or team answers for the system's decisions and can correct them.
NIST: seven characteristics of trustworthy AI
The US National Institute of Standards and Technology (NIST) released its AI Risk Management Framework on 26 January 2023. It lists the characteristics of trustworthy AI:
| Characteristic | In this episode |
|---|---|
| Valid and reliable | works, and keeps working (evals, episode 36) |
| Safe | doesn't put people in danger |
| Secure and resilient | survives attacks (episode 35) |
| Accountable and transparent | audit log, a responsible owner |
| Explainable and interpretable | Meera's reasons |
| Privacy-enhanced | the redactor |
| Fair, with harmful bias managed | swap test, group rates |
Its core asks you to map, measure, manage and govern AI risks across the system's life.
The law is catching up: the EU AI Act
The European Union's AI Act is risk-based: the riskier the use, the stricter the rules.
| Risk level | Rule | Example |
|---|---|---|
| Unacceptable | banned | social scoring |
| High | strict checks before use | deciding access to credit; sorting job applications |
| Limited | transparency | chatbots must make clear you're talking to an AI |
| Minimal | mostly left alone | spam filters |
The high-risk obligations were due on 2 August 2026. On 7 May 2026 EU lawmakers reached a provisional agreement to delay them to 2 December 2027 for stand-alone systems such as hiring tools (and 2 August 2028 for AI built into regulated products); Parliament voted for it on 16 June 2026, with Council approval pending at that time. Don't wait for deadlines: bad AI hurts people today.
Responsible AI checklist (before you ship)
- [ ] Fairness: swap tests and selection rates per group, on every model change
- [ ] Privacy: collect less, redact before sending/logging, delete when done; test the redactor
- [ ] Honesty: sources verified in code; "I don't know" allowed; users told it's an AI; AI media labelled
- [ ] Accountability: reasons, human review, audit log for decisions about people
- [ ] Ask the people affected: test with real, varied users
Glossary
| Term | Meaning | Example from this episode |
|---|---|---|
| Responsible AI | building AI that is fair, private, honest and accountable | the checklist |
| Bias (in AI) | systematic unfair treatment of people or groups | Priya 3, Kabir 5 |
| Training data bias | unfair patterns in the data a model learns from | 10 years of mostly-male hires |
| Weight | a number a model learned for a feature | "women's": -2 |
| Feature | an input the model uses | a word in the CV |
| Proxy | a feature that stands in for a protected group | a girls' college name, a PIN code |
| Counterfactual / swap test | change one irrelevant thing, check the output stays the same | he ↔ she, adding "women's" |
AssertionError |
the error assert raises when a check fails |
score changed 5 -> 3 |
| Selection rate | share of a group given the positive outcome | men 80%, women 20% |
| Fairness | equal treatment for equal cases; harmful bias managed | 3 = 3 |
| Personal data / PII | data that identifies a person | phone, Aadhaar, card |
| Redaction | replacing sensitive text with labels | [CARD] |
| Regular expression | a text pattern | \b[6-9]\d{9}\b |
| Data minimisation | collecting only what the task needs | no PIN code for a loan model |
| Consent | permission given for a stated purpose | "we use your number to call you back" |
| Retention | how long data is kept | delete after the refund |
| Hallucination | confident, fluent, false output | "Mehta v. SkyAir (2019)" |
| Citation | a reference to a source | "Rao v. JetFast (2021)" |
| Grounding | answering only from checked sources | LIBRARY |
| Transparency | being open about what a system is and how it works | "you are talking to an AI" |
| Black box | a system that decides without understandable reasons | REJECTED, why? None |
| Explainability | decisions with readable reasons | 3 missed EMIs this year |
| Human in the loop | a person reviews or approves decisions | REVIEW BY OFFICER |
| Audit log | a record of decisions, reasons and model version | the JSON line |
| Accountability | someone answers for the system and can correct it | the loan officer |
| NIST AI RMF | US framework for managing AI risks (Jan 2023) | seven characteristics |
| EU AI Act | the EU's risk-based AI law | hiring = high risk |
| DPDP Act | India's Digital Personal Data Protection Act, 2023 | rules notified 13 Nov 2025 |
Common mistakes
| Mistake | What you see | Why | Fix |
|---|---|---|---|
| Train on past decisions as they are | Kabir: 5, Priya: 3 |
the model copies old bias | swap tests + group rates |
| Delete one word and call it fixed | proxy features still biased | other features stand in | keep testing every release |
| Only check overall accuracy | one group at 20% | averages hide groups | compare per group |
| Log raw user messages | Aadhaar 1234 5678 9012 in the log |
stored data leaks | redact before logging |
| Short pattern before long one | card [AADHAAR] 1111 |
wrong rule matched first | longest first + tests |
| Log before redacting | raw text in the log | wrong order | redact, then log and send |
| Trust the model's citations | NOT FOUND: Mehta v. SkyAir (2019) |
hallucinated source | verify sources in code |
| Yes/no decisions with no reasons | why? None |
black box | reasons + human review + audit log |
| Feed data the model doesn't need | PIN code in a loan model | proxy for a group | data minimisation |
Mini project: a responsible support bot
Build a support bot for a train app that: (1) redacts phone, Aadhaar, card and email before logging or calling the model; (2) answers only from a small library of policy documents and lists the document for every answer, or says "I don't know"; (3) opens with "You're chatting with an AI assistant"; (4) sends refunds above Rs 500 to a human with the reasons; (5) writes an audit log line for every decision; (6) has tests: redactor samples, a swap test on names, and a citation check.
Practice
1. Add a swap test for names: the same CV with different names must get the same score.
Solution
SKILLS = {"python": 2, "sql": 1, "java": 2, "aws": 1}
def score(cv):
return sum(SKILLS.get(word, 0) for word in cv.lower().split())
def swap_test(cv_a, cv_b):
a, b = score(cv_a), score(cv_b)
assert a == b, f"score changed {a} -> {b}: '{cv_b}'"
print("PASS:", a, "=", b)
base = "NAME knows python sql aws"
for name_a, name_b in [("Rahul", "Rehana"), ("Kabir", "Priya"), ("Arjun", "Ayesha")]:
swap_test(base.replace("NAME", name_a), base.replace("NAME", name_b))
Output:
PASS: 4 = 4
PASS: 4 = 4
PASS: 4 = 4
2. Redact email addresses too.
Solution
import re
RULES = [(r"[\w.+-]+@[\w-]+\.[\w.]+", "[EMAIL]"),
(r"\b\d{4} \d{4} \d{4} \d{4}\b", "[CARD]"),
(r"\b\d{4} \d{4} \d{4}\b", "[AADHAAR]"),
(r"\b[6-9]\d{9}\b", "[PHONE]")]
def redact(text):
for pattern, label in RULES:
text = re.sub(pattern, label, text)
return text
print(redact("Mail me at asha.k@example.com or call 9876543210"))
Output:
Mail me at [EMAIL] or call [PHONE]
3. Fix the bot from quiz 2 so personal data never reaches the log.
Solution
import re
def redact(text): return re.sub(r"\b[6-9]\d{9}\b", "[PHONE]", text)
def llm(msg): return "Sure, I'll check."
def ask(msg):
clean = redact(msg) # 1. redact first
print("LOG", clean) # 2. log only the clean text
return llm(clean) # 3. send only the clean text
print(ask("My number is 9876543210, where is my refund?"))
Output:
LOG My number is [PHONE], where is my refund?
Sure, I'll check.
4. Let a person appeal a decision, and record the appeal in the audit log.
Solution
import json
audit_log = []
def decide(applicant):
reasons = []
if applicant["income"] * 3 < applicant["loan"]:
reasons.append("loan is more than 3x yearly income")
if applicant["missed_emis"] > 2:
reasons.append(f"{applicant['missed_emis']} missed EMIs this year")
decision = "REVIEW BY OFFICER" if reasons else "APPROVED"
audit_log.append({"who": applicant["name"], "decision": decision,
"reasons": reasons, "appeal": None})
return decision
def appeal(name, why):
for entry in audit_log:
if entry["who"] == name:
entry["appeal"] = why
decide({"name": "Meera", "income": 300000, "loan": 600000, "missed_emis": 3})
appeal("Meera", "2 of the 3 EMIs were bank errors, letter attached")
print(json.dumps(audit_log[0], indent=1))
Output:
{
"who": "Meera",
"decision": "REVIEW BY OFFICER",
"reasons": [
"3 missed EMIs this year"
],
"appeal": "2 of the 3 EMIs were bank errors, letter attached"
}
Quick quiz 1
Two loan applicants with the same income and history but different names ("Rahul Sharma": 720, "Rehana Shaikh": 640). Which check catches it?
Answer
B: a swap test on the name. Change only the name; the score must stay the same. Then remove names from the inputs and keep testing for proxies.
Quick quiz 2
def ask(msg):
print('LOG', msg)
return llm(redact(msg))
What's the privacy bug?
Answer
B: it logs before redacting. The raw message is stored anyway. Redact first, then log and send only the redacted text.
Interview questions
- How can a model become biased if nobody programs bias into it? It learns patterns from data; if past decisions were unfair, the model reproduces them, including through proxy features.
- How would you test a model for fairness? Counterfactual (swap) tests on protected attributes and proxies, and comparing outcomes such as selection rates and error rates per group, on every release.
- Why isn't removing the sensitive column enough? Other features (names, colleges, locations) can be proxies for it.
- How do you keep personal data out of LLM prompts and logs? Data minimisation, redaction before sending or logging (with tested patterns), access control, and retention limits.
- How do you reduce hallucinated citations? Ground answers in retrieved sources, verify every citation and quote in code, and allow "I don't know".
- What makes an AI decision accountable? Readable reasons, a human who can review and overrule, an audit log with the model version, and a way to appeal.
- Name the NIST AI RMF trustworthiness characteristics. Valid and reliable; safe; secure and resilient; accountable and transparent; explainable and interpretable; privacy-enhanced; fair with harmful bias managed.
- How does the EU AI Act classify AI? By risk: unacceptable (banned), high (strict requirements, e.g. credit and hiring), limited (transparency, e.g. chatbots), minimal.
Summary
- Models copy the patterns in their data, including bias; catch it with swap tests and per-group rates, and watch for proxies.
- Privacy: collect less, ask consent, redact before data leaves, delete when done; test the redactor (longest patterns first).
- Honesty: verify sources in code, allow "I don't know", say it's an AI, label AI media.
- Accountability: reasons, a human in the loop, an audit log.
- Frameworks and laws: NIST AI RMF (7 characteristics), the EU AI Act (risk levels; high-risk rules delayed to Dec 2027 by a 2026 agreement), India's DPDP Act (most duties from May 2027).
Next: episode 39, Capstone: a research assistant agent.
Sources (fetched 6 Nov 2026): Reuters via Euronews, "Amazon scraps secret AI recruiting tool that showed bias against women" (10 Oct 2018); LawNext, "Court imposes sanctions on lawyers who filed bogus cases after relying on ChatGPT" (June 2023); Davis Wright Tremaine, "NIST Releases Final Risk Management Framework for Developing Trustworthy AI" (Jan 2023); Bundesnetzagentur, "Risk levels" (EU AI Act); Sidley Data Matters, "EU Lawmakers Reach Provisional Agreement to Delay Key EU AI Act Obligations" (22 Jun 2026); AZB & Partners, "India's Digital Personal Data Protection Act: phased rollout and key compliance milestones" (14 Nov 2025).