SyntaxPilot
Courses/AI: Zero to Agents/Lesson 38
Video coming soonThis lesson’s video is being made

Start with the notes below; the video appears here as soon as it is on YouTube.

AI: Zero to Agents · Lesson 38 of 40

Responsible AI

queued

These are the notes for AI episode 38. An AI system can work perfectly in a demo and still hurt people: it can treat equal people unequally, leak their personal data, state made-up facts with confidence, or make decisions nobody can explain or appeal. In this episode you watch each of these happen in code and invent the fix. Every Python example runs as-is in the SyntaxPilot compiler; outputs and errors are real (Python 3.13). The models are toys (word weights and if-rules) standing in for real ML models and LLM calls.

Real-life example: why you trust a good doctor

A good doctor Responsible AI
Treats you by your symptoms, not your surname or gender Fairness: same skills, same score
Keeps your file locked Privacy: personal data protected
Says "let's run a test" when unsure Honesty: no made-up facts, "I don't know" is allowed
Signs the prescription and explains why Accountability: reasons, a responsible person, a record

Responsible AI means building AI systems that are fair, protect privacy, are honest about what they know, and have people who answer for their decisions.

Problem 1: a CV screener learns bias

A company trains a screener on ten years of past hiring. The toy "training" gives every word in a hired CV +1 and every word in a rejected CV −1.

past_hires = [                       # 10 years of past CVs (mostly men hired)
    ("python sql cricket club", 1), ("java sql chess club", 1),
    ("python aws chess club", 1),   ("java python football", 1),
    ("python sql women's chess club", 0), ("java aws women's coding club", 0),
]
weights = {}                         # toy "training": words seen in hires go up
for cv, hired in past_hires:
    for word in cv.split():
        weights[word] = weights.get(word, 0) + (1 if hired else -1)

def score(cv):
    return sum(weights.get(word, 0) for word in cv.split())

print("Kabir:", score("python sql chess club captain"))
print("Priya:", score("python sql women's chess club captain"))

Output:

Kabir: 5
Priya: 3

Kabir and Priya have the same skills. Priya scores lower only because her CV says "women's chess club". Nobody wrote a rule against women: in the past data most hires were men, so "women's" appeared mostly in rejected CVs and got a negative weight (−2).

  • Bias (in AI): a model treating people or groups unfairly in a systematic way.
  • Training data bias: the data reflects old unfair decisions, so the model learns them.

This really happened

Reuters reported on 10 October 2018 that Amazon had been building a CV screener since 2014. It gave candidates one to five stars and was trained on patterns in CVs submitted to the company over a 10-year period. It penalised CVs containing the word "women's" (as in "women's chess club captain"). Amazon edited the programs to be neutral to those terms; the team was later disbanded.

Fix 1a: the swap test (counterfactual test)

Take two inputs that differ only in something that should not matter, and check the output does not change.

weights = {"python": 2, "sql": 1, "chess": 1, "club": 1,
           "women's": -2}                      # learned from the past

def score(cv):
    return sum(weights.get(word, 0) for word in cv.split())

def swap_test(cv_a, cv_b):                     # two CVs, ONE word different
    a, b = score(cv_a), score(cv_b)
    assert a == b, f"score changed {a} -> {b}: '{cv_b}'"
    print("PASS:", cv_a, "=", cv_b)

swap_test("he led python sql chess club", "she led python sql chess club")
swap_test("python sql chess club", "python sql women's chess club")

Output:

PASS: he led python sql chess club = she led python sql chess club
Traceback (most recent call last):
  File "main.py", line 13, in <module>
    swap_test("python sql chess club", "python sql women's chess club")
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "main.py", line 9, in swap_test
    assert a == b, f"score changed {a} -> {b}: '{cv_b}'"
           ^^^^^^
AssertionError: score changed 5 -> 3: 'python sql women's chess club'

The AssertionError is good news: the bug is caught in a test, not in a real person's life.

  • Counterfactual test ("swap test"): change one attribute that should be irrelevant and check the decision stays the same.
  • assert condition, message: raises AssertionError: message when the condition is false.

Fix 1b: compare selection rates per group

results = [("men", True), ("men", True), ("men", False), ("men", True),
           ("men", True), ("women", False), ("women", True), ("women", False),
           ("women", False), ("women", False)]   # who the screener shortlisted

for group in ["men", "women"]:
    picks = [ok for g, ok in results if g == group]
    rate = sum(picks) / len(picks)
    print(f"{group:6} shortlisted {sum(picks)} of {len(picks)} = {rate:.0%}")

Output:

men    shortlisted 4 of 5 = 80%
women  shortlisted 1 of 5 = 20%
  • Selection rate: the share of a group that gets the positive outcome (shortlisted, approved).
  • A big gap (80% vs 20%) does not prove unfairness on its own, but it is an alarm: someone must find out why before the system goes live. Overall accuracy can look fine while one group is treated badly.

Fix 1c: score only what the job needs

SKILLS = {"python": 2, "sql": 1, "java": 2, "aws": 1}   # job skills ONLY

def score(cv):
    return sum(SKILLS.get(word, 0) for word in cv.split())

def swap_test(cv_a, cv_b):
    a, b = score(cv_a), score(cv_b)
    assert a == b, f"score changed {a} -> {b}: '{cv_b}'"
    print("PASS:", a, "=", b)

swap_test("python sql chess club", "python sql women's chess club")
swap_test("he knows java aws", "she knows java aws")
print("Kabir:", score("python sql chess club captain"))
print("Priya:", score("python sql women's chess club captain"))

Output:

PASS: 3 = 3
PASS: 3 = 3
Kabir: 3
Priya: 3

Warning: deleting one word is not enough in real models. Other features can stand in for a group (the name of a girls' college, a sport, a PIN code). These are called proxies. Keep the swap tests and group checks in your test suite and run them every time the model or data changes.

Problem 2: a log that leaks personal data

log = []                                 # every prompt is saved for "debugging"

def ask_bot(message):
    log.append(message)                  # sent to the model AND stored
    return "Sure, I'll check your refund."

ask_bot("Refund please. My phone is 9876543210, "
        "Aadhaar 1234 5678 9012, card 4111 1111 1111 1111")
print("log file:", log[0])

Output:

log file: Refund please. My phone is 9876543210, Aadhaar 1234 5678 9012, card 4111 1111 1111 1111

Phone, Aadhaar and full card number are sent to the model and stored in plain text, where many people can read them. The refund only needed the booking.

  • Personal data / PII (personally identifiable information): data that identifies a person: phone, Aadhaar, card number, email, address.

Fix 2: redact before data leaves your code

import re

RULES = [(r"\b\d{4} \d{4} \d{4} \d{4}\b", "[CARD]"),   # longest first
         (r"\b\d{4} \d{4} \d{4}\b", "[AADHAAR]"),
         (r"\b[6-9]\d{9}\b", "[PHONE]")]

def redact(text):                        # hide personal data BEFORE it leaves
    for pattern, label in RULES:
        text = re.sub(pattern, label, text)
    return text

msg = ("Refund please. My phone is 9876543210, "
       "Aadhaar 1234 5678 9012, card 4111 1111 1111 1111")
print("sent :", redact(msg))

Output:

sent : Refund please. My phone is [PHONE], Aadhaar [AADHAAR], card [CARD]
  • Redaction: replacing sensitive parts of text with labels before sending, storing or showing it.
  • Regular expression (regex): a pattern for matching text. \d{4} = four digits, \b = word boundary, [6-9]\d{9} = ten digits starting with 6-9 (an Indian mobile number).

The rule-order bug

import re

RULES = [(r"\b\d{4} \d{4} \d{4}\b", "[AADHAAR]"),     # Aadhaar rule first...
         (r"\b\d{4} \d{4} \d{4} \d{4}\b", "[CARD]"),
         (r"\b[6-9]\d{9}\b", "[PHONE]")]

def redact(text):
    for pattern, label in RULES:
        text = re.sub(pattern, label, text)
    return text

print(redact("card 4111 1111 1111 1111"))

Output:

card [AADHAAR] 1111

The 12-digit Aadhaar rule matched the first 12 digits of the 16-digit card first, so the card rule never matched and the last four digits leaked with the wrong label. Put the longest patterns first and test the redactor with samples of every kind.

Privacy habits

  1. Collect less (data minimisation): only what the task needs.
  2. Consent: ask, and say clearly what the data is for.
  3. Redact before sending to a model or writing to a log.
  4. Delete it when the job is done (retention limits).

India's law: the Digital Personal Data Protection (DPDP) Act, 2023. Its rules were notified on 13 November 2025 and apply in phases: the Data Protection Board at once; consent managers after one year (12 November 2026); most duties (notice and consent, children's data, people's rights, breach notification, security safeguards, erasure) after 18 months (12 May 2027).

Problem 3: confident and wrong

def llm(question):                       # toy LLM: always sounds confident
    return {"answer": "Airlines must pay for late bags.",
            "cites": ["Mehta v. SkyAir (2019)", "Rao v. JetFast (2021)"]}

reply = llm("Find court cases about lost airline luggage")
print("answer:", reply["answer"])
for case in reply["cites"]:
    print("  source:", case)

Output:

answer: Airlines must pay for late bags.
  source: Mehta v. SkyAir (2019)
  source: Rao v. JetFast (2021)

One of these cases does not exist. A language model predicts likely words (episode 19), and a realistic case name is very likely words. This is a hallucination: fluent, confident, false output.

This happened in a real court

In Mata v. Avianca (New York federal court), lawyers filed a brief citing court opinions that did not exist, with fake quotes and citations, produced with ChatGPT. On 22 June 2023, Judge P. Kevin Castel sanctioned the two lawyers and their firm $5,000. The people who used the tool without checking were held responsible.

Fix 3: check every source in code

LIBRARY = {"Rao v. JetFast (2021)": "court record #5531"}   # sources we can check

def llm(question):
    return {"answer": "Airlines must pay for late bags.",
            "cites": ["Mehta v. SkyAir (2019)", "Rao v. JetFast (2021)"]}

def check(reply):                        # never trust a citation you can't find
    real = [c for c in reply["cites"] if c in LIBRARY]
    fake = [c for c in reply["cites"] if c not in LIBRARY]
    for c in fake:
        print("  NOT FOUND:", c)
    if not real:
        return "I couldn't find a real source for this."
    return reply["answer"] + " (source: " + ", ".join(real) + ")"

print(check(llm("Find court cases about lost airline luggage")))

Output:

  NOT FOUND: Mehta v. SkyAir (2019)
Airlines must pay for late bags. (source: Rao v. JetFast (2021))

A real system also checks that the source actually says what the answer claims (episode 39 checks quotes word for word). If nothing checks out, "I couldn't find a real source" is the honest answer.

Honesty habits

  • Show sources and verify them in code; ground answers in retrieved documents (RAG, episode 24).
  • Allow "I don't know" in the prompt and in the user interface.
  • Tell people when they are talking to an AI, not a human.
  • Label AI-generated images, voices and videos.

Problem 4: a black box

def model(applicant):                    # a black box: just yes or no
    return applicant["income"] * 3 > applicant["loan"] + 400000

meera = {"name": "Meera", "income": 300000, "loan": 600000, "pin": "110092"}
print(meera["name"], "->", "APPROVED" if model(meera) else "REJECTED")
print("why?", meera.get("reason"))

Output:

Meera -> REJECTED
why? None

Meera can't fix anything, appeal, or know whether it was a mistake. And the model was given her PIN code, which it doesn't need and which can act as a proxy for a group.

  • Black box: a system whose decisions come without reasons people can understand.

Fix 4: reasons, a human in the loop, an audit log

import json
audit_log = []

def decide(applicant):                   # reasons a person can read and check
    reasons = []
    if applicant["income"] * 3 < applicant["loan"]:
        reasons.append("loan is more than 3x yearly income")
    if applicant["missed_emis"] > 2:
        reasons.append(f"{applicant['missed_emis']} missed EMIs this year")
    decision = "REVIEW BY OFFICER" if reasons else "APPROVED"   # human in the loop
    audit_log.append({"who": applicant["name"], "decision": decision,
                      "reasons": reasons, "model": "loan-rules-v2"})
    return decision, reasons

print(decide({"name": "Meera", "income": 300000, "loan": 600000, "missed_emis": 3}))
print(decide({"name": "Arjun", "income": 500000, "loan": 900000, "missed_emis": 0}))
print(json.dumps(audit_log[0]))

Output:

('REVIEW BY OFFICER', ['3 missed EMIs this year'])
('APPROVED', [])
{"who": "Meera", "decision": "REVIEW BY OFFICER", "reasons": ["3 missed EMIs this year"], "model": "loan-rules-v2"}
  • Explainability: decisions come with reasons a person can read and check.
  • Human in the loop: a person reviews or approves important decisions (also used for risky agent actions in episode 35).
  • Audit log: a record of who was affected, what was decided, why, and by which model version.
  • Accountability: a named person or team answers for the system's decisions and can correct them.

NIST: seven characteristics of trustworthy AI

The US National Institute of Standards and Technology (NIST) released its AI Risk Management Framework on 26 January 2023. It lists the characteristics of trustworthy AI:

Characteristic In this episode
Valid and reliable works, and keeps working (evals, episode 36)
Safe doesn't put people in danger
Secure and resilient survives attacks (episode 35)
Accountable and transparent audit log, a responsible owner
Explainable and interpretable Meera's reasons
Privacy-enhanced the redactor
Fair, with harmful bias managed swap test, group rates

Its core asks you to map, measure, manage and govern AI risks across the system's life.

The law is catching up: the EU AI Act

The European Union's AI Act is risk-based: the riskier the use, the stricter the rules.

Risk level Rule Example
Unacceptable banned social scoring
High strict checks before use deciding access to credit; sorting job applications
Limited transparency chatbots must make clear you're talking to an AI
Minimal mostly left alone spam filters

The high-risk obligations were due on 2 August 2026. On 7 May 2026 EU lawmakers reached a provisional agreement to delay them to 2 December 2027 for stand-alone systems such as hiring tools (and 2 August 2028 for AI built into regulated products); Parliament voted for it on 16 June 2026, with Council approval pending at that time. Don't wait for deadlines: bad AI hurts people today.

Responsible AI checklist (before you ship)

  • [ ] Fairness: swap tests and selection rates per group, on every model change
  • [ ] Privacy: collect less, redact before sending/logging, delete when done; test the redactor
  • [ ] Honesty: sources verified in code; "I don't know" allowed; users told it's an AI; AI media labelled
  • [ ] Accountability: reasons, human review, audit log for decisions about people
  • [ ] Ask the people affected: test with real, varied users

Glossary

Term Meaning Example from this episode
Responsible AI building AI that is fair, private, honest and accountable the checklist
Bias (in AI) systematic unfair treatment of people or groups Priya 3, Kabir 5
Training data bias unfair patterns in the data a model learns from 10 years of mostly-male hires
Weight a number a model learned for a feature "women's": -2
Feature an input the model uses a word in the CV
Proxy a feature that stands in for a protected group a girls' college name, a PIN code
Counterfactual / swap test change one irrelevant thing, check the output stays the same he ↔ she, adding "women's"
AssertionError the error assert raises when a check fails score changed 5 -> 3
Selection rate share of a group given the positive outcome men 80%, women 20%
Fairness equal treatment for equal cases; harmful bias managed 3 = 3
Personal data / PII data that identifies a person phone, Aadhaar, card
Redaction replacing sensitive text with labels [CARD]
Regular expression a text pattern \b[6-9]\d{9}\b
Data minimisation collecting only what the task needs no PIN code for a loan model
Consent permission given for a stated purpose "we use your number to call you back"
Retention how long data is kept delete after the refund
Hallucination confident, fluent, false output "Mehta v. SkyAir (2019)"
Citation a reference to a source "Rao v. JetFast (2021)"
Grounding answering only from checked sources LIBRARY
Transparency being open about what a system is and how it works "you are talking to an AI"
Black box a system that decides without understandable reasons REJECTED, why? None
Explainability decisions with readable reasons 3 missed EMIs this year
Human in the loop a person reviews or approves decisions REVIEW BY OFFICER
Audit log a record of decisions, reasons and model version the JSON line
Accountability someone answers for the system and can correct it the loan officer
NIST AI RMF US framework for managing AI risks (Jan 2023) seven characteristics
EU AI Act the EU's risk-based AI law hiring = high risk
DPDP Act India's Digital Personal Data Protection Act, 2023 rules notified 13 Nov 2025

Common mistakes

Mistake What you see Why Fix
Train on past decisions as they are Kabir: 5, Priya: 3 the model copies old bias swap tests + group rates
Delete one word and call it fixed proxy features still biased other features stand in keep testing every release
Only check overall accuracy one group at 20% averages hide groups compare per group
Log raw user messages Aadhaar 1234 5678 9012 in the log stored data leaks redact before logging
Short pattern before long one card [AADHAAR] 1111 wrong rule matched first longest first + tests
Log before redacting raw text in the log wrong order redact, then log and send
Trust the model's citations NOT FOUND: Mehta v. SkyAir (2019) hallucinated source verify sources in code
Yes/no decisions with no reasons why? None black box reasons + human review + audit log
Feed data the model doesn't need PIN code in a loan model proxy for a group data minimisation

Mini project: a responsible support bot

Build a support bot for a train app that: (1) redacts phone, Aadhaar, card and email before logging or calling the model; (2) answers only from a small library of policy documents and lists the document for every answer, or says "I don't know"; (3) opens with "You're chatting with an AI assistant"; (4) sends refunds above Rs 500 to a human with the reasons; (5) writes an audit log line for every decision; (6) has tests: redactor samples, a swap test on names, and a citation check.

Practice

1. Add a swap test for names: the same CV with different names must get the same score.

Solution
SKILLS = {"python": 2, "sql": 1, "java": 2, "aws": 1}

def score(cv):
    return sum(SKILLS.get(word, 0) for word in cv.lower().split())

def swap_test(cv_a, cv_b):
    a, b = score(cv_a), score(cv_b)
    assert a == b, f"score changed {a} -> {b}: '{cv_b}'"
    print("PASS:", a, "=", b)

base = "NAME knows python sql aws"
for name_a, name_b in [("Rahul", "Rehana"), ("Kabir", "Priya"), ("Arjun", "Ayesha")]:
    swap_test(base.replace("NAME", name_a), base.replace("NAME", name_b))

Output:

PASS: 4 = 4
PASS: 4 = 4
PASS: 4 = 4

2. Redact email addresses too.

Solution
import re

RULES = [(r"[\w.+-]+@[\w-]+\.[\w.]+", "[EMAIL]"),
         (r"\b\d{4} \d{4} \d{4} \d{4}\b", "[CARD]"),
         (r"\b\d{4} \d{4} \d{4}\b", "[AADHAAR]"),
         (r"\b[6-9]\d{9}\b", "[PHONE]")]

def redact(text):
    for pattern, label in RULES:
        text = re.sub(pattern, label, text)
    return text

print(redact("Mail me at asha.k@example.com or call 9876543210"))

Output:

Mail me at [EMAIL] or call [PHONE]

3. Fix the bot from quiz 2 so personal data never reaches the log.

Solution
import re
def redact(text): return re.sub(r"\b[6-9]\d{9}\b", "[PHONE]", text)
def llm(msg): return "Sure, I'll check."

def ask(msg):
    clean = redact(msg)            # 1. redact first
    print("LOG", clean)            # 2. log only the clean text
    return llm(clean)              # 3. send only the clean text

print(ask("My number is 9876543210, where is my refund?"))

Output:

LOG My number is [PHONE], where is my refund?
Sure, I'll check.

4. Let a person appeal a decision, and record the appeal in the audit log.

Solution
import json
audit_log = []

def decide(applicant):
    reasons = []
    if applicant["income"] * 3 < applicant["loan"]:
        reasons.append("loan is more than 3x yearly income")
    if applicant["missed_emis"] > 2:
        reasons.append(f"{applicant['missed_emis']} missed EMIs this year")
    decision = "REVIEW BY OFFICER" if reasons else "APPROVED"
    audit_log.append({"who": applicant["name"], "decision": decision,
                      "reasons": reasons, "appeal": None})
    return decision

def appeal(name, why):
    for entry in audit_log:
        if entry["who"] == name:
            entry["appeal"] = why

decide({"name": "Meera", "income": 300000, "loan": 600000, "missed_emis": 3})
appeal("Meera", "2 of the 3 EMIs were bank errors, letter attached")
print(json.dumps(audit_log[0], indent=1))

Output:

{
 "who": "Meera",
 "decision": "REVIEW BY OFFICER",
 "reasons": [
  "3 missed EMIs this year"
 ],
 "appeal": "2 of the 3 EMIs were bank errors, letter attached"
}

Quick quiz 1

Two loan applicants with the same income and history but different names ("Rahul Sharma": 720, "Rehana Shaikh": 640). Which check catches it?

Answer

B: a swap test on the name. Change only the name; the score must stay the same. Then remove names from the inputs and keep testing for proxies.

Quick quiz 2

def ask(msg):
    print('LOG', msg)
    return llm(redact(msg))

What's the privacy bug?

Answer

B: it logs before redacting. The raw message is stored anyway. Redact first, then log and send only the redacted text.

Interview questions

  1. How can a model become biased if nobody programs bias into it? It learns patterns from data; if past decisions were unfair, the model reproduces them, including through proxy features.
  2. How would you test a model for fairness? Counterfactual (swap) tests on protected attributes and proxies, and comparing outcomes such as selection rates and error rates per group, on every release.
  3. Why isn't removing the sensitive column enough? Other features (names, colleges, locations) can be proxies for it.
  4. How do you keep personal data out of LLM prompts and logs? Data minimisation, redaction before sending or logging (with tested patterns), access control, and retention limits.
  5. How do you reduce hallucinated citations? Ground answers in retrieved sources, verify every citation and quote in code, and allow "I don't know".
  6. What makes an AI decision accountable? Readable reasons, a human who can review and overrule, an audit log with the model version, and a way to appeal.
  7. Name the NIST AI RMF trustworthiness characteristics. Valid and reliable; safe; secure and resilient; accountable and transparent; explainable and interpretable; privacy-enhanced; fair with harmful bias managed.
  8. How does the EU AI Act classify AI? By risk: unacceptable (banned), high (strict requirements, e.g. credit and hiring), limited (transparency, e.g. chatbots), minimal.

Summary

  • Models copy the patterns in their data, including bias; catch it with swap tests and per-group rates, and watch for proxies.
  • Privacy: collect less, ask consent, redact before data leaves, delete when done; test the redactor (longest patterns first).
  • Honesty: verify sources in code, allow "I don't know", say it's an AI, label AI media.
  • Accountability: reasons, a human in the loop, an audit log.
  • Frameworks and laws: NIST AI RMF (7 characteristics), the EU AI Act (risk levels; high-risk rules delayed to Dec 2027 by a 2026 agreement), India's DPDP Act (most duties from May 2027).

Next: episode 39, Capstone: a research assistant agent.

Sources (fetched 6 Nov 2026): Reuters via Euronews, "Amazon scraps secret AI recruiting tool that showed bias against women" (10 Oct 2018); LawNext, "Court imposes sanctions on lawyers who filed bogus cases after relying on ChatGPT" (June 2023); Davis Wright Tremaine, "NIST Releases Final Risk Management Framework for Developing Trustworthy AI" (Jan 2023); Bundesnetzagentur, "Risk levels" (EU AI Act); Sidley Data Matters, "EU Lawmakers Reach Provisional Agreement to Delay Key EU AI Act Obligations" (22 Jun 2026); AZB & Partners, "India's Digital Personal Data Protection Act: phased rollout and key compliance milestones" (14 Nov 2025).