SyntaxPilot
Courses/Machine Learning/Lesson 9

Machine Learning · Lesson 9 of 90 · 11 min

Maths 7: probability basics

experimentsample spaceeventprobability rulesrandom variablejoint probabilitymarginal probabilityconditional probabilityindependenceBayes' theorempriorposteriorbase-rate fallacy

Machine Learning Full Course · Episode 9. Every code sample below was run with Python 3.13 using only the standard library (fractions, random, collections); the output shown is the real output. The medical-test numbers are made up for teaching; no real test is described.

The puzzle: a positive result on a 99% test

A health camp screens everyone for a rare illness.

  • 1 person in 1,000 has it.
  • The test catches 99% of sick people.
  • It wrongly flags 1% of healthy people.

Your result is positive. How likely is it that you are actually sick? Most people answer 99%. To check, we build probability from scratch, and the real answer turns out to be about 9%.

Experiment, outcome, sample space, event

Term Meaning Ludo die example
Experiment an action whose result can't be known in advance rolling a die
Outcome one possible result a 4
Sample space the set of all possible outcomes {1, 2, 3, 4, 5, 6}
Event a group of outcomes we care about "even" = {2, 4, 6}; "big" (> 3) = {4, 5, 6}

When every outcome is equally likely, the probability of an event A is

P(A) = (number of favourable outcomes) ÷ (total number of outcomes)

from fractions import Fraction

space = [1, 2, 3, 4, 5, 6]               # sample space of one die roll
even = [x for x in space if x % 2 == 0]  # an event
big = [x for x in space if x > 3]        # another event

def P(event):                            # equally likely outcomes
    return Fraction(len(event), len(space))

print("P(even) =", P(even), " P(big) =", P(big))
print("P(7) =", P([x for x in space if x == 7]), " P(any) =", P(space))
print("P(not 6) =", 1 - P([6]))

Output:

P(even) = 1/2  P(big) = 1/2
P(7) = 0  P(any) = 1
P(not 6) = 5/6
  • Fraction(3, 6) keeps exact fractions (shown as 1/2) instead of decimals.
  • [x for x in space if x % 2 == 0] is a list comprehension: keep the x's that pass the test (% is the remainder).

The probability rules

  1. Range: 0 ≤ P(A) ≤ 1. An impossible event (a 7) has probability 0; a certain event (the whole sample space) has probability 1.
  2. Complement: P(not A) = 1 − P(A). "Not A" (the complement) is every outcome outside A. P(not 6) = 5/6.
  3. Addition rule: P(A or B) = P(A) + P(B) − P(A and B).

Why the addition rule subtracts (the obvious way fails)

from fractions import Fraction

space = [1, 2, 3, 4, 5, 6]
even = {x for x in space if x % 2 == 0}
big = {x for x in space if x > 3}

def P(event):
    return Fraction(len(event), len(space))

print("just add:     ", P(even) + P(big))
print("count it:     ", P(even | big), sorted(even | big))
print("subtract both:", P(even) + P(big) - P(even & big))

Output:

just add:      1
count it:      2/3 [2, 4, 5, 6]
subtract both: 2/3

Adding gives 1 ("certain"), but a 1 or a 3 is neither even nor big. 4 and 6 are in both events, so adding counted them twice. Subtract the overlap once. If two events can't both happen (mutually exclusive, e.g. "roll a 1" and "roll a 2"), the overlap is 0 and you may simply add.

  • {...} makes a set; | is set union (OR), & is intersection (AND).

Random variables

A random variable is a rule that gives a number for each outcome. Roll two dice in Ludo: 36 equally likely pairs, but your token moves by the sum, X = 2 … 12. X is "random" because it depends on the roll. Each value has a probability: a 7 can come 6 ways (1+6, 2+5, 3+4, 4+3, 5+2, 6+1), so P(X = 7) = 6/36; a 2 only one way (1/36).

import random
from collections import Counter

random.seed(7)                     # same "random" numbers every run
N = 100_000
sums = Counter(random.randint(1, 6) + random.randint(1, 6) for _ in range(N))

def exact(t):                      # count the 36 equally likely pairs
    return sum(1 for a in range(1, 7) for b in range(1, 7) if a + b == t) / 36

for t in [2, 7, 12]:
    print(t, "simulated:", round(sums[t] / N, 4), " exact:", round(exact(t), 4))

Output:

2 simulated: 0.0277  exact: 0.0278
7 simulated: 0.1682  exact: 0.1667
12 simulated: 0.0274  exact: 0.0278
  • random.randint(1, 6) gives a whole number from 1 to 6 inclusive.
  • random.seed(7) fixes the starting point of Python's pseudo-random numbers, so everyone gets the same output (reproducibility, episode 2).
  • Counter counts how many times each sum appeared.
  • Simulation (the long-run frequency view of probability): the more rolls, the closer the frequency gets to the exact probability. When unsure of an answer, simulate it.

Joint and marginal probability

Raju's order book for one week, 250 orders:

Tea Coffee Total
Morning 120 30 150
Evening 60 40 100
Total 180 70 250
  • Joint probability, two things together: P(morning and coffee) = 30 / 250 = 0.12.
  • Marginal probability, one thing whatever the other: P(coffee) = (30 + 40) / 250 = 0.28; P(morning) = 150 / 250 = 0.6. Found by adding the joint table over the variable you don't care about; the name comes from the totals written in the table's margin.

Conditional probability

Given that it is morning, what is the chance of coffee? Look only at the morning row: 30 / 150 = 0.2. A probability computed when something is already known is a conditional probability, written P(A | B), read "A given B":

P(A | B) = P(A and B) ÷ P(B)

orders = {("morning", "tea"): 120, ("morning", "coffee"): 30,
          ("evening", "tea"): 60, ("evening", "coffee"): 40}
total = sum(orders.values())

def P(time=None, drink=None):             # probability from counts
    n = sum(c for (t, d), c in orders.items()
            if (time is None or t == time) and (drink is None or d == drink))
    return n / total

print("joint    P(morning and coffee):", P("morning", "coffee"))
print("marginal P(coffee):", P(drink="coffee"), " P(morning):", P("morning"))
print("conditional P(coffee | morning):", P("morning", "coffee") / P("morning"))
print("conditional P(morning | coffee):", round(P("morning", "coffee") / P(drink="coffee"), 4))

Output:

joint    P(morning and coffee): 0.12
marginal P(coffee): 0.28  P(morning): 0.6
conditional P(coffee | morning): 0.2
conditional P(morning | coffee): 0.4286
  • None as a default means "don't filter on this". orders.items() gives each ((time, drink), count) pair.
  • P(A | B) ≠ P(B | A). Of the morning orders, 20% are coffee; of the coffee orders, 43% are in the morning. Different questions, different denominators. Swapping them is the most common probability mistake, and it is the trap in the health-camp puzzle.

Independence

Two events are independent when knowing one tells you nothing about the other (two separate coin tosses). Then, and only then:

P(A and B) = P(A) × P(B)

import random

random.seed(1)
N = 100_000
a = [random.random() < 0.5 for _ in range(N)]   # coin 1 lands heads?
b = [random.random() < 0.5 for _ in range(N)]   # coin 2 lands heads?
both = sum(x and y for x, y in zip(a, b)) / N
print("coins: P(A and B) =", round(both, 3), " P(A)P(B) =", round(sum(a) / N * sum(b) / N, 3))

p_coffee, p_morning, p_both = 0.28, 0.6, 0.12
print("chai:  P(A and B) =", p_both, " P(A)P(B) =", round(p_coffee * p_morning, 3))

Output:

coins: P(A and B) = 0.251  P(A)P(B) = 0.25
chai:  P(A and B) = 0.12  P(A)P(B) = 0.168
  • random.random() gives a decimal in [0, 1); < 0.5 is True half the time. sum of True/False counts the Trues.
  • Coins: they match (independent). Chai: 0.12 ≠ 0.168, so time of day and drink are dependent (equivalently, P(coffee | morning) = 0.2 ≠ P(coffee) = 0.28).

Back to the puzzle: simulate a million people

import random

random.seed(42)
N = 1_000_000
sick_pos = healthy_pos = 0
for _ in range(N):
    sick = random.random() < 0.001          # 1 in 1,000 people have it
    if sick:
        positive = random.random() < 0.99   # catches 99% of sick people
    else:
        positive = random.random() < 0.01   # wrongly flags 1% of healthy
    if positive:
        sick_pos += sick
        healthy_pos += not sick
print("positives:", sick_pos + healthy_pos, "sick:", sick_pos, "healthy:", healthy_pos)
print("P(sick | positive) =", round(sick_pos / (sick_pos + healthy_pos), 3))

Output:

positives: 10969 sick: 935 healthy: 10034
P(sick | positive) = 0.085

Of 10,969 positives, only 935 are sick. Why the gut fails: healthy people outnumber sick people 999 to 1, so 1% of the huge healthy crowd (≈ 10,000 false alarms) swamps 99% of the tiny sick group (≈ 990 true alarms). The 99% was P(positive | sick); the question asked P(sick | positive).

Bayes' theorem

P(H | E) = P(E | H) × P(H) ÷ P(E)

Piece Meaning Puzzle value
H the hypothesis "I am sick"
E the evidence "the test is positive"
Prior P(H) belief before the evidence 0.001
Likelihood P(E | H) how likely the evidence is if H is true 0.99
Evidence P(E) how likely the evidence is overall 0.99 × 0.001 + 0.01 × 0.999
Posterior P(H | E) belief after the evidence ≈ 0.09

P(E) uses the law of total probability: a positive can come from a sick person or from a healthy person, two separate ways, so add them.

prior = 0.001                 # P(sick) before the test
p_pos_sick = 0.99             # P(positive | sick)
p_pos_healthy = 0.01          # P(positive | healthy)

p_pos = p_pos_sick * prior + p_pos_healthy * (1 - prior)   # all positives
posterior = p_pos_sick * prior / p_pos                       # Bayes' theorem
print("P(positive) =", round(p_pos, 5))
print("P(sick | positive) =", round(posterior, 4))

second = p_pos_sick * posterior / (p_pos_sick * posterior + p_pos_healthy * (1 - posterior))
print("after a 2nd positive test:", round(second, 4))

Output:

P(positive) = 0.01098
P(sick | positive) = 0.0902
after a 2nd positive test: 0.9075
  • The exact posterior 0.0902 matches the simulation (0.085 is within sampling noise for ~1,000 sick people).
  • Today's posterior is tomorrow's prior: a second independent positive test lifts the belief to 90.75%. This is why rare diagnoses are confirmed with a second test.
  • Base-rate fallacy: ignoring how common something is to begin with (the base rate, here 1 in 1,000). A fallacy is a common thinking mistake.
  • Where it shows up in ML: spam filters (episode 28, naive Bayes), fraud alerts and any detector of rare events: with a rare positive class, even an accurate model raises mostly false alarms (precision, episode 24).

Common mistakes

Mistake Result Why Fix
P(even or big) = 1/2 + 1/2 1 instead of 2/3 {4, 6} counted twice subtract P(A and B)
P(coffee | morning) = P(morning | coffee) 0.2 vs 0.4286 different denominators divide by the probability of what you KNOW
P(A and B) = P(A) × P(B) always 0.168 instead of 0.12 only true for independent events check independence first
positive on a 99% test ⇒ 99% sick really ≈ 0.09 base-rate fallacy Bayes: include the prior
a probability above 1 or below 0 impossible answer a counting or formula error check 0 ≤ P ≤ 1
simulation without random.seed different numbers every run no fixed starting point set a seed for repeatable results
random.randint(1, 6) thought to exclude 6 wrong sample space randint includes both ends (unlike range) randint(a, b) → a … b

Mini project: the three-box puzzle

A prize is hidden in one of three boxes. You pick one. The host, who knows where the prize is, opens one of the other boxes and shows it is empty, then offers you a switch. Stay or switch?

import random

random.seed(11)
N = 100_000
stay_wins = switch_wins = 0
for _ in range(N):
    prize = random.randint(1, 3)          # the box with the prize
    pick = random.randint(1, 3)           # your first choice
    # the host opens an empty box that you didn't pick
    opened = random.choice([b for b in (1, 2, 3) if b != pick and b != prize])
    switched = next(b for b in (1, 2, 3) if b != pick and b != opened)
    stay_wins += pick == prize
    switch_wins += switched == prize
print("stay:", stay_wins / N, " switch:", switch_wins / N)

Output:

stay: 0.33531  switch: 0.66469

Switching wins 2/3 of the time. Your first pick is right with probability 1/3 (the prior), and the host's choice is not independent of where the prize is: he must avoid it. That information moves the remaining 2/3 onto the other closed box. random.choice(list) picks one item; next(...) takes the first item that matches.

Practice

  1. P(sum of two dice ≥ 10), exact and simulated.
Solution

Pairs: (4,6), (5,5), (6,4), (5,6), (6,5), (6,6) = 6 of 36 = 1/6.

import random
from fractions import Fraction

exact = Fraction(sum(1 for a in range(1, 7) for b in range(1, 7) if a + b >= 10), 36)
random.seed(5)
N = 100_000
hits = sum(random.randint(1, 6) + random.randint(1, 6) >= 10 for _ in range(N))
print("exact:", exact, "=", round(float(exact), 4), " simulated:", hits / N)

Output:

exact: 1/6 = 0.1667  simulated: 0.16637
  1. P(tea | evening) and P(evening | tea) from Raju's table.
Solution

P(tea | evening) = 60 / 100 = 0.6; P(evening | tea) = 60 / 180 = 0.3333. Different denominators, different answers.

  1. Redo the test puzzle with 1 in 100 people sick.
Solution

posterior = 0.99 × 0.01 ÷ (0.99 × 0.01 + 0.01 × 0.99) = 0.5. A 10× higher base rate turns 9% into 50%: the prior matters enormously.

def bayes(prior, p_e_h, p_e_not_h):
    return p_e_h * prior / (p_e_h * prior + p_e_not_h * (1 - prior))

print(round(bayes(0.01, 0.99, 0.01), 4))

prints 0.5.

  1. On one die, are "even" and "big" independent?
Solution

P(even and big) = P({4, 6}) = 1/3, but P(even) × P(big) = 1/2 × 1/2 = 1/4. Not equal, so not independent: knowing the roll is big makes even more likely (2 of 3 big numbers are even). In Python, P(even & big), P(even) * P(big) prints 1/3 1/4.

Quick quiz

from fractions import Fraction

space = range(1, 7)
A = {x for x in space if x <= 2}
B = {x for x in space if x % 2 == 1}

P = lambda e: Fraction(len(e), 6)
print(P(A | B), P(A & B) / P(B))

A) 2/3 1/3 B) 5/6 1/3 C) 2/3 1/2 D) 1 1/3

Answer

A) 2/3 1/3. A = {1, 2}, B = {1, 3, 5}. A or B = {1, 2, 3, 5}: 4/6 = 2/3. A and B = {1}: 1/6, so P(A | B) = (1/6) ÷ (3/6) = 1/3.

Interview questions

  1. What is conditional probability? The probability of A when B is known to have happened: P(A | B) = P(A and B) / P(B).
  2. State Bayes' theorem and name its parts. P(H | E) = P(E | H)·P(H) / P(E): posterior = likelihood × prior / evidence.
  3. What does independence mean, and how do you test it? Knowing one event doesn't change the other's probability: P(A | B) = P(A), equivalently P(A and B) = P(A)·P(B).
  4. Independent vs mutually exclusive? Mutually exclusive events can't happen together (P(A and B) = 0); independent events don't influence each other. Two mutually exclusive events with non-zero probabilities are always dependent.
  5. What is the base-rate fallacy, and where does it hurt in ML? Ignoring the prior frequency. With rare positives (fraud, disease), even an accurate classifier's alerts are mostly false positives, so accuracy alone misleads.
  6. Joint vs marginal vs conditional? Joint: P(A and B). Marginal: P(A), summing the joint over the other variable. Conditional: P(A | B), the joint divided by the marginal of B.
  7. How would you check a probability answer quickly? Simulate it with a fixed seed and compare the frequency with the formula.

Glossary

Term Meaning Example
Probability A number from 0 to 1 saying how likely something is P(even) = 1/2
Experiment An action with an uncertain result rolling a die
Outcome One possible result a 4
Sample space The set of all possible outcomes {1, …, 6}
Event A group of outcomes even = {2, 4, 6}
Equally likely Every outcome has the same chance a fair die
Complement Everything outside an event; P(not A) = 1 − P(A) not 6: 5/6
Addition rule P(A or B) = P(A) + P(B) − P(A and B) 2/3
Mutually exclusive Events that can't happen together roll a 1 / roll a 2
Random variable A number assigned to each outcome X = sum of two dice
Simulation Estimating probabilities by repeating random trials 100,000 rolls
Seed The starting point of pseudo-random numbers; same seed, same numbers random.seed(7)
Joint probability P(A and B) P(morning and coffee) = 0.12
Marginal probability P(A) whatever the other variable is P(coffee) = 0.28
Conditional probability P(A | B) = P(A and B) / P(B) P(coffee | morning) = 0.2
Independent events Knowing one doesn't change the other; P(A and B) = P(A)·P(B) two coins
Dependent events Knowing one changes the other morning and coffee
Hypothesis A claim we want the probability of "I am sick"
Evidence What we observed a positive test
Prior Belief before the evidence 0.001
Likelihood P(evidence | hypothesis) 0.99
Posterior Belief after the evidence 0.0902
Law of total probability Add up every separate way an event can happen P(positive) = 0.01098
Bayes' theorem P(H | E) = P(E | H)·P(H) / P(E) 9% sick
Base rate How common something is in the whole population 1 in 1,000
Base-rate fallacy Ignoring the base rate when judging evidence "99% sick"
False positive A positive result for someone who doesn't have it 10,034 healthy positives
Fraction Python's exact fraction type Fraction(1, 2)
Set, \|, & Python set; union (or); intersection (and) even \| big

Summary

  • An experiment has a sample space of outcomes; an event is a group of them; P = favourable ÷ total when outcomes are equally likely.
  • Rules: 0 ≤ P ≤ 1, complement 1 − P(A), addition rule subtracts the overlap.
  • A random variable puts a number on each outcome; simulation with a seed checks any answer.
  • From a table: joint, marginal and conditional probability; P(A | B) ≠ P(B | A).
  • Independent events multiply; dependent ones don't.
  • Bayes' theorem turns a prior into a posterior; ignoring the base rate (the base-rate fallacy) turns 9% into a mistaken 99%.

Next episode: Maths for ML 8, probability distributions: PMF, PDF, CDF, Bernoulli, binomial, Poisson, uniform and normal, mean and variance, sampled with NumPy.