Machine Learning Full Course · Episode 82. Every program below was run on a Linux machine with Python 3.13, NumPy 2.5.3, SciPy 1.18.1 and scikit-learn 1.9.1, and the output shown is the real output. All loan and user data is made up: random numbers with a hidden rule.
The big idea: a live model can fail while the server looks perfect. So we monitor four things: system health, data (input) drift, prediction drift and performance. Performance needs the true answers (labels), and those usually arrive late, so we watch the inputs and predictions today with the KS test and the Population Stability Index (PSI), and check accuracy when the labels catch up. Drift comes in three kinds: covariate shift, label (prior) shift and concept drift. Numbers become alerts and retraining triggers, and a new model is released safely with shadow deployment, a canary release or an A/B test.
Sources: SciPy docs, scipy.stats.ks_2samp ("Performs the two-sample Kolmogorov-Smirnov test for goodness of fit."; the two-sided statistic is the maximum absolute difference between the two empirical distribution functions). du Pisanie, Allison, Budde and Visagie, A critical review of existing and new population stability testing procedures in credit risk scoring (PSI = Σ (P_j − q_j) · ln(P_j / q_j); the rule of thumb they cite: below 0.1 "no substantial change", 0.1 to 0.25 "a small change", 0.25 or more "a substantial change"). Martin Fowler, Canary Release ("a technique to reduce the risk of introducing a new software version in production" by rolling it out to a small group of users first; named after the canaries miners carried into coal mines as an early warning of toxic gas).
Words you need first
- Model: the learned rule that turns inputs into a prediction (Episode 1). Ours turns income and loan size into "risky / not risky".
- Label: the true answer for a row, e.g. "did this person repay?" (Episode 1).
- Accuracy: the share of predictions that were right (Episode 24).
- Distribution: how a feature's values are spread out: where most people are, and how wide the spread is (Episode 10).
- p-value: the chance of seeing a difference at least this big if nothing had really changed (Episode 11). A tiny p-value means "this is unlikely to be luck".
- Chi-square test: a test of whether two categorical variables are related, e.g. "model A/B" and "took the offer / didn't" (Episode 11).
The story: all green, still wrong
Our loan model is live. Model monitoring means checking a live model all the time, so problems are noticed early. The obvious attempt is to watch what every web team already watches: the status code (200 = "request worked"), the latency (how long one prediction takes) and the error count. If all three are fine, the model must be fine. Let's test that.
import numpy as np
from sklearn.linear_model import LogisticRegression
rng = np.random.default_rng(82)
def loans(n, cut): # made-up: income (thousand ₹/month), loan (lakh ₹)
X = np.c_[rng.normal(60, 15, n), rng.normal(8, 3, n)]
y = (X[:, 1] * 6 - X[:, 0] * 0.4 + rng.normal(0, 4, n) > cut).astype(int)
return X, y
model = LogisticRegression().fit(*loans(5000, cut=20))
print("week | status | latency | dashboard | accuracy, 90 days later")
for week, cut in enumerate([20, 20, 18, 12, 6, 2, 0], 1): # rates rise: the rule shifts
X, y = loans(1000, cut)
latency = int(rng.normal(14, 2)) # the server is perfectly healthy
print(f" {week} | 200 | {latency} ms | ALL GREEN | {model.score(X, y):.2f}")
week | status | latency | dashboard | accuracy, 90 days later
1 | 200 | 14 ms | ALL GREEN | 0.94
2 | 200 | 14 ms | ALL GREEN | 0.94
3 | 200 | 13 ms | ALL GREEN | 0.93
4 | 200 | 14 ms | ALL GREEN | 0.84
5 | 200 | 13 ms | ALL GREEN | 0.77
6 | 200 | 14 ms | ALL GREEN | 0.70
7 | 200 | 16 ms | ALL GREEN | 0.70
Seven weeks of "ALL GREEN" while the real accuracy falls from 0.94 to 0.70, and we only learn those accuracy numbers when the repayment answers arrive, months later. Server health is not model health: a server can answer quickly and correctly (in the HTTP sense) with wrong predictions.
Real-life example: running a dhaba
| In a dhaba | In ML |
|---|---|
| The gas is on and orders go out fast, but that says nothing about taste | System health: status, latency, errors |
| Customer reviews arrive next week, too late for today's dal | Delayed labels: the true answers come late |
| Today the customers are college students, not office staff: visible at the door right now | Data drift: watch today's inputs |
Four things to watch
- System health: status codes, latency, errors, memory.
- Data (input) drift: do today's inputs look like the training inputs?
- Prediction drift: has the share of "risky" answers changed? (Available today, no labels needed.)
- Performance: accuracy, precision, recall. What we care about most, but it needs labels.
- Delayed labels: the true answer arrives weeks or months later. Whether a loan is repaid is known only at the end of the loan; whether a delivery was late is known in an hour.
- Performance decay: a model slowly getting worse over time because the world it learned from has changed.
Three kinds of drift
- Data drift: the umbrella word: live data stops looking like the training data.
- Covariate shift: the inputs X change, the rule from X to y stays. Example: suddenly many more young people apply, but for any one person the chance of repaying is the same. (Covariate is another word for an input feature.)
- Label shift (prior shift): the share of each answer changes. Example: risky applicants go from 10% to 30% of all applicants.
- Concept drift: the rule itself changes: the very same person now has a different true answer. Example: interest rates rise, so the same 8-lakh loan is now riskier. The inputs look normal, which makes it the hardest to spot.
In symbols, with X = inputs and y = answer: covariate shift changes P(X), label shift changes P(y), concept drift changes P(y | X).
Measuring drift: why the mean is not enough
import numpy as np
rng = np.random.default_rng(82)
train_income = rng.normal(60, 15, 5000) # thousand rupees a month, at training time
live_income = rng.normal(60, 35, 5000) # this month: same average, far more spread
print(f"mean train {train_income.mean():.1f} live {live_income.mean():.1f}")
gap = abs(train_income.mean() - live_income.mean())
print("mean check says:", "DRIFT" if gap > 3 else "no drift")
print(f"lowest train {train_income.min():.0f} live {live_income.min():.0f}")
mean train 60.3 live 60.0
mean check says: no drift
lowest train 7 live -83
The averages match (60.3 vs 60.0), so "no drift". But the lowest live value is −83: the live incomes are spread far wider, and some are impossible (negative income, a data error upstream). One number cannot describe a whole shape.
The KS test
KS stands for Kolmogorov-Smirnov. For each sample, draw the curve "what share of people are below x" (the empirical cumulative distribution function). If nothing changed, the two curves lie on top of each other. The KS statistic is the biggest vertical gap between them: 0 means identical, 1 means completely different. scipy.stats.ks_2samp also returns a p-value: the chance of a gap this big if both samples came from the same distribution.
PSI: the Population Stability Index
Cut the training data into 10 buckets with 10% of people each (using quantiles). For each bucket compare the live share with the training share:
PSI = Σ over buckets (live% − train%) × ln(live% ÷ train%)
Each term is never negative (when live > train both factors are positive; when live < train both are negative), so PSI only grows with drift. Rule of thumb from credit scoring: below 0.1 no real change, 0.1 to 0.25 a small change (watch), 0.25 or more a big change (act).
import numpy as np
from scipy.stats import ks_2samp
rng = np.random.default_rng(82)
train_income = rng.normal(60, 15, 5000) # thousand rupees a month, at training time
live_income = rng.normal(60, 35, 5000) # this month: same average, far more spread
res = ks_2samp(train_income, live_income) # biggest gap between the two "% below" curves
print(f"KS statistic {res.statistic:.3f} p-value {res.pvalue:.2e}")
def psi(expected, actual, bins=10):
edges = np.quantile(expected, np.linspace(0, 1, bins + 1)) # 10 equal-count buckets
edges[0], edges[-1] = -np.inf, np.inf
e = np.histogram(expected, edges)[0] / len(expected) # share per bucket, train
a = np.histogram(actual, edges)[0] / len(actual) # share per bucket, live
print("train share:", np.round(e, 2))
print("live share: ", np.round(a, 2))
return np.sum((a - e) * np.log(a / e))
print(f"PSI {psi(train_income, live_income):.3f}")
KS statistic 0.196 p-value 3.31e-84
train share: [0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1]
live share: [0.29 0.07 0.04 0.05 0.05 0.04 0.04 0.06 0.07 0.29]
PSI 0.684
KS = 0.196 with p ≈ 3 × 10⁻⁸⁴: real drift. PSI = 0.684, far above 0.25, and the bucket shares show where: 29% of live incomes fall in each end bucket instead of 10%.
Mistake: an empty bucket
import numpy as np
rng = np.random.default_rng(82)
train_income = rng.normal(60, 15, 5000)
live_income = rng.normal(70, 15, 5000).clip(45) # new rule: only incomes above 45 may apply
def psi(expected, actual, bins=10):
edges = np.quantile(expected, np.linspace(0, 1, bins + 1))
edges[0], edges[-1] = -np.inf, np.inf
e = np.histogram(expected, edges)[0] / len(expected)
a = np.histogram(actual, edges)[0] / len(actual)
print("live share:", np.round(a, 2))
return np.sum((a - e) * np.log(a / e))
print(f"PSI {psi(train_income, live_income):.3f}")
live share: [0. 0.07 0.05 0.07 0.08 0.09 0.1 0.13 0.16 0.26]
PSI inf
m2b.py:11: RuntimeWarning: divide by zero encountered in log
return np.sum((a - e) * np.log(a / e))
With the new "income above 45" rule, the lowest bucket is empty: live share 0. ln(0 / 0.1) is −∞, so one bucket makes PSI infinite. The fix is a tiny floor for empty buckets:
import numpy as np
rng = np.random.default_rng(82)
train_income = rng.normal(60, 15, 5000)
live_income = rng.normal(70, 15, 5000).clip(45) # new rule: only incomes above 45 may apply
def psi(expected, actual, bins=10):
edges = np.quantile(expected, np.linspace(0, 1, bins + 1))
edges[0], edges[-1] = -np.inf, np.inf
e = np.histogram(expected, edges)[0] / len(expected)
a = np.histogram(actual, edges)[0] / len(actual)
a = np.clip(a, 1e-4, None) # fix: an empty bucket counts as 0.01%
print("live share:", np.round(a, 2))
return np.sum((a - e) * np.log(a / e))
print(f"PSI {psi(train_income, live_income):.3f}")
live share: [0. 0.07 0.05 0.07 0.08 0.09 0.1 0.13 0.16 0.26]
PSI 0.954
Does input drift mean the model is wrong?
import numpy as np
from scipy.stats import ks_2samp
from sklearn.linear_model import LogisticRegression
rng = np.random.default_rng(7)
def loans(n, mean_income=60, cut=20): # made-up: income, loan size
X = np.c_[rng.normal(mean_income, 15, n), rng.normal(8, 3, n)]
y = (X[:, 1] * 6 - X[:, 0] * 0.4 + rng.normal(0, 4, n) > cut).astype(int)
return X, y
Xtr, ytr = loans(5000)
model = LogisticRegression().fit(Xtr, ytr)
worlds = {"no change": loans(3000),
"covariate shift": loans(3000, mean_income=40), # inputs move, rule same
"concept drift": loans(3000, cut=5)} # inputs same, rule changes
print(f"{'world':16} {'income KS p':>11} {'default rate':>12} {'accuracy':>8}")
for name, (X, y) in worlds.items():
p = ks_2samp(Xtr[:, 0], X[:, 0]).pvalue
print(f"{name:16} {p:11.3f} {y.mean():12.2f} {model.score(X, y):8.2f}")
print(f"{'training':16} {'':>11} {ytr.mean():12.2f}")
world income KS p default rate accuracy
no change 0.558 0.58 0.94
covariate shift 0.000 0.74 0.95
concept drift 0.977 0.83 0.77
training 0.59
- Covariate shift: KS fires (p = 0.000), the risky share rose from 0.59 to 0.74 (label shift too), yet accuracy is 0.95: the rule didn't change, so the model still applies.
- Concept drift: KS is calm (p = 0.977) but accuracy fell to 0.77. Input checks cannot see it; only labels can.
So: input/prediction checks give early warnings; label-based performance gives the final verdict. You need both.
Alerts and retraining triggers
- Alert: an automatic message to a person (chat, SMS, phone call) when a number crosses a line, e.g. PSI > 0.25.
- Two levels: "watch" at 0.1, "ALERT" at 0.25.
- Retraining trigger: a rule that starts the training pipeline (Episode 81), e.g. "accuracy below 0.85 once labels arrive".
- Alert fatigue: so many alarms that people ignore them all, including the real one. Fewer, meaningful alerts beat many noisy ones.
import numpy as np
from sklearn.linear_model import LogisticRegression
rng = np.random.default_rng(4)
def loans(n, week): # made-up: incomes fall and the risk rule shifts
X = np.c_[rng.normal(60 - 2 * week, 15, n), rng.normal(8, 3, n)]
return X, (X[:, 1] * 6 - X[:, 0] * 0.4 + rng.normal(0, 4, n) > 20 - 3 * week).astype(int)
def psi(e, a):
edges = np.quantile(e, np.linspace(0, 1, 11)); edges[[0, -1]] = -np.inf, np.inf
pe, pa = [np.clip(np.histogram(v, edges)[0] / len(v), 1e-4, None) for v in (e, a)]
return np.sum((pa - pe) * np.log(pa / pe))
Xtr, ytr = loans(5000, 0); model = LogisticRegression().fit(Xtr, ytr)
LAG, weeks = 3, {} # repayment answers arrive 3 weeks late
for w in range(1, 9):
weeks[w] = loans(1000, w); d = psi(Xtr[:, 0], weeks[w][0][:, 0]) # input drift: today
flag = "ALERT" if d > 0.25 else "watch" if d > 0.1 else "ok"
acc = model.score(*weeks[w - LAG]) if w > LAG else None # performance: late
act = "RETRAIN" if acc is not None and acc < 0.85 else ""
late = "labels not in yet" if acc is None else f"week {w - LAG} accuracy {acc:.2f} {act}"
print(f"week {w}: income PSI {d:.2f} {flag:5} | {late}")
week 1: income PSI 0.03 ok | labels not in yet
week 2: income PSI 0.06 ok | labels not in yet
week 3: income PSI 0.16 watch | labels not in yet
week 4: income PSI 0.30 ALERT | week 1 accuracy 0.92
week 5: income PSI 0.46 ALERT | week 2 accuracy 0.88
week 6: income PSI 0.64 ALERT | week 3 accuracy 0.85
week 7: income PSI 0.85 ALERT | week 4 accuracy 0.79 RETRAIN
week 8: income PSI 1.11 ALERT | week 5 accuracy 0.77 RETRAIN
PSI moves from "ok" to "watch" in week 3 and "ALERT" in week 4, weeks before the labels confirm the damage (0.92 → 0.79). In week 7 the retraining trigger fires.
Releasing the new model safely
| In a restaurant | In ML |
|---|---|
| The new cook makes every dish too, but his plates are thrown away; the head cook just tastes them | Shadow deployment |
| His dish goes to one table first, then two, then more | Canary release |
| Half the tables (chosen at random) get the old menu, half the new; count which sells more | A/B testing |
- Shadow deployment: the new model receives a copy of every live request; its answers are logged and compared but never shown to customers. Zero customer risk, real data.
- Canary release: send the new version to a small slice of users first (say 5%), watch errors and drift, then widen. Martin Fowler: "a technique to reduce the risk of introducing a new software version in production". Named after the canary miners carried as an early gas warning.
- A/B testing: split users at random into A (old model) and B (new model) and compare a business metric (offers accepted, loans repaid, revenue) with a statistical test.
- Champion / challenger: the current production model is the champion; the new one trying to replace it is the challenger.
A/B after one hour (mistake)
import numpy as np
from scipy.stats import chi2_contingency
rng = np.random.default_rng(5)
users = 400 # after one hour
group = rng.choice(["A", "B"], users) # random split: A = old model, B = new model
rate = np.where(group == "A", 0.300, 0.315) # made-up true rates of offers taken
took = rng.random(users) < rate
for g in "AB":
print(f"model {g}: {np.sum(group == g)} users, offer taken {took[group == g].mean():.3f}")
table = [[np.sum(took & (group == g)), np.sum(~took & (group == g))] for g in "AB"]
chi2, p, dof, _ = chi2_contingency(table)
print(f"chi-square p-value {p:.3f} ->", "B is really better" if p < 0.05 else "could be luck")
model A: 213 users, offer taken 0.343
model B: 187 users, offer taken 0.316
chi-square p-value 0.638 -> could be luck
With 400 users, A even looks better, but p = 0.638: the difference is easily luck. Stopping now would cancel a better model.
A/B with enough users
import numpy as np
from scipy.stats import chi2_contingency
rng = np.random.default_rng(5)
users = 20000
group = rng.choice(["A", "B"], users) # random split: A = old model, B = new model
rate = np.where(group == "A", 0.300, 0.315) # made-up true rates of offers taken
took = rng.random(users) < rate
for g in "AB":
print(f"model {g}: {np.sum(group == g)} users, offer taken {took[group == g].mean():.3f}")
table = [[np.sum(took & (group == g)), np.sum(~took & (group == g))] for g in "AB"]
chi2, p, dof, _ = chi2_contingency(table)
print(f"chi-square p-value {p:.3f} ->", "B is really better" if p < 0.05 else "could be luck")
model A: 9992 users, offer taken 0.303
model B: 10008 users, offer taken 0.319
chi-square p-value 0.014 -> B is really better
With 20,000 users, B wins (0.319 vs 0.303) with p = 0.014 < 0.05. Decide the sample size before the test and don't stop early just because a dashboard looks exciting.
Common mistakes
| Mistake | What you see | Why | Fix |
|---|---|---|---|
| Watching only the server | ALL GREEN while accuracy 0.94 → 0.70 |
server health ≠ model health | watch inputs, predictions, accuracy |
| Comparing only the mean | "no drift" with twice the spread | one number hides the shape | KS test or PSI |
| PSI with an empty bucket | RuntimeWarning: divide by zero encountered in log, PSI inf |
ln(0) = −∞ | np.clip(share, 1e-4, None) |
| Trusting input checks alone | KS p = 0.977 but accuracy 0.77 | concept drift lives in the labels | track accuracy when labels arrive |
| An alarm on every wiggle | alert fatigue | lines set too tight | "watch" and "alert" levels |
| Ending an A/B test early | p = 0.638 after 400 users | too few users, luck dominates | fix the sample size first |
Mini project: a monitoring report
Extend the weekly job: (1) PSI for every feature, (2) PSI of the model's predicted risk (prediction drift), (3) accuracy, precision and recall when labels arrive, (4) one line per week saying ok / watch / ALERT / RETRAIN, and (5) only send a message when the status changes. Run it for 12 weeks with a concept drift starting in week 6 and check which signal fires first.
Practice
1. Write the PSI for two buckets: train 50% / 50%, live 70% / 30%.
Solution
(0.7 − 0.5)·ln(0.7/0.5) + (0.3 − 0.5)·ln(0.3/0.5) = 0.2·0.336 + (−0.2)·(−0.511) = 0.067 + 0.102 = 0.169: a small change ("watch").
2. A KS test on 2 million rows gives p = 10⁻¹² but the KS statistic is 0.004. Should you panic?
Solution
No. With huge samples even tiny, harmless differences become "significant". Look at the size of the change (KS statistic, PSI) not just the p-value, and confirm with performance when labels arrive.
3. Add prediction drift to the weekly job.
Solution
Store ref = model.predict_proba(Xtr)[:, 1] once, then each week compute psi(ref, model.predict_proba(X)[:, 1]) with the same psi function and the same watch/alert levels.
4. Which release lets you test a new model on real traffic with zero customer impact?
Solution
Shadow deployment: the challenger gets a copy of each request but its answers are only logged.
Quick quiz
A food-delivery time model: inputs look unchanged (KS p = 0.8), but after the monsoon arrives the same distance at the same time takes 40% longer. Which drift? Concept drift: same inputs, a new true answer. Only the real delivery times (labels) reveal it.
Interview questions
- What is the difference between data drift and concept drift? Give an example of each.
- Labels arrive 90 days late. How do you monitor the model in the meantime?
- How is PSI computed, and what do 0.1 and 0.25 mean?
- When would a KS test give a misleading alarm?
- Shadow deployment vs canary release vs A/B testing: when do you use each?
- What is alert fatigue and how do you avoid it?
- What should trigger retraining, and why shouldn't you auto-ship every retrained model?
Glossary
| Term | Meaning, with an example |
|---|---|
| Model monitoring | Checking a live model all the time so problems are noticed early. |
| System health | Server numbers: status code 200, latency 14 ms, error count. |
| Status code 200 | The web server's code for "request worked". It says nothing about whether the prediction was right. |
| Latency | How long one prediction takes, e.g. 14 milliseconds. |
| Data drift (input drift) | Live inputs stop looking like the training inputs: incomes spread twice as wide. |
| Covariate shift | Inputs X change, the rule X → y stays: more young applicants, same repayment behaviour per person. |
| Label shift (prior shift) | The share of each answer changes: risky applicants 10% → 30%. |
| Concept drift | The rule X → y changes: same loan, same income, now riskier because rates rose. |
| Prediction drift | The model's outputs change: it suddenly calls far more people risky. |
| Performance decay | A model getting slowly worse over time as the world changes. |
| Delayed labels | True answers that arrive late: loan repayment known months later. |
| Distribution | How a feature's values are spread: centre, width, shape. |
| KS test (Kolmogorov-Smirnov) | Compares two samples' "share below x" curves; statistic = biggest gap (0.196 for our incomes). |
| p-value | Chance of a difference this big if nothing changed; 3.31e-84 means "not luck". |
| PSI (Population Stability Index) | Σ (live% − train%)·ln(live%/train%) over buckets; 0.684 for our incomes. |
| Bucket (bin) | A range of values; here 10 ranges with 10% of training people each. |
| Alert | An automatic message to a person when a number crosses a line (PSI > 0.25). |
| Alert fatigue | Too many alarms, so people ignore them all. |
| Retraining trigger | A rule that starts retraining: accuracy < 0.85 once labels arrive. |
| Shadow deployment | New model gets copies of live requests; answers logged, never used. |
| Canary release | New version to a small slice of users first (5%), then more. |
| A/B testing | Random split of users into old (A) and new (B), compare a business metric with a test. |
| Champion / challenger | The current production model / the new model trying to replace it. |
| Chi-square test | Tests whether group (A/B) and outcome (took offer or not) are related; p = 0.014 here. |
Summary
- A green server dashboard can hide a failing model: watch system health, input drift, prediction drift and performance.
- Labels arrive late, so inputs and predictions give the early warning; labels give the verdict.
- Covariate shift (X changes), label shift (y shares change), concept drift (the rule changes, invisible to input checks).
- Measure whole distributions with the KS test and PSI (floor empty buckets!); 0.1 = watch, 0.25 = act.
- Turn numbers into alerts and retraining triggers, without alert fatigue.
- Release new models with shadow, canary and A/B tests, and never stop an A/B test early.
Next episode: Explainability: interpretable models, global vs local explanations, PDP and ICE, Shapley values, SHAP, LIME and counterfactuals.