SyntaxPilot
Courses/Machine Learning/Lesson 25

Machine Learning · Lesson 25 of 90 · 15 min

Classification metrics 2

threshold tuningROC curveAUC (ROC-AUC)precision-recall curveaverage precision (PR-AUC)probability calibrationcalibration curve (reliability diagram)Brier scorePlatt scalingisotonic regressionchoosing a metric from business cost

Machine Learning Full Course · Episode 25. Every code sample below was run with Python 3.13, NumPy 2.5.3 and scikit-learn 1.9.1; the output shown is the real output. All data (card payments) and all costs are made up for teaching.

The big idea: last episode every metric depended on one threshold (the cut-off above which a probability becomes "yes"). A model's scores can be judged at every threshold at once: the ROC curve and its AUC measure how well a model ranks positives above negatives; the precision-recall curve and average precision do the same when positives are rare. Separately, a model's probabilities may not mean what they say: calibration checks that "30%" really happens 30% of the time (the Brier score measures it, Platt scaling and isotonic regression fix it). Finally, with calibrated probabilities, the costs of each mistake give the best threshold directly: C_FP / (C_FP + C_FN).

Words you need first

  • Classifier: a model that predicts a class (fraud / genuine). Most classifiers first produce a score or probability (predict_proba), then compare it with a threshold.
  • Threshold: the cut-off. predict() in scikit-learn uses 0.5 for binary classifiers: probability ≥ 0.5 → class 1.
  • Positive class: the class we hunt for, coded 1 (fraud). Negative class: the other one, coded 0 (genuine).
  • TP, FP, FN, TN: true positive, false positive (false alarm), false negative (miss), true negative (Episode 24).
  • Recall (true positive rate): TP / (TP + FN), the share of real positives we caught. Precision: TP / (TP + FP), the share of our flags that were right.

The problem: two analysts, one argument

Same data as Episode 24: 20,000 made-up card payments, 3 features, 1.8% fraud, split half for training and half for testing (stratified). Analyst A trains a plain LogisticRegression. Analyst B uses class_weight="balanced" (Episode 23: errors on the rare class count more). At threshold 0.5, B catches far more fraud and claims "my model is five times better".

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
rng = np.random.default_rng(24); n = 20000           # made-up card payments, 1 = fraud
X = rng.normal(0, 1, (n, 3))
y = (rng.random(n) < 1 / (1 + np.exp(6 - 1.8 * X[:, 0] - 1.2 * X[:, 1]))).astype(int)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.5, stratify=y, random_state=0)
from sklearn.metrics import recall_score, precision_score, roc_auc_score
A = LogisticRegression().fit(X_tr, y_tr)
B = LogisticRegression(class_weight="balanced").fit(X_tr, y_tr)
for name, m in [("A plain", A), ("B balanced", B)]:
    pred = m.predict(X_te)                           # default threshold 0.5
    print(f"{name:<10}  at 0.5: recall {recall_score(y_te, pred):.3f}"
          f"  precision {precision_score(y_te, pred):.3f}")
for name, m in [("A plain", A), ("B balanced", B)]:
    print(f"{name:<10}  ROC-AUC {roc_auc_score(y_te, m.predict_proba(X_te)[:, 1]):.3f}")

Output:

A plain     at 0.5: recall 0.158  precision 0.690
B balanced  at 0.5: recall 0.864  precision 0.095
A plain     ROC-AUC 0.930
B balanced  ROC-AUC 0.930

B catches more fraud and flags far more innocent customers. Their ROC-AUC (explained below) is identical. Both models sort the payments in almost the same order (the rank correlation of their scores is 0.998, see Practice 1); class weights only pushed all of B's probabilities upwards (average 0.229 vs 0.019), so at a fixed cut of 0.5 B flags more. Comparing models at one threshold compares their thresholds, not their skill.

Threshold tuning = choosing the cut-off on purpose instead of accepting 0.5. Model A at a lower threshold behaves just like B.

Threshold tuning and the ROC curve

Real-life example: an airport metal detector has a sensitivity knob (the threshold). Turn it up and it catches more real knives (true positive rate) but also beeps for more belt buckles (false positive rate). Try every knob setting and plot hits against false alarms: that plot is the ROC curve.

  • True positive rate (TPR) = TP / (TP + FN) = recall = sensitivity.
  • False positive rate (FPR) = FP / (FP + TN), the share of real negatives wrongly flagged (= 1 − specificity).
  • ROC curve = TPR (up) against FPR (across), one point per threshold. ROC stands for receiver operating characteristic, a name from early radar work.
# (first 7 lines: the same data set-up as above)
from sklearn.metrics import roc_curve, roc_auc_score
p = LogisticRegression().fit(X_tr, y_tr).predict_proba(X_te)[:, 1]
for t in [0.9, 0.5, 0.1, 0.03, 0.01]:
    pred = p >= t
    tpr = pred[y_te == 1].mean(); fpr = pred[y_te == 0].mean()
    print(f"threshold {t:<4}  TPR (recall) {tpr:.3f}  FPR {fpr:.3f}")
fpr, tpr, th = roc_curve(y_te, p)
print("points on the ROC curve:", len(th))
print(f"ROC-AUC {roc_auc_score(y_te, p):.3f}")
pos, neg = p[y_te == 1], p[y_te == 0]
print(f"P(random fraud scores above random genuine) {(pos[:, None] > neg).mean():.3f}")

Output:

threshold 0.9   TPR (recall) 0.011  FPR 0.000
threshold 0.5   TPR (recall) 0.158  FPR 0.001
threshold 0.1   TPR (recall) 0.630  FPR 0.033
threshold 0.03  TPR (recall) 0.810  FPR 0.108
threshold 0.01  TPR (recall) 0.908  FPR 0.236
points on the ROC curve: 280
ROC-AUC 0.930
P(random fraud scores above random genuine) 0.930
  • Lowering the threshold raises both rates: more fraud caught, more genuine customers flagged.
  • roc_curve returns every threshold where the curve changes (280 here) with its FPR and TPR.
  • (pos[:, None] > neg) uses broadcasting (Episode 2) to compare every fraud score with every genuine score (184 × 9,816 pairs).

AUC: area under the curve

AUC (here ROC-AUC) is the area under the ROC curve, between 0 and 1.

  • 1.0 = perfect ranking: every positive scored above every negative.
  • 0.5 = random guessing (the diagonal line).
  • Below 0.5 = worse than guessing (often a sign the labels or the score direction are flipped).
  • Meaning: pick one random positive and one random negative; AUC is the probability the model gives the positive the higher score. Our last line computed exactly that: 0.930.
  • AUC ignores the threshold completely. It measures ranking quality, which is why models A and B tie.
  • It needs scores (predict_proba(X)[:, 1] or decision_function), not 0/1 predictions: with labels there is only one threshold, so only one point.

ROC and precision-recall curves

Left: ROC curves of the good model (3 features, AUC 0.930) and a weak one (1 feature, AUC 0.772). Right: their precision-recall curves (next section).

Plotting code:

# (first 7 lines: the same data set-up as above)
import matplotlib.pyplot as plt
from sklearn.metrics import roc_curve, precision_recall_curve, roc_auc_score, average_precision_score
good = LogisticRegression().fit(X_tr, y_tr).predict_proba(X_te)[:, 1]
weak = LogisticRegression().fit(X_tr[:, [1]], y_tr).predict_proba(X_te[:, [1]])[:, 1]
fig, ax = plt.subplots(1, 2, figsize=(12, 5))
for name, p in [("good, 3 features", good), ("weak, 1 feature", weak)]:
    f, t, _ = roc_curve(y_te, p); ax[0].plot(f, t, lw=2.5, label=f"{name}: AUC {roc_auc_score(y_te, p):.3f}")
    pr, rc, _ = precision_recall_curve(y_te, p); ax[1].plot(rc, pr, lw=2.5, label=f"{name}: AP {average_precision_score(y_te, p):.3f}")
ax[0].plot([0, 1], [0, 1], "k--", label="random guessing: AUC 0.5")
ax[0].set_xlabel("false positive rate"); ax[0].set_ylabel("true positive rate (recall)"); ax[0].legend()
ax[1].axhline(y_te.mean(), c="k", ls="--", label=f"random guessing: AP {y_te.mean():.3f}")
ax[1].set_xlabel("recall"); ax[1].set_ylabel("precision"); ax[1].legend()
plt.tight_layout(); plt.savefig("roc_pr.png", dpi=100)

The precision-recall curve

AUC 0.930 sounds excellent, but every flagged payment means a call to the customer. The call centre's question is about precision, which ROC never shows.

# (first 7 lines: the same data set-up as above)
from sklearn.metrics import roc_auc_score, average_precision_score, precision_recall_curve
good = LogisticRegression().fit(X_tr, y_tr).predict_proba(X_te)[:, 1]
weak = LogisticRegression().fit(X_tr[:, [1]], y_tr).predict_proba(X_te[:, [1]])[:, 1]
for name, p in [("good (3 features)", good), ("weak (1 feature) ", weak)]:
    print(f"{name}  ROC-AUC {roc_auc_score(y_te, p):.3f}  AP {average_precision_score(y_te, p):.3f}")
prec, rec, th = precision_recall_curve(y_te, good)
i = np.argmin(np.abs(rec - 0.8))
print(f"good model at recall {rec[i]:.2f}: precision {prec[i]:.3f}")
print(f"random guessing: AP = fraud rate = {y_te.mean():.3f}")

Output:

good (3 features)  ROC-AUC 0.930  AP 0.403
weak (1 feature)   ROC-AUC 0.772  AP 0.073
good model at recall 0.80: precision 0.127
random guessing: AP = fraud rate = 0.018
  • At 80% recall the precision is only 0.127: about 7 of every 8 calls go to an innocent customer.
  • Why ROC hid it: FPR divides false alarms by all 9,816 genuine payments. About a thousand false alarms is only ~10% FPR, so the ROC curve barely moves; precision compares them with the ~150 frauds caught.
  • Precision-recall (PR) curve: precision (up) against recall (across), one point per threshold.
  • Average precision (AP), also called PR-AUC: summarises the PR curve as the average precision over all recall levels (scikit-learn computes Σ (Rₙ − Rₙ₋₁)·Pₙ, a step-wise area).
  • Baseline: random guessing gets precision equal to the share of positives, so its AP ≈ 0.018 here, not 0.5. An AP of 0.403 is over 20 times better than chance.
  • Rule: when positives are rare, report the PR curve and AP alongside (or instead of) ROC-AUC.

Calibration: can the probabilities be trusted?

Real-life example: a weather forecaster says "30% chance of rain" on many days. If it rained on about 30% of those days, her numbers are calibrated. A forecaster who says 80% when it rains one day in ten is miscalibrated: she may still rank days well (good AUC) while her numbers are wrong.

  • Calibration: among all cases given probability p, about a fraction p are positive.
  • Calibration curve (reliability diagram): group cases by predicted probability, then plot the average prediction (across) against the real positive rate (up). A perfect model lies on the diagonal.
# (first 7 lines: the same data set-up as above)
from sklearn.calibration import calibration_curve, CalibratedClassifierCV
from sklearn.metrics import brier_score_loss
B = LogisticRegression(class_weight="balanced")
p = B.fit(X_tr, y_tr).predict_proba(X_te)[:, 1]
print(f"average predicted fraud chance {p.mean():.3f}   real fraud rate {y_te.mean():.3f}")
frac, mean_p = calibration_curve(y_te, p, n_bins=5, strategy="quantile")
print("model said:", mean_p.round(3)); print("reality:   ", frac.round(3))
print(f"{'uncalibrated':<15}  average {p.mean():.3f}  Brier {brier_score_loss(y_te, p):.4f}")
for name, method in [("Platt (sigmoid)", "sigmoid"), ("isotonic", "isotonic")]:
    q = CalibratedClassifierCV(B, method=method, cv=5).fit(X_tr, y_tr).predict_proba(X_te)[:, 1]
    print(f"{name:<15}  average {q.mean():.3f}  Brier {brier_score_loss(y_te, q):.4f}")

Output:

average predicted fraud chance 0.229   real fraud rate 0.018
model said: [0.009 0.042 0.119 0.291 0.685]
reality:    [0.    0.002 0.003 0.006 0.082]
uncalibrated     average 0.229  Brier 0.1098
Platt (sigmoid)  average 0.019  Brier 0.0137
isotonic         average 0.018  Brier 0.0137
  • The balanced model says 22.9% fraud on average; reality is 1.8%. Where it said 68.5%, only 8.2% were fraud. Class weights bought recall by inflating every probability.
  • calibration_curve(..., n_bins=5, strategy="quantile") makes 5 groups with equal numbers of payments.

Calibration curve

Blue: the balanced model sags far below the diagonal (over-confident). Orange: after Platt scaling it follows the diagonal.

The Brier score

Brier score = mean of (p − y)², where p is the predicted probability and y the true 0/1 label. Lower is better; 0 is perfect.

Forecast p What happened y (p − y)²
0.9 1 (it rained) 0.01
0.9 0 (dry) 0.81
0.1 0 (dry) 0.01
0.5 either 0.25

Confident and wrong is punished hardest. Brier mixes calibration and ranking, so compare it between versions of the same model, as above (0.1098 → 0.0137).

Fixing calibration: Platt scaling and isotonic regression

  • Platt scaling (method="sigmoid"): take the model's scores on data it was not trained on and fit a small logistic regression, an S-shaped sigmoid, that maps each score to a corrected probability. Two parameters; works with little data.
  • Isotonic regression (method="isotonic"): fits a free-form, never-decreasing staircase from score to probability. More flexible, needs more data (roughly a thousand or more calibration examples) or it overfits.
  • CalibratedClassifierCV(model, method=..., cv=5): splits the training data into 5 parts; 5 times, it trains the model on 4 parts and fits the calibrator on the 5th, then averages. It never calibrates on rows the model was trained on.
  • Calibration keeps the ranking (almost) unchanged, so AUC barely moves; it fixes the numbers.
  • When it matters: whenever the probability itself is used: expected-cost decisions, showing "risk 12%" to a user, combining models, or comparing against a threshold computed from costs. For ranking only, it doesn't.

Choosing a metric (and threshold) from business cost

Made-up costs: a missed fraud costs the bank about ₹5,000 in refunds; a false alarm costs about ₹100 (an agent's check-up call).

# (first 7 lines: the same data set-up as above)
from sklearn.calibration import CalibratedClassifierCV
from sklearn.metrics import confusion_matrix
B = LogisticRegression(class_weight="balanced")
p = CalibratedClassifierCV(B, method="sigmoid", cv=5).fit(X_tr, y_tr).predict_proba(X_te)[:, 1]
COST_FN, COST_FP = 5000, 100        # missed fraud: Rs 5000 lost; false alarm: Rs 100 call
def cost(t):
    tn, fp, fn, tp = confusion_matrix(y_te, p >= t).ravel(); return COST_FN * fn + COST_FP * fp, fn, fp
for t in [0.5, 0.2, 0.05]:
    c, fn, fp = cost(t); print(f"threshold {t:<5} misses {fn:>3}  alarms {fp:>4}  cost Rs {c:>7,}")
ts = np.arange(0.005, 0.5, 0.005); best = ts[np.argmin([cost(t)[0] for t in ts])]
c, fn, fp = cost(best); print(f"best {best:.3f}      misses {fn:>3}  alarms {fp:>4}  cost Rs {c:>7,}")
print(f"theory: C_FP / (C_FP + C_FN) = {COST_FP / (COST_FP + COST_FN):.3f}")

Output:

threshold 0.5   misses 160  alarms   11  cost Rs 801,100
threshold 0.2   misses 103  alarms  101  cost Rs 525,100
threshold 0.05  misses  47  alarms  669  cost Rs 301,900
best 0.020      misses  25  alarms 1505  cost Rs 275,500
theory: C_FP / (C_FP + C_FN) = 0.020
  • Total cost = C_FN · FN + C_FP · FP.
  • The default 0.5 costs ₹8,01,100; the best threshold, 0.020, costs ₹2,75,500, about a third.
  • Why the formula works: for one payment with calibrated fraud chance p, flagging costs C_FP for sure, not flagging costs C_FN · p on average. Flag when C_FN · p > C_FP, i.e. when p > C_FP / (C_FP + C_FN) = 100 / 5,100 ≈ 0.020. This only holds if p is honest, so calibrate first.
  • A shortcut we took: the search above ran on the test set to keep the video short. That leaks test information. In real work, search on a validation set (or with cross-validation, next episode) and check the test set once (Practice 4 and the mini project do it properly).

Which tool when?

Question Use
Which model ranks better, classes roughly balanced? ROC curve, ROC-AUC
Which model ranks better, positives rare? PR curve, average precision (and ROC-AUC as a second view)
Can I trust "p = 0.3" as a risk? Calibration curve, Brier score
Probabilities over- or under-confident? CalibratedClassifierCV, sigmoid (small data) or isotonic (large data)
Which threshold to deploy? From costs: C_FP / (C_FP + C_FN) with calibrated p, or a cost search on validation data
One number for a report? The metric that matches the cost: recall at a fixed precision, cost in ₹, F-beta (Episode 24)

Common mistakes

Mistake What you see Why Fix
Comparing models at threshold 0.5 recall 0.158 vs 0.864, yet ROC-AUC 0.930 for both Different thresholds, same ranking Compare ROC-AUC / AP, then tune each threshold
AUC from predict() labels A "curve" with one corner point; AUC 0.578 instead of 0.930 Needs scores to rank roc_auc_score(y, model.predict_proba(X)[:, 1])
Trusting ROC on rare positives AUC 0.93 but precision 0.127 at recall 0.8 FPR divides by a huge number of negatives PR curve, average precision
Reading a class-weighted model's probability as risk says 22.9%, reality 1.8% Class weights shift probabilities Calibrate; check the calibration curve and Brier
Calibrating on the training rows Calibration looks perfect, fails on new data Model memorised those rows CalibratedClassifierCV(cv=5) or a separate calibration set
Isotonic on tiny data Jagged, overfitted probabilities Too flexible for few points method="sigmoid"
Tuning the threshold on the test set Optimistic cost (₹2,75,500 on test vs ₹2,85,500 when chosen honestly) The test set leaked into a decision Validation set / cross-validation
One class only in y_true UndefinedMetricWarning: Only one class is present in y_true. ROC AUC score is not defined in that case. and the score is nan ROC needs both classes Stratified splits; more data

Mini project: let scikit-learn tune the threshold for your costs

TunedThresholdClassifierCV searches the threshold with cross-validation on the training data, using any scorer, so the test set stays untouched.

# (first 7 lines: the same data set-up as above)
from sklearn.model_selection import TunedThresholdClassifierCV
from sklearn.metrics import make_scorer, confusion_matrix
def neg_cost(y_true, y_pred):
    tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()
    return -(5000 * fn + 100 * fp)                    # higher is better, so negate the cost
tuned = TunedThresholdClassifierCV(LogisticRegression(), scoring=make_scorer(neg_cost), cv=5).fit(X_tr, y_tr)
print(f"tuned threshold {tuned.best_threshold_:.3f}")
tn, fp, fn, tp = confusion_matrix(y_te, tuned.predict(X_te)).ravel()
print(f"test: misses {fn}  false alarms {fp}  cost Rs {5000 * fn + 100 * fp:,}")

Output:

tuned threshold 0.018
test: misses 25  false alarms 1552  cost Rs 280,200

It picked 0.018 from the training data alone, close to the theoretical 0.020, and the honest test cost is ₹2,80,200. Try it: change the costs to ₹2,000 and ₹200 and predict the threshold before you run it (200 / 2,200 ≈ 0.091).

Practice

  1. Are models A and B really ranking the same way? Compare their AUCs to 4 decimals and the rank correlation of their scores.
  2. What average precision and ROC-AUC does a DummyClassifier(strategy="stratified") get? Compare with the fraud rate.
  3. Is model A (no class weights) calibrated? Print its calibration curve and Brier score.
  4. Redo the cost search properly: choose the threshold on a validation set carved out of the training data, then evaluate on the test set once.
Solution 1
# (first 7 lines: the same data set-up as above)
from sklearn.metrics import roc_curve, roc_auc_score
from scipy.stats import spearmanr
pA = LogisticRegression().fit(X_tr, y_tr).predict_proba(X_te)[:, 1]
pB = LogisticRegression(class_weight="balanced").fit(X_tr, y_tr).predict_proba(X_te)[:, 1]
print(f"AUC A {roc_auc_score(y_te, pA):.4f}   AUC B {roc_auc_score(y_te, pB):.4f}")
print(f"rank correlation of the two score lists: {spearmanr(pA, pB).statistic:.4f}")
print(f"average score A {pA.mean():.3f}   B {pB.mean():.3f}")

Output:

AUC A 0.9301   AUC B 0.9297
rank correlation of the two score lists: 0.9982
average score A 0.019   B 0.229

The rankings are almost identical (Spearman rank correlation, Episode 11: 0.998); only the level of the scores differs.

Solution 2
# (first 7 lines: the same data set-up as above)
from sklearn.dummy import DummyClassifier
from sklearn.metrics import average_precision_score, roc_auc_score
d = DummyClassifier(strategy="stratified", random_state=0).fit(X_tr, y_tr)
p = d.predict_proba(X_te)[:, 1]
print(f"dummy: AP {average_precision_score(y_te, p):.3f}  ROC-AUC {roc_auc_score(y_te, p):.3f}  fraud rate {y_te.mean():.3f}")

Output:

dummy: AP 0.018  ROC-AUC 0.502  fraud rate 0.018

Random guessing: ROC-AUC ≈ 0.5, AP ≈ the share of positives. These are the floors to beat.

Solution 3
# (first 7 lines: the same data set-up as above)
from sklearn.calibration import calibration_curve
from sklearn.metrics import brier_score_loss
p = LogisticRegression().fit(X_tr, y_tr).predict_proba(X_te)[:, 1]
frac, mean_p = calibration_curve(y_te, p, n_bins=5, strategy="quantile")
print("model said:", mean_p.round(3)); print("reality:   ", frac.round(3))
print(f"average {p.mean():.3f}  real {y_te.mean():.3f}  Brier {brier_score_loss(y_te, p):.4f}")

Output:

model said: [0.    0.001 0.002 0.008 0.082]
reality:    [0.    0.001 0.004 0.006 0.082]
average 0.019  real 0.018  Brier 0.0137

Plain logistic regression is already well calibrated here (its log-loss training targets honest probabilities); the balanced version is what broke it.

Solution 4
# (first 7 lines: the same data set-up as above)
from sklearn.calibration import CalibratedClassifierCV
from sklearn.metrics import confusion_matrix
X_fit, X_val, y_fit, y_val = train_test_split(X_tr, y_tr, test_size=0.4, stratify=y_tr, random_state=1)
model = CalibratedClassifierCV(LogisticRegression(class_weight="balanced"), method="sigmoid", cv=5).fit(X_fit, y_fit)
def cost(y_true, p, t):
    tn, fp, fn, tp = confusion_matrix(y_true, p >= t).ravel()
    return 5000 * fn + 100 * fp
p_val, p_te = model.predict_proba(X_val)[:, 1], model.predict_proba(X_te)[:, 1]
ts = np.arange(0.005, 0.5, 0.005)
best = ts[np.argmin([cost(y_val, p_val, t) for t in ts])]
print(f"threshold chosen on validation: {best:.3f}")
print(f"test cost at that threshold: Rs {cost(y_te, p_te, best):,}   at 0.5: Rs {cost(y_te, p_te, 0.5):,}")

Output:

threshold chosen on validation: 0.025
test cost at that threshold: Rs 285,500   at 0.5: Rs 825,700

The honest test cost (₹2,85,500) is a little worse than the ₹2,75,500 we got by peeking at the test set: that gap is exactly the optimism a leak creates.

Quick quiz

a calibrated model flags faulty pressure cookers:
missing a faulty one costs   Rs 900
a needless inspection costs  Rs 100
best threshold = ?

A) 0.10 B) 0.50 C) 0.90 D) 0.11

Answer

A) 0.10. Threshold = C_FP / (C_FP + C_FN) = 100 / (100 + 900). D is the trap 100 / 900, which forgets to add C_FP to the denominator.

Interview questions

  1. What does the ROC curve plot? True positive rate against false positive rate for every threshold.
  2. Interpret an AUC of 0.93. A random positive gets a higher score than a random negative 93% of the time; 0.5 is random, 1.0 perfect.
  3. Why can ROC-AUC look great on imbalanced data while the model is useless in practice? FPR divides false alarms by the (huge) number of negatives, so many false alarms still give a small FPR; precision exposes them. Use PR curves and average precision.
  4. What is the baseline average precision? The positive rate (prevalence), e.g. 0.018, not 0.5.
  5. What is calibration and how do you check it? Predicted probabilities match observed frequencies; check with a calibration curve (reliability diagram) and the Brier score.
  6. Platt scaling vs isotonic regression? Platt fits a sigmoid (2 parameters, good for small data, assumes an S-shaped distortion); isotonic fits a monotonic step function (flexible, needs more data).
  7. Does calibration change AUC? Barely: it is a (near) monotonic transform of the scores, so the ranking stays; it changes the numbers.
  8. How do you pick a threshold from costs? With calibrated probabilities, flag when p > C_FP / (C_FP + C_FN); otherwise minimise total cost over thresholds on validation data.
  9. Why not tune the threshold on the test set? It leaks test information into a modelling decision, so the reported score is optimistic.
  10. Why might class_weight="balanced" hurt probability estimates? It reweights the loss, shifting predicted probabilities towards the rare class; ranking is kept, calibration is lost.

Glossary

Term Meaning Example
Threshold Cut-off above which a probability becomes class 1 0.5 by default; 0.020 from costs
Threshold tuning Choosing the threshold on purpose, from validation data or costs 0.5 → 0.02
Score / predicted probability The model's number before thresholding (predict_proba) 0.229 average for model B
True positive rate (TPR) TP / (TP + FN) = recall = sensitivity 0.630 at threshold 0.1
False positive rate (FPR) FP / (FP + TN) = 1 − specificity 0.033 at threshold 0.1
ROC curve TPR vs FPR over all thresholds (receiver operating characteristic) 280 points
AUC / ROC-AUC Area under the ROC curve; P(random positive outranks random negative) 0.930
Ranking Ordering cases from least to most likely positive A and B rank alike (0.998)
Precision-recall (PR) curve Precision vs recall over all thresholds precision 0.127 at recall 0.8
Average precision (AP) / PR-AUC Summary of the PR curve; baseline = positive rate 0.403 vs 0.073 vs 0.018
Prevalence Share of positives in the data 1.8% fraud
Calibration Predicted probabilities match observed frequencies "2%" happens 2% of the time
Miscalibrated / over-confident Probabilities systematically too high (or low) says 22.9%, reality 1.8%
Calibration curve / reliability diagram Average prediction vs real rate per group calibration_curve(y, p, n_bins=5)
Brier score Mean of (p − y)²; lower is better 0.1098 → 0.0137
Platt scaling Calibrate by fitting a sigmoid to held-out scores method="sigmoid"
Isotonic regression Calibrate with a never-decreasing step function method="isotonic"
CalibratedClassifierCV scikit-learn wrapper that calibrates with cross-validation cv=5
Cost-sensitive decision Choosing actions by the cost of each error type ₹5,000 miss vs ₹100 call
Expected cost Average cost of a decision given a probability C_FN · p vs C_FP
Cost-optimal threshold C_FP / (C_FP + C_FN) for calibrated probabilities 100 / 5,100 = 0.020
Validation set Data held out from training to make choices on, not the test set Practice 4
TunedThresholdClassifierCV scikit-learn tool that tunes the threshold by cross-validation mini project: 0.018
Broadcasting NumPy applying an operation across arrays of different shapes (Episode 2) pos[:, None] > neg

Summary

  • 0.5 is not special. Models A and B differed only in where their probabilities sat; at one threshold B "won", but both have ROC-AUC 0.930.
  • The ROC curve plots TPR vs FPR at every threshold; AUC is the chance a random positive outranks a random negative.
  • With rare positives, ROC looks rosy (precision only 0.127 at recall 0.8): use the PR curve and average precision (baseline = prevalence).
  • Calibration checks that probabilities mean what they say; the calibration curve shows it and the Brier score measures it (0.1098 → 0.0137 after Platt scaling or isotonic regression).
  • With calibrated p, costs give the threshold: C_FP / (C_FP + C_FN) = 0.020, cutting cost from ₹8,01,100 to ₹2,75,500. Tune on validation data, never on the test set.

Next episode: cross-validation: k-fold, stratified, group and time-series splits, leave-one-out, cross_validate, repeated and nested cross-validation.