Machine Learning Full Course · Episode 24. Every code sample below was run with Python 3.13, NumPy 2.5.3 and scikit-learn 1.9.1; the output shown is the real output. All data (card payments, café orders) is made up for teaching.
The big idea: accuracy treats every mistake the same, so when one class is rare a useless model can score 98%. Count each kind of right and wrong separately in a confusion matrix (TP, FP, FN, TN). From it come precision (of what we flagged, how much was right), recall (of what was really there, how much we found), specificity and the false positive rate. Combine precision and recall with the F1 score (a harmonic mean) or F-beta when one matters more, and average over many classes with macro, micro or weighted averaging.
The problem: catching card fraud
A bank wants to flag fraud (payments made with a stolen card). We have 20,000 made-up payments with 3 features each; 1 = fraud, 0 = genuine. Fewer than 2 in 100 are fraud: imbalanced data (Episode 23).
Accuracy = correct predictions / all predictions. Someone says "our model is 98% accurate". Let's compare it with the laziest model possible: DummyClassifier(strategy="most_frequent") ignores the features and always predicts the most common class (a baseline, Episode 17).
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
rng = np.random.default_rng(24); n = 20000 # made-up card payments, 1 = fraud
X = rng.normal(0, 1, (n, 3))
y = (rng.random(n) < 1 / (1 + np.exp(6 - 1.8 * X[:, 0] - 1.2 * X[:, 1]))).astype(int)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.5, stratify=y, random_state=0)
from sklearn.dummy import DummyClassifier
print(f"fraud: {y_te.sum()} of {len(y_te)} test payments ({y_te.mean():.1%})")
lazy = DummyClassifier(strategy="most_frequent").fit(X_tr, y_tr) # always says "not fraud"
print(f"lazy model: accuracy {lazy.score(X_te, y_te):.3f} frauds caught {lazy.predict(X_te)[y_te == 1].sum()}")
m = LogisticRegression().fit(X_tr, y_tr)
print(f"real model: accuracy {m.score(X_te, y_te):.3f} frauds caught {m.predict(X_te)[y_te == 1].sum()}")
Output:
fraud: 184 of 10000 test payments (1.8%)
lazy model: accuracy 0.982 frauds caught 0
real model: accuracy 0.983 frauds caught 29
The accuracy paradox
The lazy model scores 98.2% and catches zero frauds. The real logistic regression scores 98.3% and catches 29. Accuracy can't tell them apart.
- With 1.8% fraud, "always genuine" is right 98.2% of the time without learning anything.
- A model with high accuracy and no value is the accuracy paradox. It appears whenever one class is rare.
- Cause: accuracy counts every mistake as equal, but missing a fraud and blocking a genuine payment are very different mistakes.
The confusion matrix
Real-life example: a kitchen smoke alarm. The thing we look for (fire, fraud) is the positive class.
| alarm silent (predicted 0) | alarm rings (predicted 1) | |
|---|---|---|
| no fire (actual 0) | true negative (TN): a normal day | false positive (FP): burnt toast, a false alarm |
| fire (actual 1) | false negative (FN): a missed fire, the dangerous one | true positive (TP): it did its job |
"True/false" says whether the prediction was right; "positive/negative" says what it predicted. This 2×2 table is the confusion matrix: it shows which classes the model confuses. In scikit-learn, rows are the truth and columns are the prediction, and .ravel() returns the cells in the order tn, fp, fn, tp.
Older statistical names:
- Type I error = false positive (crying wolf when there is none).
- Type II error = false negative (missing the wolf).
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
rng = np.random.default_rng(24); n = 20000 # made-up card payments, 1 = fraud
X = rng.normal(0, 1, (n, 3))
y = (rng.random(n) < 1 / (1 + np.exp(6 - 1.8 * X[:, 0] - 1.2 * X[:, 1]))).astype(int)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.5, stratify=y, random_state=0)
from sklearn.metrics import confusion_matrix, precision_score, recall_score
m = LogisticRegression().fit(X_tr, y_tr); pred = m.predict(X_te) # threshold 0.5
print(confusion_matrix(y_te, pred)) # rows = truth, columns = prediction
tn, fp, fn, tp = confusion_matrix(y_te, pred).ravel()
print(f"TP {tp} FP {fp} FN {fn} TN {tn}")
print(f"precision {tp / (tp + fp):.3f} (sklearn {precision_score(y_te, pred):.3f})")
print(f"recall {tp / (tp + fn):.3f} (sklearn {recall_score(y_te, pred):.3f})")
print(f"specificity {tn / (tn + fp):.3f} false positive rate {fp / (fp + tn):.3f}")
Output:
[[9803 13]
[ 155 29]]
TP 29 FP 13 FN 155 TN 9803
precision 0.690 (sklearn 0.690)
recall 0.158 (sklearn 0.158)
specificity 0.999 false positive rate 0.001
At the default threshold 0.5: 9,803 genuine payments correctly let through, 13 false alarms, 155 frauds missed and only 29 caught. The matrix shows at once what accuracy hid.
Precision, recall, specificity and false positive rate
Real-life example: fishing with a net. Precision: of everything in my net, how much is fish (not old shoes)? Recall: of all the fish in the lake, how many did I catch? A bigger net catches more fish and more shoes.
| Metric | Formula | Question it answers | Ours (threshold 0.5) |
|---|---|---|---|
| Precision | TP / (TP + FP) | When the model says fraud, how often is it right? | 0.690 |
| Recall (sensitivity, true positive rate, TPR) | TP / (TP + FN) | Of all frauds, how many did we catch? | 0.158 |
| Specificity (true negative rate) | TN / (TN + FP) | Of all genuine payments, how many did we leave alone? | 0.999 |
| False positive rate (FPR) | FP / (FP + TN) = 1 − specificity | Of all genuine payments, how many did we wrongly flag? | 0.001 |
Our hand formulas match precision_score and recall_score. This model is very cautious: precise and almost never bothers genuine customers, but it catches under 16% of fraud.
Which matters? It depends on the cost of each mistake: a cancer screening cares most about recall (a miss is terrible); a spam filter that deletes mail cares most about precision (a false alarm loses real mail).
The precision-recall trade-off
Lowering the threshold (Episode 22) flags more payments: recall goes up, precision goes down. At threshold 0.1 (next program): precision 0.265, recall 0.630. You usually can't raise both for free.
F1: one number for both
To compare models with one number, the obvious idea is the plain average (P + R) / 2. Test it against a silly model that flags every payment:
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
rng = np.random.default_rng(24); n = 20000 # made-up card payments, 1 = fraud
X = rng.normal(0, 1, (n, 3))
y = (rng.random(n) < 1 / (1 + np.exp(6 - 1.8 * X[:, 0] - 1.2 * X[:, 1]))).astype(int)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.5, stratify=y, random_state=0)
from sklearn.metrics import precision_score, recall_score, f1_score, fbeta_score
p = LogisticRegression().fit(X_tr, y_tr).predict_proba(X_te)[:, 1]
models = {"flag all": np.ones_like(y_te), "thresh 0.5": p >= 0.5, # "flag all" says fraud every time
"thresh 0.1": p >= 0.1, "thresh 0.03": p >= 0.03}
for name, pred in models.items():
P, R = precision_score(y_te, pred), recall_score(y_te, pred)
print(f"{name:>11} P {P:.3f} R {R:.3f} mean {(P + R) / 2:.3f} F1 {f1_score(y_te, pred):.3f}"
f" F2 {fbeta_score(y_te, pred, beta=2):.3f}")
Output:
flag all P 0.018 R 1.000 mean 0.509 F1 0.036 F2 0.086
thresh 0.5 P 0.690 R 0.158 mean 0.424 F1 0.257 F2 0.186
thresh 0.1 P 0.265 R 0.630 mean 0.448 F1 0.374 F2 0.494
thresh 0.03 P 0.123 R 0.810 mean 0.466 F1 0.214 F2 0.383
The silly model has recall 1.000 and precision 0.018 (98% of its flags are innocent customers), yet the plain mean gives it 0.509, the best of all four. The plain average picked the silliest model.
F1 = 2 · P · R / (P + R) the harmonic mean of precision and recall
- The harmonic mean is pulled towards the smaller number: P 0.018, R 1.0 → mean 0.509 but F1 0.036.
- Like a chain that is only as strong as its weakest link, F1 is high only when both precision and recall are high.
- Among the four, threshold 0.1 has the best F1 (0.374).
F-beta: when one matters more
Fβ = (1 + β²) · P · R / (β² · P + R)
- β says how many times more important recall is than precision.
- β = 2 (F2): recall-heavy, for fraud, disease, faulty products. Our F2 column also picks threshold 0.1 (0.494).
- β = 0.5 (F0.5): precision-heavy, for spam filters or blocking cards (practice 3).
- β = 1: F1, both equally important.
Averaging over many classes: macro, micro, weighted
With K classes, each class gets its own precision, recall and F1 (that class as "positive", all others as "negative"). How to combine them?
Real-life example: a school with three sections of very different sizes.
| Average | School version | Classifier version | Use when |
|---|---|---|---|
| Macro | average of the three section averages | plain mean of the per-class scores | every class matters equally (rare ones too) |
| Micro | pool all students, then average | add up all TP, FP, FN over classes, then compute | every prediction matters equally |
| Weighted | section averages weighted by section size | per-class scores weighted by support | in between |
Support = the number of real examples of a class.
A café like Episode 23's, with made-up features and a rare drink, badam milk (60 of 1,000 test orders):
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import f1_score
rng = np.random.default_rng(4); n = 3000 # made-up café orders: 0 chai, 1 coffee, 2 badam milk
X = rng.normal(0, 1, (n, 2))
Z = np.c_[np.full(n, 2.5), 1.2 * X[:, 0] + 0.8, 1.0 * X[:, 1] - 0.6]
P = np.exp(Z) / np.exp(Z).sum(1, keepdims=True)
y = np.array([rng.choice(3, p=row) for row in P])
pred = LogisticRegression().fit(X[:2000], y[:2000]).predict(X[2000:]); yt = y[2000:]
print("orders per class:", np.bincount(yt))
print("F1 per class: ", f1_score(yt, pred, average=None).round(3))
for avg in ["macro", "micro", "weighted"]:
print(f"{avg:>8} F1 {f1_score(yt, pred, average=avg):.3f}")
print(f"accuracy {np.mean(pred == yt):.3f}")
Output:
orders per class: [740 200 60]
F1 per class: [0.852 0.383 0.033]
macro F1 0.423
micro F1 0.756
weighted F1 0.709
accuracy 0.756
- F1 per class: chai 0.852, coffee 0.383, badam milk 0.033: the model almost never gets badam milk right.
- Micro F1 = 0.756 = accuracy. For single-label multiclass problems, micro-averaged precision, recall and F1 all equal accuracy, so micro hides the rare class too.
- Weighted F1 0.709: still dominated by chai.
- Macro F1 0.423 exposes the failure. If every class matters, report macro.
Which metric when?
| Situation | Watch | Compare models with |
|---|---|---|
| misses are costly (fraud, disease, faulty cookers) | recall (sensitivity) | F2, recall at a fixed precision |
| false alarms are costly (spam deleting mail, blocking genuine cards) | precision, specificity / FPR | F0.5 |
| both matter roughly equally | precision and recall | F1 |
| many classes, all important | per-class scores | macro F1 |
| balanced classes, equal costs | accuracy is fine | accuracy |
Always look at the confusion matrix, never only one number.
Common mistakes
| Mistake | Real symptom | Why | Fix |
|---|---|---|---|
| reporting accuracy on a rare class | lazy model: accuracy 0.982, 0 frauds caught | accuracy paradox | confusion matrix, recall, precision |
| plain average of precision and recall | "flag everything": mean 0.509, the best | the arithmetic mean hides a near-zero | F1 (harmonic mean) |
| reading the matrix the wrong way round | FP and FN swapped | scikit-learn: rows = truth, columns = prediction | tn, fp, fn, tp = confusion_matrix(y, pred).ravel() |
| micro F1 when the rare class matters | micro F1 0.756 = accuracy, badam milk F1 0.033 | big classes dominate | macro F1, per-class report |
| wrong positive class | precision of the class you didn't mean | pos_label defaults to 1 |
code the class you hunt for as 1, or pass pos_label |
| precision when nothing is flagged | UndefinedMetricWarning and precision 0.0 |
TP + FP = 0, division by zero | zero_division=0 (and question the model) |
| comparing models at different thresholds | "model A has better recall" | recall depends on the threshold | compare at equal thresholds or over all thresholds (Episode 25) |
Mini project: choose the threshold from the cost
Suppose (made-up numbers) a missed fraud costs 50 units and a false alarm costs 1 unit. Total cost = 50 · FN + 1 · FP:
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
rng = np.random.default_rng(24); n = 20000 # made-up card payments, 1 = fraud
X = rng.normal(0, 1, (n, 3))
y = (rng.random(n) < 1 / (1 + np.exp(6 - 1.8 * X[:, 0] - 1.2 * X[:, 1]))).astype(int)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.5, stratify=y, random_state=0)
from sklearn.metrics import confusion_matrix
p = LogisticRegression().fit(X_tr, y_tr).predict_proba(X_te)[:, 1]
MISS, ALARM = 50, 1 # made-up costs: a missed fraud costs 50 units, a false alarm 1 unit
for t in [0.5, 0.3, 0.2, 0.1, 0.05, 0.03]:
tn, fp, fn, tp = confusion_matrix(y_te, p >= t).ravel()
print(f"threshold {t:<4} FN {fn:3} FP {fp:4} cost {MISS * fn + ALARM * fp:5}")
Output:
threshold 0.5 FN 155 FP 13 cost 7763
threshold 0.3 FN 126 FP 49 cost 6349
threshold 0.2 FN 101 FP 114 cost 5164
threshold 0.1 FN 68 FP 321 cost 3721
threshold 0.05 FN 47 FP 678 cost 3028
threshold 0.03 FN 35 FP 1062 cost 2812
The cheapest of these thresholds is 0.03, far below 0.5, because a miss is 50 times worse than a false alarm. Episode 25 does this properly with curves over every threshold.
Practice
- Print the confusion matrix, recall and precision of the lazy model.
- Print F1 for thresholds from 0.05 to 0.5. Which threshold is best?
- Compute F0.5 for the four models of the F1 program. Which one does it prefer, and why?
- Run
classification_reporton the café data and read the badam milk row.
Solution 1
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
rng = np.random.default_rng(24); n = 20000 # made-up card payments, 1 = fraud
X = rng.normal(0, 1, (n, 3))
y = (rng.random(n) < 1 / (1 + np.exp(6 - 1.8 * X[:, 0] - 1.2 * X[:, 1]))).astype(int)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.5, stratify=y, random_state=0)
from sklearn.dummy import DummyClassifier
from sklearn.metrics import confusion_matrix, precision_score, recall_score
pred = DummyClassifier(strategy="most_frequent").fit(X_tr, y_tr).predict(X_te)
print(confusion_matrix(y_te, pred))
print("recall", recall_score(y_te, pred), " precision", precision_score(y_te, pred, zero_division=0))
Output:
[[9816 0]
[ 184 0]]
recall 0.0 precision 0.0
All 184 frauds are false negatives, and nothing is flagged, so precision is 0 / 0: undefined. Without zero_division=0, scikit-learn prints an UndefinedMetricWarning and returns 0.0.
Solution 2
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
rng = np.random.default_rng(24); n = 20000 # made-up card payments, 1 = fraud
X = rng.normal(0, 1, (n, 3))
y = (rng.random(n) < 1 / (1 + np.exp(6 - 1.8 * X[:, 0] - 1.2 * X[:, 1]))).astype(int)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.5, stratify=y, random_state=0)
from sklearn.metrics import f1_score
p = LogisticRegression().fit(X_tr, y_tr).predict_proba(X_te)[:, 1]
for t in [0.05, 0.1, 0.15, 0.2, 0.3, 0.4, 0.5]:
print(f"threshold {t:<4} F1 {f1_score(y_te, p >= t):.3f}")
Output:
threshold 0.05 F1 0.274
threshold 0.1 F1 0.374
threshold 0.15 F1 0.429
threshold 0.2 F1 0.436
threshold 0.3 F1 0.399
threshold 0.4 F1 0.344
threshold 0.5 F1 0.257
Threshold 0.2 gives the best F1 (0.436), better than the 0.1 we used in the episode: F1 peaks where precision and recall are balanced.
Solution 3
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
rng = np.random.default_rng(24); n = 20000 # made-up card payments, 1 = fraud
X = rng.normal(0, 1, (n, 3))
y = (rng.random(n) < 1 / (1 + np.exp(6 - 1.8 * X[:, 0] - 1.2 * X[:, 1]))).astype(int)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.5, stratify=y, random_state=0)
from sklearn.metrics import fbeta_score
p = LogisticRegression().fit(X_tr, y_tr).predict_proba(X_te)[:, 1]
models = {"flag everything": np.ones_like(y_te), "threshold 0.5": p >= 0.5,
"threshold 0.1": p >= 0.1, "threshold 0.03": p >= 0.03}
for name, pred in models.items():
print(f"{name:>15}: F0.5 {fbeta_score(y_te, pred, beta=0.5):.3f}")
Output:
flag everything: F0.5 0.023
threshold 0.5: F0.5 0.412
threshold 0.1: F0.5 0.300
threshold 0.03: F0.5 0.148
F0.5 weights precision more, so it prefers the cautious threshold 0.5 (0.412), the opposite of F2's choice.
Solution 4
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import f1_score
rng = np.random.default_rng(4); n = 3000 # made-up café orders: 0 chai, 1 coffee, 2 badam milk
X = rng.normal(0, 1, (n, 2))
Z = np.c_[np.full(n, 2.5), 1.2 * X[:, 0] + 0.8, 1.0 * X[:, 1] - 0.6]
P = np.exp(Z) / np.exp(Z).sum(1, keepdims=True)
y = np.array([rng.choice(3, p=row) for row in P])
pred = LogisticRegression().fit(X[:2000], y[:2000]).predict(X[2000:]); yt = y[2000:]
print("orders per class:", np.bincount(yt))
print("F1 per class: ", f1_score(yt, pred, average=None).round(3))
for avg in ["macro", "micro", "weighted"]:
print(f"{avg:>8} F1 {f1_score(yt, pred, average=avg):.3f}")
print(f"accuracy {np.mean(pred == yt):.3f}")
from sklearn.metrics import classification_report
print(classification_report(yt, pred, target_names=["chai", "coffee", "badam milk"], digits=3))
Output:
orders per class: [740 200 60]
F1 per class: [0.852 0.383 0.033]
macro F1 0.423
micro F1 0.756
weighted F1 0.709
accuracy 0.756
precision recall f1-score support
chai 0.778 0.942 0.852 740
coffee 0.563 0.290 0.383 200
badam milk 1.000 0.017 0.033 60
accuracy 0.756 1000
macro avg 0.780 0.416 0.423 1000
weighted avg 0.748 0.756 0.709 1000
Badam milk: precision 1.000 but recall 0.017. The model predicted badam milk only once (correctly) and missed the other 59. "macro avg" and "weighted avg" are the averages from the episode; the "accuracy" row is the micro average.
Quick quiz
a disease screening test, 1,000 people:
TP = 40 FN = 10
FP = 60 TN = 890
recall = ?
A) 0.80 B) 0.40 C) 0.93 D) 0.94
Answer
A) 0.80. Recall = TP / (TP + FN) = 40 / 50. B is the precision (40 / 100) and C the accuracy (930 / 1,000). Specificity is 890 / 950 = 0.937.
Interview questions
- What is the accuracy paradox? On imbalanced data a trivial majority-class model reaches high accuracy with no value (98.2% while catching no fraud).
- Define precision and recall. Precision = TP / (TP + FP): how many flagged are truly positive. Recall = TP / (TP + FN): how many positives were found.
- What are type I and type II errors? Type I = false positive; type II = false negative.
- Sensitivity vs specificity? Sensitivity = recall = TPR; specificity = TN / (TN + FP) = 1 − FPR.
- Why is F1 a harmonic mean? So that it is high only when both precision and recall are high; it heavily penalises a model that sacrifices one for the other.
- When would you use F2 or F0.5? F2 when misses (false negatives) are costlier; F0.5 when false alarms (false positives) are costlier.
- Macro vs micro vs weighted averaging? Macro: unweighted mean over classes (rare classes count equally). Micro: pool TP/FP/FN over classes (equals accuracy for single-label multiclass). Weighted: mean weighted by support.
- How do you choose a threshold? From the costs of each error type, or by maximising a chosen metric on validation data (never on the test set).
- What is the layout of scikit-learn's confusion matrix? Rows = true labels, columns = predicted labels, classes in sorted order; for binary,
ravel()gives tn, fp, fn, tp.
Glossary
| Term | Meaning | Example |
|---|---|---|
| Accuracy | Correct predictions / all predictions | 0.983 |
| Accuracy paradox | High accuracy, no value, because one class is rare | lazy model: 98.2%, 0 frauds caught |
Baseline / DummyClassifier |
A trivial model to beat (Episode 17) | strategy="most_frequent" |
| Positive class | The class we are looking for, coded 1 | fraud |
| True positive (TP) | Actual 1, predicted 1 | fraud flagged (29) |
| False positive (FP) | Actual 0, predicted 1: a false alarm | genuine flagged (13) |
| False negative (FN) | Actual 1, predicted 0: a miss | fraud let through (155) |
| True negative (TN) | Actual 0, predicted 0 | genuine let through (9,803) |
| Confusion matrix | Table of counts: rows = truth, columns = prediction | [[9803 13] [155 29]] |
| Type I error | A false positive | burnt-toast alarm |
| Type II error | A false negative | silent alarm in a fire |
| Precision | TP / (TP + FP) | 0.690 |
| Recall / sensitivity / true positive rate (TPR) | TP / (TP + FN) | 0.158 |
| Specificity (true negative rate) | TN / (TN + FP) | 0.999 |
| False positive rate (FPR) | FP / (FP + TN) = 1 − specificity | 0.001 |
| Precision-recall trade-off | Raising one usually lowers the other | threshold 0.5 → 0.1 |
| Harmonic mean | 2ab / (a + b), pulled towards the smaller value | 0.018 and 1.0 → 0.036 |
| F1 score | Harmonic mean of precision and recall | 0.374 at threshold 0.1 |
| F-beta (Fβ) | Weighted harmonic mean; β > 1 favours recall | F2 0.494, F0.5 0.412 |
| Support | Number of true examples of a class | badam milk: 60 |
| Macro average | Plain mean of per-class scores | macro F1 0.423 |
| Micro average | Pool all TP/FP/FN, then compute | micro F1 0.756 = accuracy |
| Weighted average | Per-class scores weighted by support | weighted F1 0.709 |
classification_report |
scikit-learn table of per-class precision, recall, F1, support | practice 4 |
zero_division |
What to return when a metric divides by 0 | precision_score(..., zero_division=0) |
pos_label |
Which label counts as positive | default 1 |
Summary
- Accuracy paradox: with 1.8% fraud, "never fraud" scores 98.2% and catches nothing.
- The confusion matrix splits predictions into TP, FP, FN, TN; FP = type I, FN = type II error.
- Precision 0.690 (flags are mostly right) but recall 0.158 (most fraud missed); specificity 0.999, FPR 0.001.
- The plain average of P and R crowned a silly model; F1 (harmonic mean) doesn't; F2 and F0.5 weight recall or precision.
- Macro averaging makes rare classes count (0.423); micro equals accuracy (0.756); weighted sits in between.
Next episode: classification metrics 2: threshold tuning, ROC curves and AUC, precision-recall curves, calibration, and choosing a metric from business cost.