SyntaxPilot
Courses/Machine Learning/Lesson 46
Video coming soonThis lesson’s video is being made

Start with the notes below; the video appears here as soon as it is on YouTube.

Machine Learning · Lesson 46 of 90

Anomaly detection

anomalyoutlier detection vs novelty detectionstatistical thresholdsisolation forestlocal outlier factor (LOF)one-class SVMcontamination parameterevaluating detectors with few labels

Machine Learning Full Course · Episode 46. Every code sample below was run with Python 3.13, NumPy 2.5.3, pandas 3.0.5 and scikit-learn 1.9.1; the output shown is the real output. Riya's 2,030 orders are made up for teaching. Because we made the data ourselves, we secretly planted 30 frauds and know exactly which rows they are, which lets us check every detector.

The big idea: an anomaly is a row that does not fit the usual pattern, like a fraud order among normal ones. Usually nobody has labelled the frauds, so detectors work without labels and give every row an anomaly score (higher = odder); people then check the top of the list. The obvious rule, "flag any value more than 3 standard deviations from its column's mean" (a z-score threshold), caught all 10 huge-amount frauds but 0 of the 20 frauds that were odd only as a combination (few items, high amount). An isolation forest cuts the data with random splits: odd rows are cut off after fewer splits (10.4 for a normal order vs 7.4 and 6.0 for frauds). Its contamination parameter only moves the cut-off: "auto" flagged 598 orders, 0.015 flagged 31 (27 frauds). The local outlier factor (LOF) compares a row's crowdedness with its neighbours': with the default 20 neighbours it caught only 12 of 30, because a group of 20 lookalike frauds hid each other; with 50 neighbours it caught all 30. With only a few labels, judge detectors by precision@k and average precision (z-score 0.37, isolation forest 0.95, LOF k=20 0.44, LOF k=50 1.00). For novelty detection (learn from clean data, judge new rows), a one-class SVM with nu = 0.01 found all 30 frauds with 14 false alarms, and LOF needs novelty=True to do the same.

Sources: F. T. Liu, K. M. Ting and Z.-H. Zhou, "Isolation Forest", ICDM 2008, pp. 413-422 (researchr.org); M. M. Breunig, H.-P. Kriegel, R. T. Ng and J. Sander, "LOF: Identifying Density-Based Local Outliers", SIGMOD 2000 (sigmodrecord.org); the docstrings of the installed scikit-learn 1.9.1: IsolationForest (contamination is 'auto' or a float in (0, 0.5]; predict returns -1 / 1; score_samples), LocalOutlierFactor (n_neighbors=20 by default; novelty=True is needed to call predict on new data; negative_outlier_factor_), OneClassSVM (nu is an upper bound on the fraction of training errors) and average_precision_score.

Words you need first

  • Row, column, feature: one row is one order; a column (feature) is one thing measured about it, here the number of items and the amount in rupees (Rs).
  • Label, supervised vs unsupervised learning (Episode 1): a label is the answer column (here: fraud or not). Supervised learning learns from labelled rows; unsupervised learning works with no answer column. Anomaly detection is usually unsupervised.
  • Mean and standard deviation (Episode 10): the average, and the typical distance of values from the average. Example: if order amounts have mean Rs 1,200 and standard deviation Rs 860, most orders are within a few hundred rupees of Rs 1,200.
  • z-score and standardising, StandardScaler (Episode 16): z = (value − mean) / standard deviation: how many standard deviations a value is from the mean. Standardising turns every column into z-scores (mean 0, standard deviation 1).
  • Distance and k-nearest neighbours (Episode 27): the distance between two rows is the straight-line distance between their points; the k nearest neighbours of a row are the k rows closest to it.
  • Precision and average precision (Episode 25): precision = of the rows you flagged, the share that really are positive. Average precision summarises a whole ranking in one number.
  • Kernel, RBF kernel, SVM (Episode 32): a support vector machine draws a boundary; the RBF kernel lets that boundary bend into any shape.
  • Density and log-density (Episode 45): how crowded the data is around a point. Last episode, the trip "2 km in 60 minutes" had log-density −382.9 under the mixture: no component explained it. That is an anomaly.
  • Random generator: np.random.default_rng(4) makes the same "random" numbers every run, so you get exactly the outputs printed here.

The real-life hook: fake orders at Riya's store

Riya's grocery app had 2,030 orders this month. For each we keep two columns: the number of items and the amount in rupees. A normal order costs about Rs 150 per item. Some orders are fraud: stolen cards, fake refund tricks. Nobody labelled them, and Riya's team has time to check about 50 orders by hand. Which 50?

We planted 30 frauds of two kinds:

  • 20 "small but pricey" orders: 1 to 3 items for Rs 1,500 to 2,500.
  • 10 "huge amount" orders: 2 to 9 items for Rs 8,000 to 9,000.

These six lines create the data; every program below starts with them (we call them HDR):

import numpy as np, pandas as pd
r = np.random.default_rng(4)                                             # Riya's orders this month (made up)
items = r.integers(1, 16, 2000); amount = 150 * items * r.lognormal(0, .15, 2000)    # 2,000 normal: ~Rs 150 per item
fi = np.r_[r.integers(1, 4, 20), r.integers(2, 10, 10)]                 # 30 frauds: 20 small but pricey,
fa = np.r_[r.uniform(1500, 2500, 20), r.uniform(8000, 9000, 10)]        # 10 with huge amounts
X = np.c_[np.r_[items, fi], np.r_[amount, fa]]; y = np.r_[np.zeros(2000, int), np.ones(30, int)]  # y = 1: fraud
  • r.integers(1, 16, 2000): 2,000 normal orders with 1 to 15 items (the upper end 16 is excluded).
  • 150 * items * r.lognormal(0, .15, 2000): Rs 150 per item times a random factor near 1 (a lognormal number is always positive; with the setting 0.15 it stays close to 1, so amounts vary by a modest percentage around Rs 150 per item).
  • fi, fa: items and amounts of the 30 frauds. np.r_[a, b] glues arrays end to end; np.c_[a, b] puts them side by side as columns.
  • X has shape (2030, 2): rows 0 to 1999 are normal, rows 2000 to 2019 are the small pricey frauds, rows 2020 to 2029 the huge ones. y is the hidden truth (1 = fraud), which no detector sees; we use it only to check.

Anomalies, outliers and novelties

  • Anomaly: a row that does not fit the usual pattern. Example: a login at 3 a.m. from another country on your bank app; an order of 2 items for Rs 2,132.
  • Outlier detection: the odd rows are already mixed into the data you have, like this month's orders. You fit on the data and score the same data.
  • Novelty detection: you have clean data (say last month's orders, all checked) and you judge each new row against it. Like a security guard who knows every regular face and notices a stranger.
  • Anomaly score: the number a detector gives each row; in these notes higher = odder. You sort by it and look at the top.
  • Inlier: a normal row (the opposite of an outlier).

Either way it is usually unsupervised: there are no fraud labels to learn from. That is the difference from the classification episodes: we are not learning "what fraud looks like", we are learning "what normal looks like" and flagging what does not fit.

The obvious attempt: z-score thresholds

From statistics (Episode 16): a value's z-score is how many standard deviations it is from its column's mean. A common rule: flag a row if any column has |z| > 3 (more than 3 standard deviations away). For a bell curve, values that far out are rare.

import numpy as np, pandas as pd
r = np.random.default_rng(4)                                             # Riya's orders this month (made up)
items = r.integers(1, 16, 2000); amount = 150 * items * r.lognormal(0, .15, 2000)    # 2,000 normal: ~Rs 150 per item
fi = np.r_[r.integers(1, 4, 20), r.integers(2, 10, 10)]                 # 30 frauds: 20 small but pricey,
fa = np.r_[r.uniform(1500, 2500, 20), r.uniform(8000, 9000, 10)]        # 10 with huge amounts
X = np.c_[np.r_[items, fi], np.r_[amount, fa]]; y = np.r_[np.zeros(2000, int), np.ones(30, int)]  # y = 1: fraud
print("rows:", len(X), "   first order (items, Rs):", X[0].round())
z = np.abs((X - X.mean(axis=0)) / X.std(axis=0))                         # z-score of every value, per column
flag = (z > 3).any(axis=1)                                               # obvious rule: 3+ std away in ANY column
print("flagged:", flag.sum(), "   real frauds caught:", (flag & (y == 1)).sum(), "of 30")
print("huge amount:", flag[2020:].sum(), "of 10    few items, high amount:", flag[2000:2020].sum(), "of 20")

Output:

rows: 2030    first order (items, Rs): [  11. 2138.]
flagged: 10    real frauds caught: 10 of 30
huge amount: 10 of 10    few items, high amount: 0 of 20
  • X.mean(axis=0), X.std(axis=0): the mean and standard deviation of each column (axis=0 = down the rows). np.abs(...) drops the sign, so z is a distance.
  • (z > 3).any(axis=1): True if any column of that row is over 3 (axis=1 = across the columns).
  • flag & (y == 1): flagged and really fraud. Summing True/False counts the Trues.
  • 10 flagged, all 10 real frauds. Looks great, but there were 30.
  • It caught all 10 huge-amount frauds and 0 of the 20 few-items-high-amount frauds.

Why per-column thresholds miss combinations

Look at a small pricey fraud: 2 items, Rs 2,132. Is 2 items unusual? No: normal orders have 1 to 15 items. Is Rs 2,132 unusual? No: the very first normal order is 11 items for Rs 2,138. Each column alone looks perfectly normal. What is odd is the combination: Rs 2,132 for 2 items is over Rs 1,000 per item, when normal is about Rs 150.

A rule that checks one column at a time can never see that. We need a method that looks at all columns together.

Isolation forest: odd things are easy to isolate

The idea: twenty questions

Play twenty questions to single out one person in a crowd with yes/no questions: "taller than 170 cm?", "wearing glasses?". An ordinary person takes many questions, because lots of people look like them. Someone in a gorilla suit takes one: nobody else is like them. Odd things are easy to isolate.

That is the idea of the isolation forest (Liu, Ting and Zhou, 2008):

  • A random split picks a random column and a random cut value between that column's min and max, and divides the rows into two sides.
  • Keep splitting the side the point is on until it is alone.
  • The number of splits needed is its path length (in a tree drawing, the depth at which the point ends up alone).
  • A short path = anomaly. Normal rows sit in crowds, so the cuts must slice thinner and thinner around them.
  • One sequence of random splits is an isolation tree; many trees, averaged, make the forest. (A "tree" because each split branches into two, like the decision trees of earlier episodes, except the cuts are random instead of chosen.)

Isolation from scratch

For one point: pick a random column and a random cut, keep the point's side, count until it is alone; repeat with 200 random trees and average.

import numpy as np, pandas as pd
r = np.random.default_rng(4)                                             # Riya's orders this month (made up)
items = r.integers(1, 16, 2000); amount = 150 * items * r.lognormal(0, .15, 2000)    # 2,000 normal: ~Rs 150 per item
fi = np.r_[r.integers(1, 4, 20), r.integers(2, 10, 10)]                 # 30 frauds: 20 small but pricey,
fa = np.r_[r.uniform(1500, 2500, 20), r.uniform(8000, 9000, 10)]        # 10 with huge amounts
X = np.c_[np.r_[items, fi], np.r_[amount, fa]]; y = np.r_[np.zeros(2000, int), np.ones(30, int)]  # y = 1: fraud
def splits_to_isolate(x, data, rng):                                     # one random tree, followed for ONE point
    n = 0
    while len(data) > 1:
        col = rng.integers(2); lo, hi = data[:, col].min(), data[:, col].max()
        if lo == hi: break
        cut = rng.uniform(lo, hi)                                        # random column, random cut
        data = data[data[:, col] < cut] if x[col] < cut else data[data[:, col] >= cut]   # keep x's side
        n += 1
    return n
rng = np.random.default_rng(0)
for name, i in [("normal order", 3), ("fraud: few, pricey", 2003), ("fraud: huge amount", 2025)]:
    depth = [splits_to_isolate(X[i], X, rng) for _ in range(200)]       # 200 random trees
    print(f"{name:19} {X[i, 0]:2.0f} items, Rs {X[i, 1]:4.0f}: {np.mean(depth):4.1f} splits")

Output:

normal order         8 items, Rs 1130: 10.4 splits
fraud: few, pricey   2 items, Rs 2132:  7.4 splits
fraud: huge amount   7 items, Rs 8648:  6.0 splits
  • rng.integers(2): column 0 or 1 at random. rng.uniform(lo, hi): a random cut between that column's min and max.
  • if lo == hi: break: all remaining rows have the same value in that column (for example, identical orders), so no cut can separate them; stop.
  • The line with data[...] keeps only the rows on the same side of the cut as our point x.
  • Normal order (8 items, Rs 1,130): 10.4 splits on average. Small pricey fraud: 7.4. Huge fraud: 6.0. Fewer splits = odder.
  • Notice the random cuts use both columns at once: a cut on items and then a cut on amount can box in "few items AND high amount", which no single-column rule could.

A single random tree is noisy (one unlucky cut can isolate a normal row early, or a fraud late), which is why we average over many trees.

From path length to an anomaly score

scikit-learn's IsolationForest averages the path length over its trees and turns it into an anomaly score between 0 and 1: a short average path gives a score near 1 (odd), a long one a lower score. (Each tree is built on a random sample of rows, by default max_samples="auto" = min(256, number of rows), and if several rows are still together when a tree stops, an estimate of the remaining path is added, according to the installed docstrings.) score_samples returns the opposite (negative) of that score, "the lower, the more abnormal", so we write -iso.score_samples(X) to get "higher = odder".

IsolationForest in scikit-learn, and the contamination parameter

predict gives −1 for anomaly and 1 for normal. Where is the line between them? That is the contamination parameter: the share of rows you expect to be anomalies. Three settings:

import numpy as np, pandas as pd
r = np.random.default_rng(4)                                             # Riya's orders this month (made up)
items = r.integers(1, 16, 2000); amount = 150 * items * r.lognormal(0, .15, 2000)    # 2,000 normal: ~Rs 150 per item
fi = np.r_[r.integers(1, 4, 20), r.integers(2, 10, 10)]                 # 30 frauds: 20 small but pricey,
fa = np.r_[r.uniform(1500, 2500, 20), r.uniform(8000, 9000, 10)]        # 10 with huge amounts
X = np.c_[np.r_[items, fi], np.r_[amount, fa]]; y = np.r_[np.zeros(2000, int), np.ones(30, int)]  # y = 1: fraud
from sklearn.ensemble import IsolationForest
for c in ["auto", 0.015, 0.10]:                                          # contamination: expected share of anomalies
    iso = IsolationForest(n_estimators=200, contamination=c, random_state=0).fit(X)
    flag = iso.predict(X) == -1                                          # -1 = anomaly, 1 = normal
    print(f"contam.={c!s:5} flagged {flag.sum():3}  caught {(flag & (y == 1)).sum():2}/30",
          f"(small {flag[2000:2020].sum():2}, huge {flag[2020:].sum():2})")
s = -iso.score_samples(X)                                                # higher = more anomalous
print("score  normal median", np.median(s[:2000]).round(3), "   fraud median", np.median(s[2000:]).round(3))

Output:

contam.=auto  flagged 598  caught 30/30 (small 20, huge 10)
contam.=0.015 flagged  31  caught 27/30 (small 17, huge 10)
contam.=0.1   flagged 203  caught 30/30 (small 20, huge 10)
score  normal median 0.473    fraud median 0.69
  • n_estimators=200: 200 isolation trees (the default is 100). random_state=0: the same random trees every run.
  • contamination="auto" (the default; the threshold is set "as in the original paper"): 598 orders flagged. Every fraud caught, but nobody can check 600 orders.
  • 0.015 (about 30 out of 2,030): 31 flagged, 27 of them fraud (17 small, 10 huge).
  • 0.10: 203 flagged, all 30 frauds inside.
  • Scores: median 0.473 for normal orders, 0.69 for frauds (the median is the middle value when sorted).
  • The scores are the same every time; contamination only moves the cut-off. Set it from your budget: how many rows can people actually check? 50 checks out of 2,030 orders ≈ 0.025.
  • Allowed values: "auto" or a float in (0, 0.5]. More than half "anomalies" makes no sense (see Common mistakes).
  • Isolation forest uses only cuts, never distances, so it does not need scaling: cutting "items < 4" works the same whatever the units.

Local outlier factor (LOF): lonelier than my neighbours

A second idea, from Breunig, Kriegel, Ng and Sander (2000).

  • Local density: look at a row's k nearest neighbours (as in Episode 27). If they are close, the row sits in a dense (crowded) area: high local density. If they are far, low density.
  • Local outlier factor: compare my density with my neighbours' densities: roughly LOF = (neighbours' average density) ÷ (my density). (The paper's exact version uses a smoothed distance called the reachability distance, and the density built from it is the local reachability density; the docstring describes LOF as "the average of the ratio of the local reachability density of a sample and those of its k-nearest neighbors".)
  • LOF ≈ 1: I am as crowded as my neighbours → normal.
  • LOF ≫ 1 (much bigger than 1): it is much emptier around me than around them → anomaly.
  • Example: a house in a village. What matters is not how far it is from the city, but whether it is lonelier than the houses around it. That is what "local" means: a sparse area of the data is fine if all its rows are sparse; only a row much sparser than its own neighbourhood is odd.
  • n_neighbors is the k: how many neighbours each row compares itself with (default 20).
  • Scale first: LOF uses distances. Items run from 1 to 15 (standard deviation 4.4) and amounts into the thousands (standard deviation 860.6), so without scaling, distance would be almost only the amount (Practice 6 shows the damage). StandardScaler puts both on the same footing.
  • In scikit-learn, negative_outlier_factor_ holds the negative LOF of every training row ("the higher, the more normal"), so -lof.negative_outlier_factor_ is the LOF itself.
import numpy as np, pandas as pd
r = np.random.default_rng(4)                                             # Riya's orders this month (made up)
items = r.integers(1, 16, 2000); amount = 150 * items * r.lognormal(0, .15, 2000)    # 2,000 normal: ~Rs 150 per item
fi = np.r_[r.integers(1, 4, 20), r.integers(2, 10, 10)]                 # 30 frauds: 20 small but pricey,
fa = np.r_[r.uniform(1500, 2500, 20), r.uniform(8000, 9000, 10)]        # 10 with huge amounts
X = np.c_[np.r_[items, fi], np.r_[amount, fa]]; y = np.r_[np.zeros(2000, int), np.ones(30, int)]  # y = 1: fraud
from sklearn.neighbors import LocalOutlierFactor; from sklearn.preprocessing import StandardScaler
Xs = StandardScaler().fit_transform(X)                                   # distances need equal units
for k in (20, 50):
    lof = LocalOutlierFactor(n_neighbors=k, contamination=0.015)
    flag = lof.fit_predict(Xs) == -1; s = -lof.negative_outlier_factor_  # LOF score: about 1 = like its neighbours
    print(f"n_neighbors={k}: flagged {flag.sum()}, caught {(flag & (y == 1)).sum()}/30",
          f"(small {flag[2000:2020].sum()}, huge {flag[2020:].sum()})")
    print(f"   LOF median: normal {np.median(s[:2000]):.2f}, small-order frauds {np.median(s[2000:2020]):.2f}")

Output:

n_neighbors=20: flagged 31, caught 12/30 (small 2, huge 10)
   LOF median: normal 1.03, small-order frauds 1.27
n_neighbors=50: flagged 31, caught 30/30 (small 20, huge 10)
   LOF median: normal 1.03, small-order frauds 4.67
  • fit_predict(Xs): fit and label the same rows in one go (−1 = anomaly). LOF also has a contamination parameter that sets the cut-off, exactly like isolation forest.
  • Normal orders have median LOF 1.03: as crowded as their neighbours, as expected.
  • 20 neighbours: only 12 of 30, just 2 of the small pricey frauds. Their median LOF is 1.27, barely above normal.
  • 50 neighbours: 30 of 30, LOF 4.67.

Masking: anomalies that come in groups

Why did 20 neighbours fail? The 20 small pricey frauds look alike (maybe one fraud ring). Each one's 20 nearest neighbours are mostly the other frauds, so the group looks like a normal little crowd: dense, and as dense as its neighbours. Its LOF stays near 1. This is masking: anomalies in a group hide each other. With 50 neighbours, each fraud's neighbourhood must reach out to the real orders, and the comparison shows how lonely the group is.

The lesson: n_neighbors must be bigger than the largest group of anomalies you want to catch. (On the other side, the 20-neighbour LOF spent its flags on ordinary orders at the edges of the normal cloud.)

Evaluating with few labels: precision@k and average precision

How do you compare detectors when you have only a few labels? Use them the way the team will work:

  • Rank every order by anomaly score, highest first.
  • A person checks the top k by hand.
  • precision@k ("precision at k") = frauds in the top k ÷ k. Example: 18 frauds among the top 50 → 18 / 50 = 0.36.
  • Average precision (AP, Episode 25): sums up the whole ranking in one number. Walk down the ranking; at each real fraud, compute the precision so far; average those values over all frauds. 1.0 means every fraud is ranked above every normal order. Unlike precision@k, it needs no k and no cut-off, and it does not depend on contamination at all, only on the scores.
  • Recall (the share of all frauds that were found) is the other side: "caught 27 of 30" is a recall of 0.9.
import numpy as np, pandas as pd
r = np.random.default_rng(4)                                             # Riya's orders this month (made up)
items = r.integers(1, 16, 2000); amount = 150 * items * r.lognormal(0, .15, 2000)    # 2,000 normal: ~Rs 150 per item
fi = np.r_[r.integers(1, 4, 20), r.integers(2, 10, 10)]                 # 30 frauds: 20 small but pricey,
fa = np.r_[r.uniform(1500, 2500, 20), r.uniform(8000, 9000, 10)]        # 10 with huge amounts
X = np.c_[np.r_[items, fi], np.r_[amount, fa]]; y = np.r_[np.zeros(2000, int), np.ones(30, int)]  # y = 1: fraud
from sklearn.ensemble import IsolationForest; from sklearn.neighbors import LocalOutlierFactor
from sklearn.preprocessing import StandardScaler; from sklearn.metrics import average_precision_score
Xs = StandardScaler().fit_transform(X)
scores = {"z-score (max)": np.abs(Xs).max(axis=1),                   # each detector gives a score: higher = odder
          "isolation forest": -IsolationForest(random_state=0).fit(X).score_samples(X),
          "LOF, k=20": -LocalOutlierFactor(n_neighbors=20).fit(Xs).negative_outlier_factor_,
          "LOF, k=50": -LocalOutlierFactor(n_neighbors=50).fit(Xs).negative_outlier_factor_}
for name, s in scores.items():
    top = np.argsort(-s)[:50]                                            # a person checks only the top 50
    ap = average_precision_score(y, s)                                   # AP: whole ranking, one number
    print(f"{name:16} top 50: {y[top].sum():2} frauds  prec@50 {y[top].mean():.2f}  AP {ap:.2f}")

Output:

z-score (max)    top 50: 10 frauds  prec@50 0.20  AP 0.37
isolation forest top 50: 30 frauds  prec@50 0.60  AP 0.95
LOF, k=20        top 50: 13 frauds  prec@50 0.26  AP 0.44
LOF, k=50        top 50: 30 frauds  prec@50 0.60  AP 1.00
  • np.abs(Xs).max(axis=1): the z-score rule turned into a score: each row's largest |z|.
  • np.argsort(-s)[:50]: the positions of the 50 highest scores (sorting -s ascending = s descending).
  • y[top].mean(): the share of frauds in those 50 = precision@50.
  • z-score: 10 frauds in the top 50, AP 0.37. LOF k=20: 13, AP 0.44.
  • Isolation forest: all 30 in the top 50, AP 0.95. LOF k=50: AP 1.00: every fraud above every normal order.
  • precision@50 is 0.60 for both, the best possible: only 30 frauds exist, so at most 30 of 50 can be fraud.

A few labels: even 30 labelled frauds are enough to compare detectors and pick settings like n_neighbors. They are far too few to train a reliable classifier. In real life the labels come from the people who check the top of the list: every check becomes a label.

Novelty detection with a one-class SVM

Suppose the first 1,500 orders are last month's, all checked and clean, and the remaining 530 are new (they contain all 30 frauds). Learn what normal looks like from the clean rows, then judge the new ones.

  • A one-class SVM draws a boundary around the normal data, like a fence around a farm. New rows outside the fence are anomalies.
  • RBF kernel (Episode 32): lets the fence take any shape instead of a straight line or circle.
  • gamma: how local the RBF kernel is. Small gamma → a smooth, loose fence; large gamma → a tight, wiggly fence that hugs the training rows. (Default "scale"; we set 1.0 on scaled data.)
  • nu: "an upper bound on the fraction of training errors" (docstring): the most training rows the fence may leave outside. Smaller nu → a looser fence. Default 0.5, which is far too many for fraud.
  • Scale first: the RBF kernel uses distances too. Fit the scaler on the training rows only and apply it to the new ones, so the new rows are measured in last month's units.
import numpy as np, pandas as pd
r = np.random.default_rng(4)                                             # Riya's orders this month (made up)
items = r.integers(1, 16, 2000); amount = 150 * items * r.lognormal(0, .15, 2000)    # 2,000 normal: ~Rs 150 per item
fi = np.r_[r.integers(1, 4, 20), r.integers(2, 10, 10)]                 # 30 frauds: 20 small but pricey,
fa = np.r_[r.uniform(1500, 2500, 20), r.uniform(8000, 9000, 10)]        # 10 with huge amounts
X = np.c_[np.r_[items, fi], np.r_[amount, fa]]; y = np.r_[np.zeros(2000, int), np.ones(30, int)]  # y = 1: fraud
from sklearn.svm import OneClassSVM; from sklearn.neighbors import LocalOutlierFactor
from sklearn.preprocessing import StandardScaler; import textwrap
sc = StandardScaler().fit(X[:1500]); train, new, y_new = sc.transform(X[:1500]), sc.transform(X[1500:]), y[1500:]
for nu in (0.01, 0.05):                                                  # nu: share of training rows allowed outside
    flag = OneClassSVM(kernel="rbf", gamma=1.0, nu=nu).fit(train).predict(new) == -1
    print(f"one-class SVM nu={nu}: flagged {flag.sum()}, frauds {(flag & (y_new == 1)).sum()}/30,",
          f"normal {(flag & (y_new == 0)).sum()}")
try: LocalOutlierFactor(n_neighbors=50).fit(train).predict(new)          # MISTAKE: LOF set up for outlier detection
except AttributeError as e: print(*textwrap.wrap("AttributeError: " + str(e), 62), sep="\n")
flag = LocalOutlierFactor(n_neighbors=50, novelty=True).fit(train).predict(new) == -1   # fix: novelty=True
print("LOF novelty=True: frauds", (flag & (y_new == 1)).sum(), "of 30, normal flagged", (flag & (y_new == 0)).sum())

Output:

one-class SVM nu=0.01: flagged 44, frauds 30/30, normal 14
one-class SVM nu=0.05: flagged 54, frauds 30/30, normal 24
AttributeError: This 'LocalOutlierFactor' has no attribute
'predict'
LOF novelty=True: frauds 30 of 30, normal flagged 15
  • StandardScaler().fit(X[:1500]) learns mean and standard deviation from last month only; sc.transform(...) applies them.
  • .fit(train).predict(new): learn the fence from clean rows, label the new rows (−1 outside, 1 inside).
  • nu = 0.01: 44 flagged: all 30 frauds plus 14 normal orders (false alarms, normal rows wrongly flagged).
  • nu = 0.05: still 30 frauds, but 24 false alarms. A tighter fence costs more checking.
  • LOF: AttributeError: This 'LocalOutlierFactor' has no attribute 'predict'. By default LOF is set up for outlier detection: it only scores the rows it was fit on (use fit_predict or negative_outlier_factor_).
  • Fix: novelty=True. Now LOF learns from the clean rows and can predict new ones: 30 of 30 frauds, 15 normal flagged. The docstring warns: with novelty=True, use predict, decision_function and score_samples only on new data, not on the training set.
  • textwrap.wrap(text, 62) only splits the long message into lines of at most 62 characters so it fits the screen; AttributeError is Python's error for "this object has no such attribute (method or value)".

Two traps with labels and contamination

import numpy as np, pandas as pd
r = np.random.default_rng(4)                                             # Riya's orders this month (made up)
items = r.integers(1, 16, 2000); amount = 150 * items * r.lognormal(0, .15, 2000)    # 2,000 normal: ~Rs 150 per item
fi = np.r_[r.integers(1, 4, 20), r.integers(2, 10, 10)]                 # 30 frauds: 20 small but pricey,
fa = np.r_[r.uniform(1500, 2500, 20), r.uniform(8000, 9000, 10)]        # 10 with huge amounts
X = np.c_[np.r_[items, fi], np.r_[amount, fa]]; y = np.r_[np.zeros(2000, int), np.ones(30, int)]  # y = 1: fraud
from sklearn.ensemble import IsolationForest; import textwrap
try: IsolationForest(contamination=0.6).fit(X)                          # MISTAKE: "60% of orders are fraud"
except ValueError as e: print(*textwrap.wrap(type(e).__name__ + ": " + str(e), 62), sep="\n")
iso = IsolationForest(contamination=0.015, random_state=0).fit(X)
print("labels it uses:", np.unique(iso.predict(X)))                     # -1 and 1, not 1 and 0
print("WRONG count of anomalies (sum of predict):", iso.predict(X).sum())
print("right count:", (iso.predict(X) == -1).sum())

Output:

InvalidParameterError: The 'contamination' parameter of
IsolationForest must be a str among {'auto'} or a float in the
range (0.0, 0.5]. Got 0.6 instead.
labels it uses: [-1  1]
WRONG count of anomalies (sum of predict): 1968
right count: 31
  • InvalidParameterError: The 'contamination' parameter of IsolationForest must be a str among {'auto'} or a float in the range (0.0, 0.5]. Got 0.6 instead. InvalidParameterError is scikit-learn's error for a bad setting; it is a kind of ValueError, which is why except ValueError caught it. If more than half your rows are "anomalies", they are not anomalies: they are a second normal.
  • np.unique(...): the distinct values: −1 and 1, not 1 and 0.
  • Summing predict gives 1968: (2030 − 31) × 1 + 31 × (−1) = 1999 − 31 = 1968. Nonsense as a count.
  • Right count: (pred == -1).sum() = 31.

Common mistakes

Mistake Exact message / symptom Why Fix
z-score per column only 0 of 20 combination frauds caught Odd only together Isolation forest / LOF on all columns
contamination='auto' blindly 598 orders flagged Cut-off ≠ your checking budget Set it from checking capacity (0.015 → 31 flagged)
n_neighbors smaller than a group of anomalies n_neighbors=20: caught 12 of 30 Anomalies mask each other Bigger n_neighbors (50 → 30 of 30)
LOF on unscaled columns AP 0.35 instead of 1.00 (Practice 6) Rupees swamp item counts in the distance StandardScaler first
lof.predict(new_rows) AttributeError: This 'LocalOutlierFactor' has no attribute 'predict' Outlier-detection mode LocalOutlierFactor(novelty=True)
contamination=0.6 InvalidParameterError: The 'contamination' parameter of IsolationForest must be a str among {'auto'} or a float in the range (0.0, 0.5]. Got 0.6 instead. Over half "anomalies" is a second normal A float in (0, 0.5], or "auto"
sum(predict(X)) as a count 1968 "anomalies" Labels are −1 / 1 (pred == -1).sum() → 31
Forgetting the sign of score_samples Sorting puts the most normal rows first scikit-learn: lower = more abnormal Use -score_samples for "higher = odder"

Mini project: Riya's daily review list

The team checks 50 orders. Combine two detectors that work in different ways (isolation forest on raw columns, LOF with 50 neighbours on scaled columns) by their average rank, build the review list, and see how good it is.

import numpy as np, pandas as pd
r = np.random.default_rng(4)                                             # Riya's orders this month (made up)
items = r.integers(1, 16, 2000); amount = 150 * items * r.lognormal(0, .15, 2000)    # 2,000 normal: ~Rs 150 per item
fi = np.r_[r.integers(1, 4, 20), r.integers(2, 10, 10)]                 # 30 frauds: 20 small but pricey,
fa = np.r_[r.uniform(1500, 2500, 20), r.uniform(8000, 9000, 10)]        # 10 with huge amounts
X = np.c_[np.r_[items, fi], np.r_[amount, fa]]; y = np.r_[np.zeros(2000, int), np.ones(30, int)]  # y = 1: fraud
from sklearn.ensemble import IsolationForest; from sklearn.neighbors import LocalOutlierFactor
from sklearn.preprocessing import StandardScaler
Xs = StandardScaler().fit_transform(X)                                   # Riya's review list: 50 orders a day
iso = -IsolationForest(n_estimators=200, random_state=0).fit(X).score_samples(X)
lof = -LocalOutlierFactor(n_neighbors=50).fit(Xs).negative_outlier_factor_
rank = lambda s: np.argsort(np.argsort(-s)) + 1                         # 1 = most anomalous
both = (rank(iso) + rank(lof)) / 2                                       # average rank of the two detectors
df = pd.DataFrame({"items": X[:, 0].astype(int), "amount": X[:, 1].round(), "per_item": (X[:, 1] / X[:, 0]).round(),
                   "iforest": iso.round(3), "lof": lof.round(2), "avg_rank": both, "fraud": y})
review = df.sort_values("avg_rank").head(50)
print(review.head(8).to_string())
for k in (10, 30, 50):
    print(f"precision@{k}: {review.fraud.head(k).mean():.2f}")
print("frauds in the review list:", review.fraud.sum(), "of 30;  normal orders to check:", (review.fraud == 0).sum())
print("median Rs per item:  normal", df.per_item[y == 0].median(), "  flagged", review.per_item.median())

Output:

      items  amount  per_item  iforest    lof  avg_rank  fraud
2028      6  8717.0    1453.0    0.779  16.12       1.5      1
2023      7  8661.0    1237.0    0.779  15.90       2.5      1
2025      7  8648.0    1235.0    0.779  15.87       3.5      1
2029      9  8914.0     990.0    0.771  16.70       3.5      1
2024      8  8420.0    1052.0    0.772  15.73       5.0      1
2021      7  8255.0    1179.0    0.772  15.07       6.0      1
2020      9  8408.0     934.0    0.766  15.61       7.5      1
2026      6  8066.0    1344.0    0.770  14.39       8.5      1
precision@10: 1.00
precision@30: 0.93
precision@50: 0.60
frauds in the review list: 30 of 30;  normal orders to check: 20
median Rs per item:  normal 150.0   flagged 842.5
  • np.argsort(np.argsort(-s)) + 1 turns scores into ranks (1 = highest score). Averaging ranks combines detectors whose scores live on different scales (0 to 1 for isolation forest, about 1 to 17 for LOF); this is called rank averaging (a simple ensemble: several models combined).
  • pd.DataFrame, sort_values, head: a pandas table, sorted by average rank, first rows.
  • The top 8 are all huge-amount frauds, with isolation forest scores around 0.77 and LOF around 15 to 17.
  • precision@10 = 1.00, @30 = 0.93, @50 = 0.60. All 30 frauds are in the 50-order list; the other 20 checks are normal orders.
  • A Rs per item column explains the flags to the team: median Rs 150 for normal orders, Rs 842.5 in the review list. (If you already know such a ratio matters, adding it as a column is feature engineering, and then even a simple rule on it would work. Detectors earn their keep when you do not know in advance which combination is odd.)

Practice

1. Scratch isolation for more rows: three normal orders, three small pricey frauds and two huge ones. Do frauds always need fewer splits?

Solution
import numpy as np, pandas as pd
r = np.random.default_rng(4)                                             # Riya's orders this month (made up)
items = r.integers(1, 16, 2000); amount = 150 * items * r.lognormal(0, .15, 2000)    # 2,000 normal: ~Rs 150 per item
fi = np.r_[r.integers(1, 4, 20), r.integers(2, 10, 10)]                 # 30 frauds: 20 small but pricey,
fa = np.r_[r.uniform(1500, 2500, 20), r.uniform(8000, 9000, 10)]        # 10 with huge amounts
X = np.c_[np.r_[items, fi], np.r_[amount, fa]]; y = np.r_[np.zeros(2000, int), np.ones(30, int)]  # y = 1: fraud
def splits_to_isolate(x, data, rng):                                     # same function as "from scratch"
    n = 0
    while len(data) > 1:
        col = rng.integers(2); lo, hi = data[:, col].min(), data[:, col].max()
        if lo == hi: break
        cut = rng.uniform(lo, hi)
        data = data[data[:, col] < cut] if x[col] < cut else data[data[:, col] >= cut]
        n += 1
    return n
rng = np.random.default_rng(0)
for i in [0, 1, 2, 2000, 2010, 2019, 2021, 2029]:                       # 3 normal, 3 small-pricey, 2 huge
    d = np.mean([splits_to_isolate(X[i], X, rng) for _ in range(200)])
    print(f"row {i:4}  fraud={y[i]}  {X[i, 0]:2.0f} items, Rs {X[i, 1]:4.0f}: {d:4.1f} splits")

Output:

row    0  fraud=0  11 items, Rs 2138: 10.0 splits
row    1  fraud=0  15 items, Rs 2445:  7.0 splits
row    2  fraud=0  14 items, Rs 2369:  8.9 splits
row 2000  fraud=1   3 items, Rs 2340:  7.1 splits
row 2010  fraud=1   1 items, Rs 2374:  6.0 splits
row 2019  fraud=1   1 items, Rs 1513:  6.5 splits
row 2021  fraud=1   7 items, Rs 8255:  5.8 splits
row 2029  fraud=1   9 items, Rs 8914:  4.4 splits

Mostly yes: the frauds need 4.4 to 7.1 splits. But row 1, a normal order of 15 items for Rs 2,445, needs only 7.0, as few as fraud row 2000 (7.1). It sits at the far edge of the normal cloud (the most items possible), where few other rows live. Isolation forest flags "far from the crowd", not "fraud": edge cases of normal behaviour get high scores too, which is why people check the list.

2. IsolationForest with 10, 100 and 500 trees, five random states each. What changes?

Solution
import numpy as np, pandas as pd
r = np.random.default_rng(4)                                             # Riya's orders this month (made up)
items = r.integers(1, 16, 2000); amount = 150 * items * r.lognormal(0, .15, 2000)    # 2,000 normal: ~Rs 150 per item
fi = np.r_[r.integers(1, 4, 20), r.integers(2, 10, 10)]                 # 30 frauds: 20 small but pricey,
fa = np.r_[r.uniform(1500, 2500, 20), r.uniform(8000, 9000, 10)]        # 10 with huge amounts
X = np.c_[np.r_[items, fi], np.r_[amount, fa]]; y = np.r_[np.zeros(2000, int), np.ones(30, int)]  # y = 1: fraud
from sklearn.ensemble import IsolationForest; from sklearn.metrics import average_precision_score
for n in (10, 100, 500):                                                 # number of random trees
    aps = [average_precision_score(y, -IsolationForest(n_estimators=n, random_state=s).fit(X).score_samples(X))
           for s in range(5)]                                            # 5 different random states
    print(f"n_estimators={n:3}: AP from 5 seeds {np.round(aps, 2)}   spread {max(aps) - min(aps):.2f}")

Output:

n_estimators= 10: AP from 5 seeds [0.97 0.95 0.97 1.   1.  ]   spread 0.05
n_estimators=100: AP from 5 seeds [0.95 0.99 0.95 0.96 0.98]   spread 0.03
n_estimators=500: AP from 5 seeds [0.93 0.95 0.94 0.95 0.94]   spread 0.02

More trees do not make the average AP higher here (10 trees even got lucky twice with 1.00); they make it steadier: the spread between random states shrinks from 0.05 to 0.03 to 0.02. Averaging more random trees removes luck, good and bad. The default of 100 is a reasonable middle.

3. LOF: average precision for n_neighbors from 5 to 100. Where does it start working?

Solution
import numpy as np, pandas as pd
r = np.random.default_rng(4)                                             # Riya's orders this month (made up)
items = r.integers(1, 16, 2000); amount = 150 * items * r.lognormal(0, .15, 2000)    # 2,000 normal: ~Rs 150 per item
fi = np.r_[r.integers(1, 4, 20), r.integers(2, 10, 10)]                 # 30 frauds: 20 small but pricey,
fa = np.r_[r.uniform(1500, 2500, 20), r.uniform(8000, 9000, 10)]        # 10 with huge amounts
X = np.c_[np.r_[items, fi], np.r_[amount, fa]]; y = np.r_[np.zeros(2000, int), np.ones(30, int)]  # y = 1: fraud
from sklearn.neighbors import LocalOutlierFactor; from sklearn.preprocessing import StandardScaler
from sklearn.metrics import average_precision_score
Xs = StandardScaler().fit_transform(X)
for k in (5, 10, 15, 20, 25, 30, 40, 50, 75, 100):
    s = -LocalOutlierFactor(n_neighbors=k).fit(Xs).negative_outlier_factor_
    print(f"n_neighbors={k:3}: AP {average_precision_score(y, s):.2f}")

Output:

n_neighbors=  5: AP 0.01
n_neighbors= 10: AP 0.05
n_neighbors= 15: AP 0.35
n_neighbors= 20: AP 0.44
n_neighbors= 25: AP 0.82
n_neighbors= 30: AP 0.94
n_neighbors= 40: AP 1.00
n_neighbors= 50: AP 1.00
n_neighbors= 75: AP 1.00
n_neighbors=100: AP 1.00

Below 20 neighbours LOF is hopeless (AP 0.01 at k = 5: the frauds' neighbours are almost all frauds, and even the huge-amount frauds, a group of 10, mask each other). It jumps once k passes the size of the 20-fraud ring (0.82 at 25, 0.94 at 30) and is perfect from 40 on. This curve is exactly how 30 labels help: they cannot train a model, but they can choose k.

4. One-class SVM with gamma 0.1, 1 and 10 (nu = 0.01). How do the false alarms change?

Solution
import numpy as np, pandas as pd
r = np.random.default_rng(4)                                             # Riya's orders this month (made up)
items = r.integers(1, 16, 2000); amount = 150 * items * r.lognormal(0, .15, 2000)    # 2,000 normal: ~Rs 150 per item
fi = np.r_[r.integers(1, 4, 20), r.integers(2, 10, 10)]                 # 30 frauds: 20 small but pricey,
fa = np.r_[r.uniform(1500, 2500, 20), r.uniform(8000, 9000, 10)]        # 10 with huge amounts
X = np.c_[np.r_[items, fi], np.r_[amount, fa]]; y = np.r_[np.zeros(2000, int), np.ones(30, int)]  # y = 1: fraud
from sklearn.svm import OneClassSVM; from sklearn.preprocessing import StandardScaler
sc = StandardScaler().fit(X[:1500]); train, new, y_new = sc.transform(X[:1500]), sc.transform(X[1500:]), y[1500:]
for g in (0.1, 1.0, 10.0):                                               # gamma: how local the fence is
    flag = OneClassSVM(kernel="rbf", gamma=g, nu=0.01).fit(train).predict(new) == -1
    print(f"gamma={g:4}: flagged {flag.sum():3}, frauds {(flag & (y_new == 1)).sum()}/30, normal {(flag & (y_new == 0)).sum()}")

Output:

gamma= 0.1: flagged  39, frauds 30/30, normal 9
gamma= 1.0: flagged  44, frauds 30/30, normal 14
gamma=10.0: flagged  87, frauds 30/30, normal 57

All three catch every fraud. A larger gamma makes the fence hug the training rows tightly, so ordinary new orders that fall in small gaps land outside: false alarms go 9 → 14 → 57. Here the smoothest fence (gamma 0.1) is best. Like n_neighbors, gamma is a setting you choose with your few labels.

5. For the isolation forest, compute precision@k for k = 10, 30, 50, 100, and recompute average precision by hand.

Solution
import numpy as np, pandas as pd
r = np.random.default_rng(4)                                             # Riya's orders this month (made up)
items = r.integers(1, 16, 2000); amount = 150 * items * r.lognormal(0, .15, 2000)    # 2,000 normal: ~Rs 150 per item
fi = np.r_[r.integers(1, 4, 20), r.integers(2, 10, 10)]                 # 30 frauds: 20 small but pricey,
fa = np.r_[r.uniform(1500, 2500, 20), r.uniform(8000, 9000, 10)]        # 10 with huge amounts
X = np.c_[np.r_[items, fi], np.r_[amount, fa]]; y = np.r_[np.zeros(2000, int), np.ones(30, int)]  # y = 1: fraud
from sklearn.ensemble import IsolationForest; from sklearn.metrics import average_precision_score
s = -IsolationForest(random_state=0).fit(X).score_samples(X)
order = np.argsort(-s); hits = y[order]                                  # 1 where the ranked row is a fraud
for k in (10, 30, 50, 100):
    print(f"precision@{k:<3} = {hits[:k].sum():2} / {k:3} = {hits[:k].mean():.2f}")
ranks = np.flatnonzero(hits) + 1                                         # positions (1, 2, ...) of the 30 frauds
prec_at_each = np.arange(1, 31) / ranks                                  # precision right at each fraud
print("last 5 fraud positions:", ranks[-5:])
print("AP by hand:", prec_at_each.mean().round(4), "  sklearn:", round(average_precision_score(y, s), 4))

Output:

precision@10  = 10 /  10 = 1.00
precision@30  = 26 /  30 = 0.87
precision@50  = 30 /  50 = 0.60
precision@100 = 30 / 100 = 0.30
last 5 fraud positions: [29 31 32 33 42]
AP by hand: 0.9543   sklearn: 0.9543
  • precision@k depends heavily on k: 1.00 at 10, 0.87 at 30, 0.60 at 50, 0.30 at 100. Past 30 it must fall, because only 30 frauds exist. Always report k with it.
  • np.flatnonzero(hits) + 1: the ranking positions of the frauds. The 30th fraud sits at position 42.
  • At the j-th fraud, precision so far is j ÷ its position (for example 30 / 42 for the last one). The mean of those 30 values is 0.9543, exactly average_precision_score. That is what AP means.

6. Does scaling matter? Run LOF (k = 50) and isolation forest on raw and on scaled columns.

Solution
import numpy as np, pandas as pd
r = np.random.default_rng(4)                                             # Riya's orders this month (made up)
items = r.integers(1, 16, 2000); amount = 150 * items * r.lognormal(0, .15, 2000)    # 2,000 normal: ~Rs 150 per item
fi = np.r_[r.integers(1, 4, 20), r.integers(2, 10, 10)]                 # 30 frauds: 20 small but pricey,
fa = np.r_[r.uniform(1500, 2500, 20), r.uniform(8000, 9000, 10)]        # 10 with huge amounts
X = np.c_[np.r_[items, fi], np.r_[amount, fa]]; y = np.r_[np.zeros(2000, int), np.ones(30, int)]  # y = 1: fraud
from sklearn.ensemble import IsolationForest; from sklearn.neighbors import LocalOutlierFactor
from sklearn.preprocessing import StandardScaler; from sklearn.metrics import average_precision_score
Xs = StandardScaler().fit_transform(X)
print("column std (items, Rs):", X.std(axis=0).round(1))
for name, data in [("raw", X), ("scaled", Xs)]:
    lof = -LocalOutlierFactor(n_neighbors=50).fit(data).negative_outlier_factor_
    iso = -IsolationForest(random_state=0).fit(data).score_samples(data)
    print(f"{name:6}  LOF k=50 AP {average_precision_score(y, lof):.2f}   isolation forest AP {average_precision_score(y, iso):.2f}")

Output:

column std (items, Rs): [  4.4 860.6]
raw     LOF k=50 AP 0.35   isolation forest AP 0.95
scaled  LOF k=50 AP 1.00   isolation forest AP 0.95

The amount's standard deviation (860.6) is about 200 times that of items (4.4). On raw columns, distance is almost only rupees, LOF cannot see "few items", and its AP falls from 1.00 to 0.35. Isolation forest gives 0.95 either way: a cut is a cut, whatever the units. Rule: scale before LOF and one-class SVM; isolation forest does not need it.

Quick quiz

1. A detector ranks 2,030 orders, a person checks the top 50, and 18 of those 50 are fraud. What is precision@50? Options: 0.18, 0.36, 0.50, 0.60.

Answer

0.36. precision@50 = frauds in the top 50 ÷ 50 = 18 ÷ 50 = 0.36.

2. IsolationForest(...).predict(X) returns 2,030 values whose sum is 1,968. How many anomalies were flagged?

Answer

31. With a anomalies (−1) and 2030 − a normals (+1), the sum is 2030 − 2a = 1968, so a = 31. Better: count (pred == -1).sum().

3. You change contamination from 0.015 to 0.10. Does the average precision of the isolation forest change?

Answer

No. Contamination only moves the cut-off between −1 and 1; the scores, and so the ranking and the AP, stay the same. Only the number flagged changes (31 → 203 in our run).

4. You fit LocalOutlierFactor(n_neighbors=50) on last month's clean orders and call .predict(new_orders). What happens?

Answer

AttributeError: This 'LocalOutlierFactor' has no attribute 'predict'. Create it with novelty=True for novelty detection.

5. A gang places 40 near-identical fraud orders. Which is the safer LOF setting: n_neighbors=20 or n_neighbors=60?

Answer

60. n_neighbors must be bigger than the group, or the 40 frauds are each other's neighbours and mask each other (our 20-fraud ring hid from k = 20 and was caught at k = 50).

6. Which of these needs the columns scaled: isolation forest, LOF, one-class SVM with RBF kernel?

Answer

LOF and one-class SVM, which use distances. Isolation forest uses only cuts within one column at a time (Practice 6: 0.95 raw and scaled).

Interview questions

  1. What is anomaly detection, and why is it usually unsupervised? Finding rows that do not fit the usual pattern (fraud, faults, glitches). Anomalies are rare, varied and rarely labelled, so detectors learn what normal looks like and score how far each row is from it.
  2. Outlier detection vs novelty detection? Outlier detection: the anomalies are inside the data you fit on, and you score that same data. Novelty detection: you fit on clean data and judge new rows. In scikit-learn, LOF needs novelty=True for the second.
  3. Why can per-column z-score thresholds miss anomalies? A row can be normal in every column alone and odd only as a combination: 2 items is normal, Rs 2,132 is normal, Rs 2,132 for 2 items is not. On Riya's data the rule caught 0 of 20 such frauds.
  4. Explain isolation forest. Build many trees with random splits (random column, random cut between min and max). Anomalies are isolated after few splits, normal rows sit in crowds and need many. Average path length over trees → anomaly score (higher = odder in the paper's score; scikit-learn's score_samples is its negative). Liu, Ting and Zhou, 2008.
  5. Why are anomalies isolated quickly? They are few and different: a random cut is likely to fall in the empty space between them and the crowd, separating them early.
  6. What does contamination do? Sets the threshold on the scores: the expected share of anomalies, 'auto' or a float in (0, 0.5]. It does not change the scores or the ranking. Set it from how many rows can be checked: 'auto' flagged 598, 0.015 flagged 31.
  7. Explain the local outlier factor. For each row, its local density from its k nearest neighbours; LOF is the neighbours' density divided by its own (with reachability distances, Breunig et al., 2000). About 1 = normal, much bigger = anomaly. It finds rows that are lonely compared with their own neighbourhood.
  8. How do you choose n_neighbors, and what is masking? Larger than the biggest group of anomalies you want to catch; otherwise the group members are each other's neighbours and look dense (masking). Riya's 20-fraud ring: 12 of 30 caught at k = 20, 30 of 30 at k = 50.
  9. Which detectors need feature scaling? Distance-based ones: LOF and one-class SVM (RBF). Isolation forest does not (it cuts one column at a time).
  10. How do you evaluate a detector with very few labels? Rank by score and compute precision@k for the k people can check, plus average precision over the whole ranking. A few labels can compare detectors and tune settings, not train a classifier.
  11. What is a one-class SVM, and what do nu and gamma do? It learns a boundary around clean training data (RBF kernel for any shape). nu is an upper bound on the fraction of training rows left outside; gamma controls how tightly the boundary follows the data. Riya's new orders: nu 0.01 found 30 of 30 with 14 false alarms.
  12. What labels does scikit-learn's predict return for these detectors? −1 for anomaly, 1 for normal. Count anomalies with (pred == -1).sum(), never by summing (1968 instead of 31).
  13. How would you combine detectors? Turn each one's scores into ranks and average them (rank averaging), since raw scores live on different scales. Riya's combined list held all 30 frauds in 50 checks.

Glossary

Term Meaning (plain words + example)
Anomaly A row that does not fit the usual pattern: 2 items for Rs 2,132
Outlier An anomaly inside the data you have
Inlier A normal row; predict gives it 1
Outlier detection Fit on data that already contains the anomalies, score that same data: this month's orders
Novelty detection Fit on clean data, judge new rows: last month's 1,500 checked orders → 530 new ones
Anomaly score A number per row, here higher = odder; you sort by it and check the top
Unsupervised learning Learning without an answer column: no fraud labels
Label The answer column: the hidden y (1 = fraud)
Fraud A dishonest order: stolen card, fake refund
Feature engineering Making a new column from old ones: Rs per item
Mean The average of a column
Median The middle value after sorting: normal score median 0.473
Standard deviation Typical distance from the mean: Rs 860.6 for amount, 4.4 for items
z-score (value − mean) / standard deviation: how many standard deviations away
Statistical threshold A fixed cut on a statistic: flag if |z| > 3
Combination anomaly Normal in each column alone, odd together: 0 of 20 caught by z-scores
axis=0 / axis=1 Down the rows (per column) / across the columns (per row)
Lognormal A random number that is always positive, used for the price factor near 1
Isolation Separating one row from all others by cuts
Random split Random column, random cut value between its min and max
Path length Number of splits until a row is alone: 10.4 normal, 7.4 and 6.0 frauds
Isolation tree One sequence of random splits drawn as a tree
Isolation forest Many isolation trees; short average path = anomaly (Liu, Ting, Zhou, 2008)
IsolationForest scikit-learn's isolation forest class
n_estimators Number of trees (default 100): more trees = steadier scores (spread 0.05 → 0.02)
max_samples Rows sampled per tree; "auto" = min(256, number of rows)
random_state Fixes the randomness so runs repeat exactly
score_samples Per-row score; for IsolationForest the opposite of the paper's score (lower = odder)
Contamination Expected share of anomalies; sets only the cut-off: "auto" 598 flagged, 0.015 31
predict Labels: −1 anomaly, 1 normal
fit_predict Fit and label the same rows in one call
Local density How close a row's k nearest neighbours are: crowded = high
Reachability distance The LOF paper's smoothed distance used to compute density
Local reachability density The density LOF builds from reachability distances
Local outlier factor (LOF) Neighbours' density ÷ my density: ≈ 1 normal, ≫ 1 anomaly (Breunig et al., 2000)
LocalOutlierFactor scikit-learn's LOF class
n_neighbors k, how many neighbours LOF compares with (default 20)
negative_outlier_factor_ The negative LOF of each training row: higher = more normal
novelty=True LOF set up for new rows, so predict works
Masking Grouped anomalies hide each other: k = 20 caught 12 of 30
Fraud ring Our 20 lookalike small pricey frauds
Scaling (StandardScaler) Columns to mean 0, std 1; needed for distance methods
Distance Straight-line gap between two rows' points
k nearest neighbours The k rows closest to a row
Ranking Rows sorted by score, highest first
precision@k Frauds in the top k ÷ k: 18 / 50 = 0.36
Recall Share of all frauds found: 27 of 30 = 0.9
Average precision (AP) Mean of the precision at each fraud's position in the ranking: 0.95 isolation forest, 1.00 LOF k=50
average_precision_score scikit-learn's AP: average_precision_score(y, scores)
False alarm A normal row flagged as anomaly: 14 with nu = 0.01
Few labels Enough to compare detectors and tune settings, too few to train a classifier
One-class SVM A boundary (fence) around clean data; new rows outside are anomalies
OneClassSVM scikit-learn's one-class SVM class
RBF kernel Lets the SVM boundary take any shape (Episode 32)
gamma How tightly the RBF boundary hugs the data: 0.1 → 9 false alarms, 10 → 57
nu Upper bound on the share of training rows left outside (default 0.5): 0.01 → 14 false alarms
−1 / 1 labels Anomaly / normal; summing them gives 1968, not a count
Rank averaging Combine detectors by averaging each row's ranks
Ensemble Several models combined into one answer
np.argsort Positions that would sort an array; argsort(-s) = highest first
np.unique The distinct values: [-1 1]
AttributeError Python error: the object has no such attribute: 'LocalOutlierFactor' has no attribute 'predict'
ValueError Python error for a bad value
InvalidParameterError scikit-learn's bad-setting error (a ValueError): contamination=0.6
textwrap.wrap Splits long text into lines of a given width

Summary

  • An anomaly is a row that does not fit; outlier detection finds them inside your data, novelty detection judges new rows against clean data. Detectors give each row a score, usually without labels.
  • Per-column z-score thresholds caught 10 of 30 frauds and 0 of the 20 that were odd only as a combination.
  • Isolation forest: random cuts isolate odd rows quickly (10.4 vs 7.4 and 6.0 splits); no scaling needed. Contamination only moves the cut-off: "auto" flagged 598, 0.015 flagged 31 (27 frauds). Set it from your checking budget.
  • LOF: a row much lonelier than its neighbours is odd. Scale first, and make n_neighbors bigger than any group of anomalies: k = 20 caught 12 of 30 (masking), k = 50 caught all 30.
  • With few labels, compare detectors by precision@k and average precision: z-score 0.37, isolation forest 0.95, LOF k=20 0.44, LOF k=50 1.00.
  • One-class SVM learns a fence around clean data (RBF kernel, nu, gamma): nu 0.01 found 30 of 30 new frauds with 14 false alarms. LOF needs novelty=True for new rows.
  • predict returns −1 / 1: count with (pred == -1).sum() (31), never sum (1968). contamination must be "auto" or in (0, 0.5].

Next episode: Association rules: what gets bought together (if a customer buys bread and eggs, how likely is butter?), with support, confidence, lift, Apriori and FP-growth.