SyntaxPilot
Courses/Machine Learning/Lesson 31
Video coming soonThis lesson’s video is being made

Start with the notes below; the video appears here as soon as it is on YouTube.

Machine Learning · Lesson 31 of 90

Support vector machines

hyperplanemarginmaximum-margin classifiersupport vectorshard margin vs soft marginslack variableshinge lossC parameterLinearSVC

Machine Learning Full Course · Episode 31. Every code sample below was run with Python 3.13, NumPy 2.5.3 and scikit-learn 1.9.1; the output shown is the real output. The mangoes are made up for teaching; the mini project uses the breast cancer dataset bundled with scikit-learn.

The big idea: when two classes can be split by a straight line, there are infinitely many such lines, and the ones that pass close to the data make more mistakes on new data. A support vector machine (SVM) picks the line (in general, the hyperplane) with the widest possible gap, the maximum margin. Only the points on the edge of that gap, the support vectors, decide where it goes. Real data has odd points, so a soft margin lets some points break the rule, each paying a penalty called slack, which is exactly the hinge loss. The parameter C sets the price of that penalty: big C = narrow road, few mistakes; small C = wide road, more mistakes allowed. Scale your features, and use LinearSVC for big data.

Words you need first

  • Feature, label, classification, binary classification: an input column; the answer we predict; predicting a category; predicting one of two categories (Episodes 1 and 22).
  • Vector, dot product, L2 norm: a list of numbers; multiply matching entries and add (w · x); the length of a vector, √(sum of squares) (Episode 3).
  • Convex function: a bowl-shaped function with a single lowest point (Episode 6).
  • Loss function, gradient descent, learning rate, hyperparameter: a number that measures how wrong a model is; repeatedly stepping downhill on the loss; the step size; a setting we choose rather than learn (Episode 8).
  • Regularisation: adding a penalty on large weights to stop overfitting (Episode 21).
  • Scaling (StandardScaler), pipeline: putting features on similar ranges; chaining preprocessing and a model so the scaler is fit on training data only (Episodes 16-17).
  • Cross-validation: scoring on several train/validation splits and averaging (Episode 26).

The problem: many lines, which one?

Raju sorts 40 made-up mangoes by two features: sweetness (0-10) and firmness (how hard it feels, 0-10). Ripe mangoes are sweet and soft (+1), raw ones sour and hard (−1).

A straight-line rule is f(x) = w · x + b: predict ripe if f(x) > 0, raw if f(x) < 0. The boundary is where f(x) = 0. When the two groups are well apart, many different lines separate them perfectly. We try three by hand.

import numpy as np
rng = np.random.default_rng(31)                            # 40 made-up mangoes at Raju's stall
ripe = rng.normal([7, 3], 0.8, (20, 2))                    # [sweetness, firmness]
raw = rng.normal([3, 7], 0.8, (20, 2))
X = np.r_[ripe, raw]; y = np.r_[np.ones(20), -np.ones(20)]  # +1 ripe, -1 raw
lines = {"A": ([1, -1], 0.0), "B": ([1, -0.2], -3.6), "C": ([0.3, -1], 2.4)}
for name, (w, b) in lines.items():                         # line: w . x + b = 0
    f = X @ np.array(w) + b
    acc = np.mean(np.sign(f) == y)
    margin = np.min(np.abs(f)) / np.linalg.norm(w)         # distance to the nearest mango
    print(f"line {name}: train accuracy {acc:.2f}  closest mango {margin:.2f} away")
new = rng.normal([7, 3], 1.3, (500, 2)), rng.normal([3, 7], 1.3, (500, 2))
Xn = np.r_[new[0], new[1]]; yn = np.r_[np.ones(500), -np.ones(500)]  # 1,000 new mangoes
for name, (w, b) in lines.items():
    print(f"line {name}: new mangoes {np.mean(np.sign(Xn @ np.array(w) + b) == yn):.3f}")

Output:

line A: train accuracy 1.00  closest mango 1.92 away
line B: train accuracy 1.00  closest mango 0.44 away
line C: train accuracy 1.00  closest mango 0.79 away
line A: new mangoes 0.984
line B: new mangoes 0.956
line C: new mangoes 0.927
  • All three lines are perfect on the training data, so training accuracy can't choose between them.
  • The line that keeps the most distance from the nearest mango (A, 1.92) does best on new mangoes; the line that almost touches a mango (B, 0.44) and C do worse.
  • Why the close lines fail: a line that hugs the data leaves no room for error, so a new mango that is slightly different lands on the wrong side.

Three separating lines vs the SVM's widest road

Hyperplane and distance

  • Hyperplane: the set of points where w · x + b = 0. With 2 features it is a line, with 3 a plane, with 100 features a 99-dimensional "flat" we can't picture. Always one dimension less than the space.
  • w (weights) points perpendicular to the hyperplane; b (bias) shifts it.
  • Distance from a point x to the hyperplane: |w · x + b| / ‖w‖, where ‖w‖ is the L2 norm.
  • Scale freedom: multiplying both w and b by the same number gives the same hyperplane. SVMs fix the scale so the closest points have w · x + b = ±1. Then the margin (distance from the centre line to the closest point) is 1 / ‖w‖, and the full road width is 2 / ‖w‖.

Analogy: two colonies of houses (the two classes). The municipality builds the widest road that knocks down no house. The centre line of the road is the boundary; the houses touching the road's edges are the support vectors.

The maximum-margin classifier (hard margin)

Widest road = largest 2 / ‖w‖ = smallest ‖w‖. For neater maths:

minimise    ½ ‖w‖²
subject to  yᵢ (w · xᵢ + b) ≥ 1   for every training point i
  • With labels +1 / −1, y · f(x) is positive only when the prediction has the right sign, and ≥ 1 means "on the right side and outside the road". One rule covers both classes. This is why SVMs use ±1 labels, not 0/1.
  • The problem is convex: there is exactly one best road, no local minima.
  • This model is the maximum-margin classifier, also called the hard-margin SVM because no point may be inside the road or on the wrong side.
import numpy as np
from sklearn.svm import SVC
rng = np.random.default_rng(31)                            # the same 40 made-up mangoes
X = np.r_[rng.normal([7, 3], 0.8, (20, 2)), rng.normal([3, 7], 0.8, (20, 2))]
y = np.r_[np.ones(20), -np.ones(20)].astype(int)           # +1 ripe, -1 raw
svm = SVC(kernel="linear", C=1e6).fit(X, y)                # huge C: no mistakes allowed (hard margin)
w, b = svm.coef_[0], svm.intercept_[0]
print("w =", w.round(3), f" b = {b:.3f}")
print("margin width =", round(2 / np.linalg.norm(w), 3))
print("support vectors:", len(svm.support_), "of", len(X), "->", svm.support_)
print("w.x+b at the support vectors:", (X[svm.support_] @ w + b).round(3))
keep = svm.support_                                        # throw away every other mango
svm2 = SVC(kernel="linear", C=1e6).fit(X[keep], y[keep])
print("refit on support vectors only: w =", svm2.coef_[0].round(3), f" b = {svm2.intercept_[0]:.3f}")

Output:

w = [ 0.351 -0.349]  b = -0.061
margin width = 4.041
support vectors: 3 of 40 -> [22 35  6]
w.x+b at the support vectors: [-1. -1.  1.]
refit on support vectors only: w = [ 0.351 -0.349]  b = -0.060
  • SVC = support vector classifier. kernel="linear" = straight boundary (kernels are Episode 32). C=1e6 makes mistakes so expensive that this is practically a hard margin.
  • The road is 4.04 wide, wider than line A's 3.84 (2 × 1.92): no straight line gives a wider road.
  • Only 3 of 40 mangoes are support vectors, and w · x + b is exactly −1, −1, +1 at them: they sit on the road's edges.
  • Refitting on those 3 points alone gives the same road (b differs only in the third decimal). The other 37 mangoes never mattered.

Support vectors

  • Support vector: a training point on the edge of the road or inside it (or, with a soft margin, on the wrong side). These are the only points with a say in the final model.
  • svm.support_ (their row numbers), svm.support_vectors_ (the points), svm.n_support_ (how many per class).
  • Moving a point deep inside its class changes nothing; moving a support vector moves the road. So a single weird point near the boundary can matter a lot.
  • The prediction can be written using only the support vectors: f(x) = Σ αᵢ yᵢ (xᵢ · x) + b, where the weights αᵢ are zero for every other point. This "dual form" is what makes kernels possible (Episode 32).

Soft margin and slack variables

One odd ripe mango arrives that is still firm (sweetness 4.2, firmness 5.2). A hard margin must squeeze past it.

import numpy as np
from sklearn.svm import SVC
rng = np.random.default_rng(31)                            # the same 40 made-up mangoes
X = np.r_[rng.normal([7, 3], 0.8, (20, 2)), rng.normal([3, 7], 0.8, (20, 2))]
y = np.r_[np.ones(20), -np.ones(20)].astype(int)           # +1 ripe, -1 raw
X = np.r_[X, [[4.2, 5.2]]]; y = np.r_[y, 1]                # one odd ripe mango, close to the raw ones
Xn = np.r_[rng.normal([7, 3], 1.3, (500, 2)), rng.normal([3, 7], 1.3, (500, 2))]
yn = np.r_[np.ones(500), -np.ones(500)]
for C in [1e6, 1, 0.01]:
    s = SVC(kernel="linear", C=C).fit(X, y); w = s.coef_[0]
    print(f"C={C:<6g} road {2/np.linalg.norm(w):.2f}  SVs {len(s.support_):>2}"
          f"  train {s.score(X, y):.3f}  new {s.score(Xn, yn):.3f}")

Output:

C=1e+06  road 1.22  SVs  2  train 1.000  new 0.943
C=1      road 1.64  SVs  2  train 1.000  new 0.957
C=0.01   road 5.66  SVs 16  train 0.976  new 0.984
  • Hard margin (C = 10⁶): one mango shrinks the road from 4.04 to 1.22, and accuracy on new mangoes drops from 0.984 to 0.943.
  • Soft margin (C = 0.01): a 5.66-wide road, the odd mango on the wrong side (train 0.976 = 40/41), and new mangoes back to 0.984.

Hard margin vs soft margin

The soft-margin SVM:

minimise    ½ ‖w‖² + C · Σ ξᵢ
subject to  yᵢ (w · xᵢ + b) ≥ 1 − ξᵢ ,   ξᵢ ≥ 0
  • Slack variable ξᵢ (xi): how much point i is allowed to break the rule. ξ = 0: fine (outside the road). 0 < ξ < 1: inside the road but on the correct side. ξ > 1: on the wrong side (a training mistake).
  • C: the price of one unit of slack.
  • Big C → mistakes are expensive → narrow road, tries hard to classify every training point (risk: overfitting to noise). C → ∞ is the hard margin.
  • Small C → mistakes are cheap → wide road, some points ignored (risk: underfitting).
  • In regularisation terms, C is the inverse of the regularisation strength: small C = strong regularisation (same as C in LogisticRegression, Episode 23).
  • C is a hyperparameter: choose it with cross-validation. Try values on a log scale (0.01, 0.1, 1, 10, 100).

Hinge loss: the slack as a loss function

The smallest slack a point needs is max(0, 1 − y · f(x)). That is the hinge loss:

y · f(x) Where the point is Hinge loss
2 right side, outside the road 0
1 exactly on the road's edge 0
0.5 inside the road, right side 0.5
0 on the centre line 1
−1 wrong side 2

So the soft-margin SVM is just regularisation + loss:

loss(w, b) = ½ ‖w‖²  +  C · Σ max(0, 1 − yᵢ (w · xᵢ + b))

Plotted against y·f, the hinge loss is flat at 0 and then a straight ramp: the shape of a door hinge. At the corner (y·f = 1) there is no single slope, so we use a sub-gradient (the slope of either side). Analogy: a parking fine. Parked well off the road: no fine. On the road's edge: a small fine. Across the centre line: a big fine that grows with distance.

A linear SVM from scratch

import numpy as np
rng = np.random.default_rng(31)                            # the same 41 made-up mangoes (with the odd one)
X = np.r_[rng.normal([7, 3], 0.8, (20, 2)), rng.normal([3, 7], 0.8, (20, 2)), [[4.2, 5.2]]]
y = np.r_[np.ones(20), -np.ones(20), [1]]                  # labels must be +1 / -1
def hinge(f, y): return np.maximum(0, 1 - y * f)           # 0 if on the right side with room to spare
print("hinge of y*f = 2, 1, 0.5, 0, -1:", hinge(np.array([2, 1, .5, 0, -1.]), 1))
C, lr = 0.01, 0.01; w = np.zeros(2); b = 0.0
for step in range(20001):                                  # sub-gradient descent on the SVM objective
    f = X @ w + b; bad = y * f < 1                         # mangoes inside the margin or wrong side
    loss = 0.5 * w @ w + C * hinge(f, y).sum()
    w -= lr * (w - C * (y[bad, None] * X[bad]).sum(0))
    b -= lr * (-C * y[bad].sum())
    if step % 5000 == 0: print(f"step {step:>5}: loss {loss:.4f}  w {w.round(3)}  b {b:.3f}")
slack = hinge(X @ w + b, y)
print("mangoes with slack > 0:", np.sum(slack > 1e-6), " largest slack:", slack.max().round(2))

Output:

hinge of y*f = 2, 1, 0.5, 0, -1: [0.  0.  0.5 1.  2. ]
step     0: loss 0.4100  w [ 0.009 -0.008]  b 0.000
step  5000: loss 0.0976  w [ 0.246 -0.252]  b -0.015
step 10000: loss 0.0976  w [ 0.248 -0.252]  b -0.020
step 15000: loss 0.0976  w [ 0.248 -0.252]  b -0.023
step 20000: loss 0.0976  w [ 0.248 -0.252]  b -0.023
mangoes with slack > 0: 16  largest slack: 1.29
  • The gradient of ½‖w‖² is w (it shrinks w each step, widening the road). Each point with y·f < 1 adds −C · y · x to the gradient of w and −C · y to the gradient of b; points with room to spare add nothing.
  • Our w = [0.248, −0.252] matches SVC(kernel="linear", C=0.01), which gives [0.24797413 -0.25171629], b = −0.0306. Our b (−0.023) is close; sub-gradient descent keeps jittering at the hinge's corner, while scikit-learn solves the problem exactly.
  • 16 mangoes have slack > 0: the same 16 support vectors as SVC(C=0.01). The largest slack (1.29 > 1) is the odd mango, on the wrong side.

The 0/1 label mistake

Same code with the raw mangoes labelled 0 instead of −1:

import numpy as np
rng = np.random.default_rng(31)                            # the same 41 made-up mangoes (with the odd one)
X = np.r_[rng.normal([7, 3], 0.8, (20, 2)), rng.normal([3, 7], 0.8, (20, 2)), [[4.2, 5.2]]]
y = np.r_[np.ones(20), np.zeros(20), [1]]                  # MISTAKE: 0/1 labels
def hinge(f, y): return np.maximum(0, 1 - y * f)           # 0 if on the right side with room to spare
C, lr = 0.01, 0.01; w = np.zeros(2); b = 0.0
for step in range(20001):                                  # sub-gradient descent on the SVM objective
    f = X @ w + b; bad = y * f < 1
    w -= lr * (w - C * (y[bad, None] * X[bad]).sum(0))
    b -= lr * (-C * y[bad].sum())
print("w", w.round(3), " b", round(b, 3))
print("accuracy:", np.mean((X @ w + b > 0) == (y == 1)))

Output:

w [0.002 0.001]  b 0.994
accuracy: 0.5121951219512195

With y = 0, y · f is always 0, so every raw mango has hinge loss 1 whatever w is, and it never pushes the boundary. The model drifts to "everything is ripe" (21 of 41 correct). scikit-learn's SVMs accept any two labels and convert them internally; your own hinge-loss code must use ±1.

LinearSVC and scaling

SVC solves the problem exactly but its solver works with pairs of training points, so training time grows faster than the number of rows (roughly between n² and n³). For straight boundaries on big data, LinearSVC uses a different solver (liblinear) that scales to hundreds of thousands of rows.

import numpy as np, time
from sklearn.svm import SVC, LinearSVC
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
rng = np.random.default_rng(32); n = 20000                 # 20,000 made-up mangoes from the mandi
weight = rng.normal(250000, 60000, n)                      # weight in milligrams
sweet = rng.normal(5, 2, n)                                # sweetness score 0-10
y = np.where(weight / 50000 + sweet + rng.normal(0, 1, n) > 10, 1, -1)
X = np.c_[weight, sweet]; X_tr, y_tr, X_te, y_te = X[:15000], y[:15000], X[15000:], y[15000:]
raw = LinearSVC().fit(X_tr, y_tr)
print(f"LinearSVC, raw units:  test {raw.score(X_te, y_te):.3f}")
t = time.time(); fast = make_pipeline(StandardScaler(), LinearSVC()).fit(X_tr, y_tr)
print(f"LinearSVC, scaled:     test {fast.score(X_te, y_te):.3f}  fit {time.time()-t:.2f} s")
t = time.time(); slow = make_pipeline(StandardScaler(), SVC(kernel="linear")).fit(X_tr, y_tr)
print(f"SVC(kernel='linear'):  test {slow.score(X_te, y_te):.3f}  fit {time.time()-t:.2f} s")

Output (timings vary by machine):

LinearSVC, raw units:  test 0.501
LinearSVC, scaled:     test 0.869  fit 0.01 s
SVC(kernel='linear'):  test 0.870  fit 1.78 s
  • Unscaled: 0.501, a coin toss, with no error and no warning. The margin is a distance and the penalty ½‖w‖² treats every weight alike, so a feature measured in hundreds of thousands (milligrams) swamps one measured in units. Always scale features for SVMs.
  • Scaled, LinearSVC matches SVC(kernel="linear") (0.869 vs 0.870) and fits far faster; the gap grows with more rows.
SVC(kernel="linear") LinearSVC
Solver libsvm, exact dual problem liblinear
Speed on big data slow (pairs of points) fast
Kernels yes (Episode 32) straight boundaries only
Default loss hinge squared hinge (loss="squared_hinge")
Intercept not penalised penalised slightly (intercept_scaling)
Support vectors stored yes (support_vectors_) no
Probabilities no predict_proba no predict_proba

Scores, not probabilities. SVMs give decision_function(X), the signed score f(x) (sign = class, size = how far from the boundary). SVC().predict_proba raises AttributeError: This 'SVC' has no attribute 'predict_proba'. For probabilities, wrap the SVM in CalibratedClassifierCV (Platt scaling or isotonic regression, Episode 25). In scikit-learn 1.9 the old SVC(probability=True) prints FutureWarning: The probability parameter was deprecated in 1.9 and will be removed in version 1.11.

from sklearn.svm import SVC
s = SVC(kernel="linear").fit([[0, 0], [1, 1], [0, 1], [1, 0]], [0, 1, 0, 1])
print("decision_function([[2, 2]]) =", s.decision_function([[2, 2]]))
s.predict_proba([[2, 2]])

Output:

decision_function([[2, 2]]) = [3.]
Traceback (most recent call last):
  ... (library frames) ...
AttributeError: This 'SVC' has no attribute 'predict_proba'

SVMs at a glance

labels +1 / −1  →  f(x) = w · x + b
hard margin:  min ½‖w‖²            s.t. y·f ≥ 1            (no mistakes; one odd point ruins it)
soft margin:  min ½‖w‖² + C·Σξ     s.t. y·f ≥ 1 − ξ, ξ ≥ 0
          =   min ½‖w‖² + C·Σ max(0, 1 − y·f)              (regularisation + hinge loss)
road width = 2 / ‖w‖ · support vectors = points with y·f ≤ 1
big C → narrow road, fewer training mistakes · small C → wide road, more regularisation

Common mistakes

Mistake What you see Why Fix
Picking any separating line Line C: train 1.00, new 0.927 No room for error Maximum margin (SVM)
Hard margin on noisy data Road 4.04 → 1.22, new 0.984 → 0.943 One odd point drags the road Soft margin; tune C by CV
0/1 labels in your own hinge loss accuracy 0.512, "all ripe" y · f = 0 for every 0-labelled point Use +1 / −1
Unscaled features LinearSVC test 0.501 Distances dominated by the biggest unit make_pipeline(StandardScaler(), LinearSVC())
Expecting probabilities AttributeError: This 'SVC' has no attribute 'predict_proba' SVMs output scores decision_function, or CalibratedClassifierCV
SVC(probability=True) in new code FutureWarning: The probability parameter was deprecated in 1.9 ... Deprecated CalibratedClassifierCV(SVC(), ensemble=False)
SVC on 100,000+ rows Training takes very long Pairs of points LinearSVC or SGDClassifier(loss="hinge")

Mini project: an SVM on the breast cancer dataset

The breast cancer Wisconsin (diagnostic) dataset is bundled with scikit-learn: 569 samples, 30 numeric features computed from images of cell nuclei (radius, texture, perimeter, area, smoothness and so on), and two classes, malignant and benign. Its features have very different ranges (mean area runs from 143.5 to 2501.0; mean smoothness from 0.053 to 0.163, according to the dataset's own description), which makes it a good test of scaling.

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import LinearSVC
X, y = load_breast_cancer(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, stratify=y, random_state=0)
print("train", X_tr.shape, " test", X_te.shape)
for C in [0.001, 0.01, 0.1, 1, 10]:
    model = make_pipeline(StandardScaler(), LinearSVC(C=C, max_iter=10000))
    cv = cross_val_score(model, X_tr, y_tr, cv=5).mean()
    print(f"C={C:<6} CV accuracy {cv:.3f}")
best = make_pipeline(StandardScaler(), LinearSVC(C=0.01, max_iter=10000)).fit(X_tr, y_tr)
print(f"C=0.01 (best CV) test accuracy {best.score(X_te, y_te):.3f}")
raw = LinearSVC(C=0.01, max_iter=10000).fit(X_tr, y_tr)
print(f"no scaling: test accuracy {raw.score(X_te, y_te):.3f}")

Output:

train (426, 30)  test (143, 30)
C=0.001  CV accuracy 0.965
C=0.01   CV accuracy 0.988
C=0.1    CV accuracy 0.972
C=1      CV accuracy 0.958
C=10     CV accuracy 0.946
C=0.01 (best CV) test accuracy 0.958
no scaling: test accuracy 0.930
  • Cross-validation over C on a log scale picks the regularisation strength; the test set is used once at the end.
  • Compare scaled vs unscaled: the scaled pipeline is the safe default for every SVM.

Practice

1. Plot (or print) the road width 2/‖w‖ against C on the 41 mangoes for C = 0.001 … 1000.

Solution
import numpy as np
from sklearn.svm import SVC
rng = np.random.default_rng(31)
X = np.r_[rng.normal([7, 3], 0.8, (20, 2)), rng.normal([3, 7], 0.8, (20, 2)), [[4.2, 5.2]]]
y = np.r_[np.ones(20), -np.ones(20), [1]].astype(int)
for C in [0.001, 0.01, 0.1, 1, 10, 100, 1000]:
    s = SVC(kernel="linear", C=C).fit(X, y)
    print(f"C={C:<6} road width {2/np.linalg.norm(s.coef_[0]):6.2f}  support vectors {len(s.support_)}")

Output:

C=0.001  road width  17.19  support vectors 40
C=0.01   road width   5.66  support vectors 16
C=0.1    road width   4.12  support vectors 5
C=1      road width   1.64  support vectors 2
C=10     road width   1.22  support vectors 2
C=100    road width   1.22  support vectors 2
C=1000   road width   1.22  support vectors 2

As C grows, mistakes get more expensive, the road narrows and fewer points are support vectors. Past a point, the hard-margin solution is reached and nothing changes.

2. Move a mango that is NOT a support vector. Does the road change? Then move a support vector.

Solution
import numpy as np
from sklearn.svm import SVC
rng = np.random.default_rng(31)
X = np.r_[rng.normal([7, 3], 0.8, (20, 2)), rng.normal([3, 7], 0.8, (20, 2))]
y = np.r_[np.ones(20), -np.ones(20)].astype(int)
base = SVC(kernel="linear", C=1e6).fit(X, y); sv = set(base.support_)
far = next(i for i in range(20) if i not in sv)            # a ripe mango that is not a support vector
X2 = X.copy(); X2[far] += [1.0, -1.0]                      # move it further from the road
X3 = X.copy(); X3[base.support_[0]] += [0.5, -0.5]         # move a support vector (a raw mango) towards the road
for name, Xs in [("original", X), ("moved non-SV", X2), ("moved SV", X3)]:
    s = SVC(kernel="linear", C=1e6).fit(Xs, y)
    print(f"{name:<13} w {s.coef_[0].round(3)}  b {s.intercept_[0]:.3f}")

Output:

original      w [ 0.351 -0.349]  b -0.061
moved non-SV  w [ 0.351 -0.349]  b -0.061
moved SV      w [ 0.474 -0.359]  b -0.820

Moving a non-support vector leaves w and b unchanged; moving a support vector changes them.

3. Choose C for the 41 mangoes with GridSearchCV (5-fold).

Solution
import numpy as np
from sklearn.svm import SVC
from sklearn.model_selection import GridSearchCV, StratifiedKFold
rng = np.random.default_rng(31)
X = np.r_[rng.normal([7, 3], 0.8, (20, 2)), rng.normal([3, 7], 0.8, (20, 2)), [[4.2, 5.2]]]
y = np.r_[np.ones(20), -np.ones(20), [1]].astype(int)
cv = StratifiedKFold(5, shuffle=True, random_state=0)
g = GridSearchCV(SVC(kernel="linear"), {"C": [0.001, 0.01, 0.1, 1, 10, 100]}, cv=cv).fit(X, y)
for C, m in zip(g.cv_results_["param_C"], g.cv_results_["mean_test_score"]):
    print(f"C={C:<6} CV {m:.3f}")
print("best:", g.best_params_)

Output:

C=0.001  CV 0.600
C=0.01   CV 0.975
C=0.1    CV 0.975
C=1.0    CV 0.975
C=10.0   CV 0.975
C=100.0  CV 0.975
best: {'C': 0.01}

With only 41 points CV scores are noisy and several C values tie; any of the tied values is reasonable, and the smallest tied C gives the widest road. (GridSearchCV is covered fully in Episode 42.)

4. Hinge loss vs log loss: compute both for y·f = −2, −1, 0, 1, 2, 3.

Solution
import numpy as np
m = np.array([-2, -1, 0, 1, 2, 3.0])                       # m = y * f(x), the "margin" of a point
print("hinge   ", np.maximum(0, 1 - m))
print("log loss", np.log(1 + np.exp(-m)).round(3))         # logistic regression's loss with ±1 labels

Output:

hinge    [3. 2. 1. 0. 0. 0.]
log loss [2.127 1.313 0.693 0.313 0.127 0.049]

Both punish wrong-side points roughly linearly. The difference: hinge loss is exactly 0 once y·f ≥ 1, so well-classified points stop mattering (that is why only support vectors count); log loss never reaches 0, so every point keeps a tiny pull.

5. By hand: w = [3, 4], b = −5. Distance of x = [3, 1] from the line? Road width if the closest points have f = ±1?

Solution

f(x) = 3·3 + 4·1 − 5 = 8. ‖w‖ = √(9 + 16) = 5. Distance = |8| / 5 = 1.6. Road width = 2 / ‖w‖ = 0.4.

Quick quiz

  1. Three lines all get 100% training accuracy. Which does an SVM pick? The one with the largest margin (widest road).
  2. What decides the position of the SVM boundary? Only the support vectors.
  3. Raw mango (y = −1), f(x) = 0.5. Hinge loss? max(0, 1 − (−1)(0.5)) = 1.5.
  4. Bigger C means a wider or narrower road? Narrower (mistakes cost more).
  5. Why scale features for an SVM? The margin is a distance; unscaled, the biggest unit dominates.

Interview questions

  1. What is a support vector machine? A classifier that finds the hyperplane with the maximum margin between classes; the solution depends only on the support vectors.
  2. What is the margin and why maximise it? The distance from the boundary to the closest training point; a larger margin leaves more room for new points to vary, which usually generalises better.
  3. What are support vectors? Training points on or inside the margin (or misclassified); all others have zero weight in the solution.
  4. Hard vs soft margin? Hard: no point may violate the margin (only for separable, noise-free data). Soft: slack variables allow violations, priced by C.
  5. What does C do? Trades margin width against training violations. Large C: few violations, narrow margin, risk of overfitting. Small C: wide margin, more violations, stronger regularisation.
  6. Write the hinge loss. Why is it called that? max(0, 1 − y·f(x)); its graph is flat then a straight ramp, like a hinge.
  7. Why do SVMs use ±1 labels? So that y·f(x) ≥ 1 expresses "correct side and outside the margin" for both classes in one inequality.
  8. SVC vs LinearSVC? SVC (libsvm) solves the exact dual, supports kernels, slow on large n; LinearSVC (liblinear) is linear only, much faster, uses squared hinge by default and penalises the intercept.
  9. How do you get probabilities from an SVM? Calibrate its decision function (Platt scaling / isotonic) with CalibratedClassifierCV.
  10. SVM vs logistic regression? Both linear; SVM uses hinge loss (ignores well-classified points, sparse support vectors), logistic regression uses log loss (gives calibrated-ish probabilities, every point contributes).

Glossary

Term Meaning (plain words) Example from this episode
Support vector machine (SVM) A model that separates classes with the widest possible gap sorting ripe vs raw mangoes
Hyperplane The flat boundary w · x + b = 0: a line in 2-D, a plane in 3-D the road's centre line
Weights w / bias b Direction and shift of the hyperplane w ≈ [0.35, −0.35], b = −0.061
Distance to the hyperplane |w · x + b| / ‖w‖ line A: closest mango 1.92 away
Margin Distance from the boundary to the closest training point 2.02 (half of 4.04)
Road width Full gap between the two edges, 2 / ‖w‖ 4.04
Maximum-margin classifier The separating hyperplane with the largest margin SVC(kernel="linear", C=1e6)
Support vectors Training points on the edge of the road, inside it, or on the wrong side 3 of 40 mangoes
Hard margin No point may be inside the road or on the wrong side C = 10⁶
Soft margin Points may break the rule, paying a penalty C = 0.01
Slack variable (ξ, "xi") How much one point breaks the margin rule the odd mango: 1.29
C Price of slack; inverse of regularisation strength C = 0.01 → road 5.66
Hinge loss max(0, 1 − y · f(x)) y·f = −1 → loss 2
Squared hinge Hinge loss squared; LinearSVC's default punishes big violations more
Sub-gradient A slope used where a function has a corner the hinge at y·f = 1
Dual form Writing the SVM using only dot products between points (weights αᵢ) basis of kernels (Episode 32)
SVC scikit-learn's SVM classifier (libsvm) SVC(kernel="linear")
LinearSVC Fast linear-only SVM (liblinear) 20,000 mangoes
decision_function The signed score f(x) s.decision_function([[2, 2]])
CalibratedClassifierCV Turns scores into calibrated probabilities instead of probability=True

Summary

  • Many lines can separate the training data; the ones close to the data fail on new data.
  • An SVM picks the hyperplane with the widest road (maximum margin): minimise ½‖w‖² subject to y·f ≥ 1, using ±1 labels.
  • Only the support vectors decide the road.
  • A soft margin adds slack priced by C; the slack is the hinge loss, so an SVM = ½‖w‖² + C·Σ hinge.
  • Small C = wide road, strong regularisation; big C = narrow road. Tune C with CV.
  • Scale features; use LinearSVC for big data; SVMs give scores, calibrate for probabilities.
  • Next episode: kernels, curved roads with the kernel trick.