SyntaxPilot
Courses/Machine Learning/Lesson 32
Video coming soonThis lesson’s video is being made

Start with the notes below; the video appears here as soon as it is on YouTube.

Machine Learning · Lesson 32 of 90

Kernels and SVMs in practice

feature mapskernel tricklinear kernelpolynomial kernelRBF (Gaussian) kernelgammasupport vector regression (SVR)epsilon-insensitive lossscaling for SVMswhen SVMs fit

Machine Learning Full Course · Episode 32. Every code sample below was run with Python 3.13, NumPy 2.5.3 and scikit-learn 1.9.1; the output shown is the real output. The homes and lassi sales are made up for teaching; the mini project uses the handwritten digits dataset bundled with scikit-learn.

The big idea: when no straight road separates the classes, add new features made from the old ones (a feature map) so that a straight road works in the new space; it becomes a curved road in the original space. Building those features explodes in size, but the SVM only ever needs dot products between points, and a kernel computes those dot products directly without building the features: the kernel trick. The polynomial and RBF (Gaussian) kernels are the common choices. gamma sets how far one point's influence reaches (big gamma overfits), C is still the price of mistakes, and features must be scaled. Support vector regression (SVR) uses the same machinery for numbers, with an ε-tube inside which errors are free.

Words you need first

  • SVM, hyperplane, margin, support vectors, C, hinge loss: Episode 31.
  • Dot product, L2 norm, distance: Episode 3.
  • Polynomial features: adding squares and products of features so a linear model can bend (Episode 19).
  • Normal (Gaussian) distribution, binomial, variance: Episode 10.
  • Overfitting / underfitting: memorising noise / missing the pattern (Episode 20).
  • Scaling, pipeline: Episodes 16-17. Cross-validation: Episode 26. MAE (mean absolute error): the average size of the errors (Episode 19).

The problem: no straight road works

Raju's sweet shop delivers to 400 made-up homes, recorded as km east and km north of the shop. Homes within about 3.5 km order (+1); farther ones don't (−1). The ordering homes form a round patch around the shop.

import numpy as np
from sklearn.svm import SVC
rng = np.random.default_rng(33); n = 400                   # 400 made-up homes around Raju's sweet shop
X = rng.uniform(-6, 6, (n, 2))                             # km east, km north of the shop
d = np.sqrt((X ** 2).sum(1))
y = np.where(d + rng.normal(0, 0.3, n) < 3.5, 1, -1)       # +1 = orders (lives close), -1 = doesn't
X_tr, y_tr, X_te, y_te = X[:300], y[:300], X[300:], y[300:]
lin = SVC(kernel="linear").fit(X_tr, y_tr)
print(f"straight road:          train {lin.score(X_tr, y_tr):.3f}  test {lin.score(X_te, y_te):.3f}")
print("homes it says will order:", (lin.predict(X_te) == 1).sum(), "of", len(X_te))
def phi(X): return np.c_[X, (X ** 2).sum(1)]               # feature map: add east^2 + north^2
lin3 = SVC(kernel="linear").fit(phi(X_tr), y_tr)
print(f"with x1^2 + x2^2 added: train {lin3.score(phi(X_tr), y_tr):.3f}  test {lin3.score(phi(X_te), y_te):.3f}")
w, b = lin3.coef_[0], lin3.intercept_[0]
print("weights:", w.round(2), " boundary radius:", np.sqrt(-b / w[2]).round(2), "km")

Output:

straight road:          train 0.730  test 0.770
homes it says will order: 0 of 100
with x1^2 + x2^2 added: train 0.960  test 0.960
weights: [ 0.04 -0.05 -0.47]  boundary radius: 3.48 km
  • The linear SVM looks like 0.77 accurate but predicts zero orders: it answers "no" for everyone, which is right for the ~73-77% of homes that really don't order. Accuracy hides it (the accuracy paradox, Episode 24).
  • Why it fails: a circle can't be cut out by one straight line.
  • Adding one feature, east² + north² (the squared distance), makes the classes separable by a flat cut: 0.960, and the boundary radius 3.48 km recovers the hidden 3.5 km rule.

Linear vs RBF kernel vs feature map

Feature maps

  • Feature map φ (phi): a function that turns each point into new features, e.g. φ(east, north) = (east, north, east² + north²).
  • A straight boundary in the new space is a curved boundary in the original space.
  • Same idea as polynomial features (Episode 19).
  • Problem: in real data you don't know which features to add, so you'd add all products up to some degree. The count explodes: with n features and degree d there are C(n + d, d) terms (see below).

The kernel trick

Key fact: solving an SVM (its "dual form") and making predictions only use the training points through dot products between pairs of points:

f(x) = Σ over support vectors of  αᵢ yᵢ (φ(xᵢ) · φ(x))  + b

The αᵢ are weights found by the solver (zero for non-support vectors).

  • Kernel: a function k(x, z) that returns φ(x) · φ(z) without computing φ.
  • Kernel trick: replace every dot product in the SVM with k(x, z). The SVM then behaves as if it worked in the big feature space, at the cost of the original small one.

Analogy: comparing two giant gift boxes of sweets. Unpacking every sweet (building φ) is slow; a formula that gives the same similarity from the outside (the kernel) is fast.

import numpy as np
from math import comb
x, z = np.array([2.0, 3.0]), np.array([1.0, 2.0])          # two homes (km east, km north)
def phi(v):                                                # degree-2 feature map: 6 new features
    a, b = v
    return np.array([a*a, b*b, np.sqrt(2)*a*b, np.sqrt(2)*a, np.sqrt(2)*b, 1])
print("phi(x) =", phi(x).round(3))
print("phi(x) . phi(z) =", round(phi(x) @ phi(z), 6))      # build features, then dot product
print("(x . z + 1)^2   =", (x @ z + 1) ** 2)               # kernel: no features built
for n_feat, deg in [(2, 2), (100, 2), (100, 5), (1000, 5)]:
    print(f"{n_feat:>5} features, degree {deg}: {comb(n_feat + deg, deg):>16,} mapped features")
gamma = 0.5                                                # RBF kernel: similarity from distance
for a, b in [(x, x), (x, x + 0.5), (x, z), (x, x + 10)]:
    print(f"rbf, distance {np.linalg.norm(a - b):5.2f}: {np.exp(-gamma * np.sum((a - b) ** 2)):.4f}")

Output:

phi(x) = [4.    9.    8.485 2.828 4.243 1.   ]
phi(x) . phi(z) = 81.0
(x . z + 1)^2   = 81.0
    2 features, degree 2:                6 mapped features
  100 features, degree 2:            5,151 mapped features
  100 features, degree 5:       96,560,646 mapped features
 1000 features, degree 5: 8,459,043,543,951 mapped features
rbf, distance  0.00: 1.0000
rbf, distance  0.71: 0.7788
rbf, distance  1.41: 0.3679
rbf, distance 14.14: 0.0000

Why (x · z + 1)² equals φ(x) · φ(z) for x = (a, b), z = (c, d):

(x · z + 1)² = (ac + bd + 1)²
             = a²c² + b²d² + 2abcd + 2ac + 2bd + 1
             = [a², b², √2ab, √2a, √2b, 1] · [c², d², √2cd, √2c, √2d, 1]  =  φ(x) · φ(z)

The √2 factors are exactly what makes the cross terms come out right.

  • With 100 features and degree 5, φ would have 96,560,646 entries per row; with 1,000 features, over 8 trillion. The kernel still costs one dot product of the original features plus a power.

The common kernels

Kernel Formula (scikit-learn) Boundary shapes Main knobs
Linear k(x, z) = x · z straight (Episode 31) C
Polynomial k(x, z) = (γ x · z + coef0)^degree circles, parabolas, curves of that degree degree, coef0, gamma, C
RBF (Gaussian) k(x, z) = exp(−γ ‖x − z‖²) any smooth shape gamma, C
Sigmoid k(x, z) = tanh(γ x · z + coef0) rarely used gamma, coef0
  • coef0 defaults to 0 in scikit-learn; with coef0 = 0 the polynomial kernel has only the highest-degree terms, so coef0=1 is often a better choice.
  • RBF = radial basis function: "radial" because it depends only on the distance between the points. It is a bell curve (Gaussian) around each support vector: similarity 1 for the same point, falling to 0 far away (1.0000 → 0.7788 → 0.3679 → 0.0000 above).
  • The RBF kernel corresponds to a feature map with infinitely many features; only the kernel can work with it.
  • SVC()'s default kernel is "rbf".
import numpy as np
from sklearn.svm import SVC
rng = np.random.default_rng(33); n = 400                   # the same 400 made-up homes
X = rng.uniform(-6, 6, (n, 2))                             # km east, km north of the shop
d = np.sqrt((X ** 2).sum(1))
y = np.where(d + rng.normal(0, 0.3, n) < 3.5, 1, -1)       # +1 = orders, -1 = doesn't
X_tr, y_tr, X_te, y_te = X[:300], y[:300], X[300:], y[300:]
for k in [SVC(kernel="poly", degree=2, coef0=1), SVC(kernel="rbf")]:
    k.fit(X_tr, y_tr)
    print(f"{k.kernel:<5}: train {k.score(X_tr, y_tr):.3f}  test {k.score(X_te, y_te):.3f}  SVs {len(k.support_)}")
for g in [0.01, 0.1, 1, 10, 100]:                          # gamma: how far one home's influence reaches
    k = SVC(kernel="rbf", gamma=g).fit(X_tr, y_tr)
    print(f"gamma={g:<5} train {k.score(X_tr, y_tr):.3f}  test {k.score(X_te, y_te):.3f}  SVs {len(k.support_):>3}")

Output:

poly : train 0.960  test 0.960  SVs 81
rbf  : train 0.953  test 0.950  SVs 83
gamma=0.01  train 0.947  test 0.930  SVs 143
gamma=0.1   train 0.957  test 0.930  SVs  74
gamma=1     train 0.970  test 0.920  SVs 156
gamma=10    train 0.990  test 0.820  SVs 287
gamma=100   train 1.000  test 0.770  SVs 300

What gamma does

gamma and C

  • gamma (γ): how far a single training point's influence reaches. Small γ = wide bumps, smooth boundary (can underfit). Big γ = narrow bumps, an island around each point (overfits: γ = 100 gives train 1.000, test 0.770, and all 300 points become support vectors).
  • C: the price of a margin violation (Episode 31). Big C also lets the boundary bend around single points, so C and γ interact.
  • Default gamma="scale" = 1 / (n_features × X.var()), where X.var() is the variance of all the input values together. Sensible only if features are on similar scales. (gamma="auto" = 1 / n_features.)
  • Tune C and γ together on a log grid with cross-validation.
small large
gamma smooth, may underfit wiggly islands, overfits
C wide margin, more violations narrow margin, fits training points

Scaling and tuning

import numpy as np
from sklearn.svm import SVC
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import GridSearchCV
rng = np.random.default_rng(33); n = 400                   # the same 400 made-up homes
X = rng.uniform(-6, 6, (n, 2))
d = np.sqrt((X ** 2).sum(1))
y = np.where(d + rng.normal(0, 0.3, n) < 3.5, 1, -1)
X = X * [1000, 1]                                          # east now in METRES, north still in km
X_tr, y_tr, X_te, y_te = X[:300], y[:300], X[300:], y[300:]
raw = SVC(kernel="rbf", gamma=0.1).fit(X_tr, y_tr)
print(f"unscaled: train {raw.score(X_tr, y_tr):.3f}  test {raw.score(X_te, y_te):.3f}")
pipe = make_pipeline(StandardScaler(), SVC(kernel="rbf"))
grid = {"svc__C": [0.1, 1, 10, 100], "svc__gamma": [0.01, 0.1, 1, 10]}
search = GridSearchCV(pipe, grid, cv=5).fit(X_tr, y_tr)
print("best:", search.best_params_, f" CV {search.best_score_:.3f}")
print(f"scaled + tuned: test {search.score(X_te, y_te):.3f}")

Output:

unscaled: train 0.997  test 0.780
best: {'svc__C': 100, 'svc__gamma': 0.1}  CV 0.960
scaled + tuned: test 0.960
  • With east in metres, distances are almost entirely about east, so each home becomes an island (train 0.997, test 0.780).
  • GridSearchCV: tries every combination in the grid (4 × 4 = 16), each with 5-fold cross-validation, and refits the best on all training data. Parameter names in a pipeline are <step>__<param>, e.g. svc__C. (Full treatment: Episode 42.)

Support vector regression (SVR)

Raju's lassi counter: predict glasses sold (a number) from temperature.

  • ε-tube (epsilon tube): a band of ±ε around the prediction. Errors inside it cost nothing.
  • ε-insensitive loss: max(0, |y − f(x)| − ε). Same flat-then-ramp shape as hinge loss.
  • Support vectors in SVR: points on or outside the tube; points inside have no effect.
  • SVR(kernel="rbf", C=..., gamma=..., epsilon=...); ε is in the units of the target (here, glasses), so scale or think about the target's size when choosing it.
import numpy as np
from sklearn.svm import SVR
from sklearn.linear_model import LinearRegression
rng = np.random.default_rng(34)                            # 200 made-up days at Raju's lassi counter
temp = np.sort(rng.uniform(10, 45, 200))                   # temperature, degrees C
glasses = 20 + 0.08 * (temp - 10) ** 2 + rng.normal(0, 5, 200)   # sales rise faster on hot days
X = ((temp - 27.5) / 10).reshape(-1, 1)                    # scaled by hand: centre 27.5, unit 10 C
X_tr, y_tr, X_te, y_te = X[::2], glasses[::2], X[1::2], glasses[1::2]
def eps_loss(err, eps): return np.maximum(0, np.abs(err) - eps)   # errors inside the tube are free
print("epsilon=5 loss of errors -8, -3, 0, 3, 8:", eps_loss(np.array([-8, -3, 0, 3, 8]), 5))
def mae(m): return np.mean(np.abs(m.predict(X_te) - y_te))
print(f"LinearRegression:        test MAE {mae(LinearRegression().fit(X_tr, y_tr)):.2f} glasses")
for eps in [0.5, 5, 15]:
    m = SVR(kernel="rbf", C=100, epsilon=eps).fit(X_tr, y_tr)
    print(f"SVR rbf, epsilon={eps:<4}:  test MAE {mae(m):.2f}  SVs {len(m.support_):>3} of 100")

Output:

epsilon=5 loss of errors -8, -3, 0, 3, 8: [3 0 0 0 3]
LinearRegression:        test MAE 8.51 glasses
SVR rbf, epsilon=0.5 :  test MAE 4.22  SVs  94 of 100
SVR rbf, epsilon=5   :  test MAE 4.05  SVs  36 of 100
SVR rbf, epsilon=15  :  test MAE 5.82  SVs   4 of 100
  • A straight line can't follow the curve (MAE 8.51). SVR with an RBF kernel can.
  • Small ε: almost every point is a support vector. ε = 5 (about the noise level): best error with 36 support vectors. ε = 15: tube too wide, real errors ignored.

The epsilon tube

When to use an SVM

Good fit Poor fit
Small to medium data (hundreds to tens of thousands of rows) Hundreds of thousands+ rows with a kernel: the kernel matrix is n × n (100,000 rows → 10 billion entries)
Many features, e.g. text word counts (linear kernel) Need probabilities out of the box
A clear margin; few support vectors Mixed tables of categories and numbers with little tuning time: try tree ensembles (Episodes 33-36)

For very large data with a linear boundary use LinearSVC or SGDClassifier(loss="hinge"); for approximate kernels on large data, scikit-learn offers Nystroem and RBFSampler (explicit approximate feature maps followed by a linear model).

Common mistakes

Mistake What you see Why Fix
Linear SVM on curved data test 0.770 but predicts 0 orders No line cuts a circle Feature map or kernel
Building huge feature maps by hand 96,560,646 features per row Grows like C(n + d, d) Kernel trick
Very large gamma train 1.000, test 0.770, 300/300 SVs An island per point Tune gamma with CV
Kernel on unscaled features metres: test 0.780 Distance = biggest unit StandardScaler in a pipeline
SVR tube too wide ε = 15: MAE 5.82 Ignores real errors Tune ε (start near the noise level)
Polynomial kernel with default coef0 = 0 Only highest-degree terms No lower-degree terms Set coef0=1
Wrong pipeline parameter name ValueError: Invalid parameter 'C' for estimator Pipeline(...) Pipeline params need the step prefix svc__C

The last mistake, reproduced:

from sklearn.svm import SVC
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import GridSearchCV
from sklearn.datasets import make_circles
X, y = make_circles(200, noise=0.1, factor=0.4, random_state=0)
GridSearchCV(make_pipeline(StandardScaler(), SVC()), {"C": [1, 10]}, cv=3).fit(X, y)

Output:

Traceback (most recent call last):
  ... (library frames) ...
ValueError: Invalid parameter 'C' for estimator Pipeline(steps=[('standardscaler', StandardScaler()), ('svc', SVC())]). Valid parameters are: ['memory', 'steps', 'transform_input', 'verbose'].

Mini project: handwritten digits with an RBF SVM

The digits dataset bundled with scikit-learn has 1,797 images of handwritten digits, each 8 × 8 pixels (64 features, values 0-16), with labels 0-9. SVMs handle many classes by training several binary SVMs (scikit-learn's SVC uses one-vs-one internally).

from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC, LinearSVC
X, y = load_digits(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, stratify=y, random_state=0)
print("train", X_tr.shape, " classes", sorted(set(y.tolist())))
lin = make_pipeline(StandardScaler(), LinearSVC(max_iter=10000)).fit(X_tr, y_tr)
print(f"linear SVM test {lin.score(X_te, y_te):.3f}")
grid = {"svc__C": [1, 10, 100], "svc__gamma": [0.001, 0.01, 0.1]}
search = GridSearchCV(make_pipeline(StandardScaler(), SVC(kernel="rbf")), grid, cv=5).fit(X_tr, y_tr)
print("best", search.best_params_, f" CV {search.best_score_:.3f}")
print(f"RBF SVM test {search.score(X_te, y_te):.3f}")
print("support vectors:", search.best_estimator_[-1].support_.size, "of", len(X_tr))

Output:

train (1347, 64)  classes [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
linear SVM test 0.951
best {'svc__C': 10, 'svc__gamma': 0.01}  CV 0.983
RBF SVM test 0.980
support vectors: 619 of 1347
  • The RBF kernel beats the linear SVM on this image data.
  • 619 of the 1,347 training images end up as support vectors: with ten classes and one-vs-one, many images sit near some pair's boundary.

Practice

1. Check (x · z + 1)² = φ(x) · φ(z) for three more random pairs.

Solution
import numpy as np
def phi(v):
    a, b = v
    return np.array([a*a, b*b, np.sqrt(2)*a*b, np.sqrt(2)*a, np.sqrt(2)*b, 1])
rng = np.random.default_rng(0)
for _ in range(3):
    x, z = rng.normal(size=2), rng.normal(size=2)
    print(f"{phi(x) @ phi(z):.6f}  {(x @ z + 1) ** 2:.6f}")

Output:

1.137769  1.137769
0.414668  0.414668
1.921898  1.921898

Equal every time (up to rounding).

2. Print a table of 5-fold CV accuracy over C × gamma on the 400 homes (scaled).

Solution
import numpy as np
from sklearn.svm import SVC
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import GridSearchCV
rng = np.random.default_rng(33); n = 400
X = rng.uniform(-6, 6, (n, 2)); d = np.sqrt((X ** 2).sum(1))
y = np.where(d + rng.normal(0, 0.3, n) < 3.5, 1, -1)
Cs, gs = [0.1, 1, 10, 100], [0.01, 0.1, 1, 10]
s = GridSearchCV(make_pipeline(StandardScaler(), SVC()), {"svc__C": Cs, "svc__gamma": gs}, cv=5).fit(X[:300], y[:300])
table = s.cv_results_["mean_test_score"].reshape(len(Cs), len(gs))
print("C \\ gamma " + "".join(f"{g:>7}" for g in gs))
for C, row in zip(Cs, table): print(f"{C:<9} " + "".join(f"{v:7.3f}" for v in row))

Output:

C \ gamma    0.01    0.1      1     10
0.1         0.730  0.730  0.887  0.747
1           0.730  0.937  0.947  0.940
10          0.730  0.953  0.953  0.923
100         0.940  0.960  0.937  0.903

Low C with low gamma underfits (too smooth); high gamma drops as it starts memorising. Good scores lie along a band of moderate C and gamma: the two interact.

3. SVR with a linear kernel on the lassi data. How does it compare?

Solution
import numpy as np
from sklearn.svm import SVR
rng = np.random.default_rng(34)
temp = np.sort(rng.uniform(10, 45, 200))
glasses = 20 + 0.08 * (temp - 10) ** 2 + rng.normal(0, 5, 200)
X = ((temp - 27.5) / 10).reshape(-1, 1)
X_tr, y_tr, X_te, y_te = X[::2], glasses[::2], X[1::2], glasses[1::2]
for k in ["linear", "rbf"]:
    m = SVR(kernel=k, C=100, epsilon=5).fit(X_tr, y_tr)
    print(f"{k:<6} test MAE {np.mean(np.abs(m.predict(X_te) - y_te)):.2f}")

Output:

linear test MAE 8.41
rbf    test MAE 4.05

A linear-kernel SVR is a straight line, so like linear regression it can't follow the curve; the RBF kernel can.

4. Time SVC on 2,000 vs 8,000 rows. How much slower is 4× the data?

Solution
import time
from sklearn.datasets import make_classification
from sklearn.svm import SVC
for n in [2000, 8000]:
    X, y = make_classification(n, 20, flip_y=0.1, random_state=0)
    t = time.time(); SVC().fit(X, y); print(f"{n} rows: {time.time() - t:.2f} s")

Output (timings vary by machine):

2000 rows: 0.09 s
8000 rows: 1.12 s

Four times the rows takes much more than four times as long: kernel SVM training grows faster than linearly with n.

5. By hand: RBF kernel with γ = 0.5 for two points at distance 2. Value?

Solution

exp(−0.5 × 2²) = exp(−2) ≈ 0.135.

Quick quiz

  1. x = [1, 2], z = [3, 0]. (x · z + 1)² = ? 16.
  2. Which kernel depends only on the distance between points? RBF (Gaussian).
  3. Train 1.000, test 0.770, every point a support vector. Raise or lower gamma? Lower.
  4. In SVR, which points are support vectors? Points on or outside the ε-tube.
  5. Why can't you skip scaling with an RBF kernel? It's built on distance; the biggest unit dominates.

Interview questions

  1. What is the kernel trick? Replacing dot products in an algorithm that only uses dot products (like the SVM dual) with a kernel k(x, z) = φ(x) · φ(z), so it works in a high- or infinite-dimensional feature space without computing it.
  2. Why can SVMs use kernels but plain gradient-descent linear models usually don't? The SVM dual and prediction depend on data only through dot products between points.
  3. Explain the RBF kernel and gamma. exp(−γ‖x − z‖²): similarity decays with distance; γ controls the width; large γ → narrow bumps → overfitting.
  4. How do C and gamma interact? Both control flexibility; high C and high γ overfit, low C and low γ underfit; tune them jointly on a log grid with CV.
  5. What is a valid kernel? One that equals a dot product in some feature space; equivalently its Gram (kernel) matrix is symmetric positive semi-definite for any set of points (Mercer's condition).
  6. What is SVR and the ε-insensitive loss? Regression that ignores errors smaller than ε and penalises the excess linearly; points inside the tube are not support vectors.
  7. Time and memory cost of kernel SVMs? Roughly between O(n²) and O(n³) time and an n × n kernel matrix, so they struggle beyond tens of thousands of rows.
  8. How do SVMs handle more than two classes? Several binary SVMs: one-vs-one (SVC) or one-vs-rest (LinearSVC).
  9. When would you choose a linear kernel? Many features relative to rows (e.g. text), or very large data.
  10. Why does the polynomial kernel need coef0? With coef0 = 0 it contains only the highest-degree terms; coef0 > 0 adds all lower-degree terms.

Glossary

Term Meaning (plain words) Example from this episode
Feature map φ (phi) A recipe that builds new features from old ones φ = (east, north, east² + north²)
Feature space The space of the new features 3-D space where a flat cut works
Kernel k(x, z) A function returning φ(x) · φ(z) without building φ (x · z + 1)² = 81
Kernel trick Swapping every dot product for a kernel curved roads at the cost of straight ones
Dual form The SVM written using only dot products between points f(x) = Σ αᵢ yᵢ k(xᵢ, x) + b
αᵢ (alpha) Solver weight of training point i; > 0 only for support vectors
Linear kernel k = x · z Episode 31's straight road
Polynomial kernel k = (γ x · z + coef0)^degree degree 2, coef0 1: test 0.960
coef0 Constant added inside the polynomial kernel default 0; we used 1
RBF / Gaussian kernel k = exp(−γ ‖x − z‖²), a bell curve on distance test 0.950
Radial basis function A function of distance from a centre the RBF kernel
gamma (γ) How far one point's influence reaches γ = 100 → islands
gamma="scale" Default: 1 / (n_features × X.var())
GridSearchCV Tries every combination of settings with CV best C 100, γ 0.1
Support vector regression (SVR) SVM for numbers lassi glasses vs temperature
ε-tube A band ±ε around the prediction with no loss ±5 glasses
ε-insensitive loss max(0, error
Kernel (Gram) matrix Table of k(xᵢ, xⱼ) for all pairs n × n numbers
One-vs-one Multiclass by a binary model for each pair of classes SVC on 10 digits

Summary

  • When no straight road works, a feature map makes one; the kernel trick computes its dot products without building it.
  • Polynomial kernels give curves of a given degree; RBF kernels give any smooth shape and are the default.
  • gamma = reach of each point (big = overfit), C = price of mistakes; scale, then tune both with CV.
  • SVR predicts numbers with an ε-tube: errors inside are free.
  • SVMs suit small/medium data and many features; for huge data use linear models or tree ensembles.
  • Next episode: ensembles: voting and bagging.