SyntaxPilot
Courses/Machine Learning/Lesson 21

Machine Learning · Lesson 21 of 90 · 11 min

Regularisation

regularisationpenalty termL2 regularisation (ridge)L1 regularisation (lasso)elastic netregularisation strength (alpha, lambda)sparsityfeature selection by lassoregularisation pathscaling before regularising

Machine Learning Full Course · Episode 21. Every code sample below was run with Python 3.13, NumPy 2.5.3, pandas 3.0.5, scikit-learn 1.9.1 and matplotlib 3.11.2; the output shown is the real output. All data is made up for teaching, and the auction purse is a made-up illustration, not a real league's rule.

The big idea: overfit models hide huge weights. Add a penalty term to the loss that makes big weights expensive, and the model keeps its capacity but loses its wildness. Ridge (L2) shrinks every weight; lasso (L1) sets useless weights to exactly zero (sparsity, free feature selection); elastic net mixes both. Always scale first, and choose the regularisation strength α on validation data.

Overfit models have huge weights

Episode 20's chai stall: 20 days of cups sold against temperature, a degree-15 polynomial. The report function fits degree 15 with whatever model we pass and prints the biggest weight:

import numpy as np
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression, Ridge
from sklearn.pipeline import make_pipeline
from sklearn.metrics import mean_squared_error as mse
def truth(t): return 150 - 4 * t + 25 * np.sin(t / 5)                 # episode 20's chai
rng = np.random.default_rng(21)
def sample(n):
    t = rng.uniform(5, 40, n)
    return ((t - 22.5) / 17.5).reshape(-1, 1), truth(t) + rng.normal(0, 8, n)
X_tr, y_tr = sample(20); X_va, y_va = sample(200)
def report(name, model):
    m = make_pipeline(PolynomialFeatures(15), model).fit(X_tr, y_tr)
    w = np.abs(m[-1].coef_).max()                                     # biggest weight
    tr, va = mse(y_tr, m.predict(X_tr)), mse(y_va, m.predict(X_va))
    print(f"{name:<12} max|w| {w:8.1f} train {tr:6.1f} val {va:9.1f}")
report("no penalty", LinearRegression())
for a in [0.0001, 0.01, 0.1, 1, 100]:
    report(f"ridge {a}", Ridge(alpha=a))

Output:

no penalty   max|w|  76208.2 train    8.6 val 4280828.2
ridge 0.0001 max|w|    182.7 train   21.2 val    2852.9
ridge 0.01   max|w|    102.9 train   24.5 val     349.4
ridge 0.1    max|w|     78.7 train   43.3 val      83.9
ridge 1      max|w|     60.7 train  149.2 val     271.7
ridge 100    max|w|      4.9 train 1995.3 val    1977.1

With no penalty the biggest weight is 76,208 for data that never exceeds 200 cups. That's how a curve wiggles through every dot: giant positive and negative weights that almost cancel, and stop cancelling between the dots.

The penalty term

A cricket team at an auction with unlimited money buys everyone at any price. Give it a fixed purse and every rupee must earn its place: big money on stars, little on fringe players. For a model, the purse is a penalty term added to the loss:

ridge loss = MSE + α · (w1² + w2² + ... + wd²)
  • Penalty term: α · Σw², an extra cost for big weights. The bias b is not penalised (shifting the line up or down isn't complexity).
  • L2 penalty: the sum of squared weights. Linear regression with it is ridge regression.
  • Regularisation strength α: how expensive weights are. scikit-learn calls it alpha; many books call it λ (lambda). It is a hyperparameter (Episode 8): you choose it.
  • α = 0: plain linear regression. α → ∞: every weight → 0, the model predicts the mean.
  • The normal equation (Episode 18) changes by one term: θ = (XᵀX + αI)⁻¹Xᵀy (with the bias left out of the penalty).
  • Regularisation in general: anything added to training that discourages a complicated model.

(Exact scikit-learn objectives: Ridge minimises ‖y − Xw‖² + α‖w‖² (a sum of squares, not a mean); Lasso minimises (1/2n)‖y − Xw‖² + α‖w‖₁; ElasticNet minimises (1/2n)‖y − Xw‖² + α·r‖w‖₁ + ½α(1 − r)‖w‖². So the same α means different strengths in different estimators.)

From the run above:

α biggest weight train MSE validation MSE
0 (no penalty) 76,208.2 8.6 4,280,828.2 overfits
0.0001 182.7 21.2 2,852.9
0.01 102.9 24.5 349.4
0.1 78.7 43.3 83.9 best here
1 60.7 149.2 271.7
100 4.9 1,995.3 1,977.1 underfits

Too little α overfits, too much underfits: α is a capacity knob, chosen with a validation curve (Episode 20) or cross-validation (Episode 26).

Ridge from scratch

The ridge normal equation in NumPy matches scikit-learn's Ridge (the features are centred so the bias can be left unpenalised):

import numpy as np
from sklearn.linear_model import Ridge
rng = np.random.default_rng(0)
X = rng.normal(size=(50, 4)); y = X @ [3, -2, 0, 1] + 5 + rng.normal(0, 1, 50)
alpha = 10
Xc, yc = X - X.mean(axis=0), y - y.mean()                  # centre: bias handled separately
w = np.linalg.solve(Xc.T @ Xc + alpha * np.eye(4), Xc.T @ yc)
b = y.mean() - X.mean(axis=0) @ w
print("NumPy :", w.round(4), round(b, 4))
m = Ridge(alpha=alpha).fit(X, y)
print("Ridge :", m.coef_.round(4), round(m.intercept_, 4))

Output:

NumPy : [ 2.3664 -1.4904  0.1092  0.8902] 4.888
Ridge : [ 2.3664 -1.4904  0.1092  0.8902] 4.888

Lasso, sparsity and feature selection

Episode 19's flats (area, metro distance, age) plus 40 columns of pure random noise. Every model is scaled first:

import numpy as np, pandas as pd, matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression, Ridge, Lasso
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
rng = np.random.default_rng(19); n = 300                     # episode 19's made-up flats
df = pd.DataFrame({"area": rng.normal(950, 250, n).round(), "metro_km": rng.uniform(0.2, 8, n).round(1),
                   "age": rng.integers(0, 30, n)})
df["rent"] = (0.018*df.area - 1.2*df.metro_km - 0.15*df.age + 6 + rng.normal(0, 2.5, n)).round(1)  # ₹ thousands
for i in range(1, 41): df[f"junk{i}"] = rng.normal(0, 1, n).round(2)          # 40 columns of pure noise
X, y = df.drop(columns="rent"), df.rent
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, random_state=0)
for name, reg in [("linear", LinearRegression()), ("ridge", Ridge(alpha=30)), ("lasso", Lasso(alpha=0.3))]:
    m = make_pipeline(StandardScaler(), reg).fit(X_tr, y_tr); w = m[-1].coef_
    print(f"{name:6} real {np.round(w[:3], 2)} junk used {np.sum(w[3:] != 0):2} R2 {m.score(X_te, y_te):.3f}")

Output:

linear real [ 4.41 -2.58 -1.21] junk used 40 R2 0.845
ridge  real [ 3.83 -2.2  -1.04] junk used 40 R2 0.833
lasso  real [ 4.1  -2.24 -0.84] junk used  0 R2 0.866
  • Linear: all 40 junk weights in use, test R² 0.845.
  • Ridge (α = 30): still 40 junk weights (small, never zero), 0.833.
  • Lasso (α = 0.3): 0 junk weights, best test R² 0.866.
lasso loss = MSE + α · (|w1| + |w2| + ... + |wd|)
  • L1 penalty: the sum of absolute values. Lasso = least absolute shrinkage and selection operator.
  • Why exact zeros? A weight of 0.01 costs 0.0001 under L2 (ridge barely cares, so tiny weights survive) but 0.01 under L1 (still hurts). Under lasso a weight keeps paying all the way to 0, so unless a feature cuts the error by more than its cost, its weight becomes exactly 0.
  • Sparsity: many weights exactly zero (a sparse model: mostly empty).
  • Feature selection: choosing which features to keep. A zero weight means the feature is unused, so lasso selects features for free.
  • Geometric picture: the L1 constraint region is a diamond with corners on the axes, and the best point under that constraint often lands on a corner, where some weights are 0. The L2 region is a circle with no corners.

The regularisation path

Fit lasso for many α values and record each weight:

import numpy as np, pandas as pd, matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression, Ridge, Lasso
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
rng = np.random.default_rng(19); n = 300                     # episode 19's made-up flats
df = pd.DataFrame({"area": rng.normal(950, 250, n).round(), "metro_km": rng.uniform(0.2, 8, n).round(1),
                   "age": rng.integers(0, 30, n)})
df["rent"] = (0.018*df.area - 1.2*df.metro_km - 0.15*df.age + 6 + rng.normal(0, 2.5, n)).round(1)  # ₹ thousands
for i in range(1, 41): df[f"junk{i}"] = rng.normal(0, 1, n).round(2)          # 40 columns of pure noise
X, y = df.drop(columns="rent"), df.rent
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, random_state=0)
cols = ["area", "metro_km", "age", "junk1", "junk2", "junk3"]; alphas = np.logspace(-2, 1, 40)  # 0.01..10
W = np.array([make_pipeline(StandardScaler(), Lasso(alpha=a)).fit(X_tr[cols], y_tr)[-1].coef_ for a in alphas])
for j, c in enumerate(cols):                                       # first alpha where the weight is 0
    print(f"{c:>8} becomes 0 at alpha {alphas[np.argmax(W[:, j] == 0)]:.2f}")
plt.figure(figsize=(10, 4.8)); plt.xscale("log"); plt.plot(alphas, W, label=cols); plt.axhline(0, color="k")
plt.xlabel("alpha"); plt.ylabel("weight"); plt.legend(); plt.savefig("lasso_path.png", dpi=110)

Output:

    area becomes 0 at alpha 4.92
metro_km becomes 0 at alpha 2.89
     age becomes 0 at alpha 1.19
   junk1 becomes 0 at alpha 0.01
   junk2 becomes 0 at alpha 0.08
   junk3 becomes 0 at alpha 0.10

Lasso regularisation path

A plot of every weight against α is the regularisation path. Junk goes first (junk1 is already 0 at α = 0.01), then age (≈1.19), metro distance (≈2.89) and area last (≈4.92): the most useful feature survives longest. scikit-learn also has lasso_path and LassoCV, which compute the path efficiently and pick α by cross-validation.

Scale before regularising

A tax of ₹1 per digit on your bill would be silly: same money, different tax, just from how it's written. An unscaled penalty is the same mistake. Area in sq ft needs a weight of 0.019; in thousands of sq ft the same information needs 19, so a penalty on weights punishes it 1,000× more purely because of units:

import numpy as np, pandas as pd, matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression, Ridge, Lasso
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
rng = np.random.default_rng(19); n = 300                     # episode 19's made-up flats
df = pd.DataFrame({"area": rng.normal(950, 250, n).round(), "metro_km": rng.uniform(0.2, 8, n).round(1),
                   "age": rng.integers(0, 30, n)})
df["rent"] = (0.018*df.area - 1.2*df.metro_km - 0.15*df.age + 6 + rng.normal(0, 2.5, n)).round(1)  # ₹ thousands
for i in range(1, 41): df[f"junk{i}"] = rng.normal(0, 1, n).round(2)          # 40 columns of pure noise
X, y = df.drop(columns="rent"), df.rent
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, random_state=0)
cols = ["area", "metro_km", "age"]; A_tr, A_te = X_tr[cols], X_te[cols]
K_tr, K_te = (A.assign(area=A.area / 1000) for A in (A_tr, A_te))       # area in 1000s of sq ft
for name, a, b in [("sq ft", A_tr, A_te), ("1000 sq ft", K_tr, K_te)]:
    raw = Lasso(alpha=0.5).fit(a, y_tr)
    scaled = make_pipeline(StandardScaler(), Lasso(alpha=0.5)).fit(a, y_tr)
    print(f"{name:>10}: raw test R2 {raw.score(b, y_te):.3f}   scaled {scaled.score(b, y_te):.3f}")

Output:

     sq ft: raw test R2 0.875   scaled 0.847
1000 sq ft: raw test R2 0.728   scaled 0.847
  • Raw: 0.875 with sq ft, 0.728 with thousands of sq ft. Same flats, a worse model, only because of units.
  • Scaled: 0.847 both times. With standardisation (Episode 16: mean 0, standard deviation 1) the penalty is fair.
  • Put the scaler inside the pipeline, so it is fitted on training data only (no leakage, Episode 15).

Elastic net

Twin features (Episode 19's area and area in m²): lasso keeps one and zeroes the other, an almost arbitrary choice; ridge shares the weight but never drops junk. Mix them:

import numpy as np, pandas as pd, matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression, Ridge, Lasso
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
rng = np.random.default_rng(19); n = 300                     # episode 19's made-up flats
df = pd.DataFrame({"area": rng.normal(950, 250, n).round(), "metro_km": rng.uniform(0.2, 8, n).round(1),
                   "age": rng.integers(0, 30, n)})
df["rent"] = (0.018*df.area - 1.2*df.metro_km - 0.15*df.age + 6 + rng.normal(0, 2.5, n)).round(1)  # ₹ thousands
for i in range(1, 41): df[f"junk{i}"] = rng.normal(0, 1, n).round(2)          # 40 columns of pure noise
X, y = df.drop(columns="rent"), df.rent
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, random_state=0)
from sklearn.linear_model import ElasticNet
T_tr = X_tr.assign(area_sqm=(X_tr.area / 10.764).round(1))           # area's twin, in sq m
for name, reg in [("lasso", Lasso(alpha=0.5)), ("elastic net", ElasticNet(alpha=0.5, l1_ratio=0.9))]:
    w = dict(zip(T_tr.columns, make_pipeline(StandardScaler(), reg).fit(T_tr, y_tr)[-1].coef_))
    junk = sum(w[f"junk{i}"] != 0 for i in range(1, 41))
    print(f"{name:>11}: area {w['area']:.2f}  area_sqm {w['area_sqm']:.2f}  junk used {junk}")

Output:

      lasso: area 3.91  area_sqm 0.00  junk used 0
elastic net: area 1.93  area_sqm 1.93  junk used 0
  • Lasso: all of area's weight on one twin, 0 on the other.
  • Elastic net (α = 0.5, l1_ratio = 0.9): 1.93 and 1.93, shared evenly, and still 0 junk weights.
elastic net penalty = α · ( r · Σ|w|  +  (1 − r)/2 · Σw² )      r = l1_ratio
  • l1_ratio = 1: pure lasso; 0: pure ridge; in between: a blend.
  • L1 part: sparsity. L2 part: shares weight between correlated features (multicollinearity).
  • Two hyperparameters (α and l1_ratio); ElasticNetCV tunes both by cross-validation.

Ridge vs lasso vs elastic net

Ridge (L2) Lasso (L1) Elastic net
Penalty α·Σw² α·Σ|w| mix
Exact zeros no yes yes
Feature selection no yes yes
Correlated twins shares weight picks one shares weight
Good when most features useful few features matter many correlated features, some junk
Needs scaling yes yes yes

Common mistakes

Mistake Real symptom Why Fix
regularising unscaled features test R² 0.875 → 0.728 just from changing units the penalty depends on units StandardScaler in the pipeline
α too big α = 100: train 1,995.3, validation 1,977.1 underfitting validation curve / cross-validation for α
expecting ridge to drop features 40 junk weights not 0 L2 never gives exact zeros lasso or elastic net
reading meaning into which twin lasso kept area kept, area_sqm = 0 arbitrary between twins elastic net, or drop a twin
scaling all the data before splitting optimistic score leakage (Episode 15) fit the scaler on train only (pipeline)
comparing α across estimators same α, very different effect Ridge uses a sum, Lasso a mean (1/2n) tune α per estimator
lasso with too few iterations ConvergenceWarning: Objective did not converge coordinate descent stopped early scale features, raise max_iter

Mini project: choose α automatically

RidgeCV and LassoCV try many α values with cross-validation (Episode 26) and keep the best:

import numpy as np, pandas as pd, matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression, Ridge, Lasso
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
rng = np.random.default_rng(19); n = 300                     # episode 19's made-up flats
df = pd.DataFrame({"area": rng.normal(950, 250, n).round(), "metro_km": rng.uniform(0.2, 8, n).round(1),
                   "age": rng.integers(0, 30, n)})
df["rent"] = (0.018*df.area - 1.2*df.metro_km - 0.15*df.age + 6 + rng.normal(0, 2.5, n)).round(1)  # ₹ thousands
for i in range(1, 41): df[f"junk{i}"] = rng.normal(0, 1, n).round(2)          # 40 columns of pure noise
X, y = df.drop(columns="rent"), df.rent
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, random_state=0)
from sklearn.linear_model import RidgeCV, LassoCV
alphas = np.logspace(-3, 3, 30)
ridge = make_pipeline(StandardScaler(), RidgeCV(alphas=alphas)).fit(X_tr, y_tr)
lasso = make_pipeline(StandardScaler(), LassoCV(cv=5, random_state=0)).fit(X_tr, y_tr)
print("RidgeCV  alpha", round(ridge[-1].alpha_, 3), " test R2", round(ridge.score(X_te, y_te), 3))
print("LassoCV  alpha", round(lasso[-1].alpha_, 3), " test R2", round(lasso.score(X_te, y_te), 3),
      " features kept:", int(np.sum(lasso[-1].coef_ != 0)), "of", X.shape[1])
print("kept:", list(X.columns[lasso[-1].coef_ != 0]))

Output:

RidgeCV  alpha 8.532  test R2 0.845
LassoCV  alpha 0.221  test R2 0.871  features kept: 7 of 43
kept: ['area', 'metro_km', 'age', 'junk4', 'junk23', 'junk28', 'junk35']

Cross-validation picks the α that predicts best, not the sparsest model: LassoCV chose a slightly smaller α than our 0.3, so a few junk columns survive with tiny weights, and its test R² is a touch higher. If you want fewer features, choose a larger α on purpose (or the largest α within one standard error of the best).

Practice

  1. Draw a validation curve for ridge's α on the chai data. Which α is best?
  2. Add 100 junk columns instead of 40. How many does lasso (α = 0.3) keep?
  3. Implement ridge with the normal equation for the degree-15 chai model and compare with Ridge.
  4. Run elastic net on the twins with l1_ratio 0.1, 0.5 and 0.9. How many junk weights survive?
Solution 1
import numpy as np
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import Ridge
from sklearn.pipeline import make_pipeline
from sklearn.metrics import mean_squared_error as mse
def truth(t): return 150 - 4 * t + 25 * np.sin(t / 5)
rng = np.random.default_rng(21)
def sample(n):
    t = rng.uniform(5, 40, n)
    return ((t - 22.5) / 17.5).reshape(-1, 1), truth(t) + rng.normal(0, 8, n)
X_tr, y_tr = sample(20); X_va, y_va = sample(200)
best = None
for a in np.logspace(-4, 2, 13):
    m = make_pipeline(PolynomialFeatures(15), Ridge(alpha=a)).fit(X_tr, y_tr)
    va = mse(y_va, m.predict(X_va)); best = min(best or (va, a), (va, a))
    print(f"alpha {a:9.4f}: train {mse(y_tr, m.predict(X_tr)):7.1f}  validation {va:8.1f}")
print("best alpha", round(best[1], 4), "validation", round(best[0], 1))

Output:

alpha    0.0001: train    21.2  validation   2852.9
alpha    0.0003: train    22.1  validation   1944.2
alpha    0.0010: train    22.8  validation   1093.5
alpha    0.0032: train    23.3  validation    611.6
alpha    0.0100: train    24.5  validation    349.4
alpha    0.0316: train    29.2  validation    142.5
alpha    0.1000: train    43.3  validation     83.9
alpha    0.3162: train    75.1  validation    211.0
alpha    1.0000: train   149.2  validation    271.7
alpha    3.1623: train   374.4  validation    285.4
alpha   10.0000: train   876.9  validation    700.1
alpha   31.6228: train  1497.2  validation   1411.7
alpha  100.0000: train  1995.3  validation   1977.1
best alpha 0.1 validation 83.9

A U again: the validation error falls, bottoms out, then rises as α crushes the weights.

Solution 2
import numpy as np, pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression, Lasso
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
rng = np.random.default_rng(19); n = 300
df = pd.DataFrame({"area": rng.normal(950, 250, n).round(), "metro_km": rng.uniform(0.2, 8, n).round(1),
                   "age": rng.integers(0, 30, n)})
df["rent"] = (0.018 * df.area - 1.2 * df.metro_km - 0.15 * df.age + 6 + rng.normal(0, 2.5, n)).round(1)
junk = pd.DataFrame(rng.normal(0, 1, (n, 100)).round(2), columns=[f"junk{i}" for i in range(1, 101)])
df = pd.concat([df, junk], axis=1)
X, y = df.drop(columns="rent"), df.rent
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, random_state=0)
for name, reg in [("linear", LinearRegression()), ("lasso", Lasso(alpha=0.3))]:
    m = make_pipeline(StandardScaler(), reg).fit(X_tr, y_tr); w = m[-1].coef_
    print(f"{name}: real kept {np.sum(w[:3] != 0)}/3, junk kept {np.sum(w[3:] != 0)}/100, test R2 {m.score(X_te, y_te):.3f}")

Output:

linear: real kept 3/3, junk kept 100/100, test R2 0.789
lasso: real kept 3/3, junk kept 9/100, test R2 0.862

With 103 features and only 225 training rows, plain linear regression overfits badly; lasso keeps the real features and almost no junk.

Solution 3
import numpy as np
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import Ridge
def truth(t): return 150 - 4 * t + 25 * np.sin(t / 5)
rng = np.random.default_rng(21)
t = rng.uniform(5, 40, 20); x = ((t - 22.5) / 17.5).reshape(-1, 1)
y = truth(t) + rng.normal(0, 8, 20)
P = PolynomialFeatures(15, include_bias=False).fit_transform(x)      # 15 columns: x .. x^15
alpha = 0.1
Pc, yc = P - P.mean(axis=0), y - y.mean()
w = np.linalg.solve(Pc.T @ Pc + alpha * np.eye(15), Pc.T @ yc)
b = y.mean() - P.mean(axis=0) @ w
m = Ridge(alpha=alpha).fit(P, y)
print("max |difference| in weights:", float(np.abs(w - m.coef_).max()))
print("intercepts:", round(b, 4), round(m.intercept_, 4))

Output:

max |difference| in weights: 1.865174681370263e-13
intercepts: 43.6457 43.6457

The one-term change to the normal equation, + αI, is exactly what Ridge solves.

Solution 4
import numpy as np, pandas as pd, matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression, Ridge, Lasso
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
rng = np.random.default_rng(19); n = 300                     # episode 19's made-up flats
df = pd.DataFrame({"area": rng.normal(950, 250, n).round(), "metro_km": rng.uniform(0.2, 8, n).round(1),
                   "age": rng.integers(0, 30, n)})
df["rent"] = (0.018*df.area - 1.2*df.metro_km - 0.15*df.age + 6 + rng.normal(0, 2.5, n)).round(1)  # ₹ thousands
for i in range(1, 41): df[f"junk{i}"] = rng.normal(0, 1, n).round(2)          # 40 columns of pure noise
X, y = df.drop(columns="rent"), df.rent
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, random_state=0)
from sklearn.linear_model import ElasticNet
T_tr = X_tr.assign(area_sqm=(X_tr.area / 10.764).round(1))
for r in [0.1, 0.5, 0.9]:
    m = make_pipeline(StandardScaler(), ElasticNet(alpha=0.5, l1_ratio=r)).fit(T_tr, y_tr)
    w = dict(zip(T_tr.columns, m[-1].coef_))
    junk = sum(w[f"junk{i}"] != 0 for i in range(1, 41))
    print(f"l1_ratio {r}: area {w['area']:.2f}  area_sqm {w['area_sqm']:.2f}  junk kept {junk}")

Output:

l1_ratio 0.1: area 1.75  area_sqm 1.75  junk kept 29
l1_ratio 0.5: area 1.83  area_sqm 1.83  junk kept 3
l1_ratio 0.9: area 1.93  area_sqm 1.93  junk kept 0

More L1 (higher l1_ratio) means more junk dropped; the L2 part keeps the twins sharing the weight evenly.

Quick quiz

weights after fitting, 6 scaled features:
  [ 2.41, 0.0, -1.10, 0.0, 0.0, 0.37 ]
alpha = 0.3

Ridge, lasso, or plain linear?

A) Lasso B) Ridge C) Plain linear regression D) Can't tell

Answer

A) Lasso. Exact zeros are the signature of an L1 penalty; ridge and plain linear regression essentially never produce weights of exactly 0.

Interview questions

  1. What is regularisation and why does it help? Adding a penalty (or constraint) that discourages complex models, trading a little bias for a large cut in variance.
  2. Ridge vs lasso? L2 (Σw²) shrinks all weights smoothly, never to 0, handles correlated features by sharing. L1 (Σ|w|) produces exact zeros: sparse models and feature selection; with correlated features it picks one.
  3. Why does L1 give exact zeros but L2 doesn't? L1's cost per unit of weight stays constant near 0 (non-smooth at 0; the diamond has corners on the axes); L2's cost vanishes near 0, so tiny weights are cheap.
  4. Why scale before regularising? The penalty acts on raw weights, whose size depends on feature units; without scaling, features are penalised unequally.
  5. How do you choose α? Validation curve or cross-validation (RidgeCV, LassoCV, GridSearchCV).
  6. What is elastic net and when would you use it? A mix of L1 and L2 (l1_ratio); many correlated features with some irrelevant ones, or more features than rows.
  7. Is the intercept regularised? No; penalising it would make predictions depend on where the target's zero is.
  8. How does ridge relate to the normal equation? θ = (XᵀX + αI)⁻¹Xᵀy; adding αI also makes the matrix invertible when features are collinear.
  9. Bayesian view? Ridge = MAP with a Gaussian prior on weights; lasso = MAP with a Laplace prior (Episode 12).

Glossary

Term Meaning Example
Regularisation Anything added to training that discourages complexity a penalty on weights
Penalty term Extra loss for big weights α·Σw²
Regularisation strength (α, λ) How expensive weights are alpha=0.1
Hyperparameter A setting you choose, not learned α, l1_ratio
L2 penalty Sum of squared weights 0.01² = 0.0001
Ridge regression Linear regression + L2 penalty Ridge(alpha=0.1)
L1 penalty Sum of absolute weights |0.01| = 0.01
Lasso regression Linear regression + L1 penalty Lasso(alpha=0.3)
Sparsity / sparse model Many weights exactly 0 40 junk weights = 0
Feature selection Choosing which features to keep lasso drops junk
Regularisation path Each weight plotted against α junk → 0 first, area last
Elastic net L1 + L2 mixed ElasticNet(alpha=0.5, l1_ratio=0.9)
l1_ratio The L1 share in elastic net 1 = lasso, 0 = ridge
Standardisation Mean 0, standard deviation 1 per feature StandardScaler
Shrinkage Pulling weights toward 0 76,208 → 78.7
Identity matrix (I) Square matrix with 1s on the diagonal (Episode 4) αI in the normal equation
RidgeCV, LassoCV Pick α by cross-validation alpha_ attribute
Multicollinearity Features that copy each other (Episode 19) area and area_sqm

Summary

  • Overfit models hide huge weights (76,208 for a 200-cup target).
  • Penalty term: loss = MSE + α·penalty. Ridge (L2) shrinks every weight; with α = 0.1 validation MSE fell from 4,280,828 to 83.9; α = 100 underfits.
  • Lasso (L1) sets useless weights to exactly zero: sparsity and feature selection (40 junk columns → 0, test R² 0.866).
  • The regularisation path shows features dropping out in order of usefulness.
  • Scale first: unscaled lasso changed from 0.875 to 0.728 just by changing units; scaled stayed 0.847.
  • Elastic net shares weight between twins and still drops junk.

Next episode: logistic regression from scratch: predicting yes or no with the sigmoid, odds, log-odds, the decision boundary and log loss.