Machine Learning Full Course · Episode 22. Every code sample below was run with Python 3.13, NumPy 2.5.3, scikit-learn 1.9.1 and matplotlib 3.11.2; the output shown is the real output. All data is made up for teaching.
The big idea: to predict yes or no, keep the straight line from linear regression, call its output z, and squash it with the sigmoid into a probability between 0 and 1. The model is a straight line in log-odds. A threshold turns the probability into a decision, and the place where the decision flips is a straight decision boundary. Train it with log loss (binary cross-entropy), whose gradient is simply (p − y) · x.
The problem: yes-or-no questions
So far (Episodes 18-21) our models predicted numbers: rent, cups of chai. Many real questions have only two answers:
- Will this student pass the test? (pass / fail)
- Is this email spam? (spam / not spam)
- Will this customer leave? (leave / stay)
Each possible answer is a class. Predicting one of two classes is binary classification ("binary" = two). We write the classes as 1 (the thing we are looking for, e.g. passed) and 0 (failed).
We want more than a yes/no: we want a probability, a number between 0 and 1 that says how likely the answer is 1. "This student studied 7 hours: 0.85 likely to pass."
Our made-up data: 40 students, the hours each studied, and whether they passed.
The obvious attempt: a straight line on 0/1 labels
We already know linear regression (Episode 18). Fit a line to the 0s and 1s and call anyone above 0.5 a pass:
import numpy as np
from sklearn.linear_model import LinearRegression
rng = np.random.default_rng(22) # made-up exam data
hours = rng.uniform(0, 10, 40).round(1)
passed = (rng.random(40) < 1 / (1 + np.exp(-0.9 * (hours - 5)))).astype(int)
def cut(h, y):
lin = LinearRegression().fit(h.reshape(-1, 1), y)
t = (0.5 - lin.intercept_) / lin.coef_[0]
print(f"line crosses 0.5 at {t:.2f} h, accuracy {np.mean((h >= t) == y):.0%}")
for x in [0, 10, 40]: print(f" {x:>2} hours -> {lin.predict([[x]])[0]:+.2f}")
cut(hours, passed)
toppers = np.array([25, 28, 30, 32, 35, 40]) # 6 toppers who studied a lot, all pass
cut(np.r_[hours, toppers], np.r_[passed, [1] * 6])
Output:
line crosses 0.5 at 5.64 h, accuracy 88%
0 hours -> -0.13
10 hours -> +0.99
40 hours -> +4.33
line crosses 0.5 at 9.04 h, accuracy 65%
0 hours -> +0.25
10 hours -> +0.53
40 hours -> +1.36
Two failures:
- Not probabilities. 0 hours gives −0.13 (a negative chance) and 40 hours gives +4.33 (433%). A line goes on forever in both directions, so its output is unbounded.
- Easy points break it. Six toppers who studied 25-40 hours and all passed are the easiest cases, yet they tilt the line: the cut jumps from 5.64 to 9.04 hours and training accuracy falls from 88% to 65%. The squared error punishes the toppers for being far above 1, so the line bends to reach them.
We need a function that takes any number and always returns something between 0 and 1, and that goes flat for the obvious cases.
The sigmoid (logistic) function
Real-life example: your phone's volume. Press volume-up again and again: it stops at 100%. Press volume-down: it stops at 0%. In the middle every press changes a lot; near the ends, hardly anything. That S-shaped behaviour is the sigmoid, also called the logistic function:
σ(z) = 1 / (1 + e^(−z)) e ≈ 2.718 (np.exp)
| z | e^(−z) | σ(z) |
|---|---|---|
| very negative (−6) | huge (403) | ≈ 0 (0.002) |
| 0 | 1 | 1 / 2 = 0.5 |
| very positive (+6) | ≈ 0 (0.0025) | ≈ 1 (0.998) |
Logistic regression = put the line inside the sigmoid:
z = w · x + b (the linear part, as in Episode 18)
p = σ(z) p = P(y = 1 | x), the predicted probability
Despite the name, it is a classification model; "regression" refers to the linear model inside.
import numpy as np
def sigmoid(z): return 1 / (1 + np.exp(-z))
for z in [-6, -2, 0, 2, 6]:
p = sigmoid(z)
print(f"z {z:+d} -> p {p:.3f} odds {p / (1 - p):7.3f} log-odds {np.log(p / (1 - p)):+.1f}")
Output:
z -6 -> p 0.002 odds 0.002 log-odds -6.0
z -2 -> p 0.119 odds 0.135 log-odds -2.0
z +0 -> p 0.500 odds 1.000 log-odds +0.0
z +2 -> p 0.881 odds 7.389 log-odds +2.0
z +6 -> p 0.998 odds 403.429 log-odds +6.0
Odds and log-odds (logit)
Odds are another way to state a chance: the chance it happens divided by the chance it doesn't.
odds = p / (1 − p)
Example: the weather app says 80% chance of rain → odds = 0.8 / 0.2 = 4, "4 to 1": on four days like this it rains, on one it stays dry.
| p | odds |
|---|---|
| 0.5 | 1 (fifty-fifty) |
| 0.8 | 4 |
| 0.99 | 99 |
| → 1 | → ∞ |
| → 0 | → 0 |
The log-odds (natural log of the odds) are also called the logit:
logit(p) = ln(p / (1 − p))
The output above shows that the logit of the sigmoid's output is exactly z. The logit undoes the sigmoid. So logistic regression says:
the log-odds of class 1 are a straight line in the features: ln(p / (1 − p)) = w · x + b
Each extra unit of a feature adds w to the log-odds (and multiplies the odds by e^w, which Episode 23 calls the odds ratio).
Threshold and decision boundary
Real-life example: the weather app gives a probability ("60% chance of rain"); you need a decision. Your rule: carry an umbrella if the chance is at least 50%. On the day you wear a wedding outfit you carry it at 20%, because getting wet costs more.
- A threshold is the cut-off that turns a probability into a class: predict 1 if p ≥ threshold. The usual default is 0.5, but it is your choice, set by the cost of each kind of mistake (Episode 25).
- With threshold 0.5: σ(z) ≥ 0.5 exactly when z ≥ 0. So the prediction flips where w · x + b = 0.
- The set of points where the prediction flips is the decision boundary:
- 1 feature: one point, x = −b / w (a number of hours)
- 2 features: a straight line w₁x₁ + w₂x₂ + b = 0
- more features: a flat sheet called a hyperplane
Because the boundary is always straight (flat), logistic regression is a linear classifier.
Which loss? Why not MSE
A loss function (Episode 8) measures how bad the predictions are; gradient descent follows its slope (gradient). Could we keep using mean squared error?
Real-life example: one forecaster said "1% chance of rain" and it poured; another said "50%" and it poured. A fair score punishes the confident mistake much harder.
One student who really passed (y = 1); the model's probability goes from nearly right (0.9) to confidently wrong (0.0001). The slope is taken with respect to z:
import numpy as np
y = 1 # this student really passed
for p in [0.9, 0.5, 0.1, 0.01, 0.0001]: # the model's predicted probability
mse, mse_grad = (p - y) ** 2, 2 * (p - y) * p * (1 - p) # slope with respect to z
ll, ll_grad = -np.log(p), p - y
print(f"p {p:<6} MSE {mse:.3f} slope {mse_grad:+.4f} | log loss {ll:5.2f} slope {ll_grad:+.4f}")
Output:
p 0.9 MSE 0.010 slope -0.0180 | log loss 0.11 slope -0.1000
p 0.5 MSE 0.250 slope -0.2500 | log loss 0.69 slope -0.5000
p 0.1 MSE 0.810 slope -0.1620 | log loss 2.30 slope -0.9000
p 0.01 MSE 0.980 slope -0.0196 | log loss 4.61 slope -0.9900
p 0.0001 MSE 1.000 slope -0.0002 | log loss 9.21 slope -0.9999
With MSE, the confidently wrong prediction (p = 0.0001) has a slope of only −0.0002: the model is as wrong as it can be and barely learns. The reason: MSE's slope is 2(p − y) · σ'(z), and the sigmoid's slope σ'(z) = p(1 − p) is almost 0 at the flat ends. Log loss's slope is p − y = −0.9999: the more wrong, the harder it pushes. (MSE on a sigmoid is also non-convex, so gradient descent can stall.)
Binary cross-entropy (log loss)
loss for one example = −[ y · ln(p) + (1 − y) · ln(1 − p) ]
total loss = the average over all n examples
Only one part is "switched on" at a time:
| true label | loss | p = 0.9 | p = 0.5 | p = 0.01 |
|---|---|---|---|---|
| y = 1 | −ln(p) | 0.11 | 0.69 | 4.61 |
| y = 0 | −ln(1 − p) | 2.30 | 0.69 | 0.01 |
As the probability of the true class goes to 0, the loss goes to infinity. This loss is called binary cross-entropy or log loss; it is the negative log-likelihood of the Bernoulli model from Episode 12, and it is convex in w and b (one valley, no wrong minimum).
The gradient of log loss
Using the chain rule (Episode 6) and σ'(z) = σ(z)(1 − σ(z)), the sigmoid's slope cancels:
∂loss/∂z = p − y
∂loss/∂w = mean((p − y) · x) many features: Xᵀ(p − y) / n
∂loss/∂b = mean(p − y)
Prediction error × input: exactly the same shape as linear regression's MSE gradient (Episode 18).
Logistic regression from scratch (one feature)
The 40 students plus the 6 toppers, gradient descent with learning rate 0.05. The last two lines fit scikit-learn's LogisticRegression with C=np.inf (no penalty; Episode 23 explains C) for comparison:
import numpy as np
from sklearn.linear_model import LogisticRegression
rng = np.random.default_rng(22) # same exam data + 6 toppers
hours = rng.uniform(0, 10, 40).round(1)
passed = (rng.random(40) < 1 / (1 + np.exp(-0.9 * (hours - 5)))).astype(int)
x = np.r_[hours, [25, 28, 30, 32, 35, 40]]; y = np.r_[passed, [1] * 6]
def sigmoid(z): return 1 / (1 + np.exp(-z))
w, b, lr = 0.0, 0.0, 0.05
for step in range(20001):
p = sigmoid(w * x + b) # predicted probabilities
loss = -np.mean(y * np.log(p) + (1 - y) * np.log(1 - p))
w -= lr * np.mean((p - y) * x) # gradient of log loss
b -= lr * np.mean(p - y)
if step % 5000 == 0: print(f"step {step:>5} loss {loss:.4f} w {w:.3f} b {b:+.3f}")
print(f"boundary at {-b / w:.2f} hours, accuracy {np.mean((p >= 0.5) == y):.0%}")
sk = LogisticRegression(C=np.inf).fit(x.reshape(-1, 1), y)
print(f"scikit-learn boundary {-sk.intercept_[0] / sk.coef_[0, 0]:.2f} hours")
Output:
step 0 loss 0.6931 w 0.118 b -0.001
step 5000 loss 0.3905 w 0.666 b -3.823
step 10000 loss 0.3905 w 0.675 b -3.874
step 15000 loss 0.3905 w 0.675 b -3.875
step 20000 loss 0.3905 w 0.675 b -3.875
boundary at 5.74 hours, accuracy 87%
scikit-learn boundary 5.74 hours
- Starting loss 0.6931 = −ln(0.5): a model that says 50-50 to everyone.
- It converges to 0.3905; the boundary is 5.74 hours and accuracy is 87% with the toppers (the straight line fell to 65%). The toppers sit on the flat top of the S, already near p = 1, so their p − y is almost 0 and they barely pull.
- scikit-learn finds the same boundary.
The log(0) trap
Same code with an impatient learning rate of 5:
import numpy as np
from sklearn.linear_model import LogisticRegression
rng = np.random.default_rng(22) # same exam data + 6 toppers
hours = rng.uniform(0, 10, 40).round(1)
passed = (rng.random(40) < 1 / (1 + np.exp(-0.9 * (hours - 5)))).astype(int)
x = np.r_[hours, [25, 28, 30, 32, 35, 40]]; y = np.r_[passed, [1] * 6]
def sigmoid(z): return 1 / (1 + np.exp(-z))
w, b, lr = 0.0, 0.0, 5
for step in range(3):
p = sigmoid(w * x + b) # predicted probabilities
loss = -np.mean(y * np.log(p) + (1 - y) * np.log(1 - p))
w -= lr * np.mean((p - y) * x) # gradient of log loss
b -= lr * np.mean(p - y)
print(f"step {step:>5} loss {loss:.4f} w {w:.3f} b {b:+.3f}")
print(f"boundary at {-b / w:.2f} hours, accuracy {np.mean((p >= 0.5) == y):.0%}")
Output:
step 0 loss 0.6931 w 11.815 b -0.109
step 1 loss nan w 2.989 b -2.717
step 2 loss nan w -5.480 b -4.873
boundary at -0.89 hours, accuracy 59%
main.py:11: RuntimeWarning: divide by zero encountered in log
loss = -np.mean(y * np.log(p) + (1 - y) * np.log(1 - p))
main.py:11: RuntimeWarning: invalid value encountered in multiply
loss = -np.mean(y * np.log(p) + (1 - y) * np.log(1 - p))
One giant step made w ≈ 11.8. For a 40-hour topper z ≈ 472, and σ(472) rounds to exactly 1.0 in 64-bit floating point, so ln(1 − p) = ln(0) = −∞ → RuntimeWarning: divide by zero encountered in log, then 0 · (−∞) = nan → invalid value encountered in multiply, and the loss is nan (not a number).
Fixes:
- a sensible learning rate on scaled features (Episode 16);
- clip p away from exactly 0 and 1 before the log:
p = np.clip(p, 1e-15, 1 - 1e-15)(libraries do this for you); - for very negative z,
np.exp(-z)can overflow: clip z too, e.g.np.clip(z, -500, 500).
Two features, vectorised
Real problems have many features. 200 made-up students with hours studied and a mock-test score, both standardised. With a weight vector w, Xs @ w + b gives every student's z at once and the whole gradient is one line (vectorisation, Episode 2):
import numpy as np
from sklearn.linear_model import LogisticRegression
rng = np.random.default_rng(7); n = 200 # made-up: hours studied, mock-test score
X = np.c_[rng.uniform(0, 10, n), rng.uniform(20, 100, n)]
y = (rng.random(n) < 1 / (1 + np.exp(7.5 - 0.8 * X[:, 0] - 0.06 * X[:, 1]))).astype(int) # 1 = passed
mu, sd = X.mean(0), X.std(0); Xs = (X - mu) / sd # standardise (episode 16)
def sigmoid(z): return 1 / (1 + np.exp(-z))
w, b = np.zeros(2), 0.0
for step in range(3000):
p = sigmoid(Xs @ w + b)
w -= 0.5 * Xs.T @ (p - y) / n # the whole gradient in one line
b -= 0.5 * np.mean(p - y)
sk = LogisticRegression(C=np.inf).fit(Xs, y)
print(f"scratch w {w.round(3)} b {b:.3f} | sklearn w {sk.coef_[0].round(3)} b {sk.intercept_[0]:.3f}")
print(f"accuracy {np.mean((p >= 0.5) == y):.1%}")
new = (np.array([6, 70]) - mu) / sd # 6 hours, 70 marks
print(f"new student: p = {sigmoid(new @ w + b):.3f}")
for t in [0.5, 0.8]: print(f"threshold {t}: {np.sum(p >= t)} of {n} predicted to pass")
Output:
scratch w [2.579 1.628] b 0.210 | sklearn w [2.579 1.629] b 0.210
accuracy 85.5%
new student: p = 0.850
threshold 0.5: 101 of 200 predicted to pass
threshold 0.8: 74 of 200 predicted to pass
- Our weights match scikit-learn's to 3 decimals.
- A new student with 6 hours and 70 marks: p = 0.850.
- Same model, different threshold, different decisions: 101 students predicted to pass at 0.5, 74 at 0.8 (for example, a scholarship only for students we are very sure about).
To draw the decision boundary, add these lines after the program above (the boundary is where z = 0; we solve for the mock score and undo the standardisation):
import matplotlib.pyplot as plt
hx = np.array([0, 10]); hy = mu[1] - sd[1] * (w[0] * (hx - mu[0]) / sd[0] + b) / w[1] # where z = 0
plt.figure(figsize=(9, 5)); plt.scatter(X[:, 0], X[:, 1], c=y, cmap="coolwarm", edgecolor="k")
plt.plot(hx, hy, "k--", label="decision boundary (p = 0.5)"); plt.ylim(20, 100); plt.legend()
plt.xlabel("hours studied"); plt.ylabel("mock-test score"); plt.savefig("boundary.png", dpi=110)
The picture: red dots passed, blue failed; the dashed line is p = 0.5. It is straight, as promised. A few dots sit on the wrong side: students the two features can't fully explain.

Common mistakes
| Mistake | Real symptom | Why | Fix |
|---|---|---|---|
| linear regression for a yes/no target | predictions −0.13 and 4.33; accuracy 88% → 65% after adding toppers | a line is unbounded and pulled by far points | logistic regression (sigmoid) |
| MSE loss with a sigmoid | slope −0.0002 when p = 0.0001 and y = 1 | flat ends of the S kill the gradient; non-convex | log loss |
| learning rate too big | RuntimeWarning: divide by zero encountered in log, loss nan |
p rounds to exactly 1.0 | smaller lr, scale features, np.clip |
| continuous labels to a classifier | ValueError: Unknown label type: continuous. Maybe you are trying to fit a classifier, which expects discrete classes on a regression target with continuous values. |
LogisticRegression needs classes |
labels 0/1 (or class names) |
| treating 0.5 as sacred | decisions that ignore costs | the threshold is a choice | pick it from costs (Episode 25) |
| unscaled features with plain gradient descent | very slow convergence or overflow | one feature dominates the steps | standardise first |
The ValueError above is real; it comes from:
from sklearn.linear_model import LogisticRegression
LogisticRegression().fit([[1], [2], [3]], [0.2, 0.5, 0.9])
Mini project: your own LogisticRegression class
A tiny class with the scikit-learn style API (fit, predict_proba, predict, Episode 17), including scaling and clipping, checked against scikit-learn on a held-out test set:
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
class MyLogisticRegression:
def __init__(self, lr=0.5, steps=3000):
self.lr, self.steps = lr, steps
def fit(self, X, y):
self.mu, self.sd = X.mean(0), X.std(0) # remember the training scaling
Xs = (X - self.mu) / self.sd
self.w, self.b = np.zeros(X.shape[1]), 0.0
for _ in range(self.steps):
p = self.predict_proba(X)
self.w -= self.lr * Xs.T @ (p - y) / len(y)
self.b -= self.lr * np.mean(p - y)
return self
def predict_proba(self, X):
z = (X - self.mu) / self.sd @ self.w + self.b
return 1 / (1 + np.exp(-np.clip(z, -500, 500))) # clip: no overflow
def predict(self, X, threshold=0.5):
return (self.predict_proba(X) >= threshold).astype(int)
rng = np.random.default_rng(7); n = 200
X = np.c_[rng.uniform(0, 10, n), rng.uniform(20, 100, n)]
y = (rng.random(n) < 1 / (1 + np.exp(7.5 - 0.8 * X[:, 0] - 0.06 * X[:, 1]))).astype(int)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, random_state=0)
mine = MyLogisticRegression().fit(X_tr, y_tr)
sk = LogisticRegression(C=np.inf).fit(X_tr, y_tr)
print(f"mine: test accuracy {np.mean(mine.predict(X_te) == y_te):.3f}")
print(f"sklearn: test accuracy {sk.score(X_te, y_te):.3f}")
print("largest probability difference:", np.abs(mine.predict_proba(X_te) - sk.predict_proba(X_te)[:, 1]).max().round(4))
Output:
mine: test accuracy 0.850
sklearn: test accuracy 0.850
largest probability difference: 0.0008
Practice
- Train the one-feature model with the MSE slope instead of the log-loss slope. Compare the boundaries.
- Plot the sigmoid for z from −8 to 8.
- Add a third feature, attendance, that has nothing to do with passing. What weight does it get?
- In the two-feature model, which threshold predicts exactly 50 passes?
Solution 1
import numpy as np
rng = np.random.default_rng(22)
hours = rng.uniform(0, 10, 40).round(1)
passed = (rng.random(40) < 1 / (1 + np.exp(-0.9 * (hours - 5)))).astype(int)
x = np.r_[hours, [25, 28, 30, 32, 35, 40]]; y = np.r_[passed, [1] * 6]
def sigmoid(z): return 1 / (1 + np.exp(-z))
for name in ["log loss", "MSE"]:
w, b = 0.0, 0.0
for step in range(1, 20001):
p = sigmoid(w * x + b)
g = (p - y) if name == "log loss" else 2 * (p - y) * p * (1 - p) # slope w.r.t. z
w -= 0.05 * np.mean(g * x); b -= 0.05 * np.mean(g)
if step in (500, 2000, 20000):
print(f"{name:>8} step {step:>5}: boundary {-b / w:5.2f} h, accuracy {np.mean((p >= 0.5) == y):.0%}")
Output:
log loss step 500: boundary 5.43 h, accuracy 85%
log loss step 2000: boundary 5.70 h, accuracy 89%
log loss step 20000: boundary 5.74 h, accuracy 87%
MSE step 500: boundary 5.02 h, accuracy 89%
MSE step 2000: boundary 5.37 h, accuracy 85%
MSE step 20000: boundary 5.33 h, accuracy 85%
The two losses settle on different boundaries (5.74 h vs 5.33 h): they are different objectives. Log loss is the one that matches a probability model (maximum likelihood) and is convex; with MSE the gradient fades for confident mistakes. Notice also that the best accuracy is not reached at the end of training: gradient descent minimises the loss, not the accuracy.
Solution 2
import numpy as np, matplotlib.pyplot as plt
z = np.linspace(-8, 8, 200)
plt.plot(z, 1 / (1 + np.exp(-z))); plt.axhline(0.5, ls="--", c="gray"); plt.axvline(0, ls="--", c="gray")
plt.xlabel("z"); plt.ylabel("sigmoid(z)"); plt.savefig("sigmoid.png")
print("saved sigmoid.png; sigmoid(-8) =", round(1 / (1 + np.exp(8)), 5), " sigmoid(8) =", round(1 / (1 + np.exp(-8)), 5))
Output:
saved sigmoid.png; sigmoid(-8) = 0.00034 sigmoid(8) = 0.99966
Solution 3
import numpy as np
from sklearn.linear_model import LogisticRegression
rng = np.random.default_rng(7); n = 200
X = np.c_[rng.uniform(0, 10, n), rng.uniform(20, 100, n)]
y = (rng.random(n) < 1 / (1 + np.exp(7.5 - 0.8 * X[:, 0] - 0.06 * X[:, 1]))).astype(int)
X3 = np.c_[X, rng.uniform(50, 100, n)] # attendance %, made up and unrelated to y
Xs = (X3 - X3.mean(0)) / X3.std(0)
def sigmoid(z): return 1 / (1 + np.exp(-z))
w, b = np.zeros(3), 0.0
for step in range(3000):
p = sigmoid(Xs @ w + b)
w -= 0.5 * Xs.T @ (p - y) / n; b -= 0.5 * np.mean(p - y)
print("weights (hours, mock, attendance):", w.round(3))
print("scikit-learn: ", LogisticRegression(C=np.inf).fit(Xs, y).coef_[0].round(3))
print(f"accuracy {np.mean((p >= 0.5) == y):.1%}")
Output:
weights (hours, mock, attendance): [ 2.574 1.626 -0.058]
scikit-learn: [ 2.574 1.626 -0.058]
accuracy 86.0%
The attendance weight is close to 0 (−0.058): the model finds that an unrelated feature barely changes the log-odds. (With so little data it isn't exactly 0; Episode 21's L1 penalty would push it to 0.)
Solution 4
Add to the end of the two-feature program:
t = np.sort(p)[::-1][49] # the 50th highest probability
print(f"threshold {t:.3f}: {np.sum(p >= t)} predicted to pass")
Output:
scratch w [2.579 1.628] b 0.210 | sklearn w [2.579 1.629] b 0.210
accuracy 85.5%
new student: p = 0.850
threshold 0.5: 101 of 200 predicted to pass
threshold 0.8: 74 of 200 predicted to pass
threshold 0.912: 50 predicted to pass
(The first five lines of that output are the program's own; the last line is the answer.)
Quick quiz
z = 1.2 * hours - 6
p = sigmoid(z)
a student studied 5 hours: what is p?
A) 0.5 B) 0.0 C) 1.0 D) 6.0
Answer
A) 0.5. z = 1.2 × 5 − 6 = 0 and σ(0) = 1 / (1 + 1) = 0.5: the student sits exactly on the decision boundary.
Interview questions
- Why not use linear regression for classification? Its output is unbounded (not a probability), and far-away, easy points pull the line and move the decision cut.
- What does the sigmoid do? Maps any real z to (0, 1): σ(z) = 1 / (1 + e^(−z)); σ(0) = 0.5.
- What are odds and log-odds? odds = p / (1 − p); log-odds (logit) = ln(odds). Logistic regression models the log-odds as a linear function of the features.
- What is the decision boundary of logistic regression? The set where w · x + b = 0 (p = 0.5 for threshold 0.5): a point, line or hyperplane, so it is a linear classifier.
- Why log loss rather than MSE? Log loss is convex for logistic regression and gives large gradients for confident mistakes; MSE on a sigmoid is non-convex and its gradient vanishes when the model is confidently wrong.
- Derive the gradient. ∂loss/∂z = p − y, so ∂loss/∂w = Xᵀ(p − y)/n and ∂loss/∂b = mean(p − y).
- What happens on perfectly separable data without regularisation? The weights grow without limit (the loss keeps decreasing towards 0); a penalty (C in scikit-learn) keeps them finite.
- How do you avoid log(0)? Clip probabilities to [ε, 1 − ε], or compute the loss from z with a numerically stable formula (log-sum-exp), as libraries do.
- Is 0.5 always the right threshold? No; choose it from the costs of false positives and false negatives (Episode 25).
Glossary
| Term | Meaning | Example |
|---|---|---|
| Class | One possible answer of a classifier | pass (1), fail (0) |
| Binary classification | Predicting one of two classes | spam / not spam |
| Probability | A number from 0 to 1 saying how likely something is | 0.85 likely to pass |
| Predicted probability | The model's p = P(y = 1 given x) | p = 0.850 for 6 h + 70 marks |
| Unbounded | Can grow without limit in either direction | a line: −0.13 … 4.33 |
| Sigmoid (logistic) function | σ(z) = 1 / (1 + e^(−z)), squashes any number into (0, 1) | σ(0) = 0.5, σ(6) = 0.998 |
| e (Euler's number) | The constant ≈ 2.718; np.exp(z) computes e^z |
e^2 ≈ 7.389 |
| Logistic regression | p = σ(w · x + b): a linear model inside a sigmoid | LogisticRegression() |
| Odds | p / (1 − p) | 80% rain → odds 4 ("4 to 1") |
| Log-odds (logit) | ln(p / (1 − p)), the inverse of the sigmoid | logit(0.881) = 2 |
| Natural logarithm (ln) | Logarithm base e; np.log |
ln(1) = 0 |
| Threshold | Cut-off that turns p into a class | predict pass if p ≥ 0.5 |
| Decision boundary | Where the prediction flips (z = 0 at threshold 0.5) | 5.74 hours; a line for 2 features |
| Hyperplane | A flat boundary in many dimensions | the boundary with 10 features |
| Linear classifier | A classifier whose decision boundary is straight/flat | logistic regression |
| Loss function | Number measuring how bad predictions are (Episode 8) | log loss 0.3905 |
| Binary cross-entropy (log loss) | −[y ln p + (1 − y) ln(1 − p)], averaged | p = 0.01 for a pass → 4.61 |
| Negative log-likelihood | Minus the log of the probability the model gave the data (Episode 12) | = log loss |
| Convex | Bowl-shaped: one minimum, no wrong valleys | log loss in w, b |
| Gradient of log loss | Xᵀ(p − y)/n for w, mean(p − y) for b | prediction error × input |
| Gradient descent | Repeatedly step against the gradient (Episode 8) | w -= lr * grad |
| Learning rate | Step size of gradient descent | 0.05 works, 5 blows up |
| Converge | The loss stops changing | 0.3905 from step 5,000 |
| nan | "Not a number": result of an invalid operation | 0 × (−∞) |
| Clipping | Forcing values into a range | np.clip(p, 1e-15, 1 - 1e-15) |
| Standardisation | Mean 0, standard deviation 1 per feature (Episode 16) | (X - mu) / sd |
| Vectorisation | Whole-array operations instead of loops (Episode 2) | Xs.T @ (p - y) / n |
C=np.inf |
scikit-learn setting for "no penalty" (Episode 23) | matches our unpenalised model |
Summary
- Binary classification predicts one of two classes, coded 1 and 0.
- A straight line fails: unbounded outputs (−0.13, 4.33) and a cut dragged by easy points (88% → 65%).
- The sigmoid squashes z = w · x + b into a probability; the log-odds (logit) are exactly that line.
- A threshold turns p into a decision; the decision boundary (z = 0) is straight.
- Log loss punishes confident mistakes (MSE's slope was −0.0002, log loss's −0.9999); its gradient is (p − y) · x.
- From scratch in NumPy we matched scikit-learn: boundary 5.74 h (87% even with the toppers), and weights [2.579 1.628] with two features.
- Watch for log(0): clip probabilities and use sensible learning rates on scaled data.
Next episode: logistic regression in practice: scikit-learn's LogisticRegression, C, odds ratios, multiclass with softmax, and class weights.