SyntaxPilot
Courses/Machine Learning/Lesson 22

Machine Learning · Lesson 22 of 90 · 15 min

Logistic regression from scratch

binary classificationsigmoid (logistic) functionoddslog-odds (logit)decision boundarypredicted probabilitythresholdbinary cross-entropy (log loss)gradient of log lossimplementing it in NumPy

Machine Learning Full Course · Episode 22. Every code sample below was run with Python 3.13, NumPy 2.5.3, scikit-learn 1.9.1 and matplotlib 3.11.2; the output shown is the real output. All data is made up for teaching.

The big idea: to predict yes or no, keep the straight line from linear regression, call its output z, and squash it with the sigmoid into a probability between 0 and 1. The model is a straight line in log-odds. A threshold turns the probability into a decision, and the place where the decision flips is a straight decision boundary. Train it with log loss (binary cross-entropy), whose gradient is simply (p − y) · x.

The problem: yes-or-no questions

So far (Episodes 18-21) our models predicted numbers: rent, cups of chai. Many real questions have only two answers:

  • Will this student pass the test? (pass / fail)
  • Is this email spam? (spam / not spam)
  • Will this customer leave? (leave / stay)

Each possible answer is a class. Predicting one of two classes is binary classification ("binary" = two). We write the classes as 1 (the thing we are looking for, e.g. passed) and 0 (failed).

We want more than a yes/no: we want a probability, a number between 0 and 1 that says how likely the answer is 1. "This student studied 7 hours: 0.85 likely to pass."

Our made-up data: 40 students, the hours each studied, and whether they passed.

The obvious attempt: a straight line on 0/1 labels

We already know linear regression (Episode 18). Fit a line to the 0s and 1s and call anyone above 0.5 a pass:

import numpy as np
from sklearn.linear_model import LinearRegression
rng = np.random.default_rng(22)                       # made-up exam data
hours = rng.uniform(0, 10, 40).round(1)
passed = (rng.random(40) < 1 / (1 + np.exp(-0.9 * (hours - 5)))).astype(int)
def cut(h, y):
    lin = LinearRegression().fit(h.reshape(-1, 1), y)
    t = (0.5 - lin.intercept_) / lin.coef_[0]
    print(f"line crosses 0.5 at {t:.2f} h, accuracy {np.mean((h >= t) == y):.0%}")
    for x in [0, 10, 40]: print(f"   {x:>2} hours -> {lin.predict([[x]])[0]:+.2f}")
cut(hours, passed)
toppers = np.array([25, 28, 30, 32, 35, 40])          # 6 toppers who studied a lot, all pass
cut(np.r_[hours, toppers], np.r_[passed, [1] * 6])

Output:

line crosses 0.5 at 5.64 h, accuracy 88%
    0 hours -> -0.13
   10 hours -> +0.99
   40 hours -> +4.33
line crosses 0.5 at 9.04 h, accuracy 65%
    0 hours -> +0.25
   10 hours -> +0.53
   40 hours -> +1.36

Two failures:

  1. Not probabilities. 0 hours gives −0.13 (a negative chance) and 40 hours gives +4.33 (433%). A line goes on forever in both directions, so its output is unbounded.
  2. Easy points break it. Six toppers who studied 25-40 hours and all passed are the easiest cases, yet they tilt the line: the cut jumps from 5.64 to 9.04 hours and training accuracy falls from 88% to 65%. The squared error punishes the toppers for being far above 1, so the line bends to reach them.

We need a function that takes any number and always returns something between 0 and 1, and that goes flat for the obvious cases.

The sigmoid (logistic) function

Real-life example: your phone's volume. Press volume-up again and again: it stops at 100%. Press volume-down: it stops at 0%. In the middle every press changes a lot; near the ends, hardly anything. That S-shaped behaviour is the sigmoid, also called the logistic function:

σ(z) = 1 / (1 + e^(−z))        e ≈ 2.718 (np.exp)
z e^(−z) σ(z)
very negative (−6) huge (403) ≈ 0 (0.002)
0 1 1 / 2 = 0.5
very positive (+6) ≈ 0 (0.0025) ≈ 1 (0.998)

Logistic regression = put the line inside the sigmoid:

z = w · x + b          (the linear part, as in Episode 18)
p = σ(z)               p = P(y = 1 | x), the predicted probability

Despite the name, it is a classification model; "regression" refers to the linear model inside.

import numpy as np
def sigmoid(z): return 1 / (1 + np.exp(-z))
for z in [-6, -2, 0, 2, 6]:
    p = sigmoid(z)
    print(f"z {z:+d} -> p {p:.3f}   odds {p / (1 - p):7.3f}   log-odds {np.log(p / (1 - p)):+.1f}")

Output:

z -6 -> p 0.002   odds   0.002   log-odds -6.0
z -2 -> p 0.119   odds   0.135   log-odds -2.0
z +0 -> p 0.500   odds   1.000   log-odds +0.0
z +2 -> p 0.881   odds   7.389   log-odds +2.0
z +6 -> p 0.998   odds 403.429   log-odds +6.0

Odds and log-odds (logit)

Odds are another way to state a chance: the chance it happens divided by the chance it doesn't.

odds = p / (1 − p)

Example: the weather app says 80% chance of rain → odds = 0.8 / 0.2 = 4, "4 to 1": on four days like this it rains, on one it stays dry.

p odds
0.5 1 (fifty-fifty)
0.8 4
0.99 99
→ 1 → ∞
→ 0 → 0

The log-odds (natural log of the odds) are also called the logit:

logit(p) = ln(p / (1 − p))

The output above shows that the logit of the sigmoid's output is exactly z. The logit undoes the sigmoid. So logistic regression says:

the log-odds of class 1 are a straight line in the features: ln(p / (1 − p)) = w · x + b

Each extra unit of a feature adds w to the log-odds (and multiplies the odds by e^w, which Episode 23 calls the odds ratio).

Threshold and decision boundary

Real-life example: the weather app gives a probability ("60% chance of rain"); you need a decision. Your rule: carry an umbrella if the chance is at least 50%. On the day you wear a wedding outfit you carry it at 20%, because getting wet costs more.

  • A threshold is the cut-off that turns a probability into a class: predict 1 if p ≥ threshold. The usual default is 0.5, but it is your choice, set by the cost of each kind of mistake (Episode 25).
  • With threshold 0.5: σ(z) ≥ 0.5 exactly when z ≥ 0. So the prediction flips where w · x + b = 0.
  • The set of points where the prediction flips is the decision boundary:
  • 1 feature: one point, x = −b / w (a number of hours)
  • 2 features: a straight line w₁x₁ + w₂x₂ + b = 0
  • more features: a flat sheet called a hyperplane

Because the boundary is always straight (flat), logistic regression is a linear classifier.

Which loss? Why not MSE

A loss function (Episode 8) measures how bad the predictions are; gradient descent follows its slope (gradient). Could we keep using mean squared error?

Real-life example: one forecaster said "1% chance of rain" and it poured; another said "50%" and it poured. A fair score punishes the confident mistake much harder.

One student who really passed (y = 1); the model's probability goes from nearly right (0.9) to confidently wrong (0.0001). The slope is taken with respect to z:

import numpy as np
y = 1                                                  # this student really passed
for p in [0.9, 0.5, 0.1, 0.01, 0.0001]:                # the model's predicted probability
    mse, mse_grad = (p - y) ** 2, 2 * (p - y) * p * (1 - p)     # slope with respect to z
    ll, ll_grad = -np.log(p), p - y
    print(f"p {p:<6} MSE {mse:.3f} slope {mse_grad:+.4f} | log loss {ll:5.2f} slope {ll_grad:+.4f}")

Output:

p 0.9    MSE 0.010 slope -0.0180 | log loss  0.11 slope -0.1000
p 0.5    MSE 0.250 slope -0.2500 | log loss  0.69 slope -0.5000
p 0.1    MSE 0.810 slope -0.1620 | log loss  2.30 slope -0.9000
p 0.01   MSE 0.980 slope -0.0196 | log loss  4.61 slope -0.9900
p 0.0001 MSE 1.000 slope -0.0002 | log loss  9.21 slope -0.9999

With MSE, the confidently wrong prediction (p = 0.0001) has a slope of only −0.0002: the model is as wrong as it can be and barely learns. The reason: MSE's slope is 2(p − y) · σ'(z), and the sigmoid's slope σ'(z) = p(1 − p) is almost 0 at the flat ends. Log loss's slope is p − y = −0.9999: the more wrong, the harder it pushes. (MSE on a sigmoid is also non-convex, so gradient descent can stall.)

Binary cross-entropy (log loss)

loss for one example = −[ y · ln(p) + (1 − y) · ln(1 − p) ]
total loss           = the average over all n examples

Only one part is "switched on" at a time:

true label loss p = 0.9 p = 0.5 p = 0.01
y = 1 −ln(p) 0.11 0.69 4.61
y = 0 −ln(1 − p) 2.30 0.69 0.01

As the probability of the true class goes to 0, the loss goes to infinity. This loss is called binary cross-entropy or log loss; it is the negative log-likelihood of the Bernoulli model from Episode 12, and it is convex in w and b (one valley, no wrong minimum).

The gradient of log loss

Using the chain rule (Episode 6) and σ'(z) = σ(z)(1 − σ(z)), the sigmoid's slope cancels:

∂loss/∂z = p − y
∂loss/∂w = mean((p − y) · x)          many features:  Xᵀ(p − y) / n
∂loss/∂b = mean(p − y)

Prediction error × input: exactly the same shape as linear regression's MSE gradient (Episode 18).

Logistic regression from scratch (one feature)

The 40 students plus the 6 toppers, gradient descent with learning rate 0.05. The last two lines fit scikit-learn's LogisticRegression with C=np.inf (no penalty; Episode 23 explains C) for comparison:

import numpy as np
from sklearn.linear_model import LogisticRegression
rng = np.random.default_rng(22)                       # same exam data + 6 toppers
hours = rng.uniform(0, 10, 40).round(1)
passed = (rng.random(40) < 1 / (1 + np.exp(-0.9 * (hours - 5)))).astype(int)
x = np.r_[hours, [25, 28, 30, 32, 35, 40]]; y = np.r_[passed, [1] * 6]
def sigmoid(z): return 1 / (1 + np.exp(-z))
w, b, lr = 0.0, 0.0, 0.05
for step in range(20001):
    p = sigmoid(w * x + b)                            # predicted probabilities
    loss = -np.mean(y * np.log(p) + (1 - y) * np.log(1 - p))
    w -= lr * np.mean((p - y) * x)                    # gradient of log loss
    b -= lr * np.mean(p - y)
    if step % 5000 == 0: print(f"step {step:>5}  loss {loss:.4f}  w {w:.3f}  b {b:+.3f}")
print(f"boundary at {-b / w:.2f} hours, accuracy {np.mean((p >= 0.5) == y):.0%}")
sk = LogisticRegression(C=np.inf).fit(x.reshape(-1, 1), y)
print(f"scikit-learn boundary {-sk.intercept_[0] / sk.coef_[0, 0]:.2f} hours")

Output:

step     0  loss 0.6931  w 0.118  b -0.001
step  5000  loss 0.3905  w 0.666  b -3.823
step 10000  loss 0.3905  w 0.675  b -3.874
step 15000  loss 0.3905  w 0.675  b -3.875
step 20000  loss 0.3905  w 0.675  b -3.875
boundary at 5.74 hours, accuracy 87%
scikit-learn boundary 5.74 hours
  • Starting loss 0.6931 = −ln(0.5): a model that says 50-50 to everyone.
  • It converges to 0.3905; the boundary is 5.74 hours and accuracy is 87% with the toppers (the straight line fell to 65%). The toppers sit on the flat top of the S, already near p = 1, so their p − y is almost 0 and they barely pull.
  • scikit-learn finds the same boundary.

The log(0) trap

Same code with an impatient learning rate of 5:

import numpy as np
from sklearn.linear_model import LogisticRegression
rng = np.random.default_rng(22)                       # same exam data + 6 toppers
hours = rng.uniform(0, 10, 40).round(1)
passed = (rng.random(40) < 1 / (1 + np.exp(-0.9 * (hours - 5)))).astype(int)
x = np.r_[hours, [25, 28, 30, 32, 35, 40]]; y = np.r_[passed, [1] * 6]
def sigmoid(z): return 1 / (1 + np.exp(-z))
w, b, lr = 0.0, 0.0, 5
for step in range(3):
    p = sigmoid(w * x + b)                            # predicted probabilities
    loss = -np.mean(y * np.log(p) + (1 - y) * np.log(1 - p))
    w -= lr * np.mean((p - y) * x)                    # gradient of log loss
    b -= lr * np.mean(p - y)
    print(f"step {step:>5}  loss {loss:.4f}  w {w:.3f}  b {b:+.3f}")
print(f"boundary at {-b / w:.2f} hours, accuracy {np.mean((p >= 0.5) == y):.0%}")

Output:

step     0  loss 0.6931  w 11.815  b -0.109
step     1  loss nan  w 2.989  b -2.717
step     2  loss nan  w -5.480  b -4.873
boundary at -0.89 hours, accuracy 59%
main.py:11: RuntimeWarning: divide by zero encountered in log
  loss = -np.mean(y * np.log(p) + (1 - y) * np.log(1 - p))
main.py:11: RuntimeWarning: invalid value encountered in multiply
  loss = -np.mean(y * np.log(p) + (1 - y) * np.log(1 - p))

One giant step made w ≈ 11.8. For a 40-hour topper z ≈ 472, and σ(472) rounds to exactly 1.0 in 64-bit floating point, so ln(1 − p) = ln(0) = −∞ → RuntimeWarning: divide by zero encountered in log, then 0 · (−∞) = nan → invalid value encountered in multiply, and the loss is nan (not a number).

Fixes:

  1. a sensible learning rate on scaled features (Episode 16);
  2. clip p away from exactly 0 and 1 before the log: p = np.clip(p, 1e-15, 1 - 1e-15) (libraries do this for you);
  3. for very negative z, np.exp(-z) can overflow: clip z too, e.g. np.clip(z, -500, 500).

Two features, vectorised

Real problems have many features. 200 made-up students with hours studied and a mock-test score, both standardised. With a weight vector w, Xs @ w + b gives every student's z at once and the whole gradient is one line (vectorisation, Episode 2):

import numpy as np
from sklearn.linear_model import LogisticRegression
rng = np.random.default_rng(7); n = 200              # made-up: hours studied, mock-test score
X = np.c_[rng.uniform(0, 10, n), rng.uniform(20, 100, n)]
y = (rng.random(n) < 1 / (1 + np.exp(7.5 - 0.8 * X[:, 0] - 0.06 * X[:, 1]))).astype(int)  # 1 = passed
mu, sd = X.mean(0), X.std(0); Xs = (X - mu) / sd     # standardise (episode 16)
def sigmoid(z): return 1 / (1 + np.exp(-z))
w, b = np.zeros(2), 0.0
for step in range(3000):
    p = sigmoid(Xs @ w + b)
    w -= 0.5 * Xs.T @ (p - y) / n                    # the whole gradient in one line
    b -= 0.5 * np.mean(p - y)
sk = LogisticRegression(C=np.inf).fit(Xs, y)
print(f"scratch w {w.round(3)} b {b:.3f} | sklearn w {sk.coef_[0].round(3)} b {sk.intercept_[0]:.3f}")
print(f"accuracy {np.mean((p >= 0.5) == y):.1%}")
new = (np.array([6, 70]) - mu) / sd                   # 6 hours, 70 marks
print(f"new student: p = {sigmoid(new @ w + b):.3f}")
for t in [0.5, 0.8]: print(f"threshold {t}: {np.sum(p >= t)} of {n} predicted to pass")

Output:

scratch w [2.579 1.628] b 0.210 | sklearn w [2.579 1.629] b 0.210
accuracy 85.5%
new student: p = 0.850
threshold 0.5: 101 of 200 predicted to pass
threshold 0.8: 74 of 200 predicted to pass
  • Our weights match scikit-learn's to 3 decimals.
  • A new student with 6 hours and 70 marks: p = 0.850.
  • Same model, different threshold, different decisions: 101 students predicted to pass at 0.5, 74 at 0.8 (for example, a scholarship only for students we are very sure about).

To draw the decision boundary, add these lines after the program above (the boundary is where z = 0; we solve for the mock score and undo the standardisation):

import matplotlib.pyplot as plt
hx = np.array([0, 10]); hy = mu[1] - sd[1] * (w[0] * (hx - mu[0]) / sd[0] + b) / w[1]   # where z = 0
plt.figure(figsize=(9, 5)); plt.scatter(X[:, 0], X[:, 1], c=y, cmap="coolwarm", edgecolor="k")
plt.plot(hx, hy, "k--", label="decision boundary (p = 0.5)"); plt.ylim(20, 100); plt.legend()
plt.xlabel("hours studied"); plt.ylabel("mock-test score"); plt.savefig("boundary.png", dpi=110)

The picture: red dots passed, blue failed; the dashed line is p = 0.5. It is straight, as promised. A few dots sit on the wrong side: students the two features can't fully explain.

Decision boundary

Common mistakes

Mistake Real symptom Why Fix
linear regression for a yes/no target predictions −0.13 and 4.33; accuracy 88% → 65% after adding toppers a line is unbounded and pulled by far points logistic regression (sigmoid)
MSE loss with a sigmoid slope −0.0002 when p = 0.0001 and y = 1 flat ends of the S kill the gradient; non-convex log loss
learning rate too big RuntimeWarning: divide by zero encountered in log, loss nan p rounds to exactly 1.0 smaller lr, scale features, np.clip
continuous labels to a classifier ValueError: Unknown label type: continuous. Maybe you are trying to fit a classifier, which expects discrete classes on a regression target with continuous values. LogisticRegression needs classes labels 0/1 (or class names)
treating 0.5 as sacred decisions that ignore costs the threshold is a choice pick it from costs (Episode 25)
unscaled features with plain gradient descent very slow convergence or overflow one feature dominates the steps standardise first

The ValueError above is real; it comes from:

from sklearn.linear_model import LogisticRegression
LogisticRegression().fit([[1], [2], [3]], [0.2, 0.5, 0.9])

Mini project: your own LogisticRegression class

A tiny class with the scikit-learn style API (fit, predict_proba, predict, Episode 17), including scaling and clipping, checked against scikit-learn on a held-out test set:

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

class MyLogisticRegression:
    def __init__(self, lr=0.5, steps=3000):
        self.lr, self.steps = lr, steps
    def fit(self, X, y):
        self.mu, self.sd = X.mean(0), X.std(0)        # remember the training scaling
        Xs = (X - self.mu) / self.sd
        self.w, self.b = np.zeros(X.shape[1]), 0.0
        for _ in range(self.steps):
            p = self.predict_proba(X)
            self.w -= self.lr * Xs.T @ (p - y) / len(y)
            self.b -= self.lr * np.mean(p - y)
        return self
    def predict_proba(self, X):
        z = (X - self.mu) / self.sd @ self.w + self.b
        return 1 / (1 + np.exp(-np.clip(z, -500, 500)))  # clip: no overflow
    def predict(self, X, threshold=0.5):
        return (self.predict_proba(X) >= threshold).astype(int)

rng = np.random.default_rng(7); n = 200
X = np.c_[rng.uniform(0, 10, n), rng.uniform(20, 100, n)]
y = (rng.random(n) < 1 / (1 + np.exp(7.5 - 0.8 * X[:, 0] - 0.06 * X[:, 1]))).astype(int)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, random_state=0)
mine = MyLogisticRegression().fit(X_tr, y_tr)
sk = LogisticRegression(C=np.inf).fit(X_tr, y_tr)
print(f"mine:    test accuracy {np.mean(mine.predict(X_te) == y_te):.3f}")
print(f"sklearn: test accuracy {sk.score(X_te, y_te):.3f}")
print("largest probability difference:", np.abs(mine.predict_proba(X_te) - sk.predict_proba(X_te)[:, 1]).max().round(4))

Output:

mine:    test accuracy 0.850
sklearn: test accuracy 0.850
largest probability difference: 0.0008

Practice

  1. Train the one-feature model with the MSE slope instead of the log-loss slope. Compare the boundaries.
  2. Plot the sigmoid for z from −8 to 8.
  3. Add a third feature, attendance, that has nothing to do with passing. What weight does it get?
  4. In the two-feature model, which threshold predicts exactly 50 passes?
Solution 1
import numpy as np
rng = np.random.default_rng(22)
hours = rng.uniform(0, 10, 40).round(1)
passed = (rng.random(40) < 1 / (1 + np.exp(-0.9 * (hours - 5)))).astype(int)
x = np.r_[hours, [25, 28, 30, 32, 35, 40]]; y = np.r_[passed, [1] * 6]
def sigmoid(z): return 1 / (1 + np.exp(-z))
for name in ["log loss", "MSE"]:
    w, b = 0.0, 0.0
    for step in range(1, 20001):
        p = sigmoid(w * x + b)
        g = (p - y) if name == "log loss" else 2 * (p - y) * p * (1 - p)   # slope w.r.t. z
        w -= 0.05 * np.mean(g * x); b -= 0.05 * np.mean(g)
        if step in (500, 2000, 20000):
            print(f"{name:>8} step {step:>5}: boundary {-b / w:5.2f} h, accuracy {np.mean((p >= 0.5) == y):.0%}")

Output:

log loss step   500: boundary  5.43 h, accuracy 85%
log loss step  2000: boundary  5.70 h, accuracy 89%
log loss step 20000: boundary  5.74 h, accuracy 87%
     MSE step   500: boundary  5.02 h, accuracy 89%
     MSE step  2000: boundary  5.37 h, accuracy 85%
     MSE step 20000: boundary  5.33 h, accuracy 85%

The two losses settle on different boundaries (5.74 h vs 5.33 h): they are different objectives. Log loss is the one that matches a probability model (maximum likelihood) and is convex; with MSE the gradient fades for confident mistakes. Notice also that the best accuracy is not reached at the end of training: gradient descent minimises the loss, not the accuracy.

Solution 2
import numpy as np, matplotlib.pyplot as plt
z = np.linspace(-8, 8, 200)
plt.plot(z, 1 / (1 + np.exp(-z))); plt.axhline(0.5, ls="--", c="gray"); plt.axvline(0, ls="--", c="gray")
plt.xlabel("z"); plt.ylabel("sigmoid(z)"); plt.savefig("sigmoid.png")
print("saved sigmoid.png; sigmoid(-8) =", round(1 / (1 + np.exp(8)), 5), " sigmoid(8) =", round(1 / (1 + np.exp(-8)), 5))

Output:

saved sigmoid.png; sigmoid(-8) = 0.00034  sigmoid(8) = 0.99966
Solution 3
import numpy as np
from sklearn.linear_model import LogisticRegression
rng = np.random.default_rng(7); n = 200
X = np.c_[rng.uniform(0, 10, n), rng.uniform(20, 100, n)]
y = (rng.random(n) < 1 / (1 + np.exp(7.5 - 0.8 * X[:, 0] - 0.06 * X[:, 1]))).astype(int)
X3 = np.c_[X, rng.uniform(50, 100, n)]                # attendance %, made up and unrelated to y
Xs = (X3 - X3.mean(0)) / X3.std(0)
def sigmoid(z): return 1 / (1 + np.exp(-z))
w, b = np.zeros(3), 0.0
for step in range(3000):
    p = sigmoid(Xs @ w + b)
    w -= 0.5 * Xs.T @ (p - y) / n; b -= 0.5 * np.mean(p - y)
print("weights (hours, mock, attendance):", w.round(3))
print("scikit-learn:                     ", LogisticRegression(C=np.inf).fit(Xs, y).coef_[0].round(3))
print(f"accuracy {np.mean((p >= 0.5) == y):.1%}")

Output:

weights (hours, mock, attendance): [ 2.574  1.626 -0.058]
scikit-learn:                      [ 2.574  1.626 -0.058]
accuracy 86.0%

The attendance weight is close to 0 (−0.058): the model finds that an unrelated feature barely changes the log-odds. (With so little data it isn't exactly 0; Episode 21's L1 penalty would push it to 0.)

Solution 4

Add to the end of the two-feature program:

t = np.sort(p)[::-1][49]                              # the 50th highest probability
print(f"threshold {t:.3f}: {np.sum(p >= t)} predicted to pass")

Output:

scratch w [2.579 1.628] b 0.210 | sklearn w [2.579 1.629] b 0.210
accuracy 85.5%
new student: p = 0.850
threshold 0.5: 101 of 200 predicted to pass
threshold 0.8: 74 of 200 predicted to pass
threshold 0.912: 50 predicted to pass

(The first five lines of that output are the program's own; the last line is the answer.)

Quick quiz

z = 1.2 * hours - 6
p = sigmoid(z)
a student studied 5 hours: what is p?

A) 0.5 B) 0.0 C) 1.0 D) 6.0

Answer

A) 0.5. z = 1.2 × 5 − 6 = 0 and σ(0) = 1 / (1 + 1) = 0.5: the student sits exactly on the decision boundary.

Interview questions

  1. Why not use linear regression for classification? Its output is unbounded (not a probability), and far-away, easy points pull the line and move the decision cut.
  2. What does the sigmoid do? Maps any real z to (0, 1): σ(z) = 1 / (1 + e^(−z)); σ(0) = 0.5.
  3. What are odds and log-odds? odds = p / (1 − p); log-odds (logit) = ln(odds). Logistic regression models the log-odds as a linear function of the features.
  4. What is the decision boundary of logistic regression? The set where w · x + b = 0 (p = 0.5 for threshold 0.5): a point, line or hyperplane, so it is a linear classifier.
  5. Why log loss rather than MSE? Log loss is convex for logistic regression and gives large gradients for confident mistakes; MSE on a sigmoid is non-convex and its gradient vanishes when the model is confidently wrong.
  6. Derive the gradient. ∂loss/∂z = p − y, so ∂loss/∂w = Xᵀ(p − y)/n and ∂loss/∂b = mean(p − y).
  7. What happens on perfectly separable data without regularisation? The weights grow without limit (the loss keeps decreasing towards 0); a penalty (C in scikit-learn) keeps them finite.
  8. How do you avoid log(0)? Clip probabilities to [ε, 1 − ε], or compute the loss from z with a numerically stable formula (log-sum-exp), as libraries do.
  9. Is 0.5 always the right threshold? No; choose it from the costs of false positives and false negatives (Episode 25).

Glossary

Term Meaning Example
Class One possible answer of a classifier pass (1), fail (0)
Binary classification Predicting one of two classes spam / not spam
Probability A number from 0 to 1 saying how likely something is 0.85 likely to pass
Predicted probability The model's p = P(y = 1 given x) p = 0.850 for 6 h + 70 marks
Unbounded Can grow without limit in either direction a line: −0.13 … 4.33
Sigmoid (logistic) function σ(z) = 1 / (1 + e^(−z)), squashes any number into (0, 1) σ(0) = 0.5, σ(6) = 0.998
e (Euler's number) The constant ≈ 2.718; np.exp(z) computes e^z e^2 ≈ 7.389
Logistic regression p = σ(w · x + b): a linear model inside a sigmoid LogisticRegression()
Odds p / (1 − p) 80% rain → odds 4 ("4 to 1")
Log-odds (logit) ln(p / (1 − p)), the inverse of the sigmoid logit(0.881) = 2
Natural logarithm (ln) Logarithm base e; np.log ln(1) = 0
Threshold Cut-off that turns p into a class predict pass if p ≥ 0.5
Decision boundary Where the prediction flips (z = 0 at threshold 0.5) 5.74 hours; a line for 2 features
Hyperplane A flat boundary in many dimensions the boundary with 10 features
Linear classifier A classifier whose decision boundary is straight/flat logistic regression
Loss function Number measuring how bad predictions are (Episode 8) log loss 0.3905
Binary cross-entropy (log loss) −[y ln p + (1 − y) ln(1 − p)], averaged p = 0.01 for a pass → 4.61
Negative log-likelihood Minus the log of the probability the model gave the data (Episode 12) = log loss
Convex Bowl-shaped: one minimum, no wrong valleys log loss in w, b
Gradient of log loss Xᵀ(p − y)/n for w, mean(p − y) for b prediction error × input
Gradient descent Repeatedly step against the gradient (Episode 8) w -= lr * grad
Learning rate Step size of gradient descent 0.05 works, 5 blows up
Converge The loss stops changing 0.3905 from step 5,000
nan "Not a number": result of an invalid operation 0 × (−∞)
Clipping Forcing values into a range np.clip(p, 1e-15, 1 - 1e-15)
Standardisation Mean 0, standard deviation 1 per feature (Episode 16) (X - mu) / sd
Vectorisation Whole-array operations instead of loops (Episode 2) Xs.T @ (p - y) / n
C=np.inf scikit-learn setting for "no penalty" (Episode 23) matches our unpenalised model

Summary

  • Binary classification predicts one of two classes, coded 1 and 0.
  • A straight line fails: unbounded outputs (−0.13, 4.33) and a cut dragged by easy points (88% → 65%).
  • The sigmoid squashes z = w · x + b into a probability; the log-odds (logit) are exactly that line.
  • A threshold turns p into a decision; the decision boundary (z = 0) is straight.
  • Log loss punishes confident mistakes (MSE's slope was −0.0002, log loss's −0.9999); its gradient is (p − y) · x.
  • From scratch in NumPy we matched scikit-learn: boundary 5.74 h (87% even with the toppers), and weights [2.579 1.628] with two features.
  • Watch for log(0): clip probabilities and use sensible learning rates on scaled data.

Next episode: logistic regression in practice: scikit-learn's LogisticRegression, C, odds ratios, multiclass with softmax, and class weights.