SyntaxPilot
Courses/Machine Learning/Lesson 85
Video coming soonThis lesson’s video is being made

Start with the notes below; the video appears here as soon as it is on YouTube.

Machine Learning · Lesson 85 of 90

Interview prep 1: ML theory questions

common theory questions (bias-variance, regularisation, metric choice, trees vs linear models, bagging vs boosting, gradient descent variants, overfitting fixes)explaining an algorithm clearlytypical traps

Machine Learning Full Course · Episode 85. Every program below was run on a Linux machine with Python 3.13, NumPy 2.5.3 and scikit-learn 1.9.1, and the output shown is the real output. All data is made up (random numbers with a fixed seed, so you get the same output).

The big idea: interviewers don't test whether you have read a definition; they test whether you can use it. So answer every theory question in five steps: define it in one plain sentence, give a small example, show the evidence (how you would detect or measure it), give the fix, and state the trade-off. This page answers the seven classic theory questions that way, each with a program you can run.

Words you need first

  • Training set / validation set: rows the model learns from / fresh rows used to check it (Episode 15).
  • MSE (mean squared error): the average of (prediction − truth)² (Episode 18). Lower is better.
  • Polynomial of degree d: a curve using x, x², …, xᵈ. Degree 1 is a straight line; degree 15 can wiggle a lot (Episode 19).
  • Cross-validation (CV): score a model on several train/validation splits and average (Episode 26).
  • Weight (coefficient): the number a linear model multiplies a feature by (Episode 18).
  • Precision / recall: of the "yes" predictions, how many were right / of the real "yes" cases, how many we caught (Episode 24).

The story: the memorised answer

Riya is asked "what is overfitting?" and recites a perfect line: the model learns the noise in the training data. The follow-up: how would you spot it in your own model, and what would you do? Silence. Memorising definitions is the obvious way to prepare, and it fails at the first follow-up, because the definition has nothing behind it.

The 5-step answer

Step What you say Overfitting example
1. Define one plain sentence The model fits the training rows, noise included, so it does worse on new rows.
2. Example a number, picture or story A degree-15 curve through 20 noisy points.
3. Evidence how to detect or measure it Training error low, validation error much higher (0.027 vs 1.290 below).
4. Fix what you would do More data, regularisation, a simpler model, early stopping.
5. Trade-off what the fix costs Too much regularisation underfits; more data costs money and time.

Question 1: explain the bias-variance trade-off

Real-life example: four archers. Arrows tightly grouped but all to the left = the aim itself is off (high bias). Arrows all over the board = every shot different (high variance). A tight group in the centre = the goal.

  • Bias: error from wrong assumptions, e.g. fitting a straight line to a curve. The model is wrong in the same way every time.
  • Variance: error from being too sensitive to the particular training rows; train on slightly different data and the model changes a lot.
  • Irreducible error (noise): randomness in the labels that no model can remove.
  • For squared error: expected error = bias² + variance + irreducible error.
  • Evidence: high bias → training and validation error both high (underfitting). High variance → training error low, validation error much higher (overfitting).
import numpy as np
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error as mse
def data(n, seed):                                    # made-up points: a smooth curve + noise
    r = np.random.default_rng(seed)
    x = r.uniform(0, 1, (n, 1))
    return x, np.sin(2 * np.pi * x).ravel() + r.normal(0, 0.2, n)
x, y = data(20, 2)                                    # small training set: 20 rows
xv, yv = data(1000, 0)                                # fresh validation points
print("degree   train MSE   validation MSE")
for d in [1, 4, 15]:                                  # simple, medium, very flexible model
    m = make_pipeline(PolynomialFeatures(d), LinearRegression()).fit(x, y)
    print(f"{d:6}   {mse(y, m.predict(x)):9.3f}   {mse(yv, m.predict(xv)):14.3f}")
print(f"noise variance (irreducible error): 0.2 ** 2 = {0.2 ** 2:.3f}")

Output:

degree   train MSE   validation MSE
     1       0.231            0.248
     4       0.034            0.061
    15       0.027            1.290
noise variance (irreducible error): 0.2 ** 2 = 0.040

Degree 1 has high error on both (high bias). Degree 15 has the lowest training error but a terrible validation error (high variance). Degree 4 is close to the noise floor 0.04, which no model can beat on average.

Question 2: how do you fix overfitting?

Fix Why it works
More training data a flexible model can't chase the noise of a few points when there are thousands
Regularisation a penalty on large weights keeps the curve smooth
A simpler model lower degree, shallower tree, fewer features = less variance
Early stopping stop training when the validation error starts rising
Ensembles (bagging), dropout, data augmentation averaging cancels wobble; dropout randomly switches off neurons; augmentation makes extra examples (e.g. flipped photos)

Trap: more data does not fix underfitting. A straight line stays a straight line with a million rows; for high bias, use a more flexible model or better features.

import numpy as np
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures, StandardScaler
from sklearn.linear_model import LinearRegression, Ridge
from sklearn.metrics import mean_squared_error as mse
def data(n, seed):                                    # made-up points: a smooth curve + noise
    r = np.random.default_rng(seed)
    x = r.uniform(0, 1, (n, 1))
    return x, np.sin(2 * np.pi * x).ravel() + r.normal(0, 0.2, n)
xv, yv = data(1000, 0)                                # fresh validation points
print("degree-15 model               validation MSE")
for n in [20, 200, 2000]:                             # fix 1: more data
    x, y = data(n, 2)
    m = make_pipeline(PolynomialFeatures(15), LinearRegression()).fit(x, y)
    print(f"{n:6} rows, no penalty        {mse(yv, m.predict(xv)):8.3f}")
x, y = data(20, 2)                                    # fix 2: same 20 rows + a penalty
m = make_pipeline(PolynomialFeatures(15), StandardScaler(), Ridge(alpha=0.1)).fit(x, y)
print(f"    20 rows, ridge alpha=0.1   {mse(yv, m.predict(xv)):8.3f}")

Output:

degree-15 model               validation MSE
    20 rows, no penalty           1.290
   200 rows, no penalty           0.046
  2000 rows, no penalty           0.042
    20 rows, ridge alpha=0.1      0.060

Question 3: L1 vs L2 regularisation

Real-life example: two ways to tax a family budget. A tax that grows with the square of each item cuts the big items hard but makes small ones cheap to keep (L2). A flat fee per rupee makes a barely useful small item not worth its fee, so it is cut to zero (L1).

  • Regularisation: loss = data error + alpha × penalty. Alpha (also written λ, lambda) is the regularisation strength.
  • L2 / ridge: penalty = Σ w². Every weight shrinks; none becomes exactly 0.
  • L1 / lasso: penalty = Σ |w|. Many weights become exactly 0 (sparsity), so lasso also does feature selection.
  • Elastic net: a mix of both penalties.
  • Trap: scale the features first (Episode 16). Otherwise a feature in rupees and the same feature in lakhs get punished differently just because of their units.
import numpy as np
from sklearn.linear_model import Ridge, Lasso
rng = np.random.default_rng(8)
X = rng.normal(size=(200, 8))                         # 8 made-up features, already scaled
true_w = np.array([3, -2, 1.5, 0, 0, 0, 0, 0])        # only the first 3 really matter
y = X @ true_w + rng.normal(0, 1, 200)
for model in [Ridge(alpha=50), Lasso(alpha=0.3)]:
    w = model.fit(X, y).coef_
    print(f"{type(model).__name__:6} weights {np.round(w, 2) + 0.0}  exact zeros: {np.sum(w == 0)}")

Output:

Ridge  weights [ 2.32 -1.6   1.25  0.05 -0.11  0.03 -0.1  -0.07]  exact zeros: 0
Lasso  weights [ 2.59 -1.71  1.23  0.    0.    0.    0.    0.  ]  exact zeros: 5

Question 4: which metric would you use?

Real-life example: a lazy airport scanner. A scanner that never beeps is right 99% of the time (almost no bag is dangerous) and completely useless: the accuracy paradox. What matters: of the beeps, how many were real dangers (precision), and of the dangers, how many beeped (recall).

Situation Metric
Rare positive class (fraud, disease) never accuracy alone
Missing a positive is costly recall
False alarms are costly precision
Both matter F1 = harmonic mean of precision and recall
Ranking quality across all thresholds ROC-AUC; with rare positives, PR-AUC (area under the precision-recall curve)
Regression MAE (typical miss) or RMSE (punishes big misses)

Always compare with a dummy baseline: a model that predicts the most common answer.

import numpy as np
from sklearn.dummy import DummyClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, precision_score, recall_score, average_precision_score
rng = np.random.default_rng(4)
n = 20000                                             # made-up card payments, about 1% fraud
X = rng.normal(size=(n, 4))
fraud = (X[:, 0] + X[:, 1] + rng.normal(0, 0.8, n) > 3.8).astype(int)
Xtr, Xte, ytr, yte = train_test_split(X, fraud, stratify=fraud, random_state=0)
print(f"fraud share: {yte.mean():.3f}")
for name, m in [("always 'not fraud'", DummyClassifier(strategy="most_frequent")),
                ("logistic regression", LogisticRegression(class_weight="balanced"))]:
    p = m.fit(Xtr, ytr).predict(Xte)
    s = m.predict_proba(Xte)[:, 1]
    print(f"{name:20} accuracy {accuracy_score(yte, p):.3f}  precision {precision_score(yte, p, zero_division=0):.2f}"
          f"  recall {recall_score(yte, p):.2f}  PR-AUC {average_precision_score(yte, s):.2f}")

Output:

fraud share: 0.010
always 'not fraud'   accuracy 0.990  precision 0.00  recall 0.00  PR-AUC 0.01
logistic regression  accuracy 0.934  precision 0.13  recall 0.98  PR-AUC 0.56

The dummy's 0.990 accuracy hides recall 0.00. Logistic regression catches 98% of frauds but only 13% of its alerts are real fraud: name that trade-off, and choose the threshold from the cost of each mistake (Episode 87 does this with rupees).

Question 5: trees or linear models?

Real-life example: an auto fare. "25 rupees base + 15 per km" works for any distance, even one nobody has travelled (a linear model: one weighted sum). A fare chart of past trips is exact for those trips but has no row for a 40 km trip (a decision tree: if-else boxes that cannot go beyond the data).

Linear models Decision trees
Shape they learn one weighted sum (a straight line or flat plane) thresholds, steps, interactions
Scaling needed? yes no
Explain weights the if-else path
Outside the training range continue the trend (extrapolate) predict a flat value

An interaction means the effect of one feature depends on another (here: class 1 only when both features have the same sign).

import numpy as np
from sklearn.linear_model import LinearRegression, LogisticRegression
from sklearn.tree import DecisionTreeRegressor, DecisionTreeClassifier
from sklearn.model_selection import cross_val_score
rng = np.random.default_rng(5)
x = rng.uniform(0, 10, (300, 1))                      # made-up: price grows steadily with size
y = 3 * x.ravel() + rng.normal(0, 1, 300)
lin, tree = LinearRegression().fit(x, y), DecisionTreeRegressor(max_depth=6).fit(x, y)
for size in [5, 20]:                                  # 20 is outside the training range 0..10
    print(f"size {size:2}: true ~{3 * size}  linear {lin.predict([[size]])[0]:5.1f}"
          f"  tree {tree.predict([[size]])[0]:5.1f}")
X = rng.uniform(-1, 1, (400, 2))                      # an interaction between two features
lab = (X[:, 0] * X[:, 1] > 0).astype(int)            # class 1 when both have the same sign
for name, m in [("logistic", LogisticRegression()), ("tree", DecisionTreeClassifier(max_depth=8))]:
    print(f"same-sign rule, {name:8} validation accuracy {cross_val_score(m, X, lab, cv=5).mean():.2f}")

Output:

size  5: true ~15  linear  15.0  tree  15.0
size 20: true ~60  linear  60.0  tree  30.8
same-sign rule, logistic validation accuracy 0.53
same-sign rule, tree     validation accuracy 0.91

Question 6: bagging vs boosting

Real-life example. Ask 100 friends separately to guess a cricket score and average: their wild errors cancel (bagging). A tutor who studies only your last test's mistakes, then the next, then the next (boosting).

  • Bootstrap sample: a random sample of the rows drawn with replacement (some rows twice, some not at all).
  • Bagging (bootstrap aggregating; random forest is bagging + random feature choice): independent models in parallel on bootstrap samples, averaged → reduces variance.
  • Boosting (AdaBoost, gradient boosting): models in sequence, each focused on the previous ones' errors → reduces bias.
  • So bag strong, wobbly models (deep trees) and boost weak, simple ones (stumps: trees with one split).
  • Trap: more trees rarely hurt a random forest, but too many boosting rounds can overfit, so boosting needs early stopping.
from sklearn.datasets import make_classification
from sklearn.model_selection import cross_val_score
from sklearn.tree import DecisionTreeClassifier as Tree
from sklearn.ensemble import BaggingClassifier, AdaBoostClassifier, RandomForestClassifier
X, y = make_classification(n_samples=2000, n_features=10, n_informative=6, flip_y=0.05, random_state=6)
models = {"1 stump (depth-1 tree)": Tree(max_depth=1),
          "bagging 100 stumps": BaggingClassifier(Tree(max_depth=1), n_estimators=100, random_state=0),
          "boosting 100 stumps": AdaBoostClassifier(Tree(max_depth=1), n_estimators=100, random_state=0),
          "1 deep tree": Tree(random_state=0),
          "bagging 100 deep trees": BaggingClassifier(Tree(), n_estimators=100, random_state=0),
          "random forest 100": RandomForestClassifier(n_estimators=100, random_state=0)}
for name, m in models.items():
    print(f"{name:24} accuracy {cross_val_score(m, X, y, cv=5).mean():.3f}")

Output:

1 stump (depth-1 tree)   accuracy 0.768
bagging 100 stumps       accuracy 0.766
boosting 100 stumps      accuracy 0.822
1 deep tree              accuracy 0.855
bagging 100 deep trees   accuracy 0.911
random forest 100        accuracy 0.922

Bagging stumps (0.766) did not beat one stump (0.768): averaging can't fix bias. Boosting the same stumps reached 0.822. Bagging deep trees lifted 0.855 to 0.911, and the random forest to 0.922: averaging cancels variance.

Question 7: gradient descent variants

Real-life example: walking down a hill in fog. Survey the whole hill before every step (sure but slow: batch), feel the ground under one foot and step at once (fast but zig-zag: SGD), or look a few metres around and step (mini-batch).

Variant Rows per update Updates per epoch (n rows)
Batch gradient descent all n 1
Stochastic gradient descent (SGD) 1 n
Mini-batch gradient descent a small batch b (e.g. 32) n ÷ b
  • Epoch: one full pass over the training rows.
  • Momentum: keep part of the previous step's direction, like a ball rolling downhill; smooths the zig-zag.
  • Adam: momentum plus a separate step size for each weight, based on how large its recent gradients were.
  • Divergence: with too large a learning rate, the loss grows instead of shrinking.
import numpy as np
rng = np.random.default_rng(7)
X = np.c_[np.ones(1000), rng.normal(size=(1000, 2))]  # bias column + 2 made-up features
y = X @ np.array([4.0, 2.0, -3.0]) + rng.normal(0, 0.5, 1000)
loss = lambda w: np.mean((X @ w - y) ** 2)            # mean squared error
def train(batch, lr, epochs=5):                       # one epoch = one pass over all rows
    w, steps = np.zeros(3), 0
    for _ in range(epochs):
        idx = rng.permutation(1000)                   # shuffle, then walk through in batches
        for s in range(0, 1000, batch):
            b = idx[s:s + batch]
            w -= lr * 2 * X[b].T @ (X[b] @ w - y[b]) / len(b)   # gradient of MSE on the batch
            steps += 1
    return loss(w), steps
for name, batch in [("batch GD", 1000), ("mini-batch 32", 32), ("SGD (1 row)", 1)]:
    l, steps = train(batch, lr=0.05)
    print(f"{name:14} updates {steps:5}   loss after 5 epochs {l:8.3f}")
print(f"batch GD, lr=1.2: loss {train(1000, lr=1.2)[0]:.3g}   <- learning rate too big")

Output:

batch GD       updates     5   loss after 5 epochs   10.270
mini-batch 32  updates   160   loss after 5 epochs    0.255
SGD (1 row)    updates  5000   loss after 5 epochs    0.270
batch GD, lr=1.2: loss 3.64e+03   <- learning rate too big

Same five passes over the data: batch made 5 updates and is still far from the answer; mini-batch and SGD made many more updates and reached the noise level (0.5² = 0.25). With learning rate 1.2 the loss grows far above its starting value: divergence.

"Explain an algorithm to me": k-nearest neighbours in layers

  1. Intuition: "you are like your closest neighbours": a new flat is priced like the 5 most similar flats.
  2. Mechanics: store the data; measure the distance to every row; take the k closest; vote (classification) or average (regression).
  3. Choices: k (small k = high variance, large k = high bias), the distance metric, feature scaling.
  4. Costs: training is free; each prediction compares with all n rows (Episode 86 codes it and states the complexity).
  5. When it fails: with very many features distances stop meaning much; unscaled features dominate the distance.

Typical traps

Trap 1: "our model scores 88% on pure noise". The answer is data leakage: information from the validation/test rows reached training. Here the leak is feature selection on all rows before cross-validation.

import numpy as np
from sklearn.feature_selection import SelectKBest
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
rng = np.random.default_rng(8)
X = rng.normal(size=(100, 5000))                      # 100 rows of pure random noise
y = rng.integers(0, 2, 100)                           # random labels: nothing to learn
best = SelectKBest(k=20).fit_transform(X, y)          # trap: pick features using ALL rows
print(f"select first, then CV: {cross_val_score(LogisticRegression(), best, y, cv=5).mean():.2f}")
pipe = make_pipeline(SelectKBest(k=20), LogisticRegression())   # selection inside each fold
print(f"selection inside CV:   {cross_val_score(pipe, X, y, cv=5).mean():.2f}")

Output:

select first, then CV: 0.88
selection inside CV:   0.51

Every step that learns from data (scaling, selection, imputation) belongs inside a Pipeline, so it is re-fitted on the training folds only (Episode 17).

Trap 2: "logistic regression is a regression model". It isn't: it is a classifier; the name comes from the logistic (sigmoid) function.

import numpy as np
from sklearn.linear_model import LogisticRegression
rng = np.random.default_rng(9)
X = rng.uniform(500, 3000, (50, 1))                   # made-up flat sizes in square feet
price = 0.08 * X.ravel() + rng.normal(0, 10, 50)      # price in lakh: a number, not a class
model = LogisticRegression().fit(X, price)            # "regression" is in the name...
print(model.predict([[1200]]))

Output (error, last line):

ValueError: Unknown label type: continuous. Maybe you are trying to fit a classifier, which expects discrete classes on a regression target with continuous values.

The fix, a regressor for a number target:

import numpy as np
from sklearn.linear_model import LinearRegression
rng = np.random.default_rng(9)
X = rng.uniform(500, 3000, (50, 1))                   # made-up flat sizes in square feet
price = 0.08 * X.ravel() + rng.normal(0, 10, 50)      # price in lakh: a number, not a class
model = LinearRegression().fit(X, price)              # fix: a regressor for a number target
print(f"predicted price for 1200 sq ft: {model.predict([[1200]])[0]:.1f} lakh")

Output:

predicted price for 1200 sq ft: 97.6 lakh

Other traps: tuning hyperparameters on the test set (the test score is then optimistic), and treating correlation as causation.

Common mistakes

Mistake What you see Why Fix
Only a definition stuck at the follow-up no evidence, fix or trade-off define, example, evidence, fix, trade-off
"More data fixes everything" degree 1 stays at 0.248 more data only cuts variance a more flexible model for bias
Accuracy on a rare class 0.990 with recall 0.00 accuracy paradox recall, precision, PR-AUC, a dummy baseline
A tree to continue a trend size 20 → 30.8, not 60 trees can't extrapolate a linear model or trend features
Regularising unscaled features penalty depends on units the penalty sees raw weight sizes scale first
Selecting features before CV 0.88 on pure noise data leakage selection inside a Pipeline
LogisticRegression().fit(X, price) ValueError: Unknown label type: continuous. Maybe you are trying to fit a classifier, which expects discrete classes on a regression target with continuous values. it is a classifier LinearRegression or another regressor
Learning rate too big loss grows (about 3.64e+03) divergence smaller rate, scaled features

Mini project: your answer bank

Write a one-page "answer bank": for each of the seven questions above, the five steps in your own words, plus the one program output you would quote as evidence. Then say each answer aloud in under 90 seconds, record yourself, and cut every sentence that is only a definition.

Practice

1. Training MSE 0.30, validation MSE 0.31, noise level 0.04. What is wrong and what do you do?

Solution

Both errors are high and close together: high bias (underfitting). Use a more flexible model (higher degree, deeper trees, more features); more data alone won't help.

2. Re-run the degree-15 model on 20 rows with ridge alpha 0.01, 1 and 10. What do you expect?

Solution

Real results (change alpha=0.1 in the program above): alpha 0.01 → 0.063, 0.1 → 0.060, 1 → 0.127, 10 → 0.143, 100 → 0.320. A tiny alpha such as 0.0000001 is almost no penalty: 2.178, high variance again. Too large an alpha flattens the curve towards a straight line (high bias), so validation error rises. The best value is chosen by validation, here around 0.01-0.1.

3. A disease affects 2% of patients. Model A: accuracy 0.98, recall 0.05. Model B: accuracy 0.90, recall 0.85. Which do you deploy for screening?

Solution

Model B. Missing a sick patient is the costly mistake, so recall matters; A's accuracy is close to what "always healthy" would score.

4. Why does bagging stumps not help, while boosting stumps does?

Solution

A stump is too simple (high bias) and stumps trained on bootstrap samples are nearly identical, so averaging them changes little. Boosting adds stumps that each fix the remaining errors, building a more complex model and lowering bias.

5. Write the update count for 10,000 rows, 3 epochs, mini-batch size 50.

Solution

10,000 ÷ 50 = 200 updates per epoch × 3 epochs = 600 updates.

Quick quiz

Training accuracy 0.99, validation accuracy 0.71, no more data. Best first move? Add regularisation or simplify the model: a big train-validation gap means high variance; a bigger model or more boosting rounds would make it worse.

Interview questions

  1. Explain the bias-variance trade-off and how you would diagnose each problem from training and validation error.
  2. List five ways to reduce overfitting and say why each works. Which one does not help with underfitting?
  3. What is the difference between L1 and L2 regularisation? Why does L1 give exact zeros?
  4. Your fraud model has 99% accuracy. Is it good? Which metrics would you report instead?
  5. When would you choose a decision tree over a linear model, and when not? What happens outside the training range?
  6. Bagging vs boosting: which reduces bias and which variance? Which base models suit each?
  7. Compare batch, stochastic and mini-batch gradient descent. What do momentum and Adam add?
  8. A model scores far above chance on random data. What happened?
  9. Explain k-nearest neighbours to a non-technical manager, then to an ML engineer.

Glossary

Term Meaning, with an example
Theory question A "why/when/compare" interview question with no code to write: "bagging vs boosting?"
5-step answer Define, example, evidence, fix, trade-off.
Overfitting Fitting the training rows, noise included: degree 15, train 0.027 vs validation 1.290.
Underfitting Too simple to capture the pattern: degree 1, 0.231 / 0.248.
Bias Error from wrong assumptions; arrows grouped but off target.
Variance Error from sensitivity to the training sample; arrows scattered.
Irreducible error (noise) Randomness in labels no model removes: 0.2² = 0.04 here.
Bias-variance trade-off More flexibility lowers bias but raises variance; pick the sweet spot (degree 4).
Early stopping Stop training when validation error starts to rise.
Data augmentation Making extra training examples from existing ones, e.g. flipped photos.
Dropout Randomly switching off neurons during training so a network can't rely on any one.
Regularisation Adding alpha × penalty on weights to the loss.
Alpha (λ) / regularisation strength How hard the penalty pushes: ridge alpha 0.1 → validation 0.060.
L2 regularisation (ridge) Penalty Σ w²: shrinks all weights, none exactly 0.
L1 regularisation (lasso) Penalty Σ |w|: sets useless weights exactly to 0 (5 zeros above).
Elastic net A mix of L1 and L2 penalties.
Sparsity Many weights exactly zero.
Accuracy paradox High accuracy, useless model on a rare class: 0.990, recall 0.00.
Dummy baseline A model that always predicts the most common answer.
PR-AUC Area under the precision-recall curve; good for rare positives (0.56 vs 0.01).
ROC-AUC Area under the ROC curve: chance a random positive scores above a random negative.
F1 score Harmonic mean of precision and recall.
MAE / RMSE Mean absolute error (typical miss) / root mean squared error (punishes big misses).
Interaction One feature's effect depends on another: the same-sign rule.
Extrapolation Predicting outside the training range: size 20 when training had 0-10.
Bootstrap sample Rows sampled with replacement, same size as the data.
Bagging Averaging models trained on bootstrap samples; cuts variance.
Random forest Bagged deep trees that also pick random features at each split: 0.922.
Boosting Models in sequence, each fixing the earlier ones' errors; cuts bias.
AdaBoost Boosting that re-weights the rows the previous models got wrong.
Gradient boosting Boosting where each new tree fits the remaining errors (gradients of the loss).
Stump A decision tree with a single split (depth 1).
Epoch One full pass over the training rows.
Batch gradient descent One update per epoch, using all rows.
Stochastic gradient descent (SGD) One update per row.
Mini-batch gradient descent One update per small batch, e.g. 32 rows.
Momentum Keeping part of the previous step's direction.
Adam Momentum + a per-weight step size from recent gradient sizes.
Divergence Loss growing because the learning rate is too big.
k-nearest neighbours (kNN) Predict from the k most similar training rows.
Data leakage Test/validation information reaching training: 0.88 on noise.
Pipeline scikit-learn object that re-fits every step on training folds only.
Correlation vs causation Two things moving together doesn't mean one causes the other.

Summary

  • Answer theory questions in five steps; steps 3-5 (evidence, fix, trade-off) win the marks.
  • Bias shows as high training and validation error; variance as a large gap between them.
  • Fix overfitting with more data, regularisation, simpler models, early stopping or bagging; more data doesn't fix bias.
  • Ridge shrinks weights; lasso shrinks and zeros them. Scale features first.
  • Pick metrics from the cost of mistakes, and beat a dummy baseline.
  • Trees learn interactions but can't extrapolate; linear models extrapolate but need hand-made interactions.
  • Bag high-variance models, boost high-bias ones; mini-batch gradient descent is the usual default.
  • Keep every fitted step inside a pipeline, and remember logistic regression is a classifier.

Next episode: Interview prep 2: coding and maths rounds.