Machine Learning Full Course · Episode 52. Every code sample below was run with Python 3.13, NumPy 2.5.3, pandas 3.0.5 and scikit-learn 1.9.1; the output shown is the real output. Ramesh's chai stall is the same made-up daily series as in Episodes 50 and 51: slow growth, busier weekends (× 1.3), busier winters, and luck where each day keeps 60% of yesterday's.
The big idea: ordinary machine learning models (linear regression, gradient boosting) can forecast, once you turn the series into a table of features that only use what is known at prediction time. The tempting feature "average of the week around each day" with a random split scores a dazzling MAE 10.2, then crashes on the newest day (ValueError: Input X contains NaN): it was look-ahead leakage. Honest lag, rolling-window and calendar features with a time split give 17.7 (seasonal naive: 30.1). Gradient boosting scores 24.9 because trees can't predict above the highest value they saw (448 vs a real 551). Walk-forward validation (backtesting) gives a mean of 14.5 where shuffled k-fold flatters with 12.6. MAPE becomes infinite on a day with zero sales; sMAPE is jumpy; MASE compares with the seasonal naive forecast (0.94 for our model). Four weeks ahead, direct forecasting (15.7) beats recursive (17.6), whose errors pile up (28.3 in week 4). A Prophet-style model (trend + Fourier waves + weekday, no lags) forecasts 92 days in one go with MAE 20.8, MAPE 5.7%.
Sources: S. J. Taylor and B. Letham, "Forecasting at Scale", The American Statistician 72(1):37-45, 2018 (ideas.repec.org); R. J. Hyndman, "Another Look at Forecast Accuracy Metrics for Intermittent Demand", Foresight: The International Journal of Applied Forecasting, issue 4, pp. 43-46, June 2006, which introduces MASE (ideas.repec.org).
Words you need first
- Time series, trend, seasonality, lag (Episode 50): values measured in time order; the long-term direction; a pattern that repeats with a fixed period (weekends, winters); "k steps back".
- Forecast, forecast horizon, baseline, naive and seasonal naive forecast (Episode 51): a prediction of future values; how far ahead; the simple bar every model must beat; "tomorrow = today"; "tomorrow = the same day last week".
- Supervised learning, feature, label (Episode 1): a model learns from rows of inputs (features) and answers (labels). Example: features "yesterday's cups, day of week" → label "today's cups".
- Train/test split, data leakage (Episode 15): learn on some rows, judge on rows the model never saw; leakage is when information from the test rows (or the future) sneaks into training.
- One-hot encoding (Episode 16): turn a category like "day of week = 3" into seven 0/1 columns.
- MAE (Episode 19): mean absolute error, the average size of the misses. Misses of +10, −20, +30 → MAE 20.
- Decision trees and boosting (Episodes 29-36): trees split the data into leaves and predict each leaf's average; boosting adds many small trees.
- k-fold cross-validation (Episode 26): split the data into k parts, test on each part in turn, average the scores.
- NaN: "not a number", how pandas and NumPy store a missing value.
The real-life hook: tomorrow's milk and next month's shifts
Every evening at 8 pm Ramesh orders tomorrow's milk: he needs a forecast with a horizon of one day. Once a month he plans his helpers' shifts: a horizon of 28 days. His nephew suggests the models from this course. They learn from a table, so we must turn one column of numbers into features.
These lines create the same daily series as Episodes 50-51, written more compactly (we call them HDR):
import numpy as np, pandas as pd
r = np.random.default_rng(50); days = pd.date_range("2023-01-01", "2025-12-31", freq="D"); t = np.arange(len(days))
week = np.where(days.dayofweek >= 5, 1.3, 1.0) # Ramesh's chai stall (made up): weekends 30% busier,
year = 1 + 0.25 * np.cos(2 * np.pi * (days.dayofyear - 15) / 365.25) # winter busier, summer quieter,
mood = np.zeros(len(t)); shock = r.normal(0, 0.05, len(t)) # random luck that carries over:
for i in range(1, len(t)): mood[i] = 0.6 * mood[i - 1] + shock[i] # 60% of yesterday's luck stays
ts = pd.Series(((200 + 0.1 * t) * week * year * np.exp(mood)).round().astype(int), index=days, name="cups") # + growth
ts is a pandas Series: one value (cups) per date, 1 Jan 2023 to 31 Dec 2025, 1,096 days.
The trap: a feature that peeks at the future
The obvious idea: describe each day by the average of the 7 days around it (3 before, the day, 3 after) plus its weekday, then the usual random 80/20 split.
# HDR (above), then:
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
d = pd.DataFrame({"y": ts, "week_avg": ts.rolling(7, center=True).mean()}) # average of the week AROUND each day
X = d[["week_avg"]].join(pd.get_dummies(ts.index.dayofweek.astype(str), prefix="dow", dtype=int).set_index(ts.index))
ok = X.notna().all(axis=1)
a, b, ya, yb = train_test_split(X[ok], d.y[ok], test_size=0.2, random_state=0) # random 80/20 split
model = LinearRegression().fit(a, ya)
print("test MAE:", np.abs(model.predict(b) - yb).mean().round(1), "cups a day")
try: print("newest day:", model.predict(X.tail(1))) # what Ramesh really needs
except ValueError as e: print("ValueError:", str(e).splitlines()[0]) # the first line of the real error
Output:
test MAE: 10.2 cups a day
ValueError: Input X contains NaN.
Without the try, Python prints the full traceback ending in:
ValueError: Input X contains NaN.
LinearRegression does not accept missing values encoded as NaN natively. ...
rolling(7, center=True).mean()averages a 7-day window centred on each day, so it uses the 3 days after it.- On the newest day those 3 days haven't happened, so the feature is NaN and
predictcrashes. - The 10.2 was a fake: during testing the "future" was already in the diary.
Look-ahead leakage is any feature or split that lets the model see information from after the moment of prediction. Two leaks here: the centred window, and the random split (a test day sits between training days, so the model has seen its neighbours). The rule: a forecasting feature may use only what you would really know when you predict.
Three kinds of honest features
| Kind | Real life | pandas | Example |
|---|---|---|---|
| Lag feature | yesterday's page in the diary | y.shift(k) |
lag1 = yesterday, lag7 = same weekday last week |
| Rolling-window feature | last week's average | y.shift(1).rolling(7).mean() |
mean of the 7 days before |
| Calendar feature | the wall calendar | index.dayofweek, index.month |
Saturday, December, a festival flag |
shift(k)moves every value k rows down, so each row sees the value from k days earlier.- Shift first, then roll: the window then ends yesterday.
rolling(7)alone would include today, the very value we predict. - Calendar features come from the date itself, so they are known for any future date.
These lines build the feature table (FEATS):
d = pd.DataFrame({"y": ts}) # y: cups sold that day (the label)
d["lag1"] = d.y.shift(1); d["lag7"] = d.y.shift(7) # lag features: yesterday, same day last week
d["roll7"] = d.y.shift(1).rolling(7).mean() # rolling window: mean of the 7 days before
d["dow"] = d.index.dayofweek; d["month"] = d.index.month # calendar features: known in advance
# HDR + FEATS, then:
print(d["2025-12-22":].round(1)) # the last 10 rows of the feature table
print(d.head(3)) # the first rows: no history yet
Output:
y lag1 lag7 roll7 dow month
date
2025-12-22 393 479.0 385.0 395.0 0 12
2025-12-23 397 393.0 348.0 396.1 1 12
2025-12-24 400 397.0 354.0 403.1 2 12
2025-12-25 418 400.0 348.0 409.7 3 12
2025-12-26 433 418.0 363.0 419.7 4 12
2025-12-27 551 433.0 488.0 429.7 5 12
2025-12-28 550 551.0 479.0 438.7 6 12
2025-12-29 396 550.0 393.0 448.9 0 12
2025-12-30 413 396.0 397.0 449.3 1 12
2025-12-31 412 413.0 400.0 451.6 2 12
y lag1 lag7 roll7 dow month
date
2023-01-01 323 NaN NaN NaN 6 1
2023-01-02 243 323.0 NaN NaN 0 1
2023-01-03 254 243.0 NaN NaN 1 1
The first rows have NaN (no history yet). We drop them before training (dropna()).
Linear regression vs trees, tested honestly
A time-based split: learn up to 30 Sep 2025, test on 1 Oct - 31 Dec 2025 (92 days). Day of week and month are one-hot encoded.
# HDR + FEATS, then:
from sklearn.linear_model import LinearRegression; from sklearn.ensemble import HistGradientBoostingRegressor
X = pd.get_dummies(d.drop(columns="y").astype({"dow": str, "month": str}), dtype=int).dropna(); y = d.y[X.index]
tr = X.index <= "2025-09-30"; te = ~tr; mae = lambda f: np.abs(np.asarray(f) - y[te].values).mean().round(1) # Oct-Dec
print("seasonal naive (lag 7):", mae(d.lag7[X.index][te]))
for model in [LinearRegression(), HistGradientBoostingRegressor(random_state=0)]:
f = model.fit(X[tr], y[tr]).predict(X[te])
print(f"{type(model).__name__}: MAE {mae(f)} highest forecast {f.max():.0f}")
print("highest in training:", y[tr].max(), " highest in test:", y[te].max())
Output:
seasonal naive (lag 7): 30.1
LinearRegression: MAE 17.7 highest forecast 520
HistGradientBoostingRegressor: MAE 24.9 highest forecast 448
highest in training: 481 highest in test: 551
- Seasonal naive (same day last week): 30.1. Linear regression on our features: 17.7.
HistGradientBoostingRegressor, usually stronger: 24.9. Its highest forecast is 448, below the training maximum 481, while December 2025 really reached 551.
Why trees get stuck: every leaf predicts the average of the training labels in it, so a tree (or forest, or boosting) can never predict above the largest label it saw. A growing series keeps setting records, so trees fall short: they cannot extrapolate. Fixes: a linear model, or let the tree predict the change (y − lag7) and add lag7 back (Practice 2).
Walk-forward validation and backtesting
One test quarter gives one number, which can be lucky. Normal k-fold shuffles the rows, so it trains on the future. Walk-forward validation: learn on everything up to a date, test on the next block, step forward, repeat; the training window grows (an expanding window). Replaying your history like this, each time knowing only what you knew then, is backtesting. In scikit-learn: TimeSeriesSplit.
# HDR + FEATS, then:
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import KFold, TimeSeriesSplit, cross_val_score
X = pd.get_dummies(d.drop(columns="y").astype({"dow": str, "month": str}), dtype=int).dropna(); y = d.y[X.index]
for cv in [KFold(5, shuffle=True, random_state=0), TimeSeriesSplit(5, test_size=91)]:
s = -cross_val_score(LinearRegression(), X, y, cv=cv, scoring="neg_mean_absolute_error")
print(f"{type(cv).__name__:15}", s.round(1), " mean", s.mean().round(1))
for i, (a, b) in enumerate(TimeSeriesSplit(5, test_size=91).split(X), start=1): # walk forward, 91 days at a time
print(f"fold {i}: learn up to {X.index[a[-1]]:%d %b %Y}, test {X.index[b[0]]:%d %b} - {X.index[b[-1]]:%d %b %Y}")
Output:
KFold [12.7 12.3 13.3 12.4 12.5] mean 12.6
TimeSeriesSplit [15.5 16.5 12. 11. 17.7] mean 14.5
fold 1: learn up to 02 Oct 2024, test 03 Oct - 01 Jan 2025
fold 2: learn up to 01 Jan 2025, test 02 Jan - 02 Apr 2025
fold 3: learn up to 02 Apr 2025, test 03 Apr - 02 Jul 2025
fold 4: learn up to 02 Jul 2025, test 03 Jul - 01 Oct 2025
fold 5: learn up to 01 Oct 2025, test 02 Oct - 31 Dec 2025
- Shuffled k-fold: 12.6 (too rosy, it peeks at neighbouring days). Walk-forward: 14.5.
- The folds range from 11.0 to 17.7: one test period alone can fool you.
| Fold | Learns up to | Tests |
|---|---|---|
| 1 | 02 Oct 2024 | 03 Oct 2024 - 01 Jan 2025 |
| 5 | 01 Oct 2025 | 02 Oct - 31 Dec 2025 |
MAPE, sMAPE and MASE
MAE is in cups: 17.7 is fine for a 400-cup stall and terrible for a 40-cup one. To compare series of different sizes we need errors without units.
| Metric | Formula | Good | Trap |
|---|---|---|---|
| MAPE (mean absolute percentage error) | mean(|a − f| / |a|) × 100% | easy to explain: "5% off" | infinite when an actual is 0; huge for small actuals |
| sMAPE (symmetric MAPE) | mean(|a − f| / ((|a| + |f|) / 2)) × 100% | no infinity | a day with actual 0 counts as 200% |
| MASE (mean absolute scaled error) | MAE of model / MAE of seasonal naive on the training data | no units, works with zeros; < 1 beats naive | needs a sensible naive baseline |
import numpy as np
actual = np.array([310, 290, 0, 330, 300]) # day 3: the stall was closed for Diwali
forecast = np.array([300, 300, 20, 300, 310])
err = np.abs(forecast - actual)
print("MAE :", err.mean(), "cups")
print("MAPE :", (100 * err / actual).mean(), "%") # divide by the actual value
print("sMAPE:", (100 * err / ((actual + forecast) / 2)).mean().round(1), "%") # divide by the average of both
Output:
MAE : 16.0 cups
MAPE : inf %
sMAPE: 43.9 %
main.py:6: RuntimeWarning: divide by zero encountered in divide
- One closed day (actual 0) and MAPE becomes
infwithRuntimeWarning: divide by zero encountered in divide. - sMAPE gives 43.9%, almost all of it from that one day (|0 − 20| / 10 = 200%).
- MASE for our linear model: 17.7 / 18.9 (seasonal naive MAE in training) = 0.94, so it beats a naive forecast.
- Rob Hyndman proposed MASE in 2006.
Multi-step forecasting: recursive vs direct
For the 28-day shift plan, lag1 for day 2 is unknown on day 0. Two strategies:
- Recursive: one one-step model; forecast day 1, write the guess into the series, use it as the lag for day 2, and so on. Simple, but errors pile up (guesses on guesses).
- Direct: one model per horizon h = 1 … 28, each using only values at least h days old (all known). No guesses fed back, but 28 models.
# HDR (above), then:
from sklearn.linear_model import LinearRegression
cal = lambda i: pd.get_dummies(pd.DataFrame({"dow": i.dayofweek, "month": i.month}, index=i).astype(str), dtype=int)
feat = lambda s, k: pd.DataFrame({"a": s.shift(k), "b": s.shift(k+6), "m": s.shift(k).rolling(7).mean()}).join(cal(s.index))
train, fut = ts[:"2025-09-30"], ts["2025-10-01":"2025-10-28"] # forecast 4 weeks (28 days) ahead
blank = pd.concat([train, fut * np.nan]); s = blank.copy(); direct = [] # the future is unknown: NaN
X = feat(train, 1).dropna(); one = LinearRegression().fit(X, train[X.index])
for day in fut.index: s[day] = one.predict(feat(s, 1).loc[[day]])[0] # recursive: guesses become lags
for h, day in enumerate(fut.index, start=1): # direct: one model per horizon h
X = feat(train, h).dropna(); m = LinearRegression().fit(X, train[X.index])
direct.append(m.predict(feat(blank, h).loc[[day]])[0])
for name, f in [("recursive", s[fut.index].values), ("direct", np.array(direct))]:
e = abs(f - fut.values); print(f"{name:9} days 1-7: {e[:7].mean():.1f} 22-28: {e[21:].mean():.1f} all: {e.mean():.1f}")
Output:
recursive days 1-7: 17.6 22-28: 28.3 all: 17.6
direct days 1-7: 16.4 22-28: 20.3 all: 15.7
- Week 1: 17.6 (recursive) vs 16.4 (direct). Week 4: 28.3 vs 20.3. All 28 days: 17.6 vs 15.7.
A Prophet-style model: no lags at all
Write the series as a sum of pieces that depend only on the date:
log(cups) = trend + yearly season + weekly season (+ holidays)
- Trend: a column counting the days.
- Yearly season with Fourier terms:
sin(k·2π·dayofyear/365.25)andcos(...)for k = 1, 2, …: smooth waves that repeat once, twice, … a year. A few waves added together can draw almost any yearly shape. - Weekly season: day-of-week dummies.
- Fitting on the log makes the seasons multiply instead of add.
This is the idea behind Prophet, Facebook's open-source forecasting tool, described by Sean J. Taylor and Benjamin Letham in "Forecasting at Scale" (2018). Our version is plain linear regression.
# HDR (above), then:
from sklearn.linear_model import LinearRegression
t = np.arange(len(ts)); yr = 2 * np.pi * ts.index.dayofyear / 365.25
X = pd.DataFrame({"trend": t}, index=ts.index) # piece 1: the trend
for k in [1, 2]: X[f"sin{k}"], X[f"cos{k}"] = np.sin(k * yr), np.cos(k * yr) # piece 2: yearly waves (Fourier)
X = X.join(pd.get_dummies(ts.index.dayofweek.astype(str), prefix="dow", dtype=int).set_index(ts.index)) # piece 3: weekday
tr = ts.index <= "2025-09-30" # learn to Sep, forecast Oct-Dec at once
m = LinearRegression().fit(X[tr], np.log(ts[tr])); f = np.exp(m.predict(X[~tr])); a = ts[~tr].values
mae = np.abs(f - a).mean(); scale = np.abs(ts[tr].diff(7)).mean() # MASE scale: seasonal naive MAE in training
print("Prophet-style, 92 days ahead: MAE", mae.round(1), " MAPE", (100 * np.abs(f - a) / a).mean().round(1), "%")
print("sMAPE", (100 * np.abs(f - a) / ((a + f) / 2)).mean().round(1), "% MASE", (mae / scale).round(2),
" (scale", scale.round(1), ") first 28 days MAE", np.abs(f - a)[:28].mean().round(1))
Output:
Prophet-style, 92 days ahead: MAE 20.8 MAPE 5.7 %
sMAPE 5.6 % MASE 1.1 (scale 18.9 ) first 28 days MAE 17.1
- 92 days ahead in one go: MAE 20.8, MAPE 5.7%, sMAPE 5.6% (no day near zero, so they agree).
- MASE 1.1: a bit worse than the in-sample one-step seasonal naive, which always knew last week's real value; this model forecasts three months blind. On the first 28 days: 17.1, between direct and recursive.
The scoreboard
| Setting | Method | MAE (cups a day) |
|---|---|---|
| Next day, random split | week-around average (leaks) | 10.2, impossible in real life |
| Next day, Oct-Dec 2025 | seasonal naive | 30.1 |
| Next day, Oct-Dec 2025 | gradient boosting | 24.9 |
| Next day, Oct-Dec 2025 | linear regression, lag + rolling + calendar | 17.7 |
| Next day, 5 folds | shuffled k-fold / walk-forward | 12.6 / 14.5 |
| 28 days ahead | recursive / Prophet-style / direct | 17.6 / 17.1 / 15.7 |
| 92 days ahead | Prophet-style | 20.8 |
Common mistakes
| Mistake | Exact message / symptom | Why | Fix |
|---|---|---|---|
| Centred rolling window | ValueError: Input X contains NaN. on the newest day |
It uses days after the one you predict | shift(1).rolling(7) |
rolling(k) without shift |
suspiciously good test score | the window includes the day being predicted | shift first, then roll |
| Random split / shuffled k-fold | 12.6 instead of an honest 14.5 | test days sit between training days | time split, TimeSeriesSplit |
| Trees on a growing series | highest forecast 448 when reality hits 551 | a leaf predicts an average of seen labels | linear model, or predict y − lag7 |
| MAPE with zeros | RuntimeWarning: divide by zero encountered in divide, inf |
dividing by an actual of 0 | MASE or MAE |
| Recursive far ahead | week-4 MAE 28.3 | guesses become lags | direct models, or check error by horizon |
| Feeding NaN lags to the model | ValueError: Input X contains NaN. |
the first rows have no history | dropna() before training |
Mini project: tomorrow's forecast for Ramesh
On the evening of 31 Dec 2025, add an empty row for 1 Jan 2026, rebuild the features (they only look back), train on all three years and forecast.
# HDR + FEATS, then:
from sklearn.linear_model import LinearRegression
tomorrow = pd.Timestamp("2026-01-01") # Ramesh, on the evening of 31 Dec 2025
d.loc[tomorrow] = np.nan # an empty row for tomorrow
d["lag1"] = d.y.shift(1); d["lag7"] = d.y.shift(7); d["roll7"] = d.y.shift(1).rolling(7).mean()
d["dow"] = d.index.dayofweek; d["month"] = d.index.month # rebuild: tomorrow's features use the past only
X = pd.get_dummies(d.drop(columns="y").astype({"dow": str, "month": str}), dtype=int)
known = X.notna().all(axis=1) & d.y.notna()
model = LinearRegression().fit(X[known], d.y[known]) # learn from all 3 years
print(X.loc[[tomorrow], ["lag1", "lag7", "roll7"]])
print("forecast for Thu 1 Jan 2026:", model.predict(X.loc[[tomorrow]])[0].round(), "cups")
Output:
lag1 lag7 roll7
date
2026-01-01 412.0 418.0 453.285714
forecast for Thu 1 Jan 2026: 413.0 cups
Practice
1. Add lag14 and lag28. Does the Oct-Dec MAE drop below 17.7?
Solution
# HDR + FEATS, then:
from sklearn.linear_model import LinearRegression
d["lag14"] = d.y.shift(14); d["lag28"] = d.y.shift(28) # two more lag features
X = pd.get_dummies(d.drop(columns="y").astype({"dow": str, "month": str}), dtype=int).dropna(); y = d.y[X.index]
tr = X.index <= "2025-09-30"
f = LinearRegression().fit(X[tr], y[tr]).predict(X[~tr])
print("MAE with lag14 and lag28:", np.abs(f - y[~tr]).mean().round(1))
MAE with lag14 and lag28: 17.3
A small gain (17.7 → 17.3): two and four weeks back add a little information.
2. Let gradient boosting predict the change y − lag7, then add lag7 back. Does it beat 24.9?
Solution
# HDR + FEATS, then:
from sklearn.ensemble import HistGradientBoostingRegressor
X = pd.get_dummies(d.drop(columns="y").astype({"dow": str, "month": str}), dtype=int).dropna(); y = d.y[X.index]
tr = X.index <= "2025-09-30"; change = y - X.lag7 # predict the change from last week
m = HistGradientBoostingRegressor(random_state=0).fit(X[tr], change[tr])
f = m.predict(X[~tr]) + X.lag7[~tr] # add last week back
print("boosting on y - lag7: MAE", np.abs(f - y[~tr]).mean().round(1), " highest forecast", f.max().round())
boosting on y - lag7: MAE 22.3 highest forecast 511.0
Better (24.9 → 22.3) and its highest forecast now reaches 511: the change is in a familiar range even when the level is new. Linear regression (17.7) still wins here.
3. Walk-forward with 10 folds of 30 days. What is the mean?
Solution
# HDR + FEATS, then:
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import TimeSeriesSplit, cross_val_score
X = pd.get_dummies(d.drop(columns="y").astype({"dow": str, "month": str}), dtype=int).dropna(); y = d.y[X.index]
s = -cross_val_score(LinearRegression(), X, y, cv=TimeSeriesSplit(10, test_size=30), scoring="neg_mean_absolute_error")
print(s.round(1), " mean", s.mean().round(1))
[13.7 15.3 12. 8.7 9.3 12.5 11.8 12.6 19.9 20.4] mean 13.6
Smaller test blocks vary more (8.7 to 20.4); the winter folds are the hardest.
4. Prophet-style with 4 Fourier pairs (for k in [1, 2, 3, 4]).
Solution
Prophet-style, 92 days ahead: MAE 20.8 MAPE 5.7 %
sMAPE 5.6 % MASE 1.1 (scale 18.9 ) first 28 days MAE 16.1
The 92-day MAE stays 20.8; the first 28 days improve from 17.1 to 16.1. Our made-up yearly season is one smooth wave, so extra waves add little.
Quick quiz
We forecast tomorrow's cups. Which feature leaks the future?
A) d.y.shift(1) B) d.index.dayofweek C) d.y.shift(1).rolling(7).mean() D) d.y.rolling(3).mean()
Answer
D. Without shift, the rolling window includes the day itself, which is the value we are trying to predict.
More questions:
- Why is a calendar feature like "month" never leakage?
Answer
It comes from the date, which is known in advance for any future day. - Shuffled k-fold scored 12.6 and walk-forward 14.5. Which do you report?
Answer
14.5: it is how the model will be used, always predicting after what it learned from. - A model has MASE 0.8. What does it mean?
Answer
Its MAE is 80% of the seasonal naive forecast's MAE on the training data: it beats the naive forecast. - Why does a random forest under-forecast a record December?
Answer
Each leaf predicts an average of training labels, so it can't go above the largest value it saw.
Interview questions
- What is look-ahead leakage? Give two examples. A feature or split that uses information from after the prediction time: a centred rolling mean; a random train/test split of a time series; scaling with statistics computed on the whole series.
- How do you cross-validate a forecasting model? Walk-forward validation (
TimeSeriesSplit): train on the past, test on the next block, move forward; optionally leave a gap between them. - Recursive vs direct multi-step forecasting? Recursive: one model, feeds its forecasts back as lags, errors compound. Direct: one model per horizon using only known lags, no compounding but more models. Hybrids exist (direct-recursive, multi-output).
- Why can tree models fail on trending data? They can't extrapolate beyond the label range seen in training; difference or detrend the target, or use a linear component.
- MAPE vs MASE? MAPE is a percentage of the actual, breaks on zeros and favours under-forecasts; MASE scales MAE by a naive forecast's in-sample MAE, works with zeros, and 1 means "as good as naive".
- What does Prophet model? An additive model of trend, seasonalities (Fourier series) and holiday effects, fitted to the date alone (Taylor & Letham, 2018).
Glossary
| Term | Meaning (plain words + example) |
|---|---|
| Lag feature | The value k steps back: lag1 = yesterday's cups, lag7 = same weekday last week |
| Rolling-window feature | A statistic over a sliding window of past values: the mean of the 7 days before |
| Calendar feature | A feature from the date itself: day of week, month, holiday flag; known in advance |
shift(k) |
pandas: move values k rows down so each row sees the value from k rows earlier |
rolling(k) |
pandas: a window of the last k rows (including the current one unless you shift first) |
center=True |
rolling option that centres the window on each row, so it uses later rows: leakage in forecasting |
| Look-ahead leakage | Using information from after the prediction moment: a centred average, a random split |
| Time-based split | Train on earlier dates, test on later ones |
| Extrapolate | Predict outside the range seen in training: a new sales record |
| Walk-forward validation | Train up to a date, test the next block, step forward, repeat |
| Expanding window | Each walk-forward step trains on all data so far, so training grows |
| Backtesting | Replaying history as if forecasting live, knowing only the past each time |
TimeSeriesSplit |
scikit-learn's walk-forward splitter: TimeSeriesSplit(5, test_size=91) |
| MAPE | Mean absolute percentage error: average of |miss| / |actual| × 100%. Breaks when actual = 0 |
| sMAPE | Symmetric MAPE: divides by the average of |actual| and |forecast|; 200% when actual = 0 |
| MASE | Mean absolute scaled error: model MAE / seasonal naive MAE on training data; < 1 beats naive |
| Multi-step forecasting | Forecasting several steps ahead: the next 28 days |
| Recursive forecasting | One model; its forecasts become the lags for the next step |
| Direct forecasting | One model per horizon h, using only values at least h steps old |
| Prophet | Facebook's open-source forecasting tool: trend + seasonalities + holidays |
| Prophet-style model | Any model of that additive form; ours is linear regression on trend, Fourier and weekday columns |
| Fourier terms | sin and cos waves that repeat k times a period; added together they draw a seasonal shape |
| Extrapolation failure (trees) | Trees predict leaf averages, so never above the largest training label |
HistGradientBoostingRegressor |
scikit-learn's fast histogram-based gradient boosting |
| NaN | "Not a number": a missing value |
Summary
- Turn a series into a table with lag, rolling-window (shift, then roll) and calendar features; never use anything you wouldn't know when predicting (look-ahead leakage: 10.2 was fake, then
ValueError). - Honest next-day test: seasonal naive 30.1, boosting 24.9 (trees can't extrapolate), linear regression 17.7.
- Walk-forward validation / backtesting (
TimeSeriesSplit): 14.5, where shuffled k-fold flatters with 12.6. - MAPE breaks on zeros, sMAPE is jumpy near zero, MASE compares with the naive forecast (0.94).
- 28 days ahead: direct 15.7 beats recursive 17.6 (28.3 vs 20.3 in week 4).
- A Prophet-style model (trend + Fourier waves + weekday) needs no lags: 92 days in one go, MAE 20.8, MAPE 5.7%.
Next episode: Recommender systems 1: explicit and implicit feedback, the user-item matrix, content-based and collaborative filtering, and the cold-start problem.