SyntaxPilot
Courses/Machine Learning/Lesson 89
Video coming soonThis lesson’s video is being made

Start with the notes below; the video appears here as soon as it is on YouTube.

Machine Learning · Lesson 89 of 90

Project 2: customer churn

imbalanced classification projectpipelinesthreshold choice from business costcalibrationSHAP explanationsfairness checkresults write-up

Machine Learning Full Course · Episode 89. Every program below was run on a Linux machine with Python 3.13, scikit-learn 1.9.1, pandas 3.0.5, NumPy 2.5.3 and SHAP 0.54.1 (one CPU thread), and the output shown is the real output. The broadband customers and the rupee costs are made up: the generator is at the end, so you can make exactly the same file. gender was not used to make churn; it is kept only for the fairness check.

The big idea: when the class you care about is rare, accuracy lies. Judge the model by how well it ranks the rare class (ROC-AUC, PR-AUC), turn its scores into decisions with a threshold chosen from business costs, make sure its probabilities are calibrated so that maths works, explain individual decisions with SHAP, check the results by group before anyone uses them, and write it all up in plain words.

Words you need first

  • Churn: a customer leaving. A customer who leaves has churned.
  • Classification: predicting a class (leaves / stays), usually via a probability.
  • Probability: a number from 0 to 1; 0.3 should mean "3 in 10 such customers leave".
  • Threshold: flag the customer if the predicted probability ≥ t. Default t = 0.5.
  • Imbalanced data: one class is much rarer than the other (9.3% churn here).

The problem

Real-life example: the broadband company that keeps losing people. 8,000 customers; about 9 in 100 leave each quarter. Each lost customer costs Rs 6,000. A retention call with a Rs 500 offer keeps about half of those who would leave. Calling everyone wastes Rs 500 on every happy customer. Who should get the call?

Step 1: the accuracy trap

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.dummy import DummyClassifier
df = pd.read_csv("/data/churn.csv")                        # 8,000 made-up broadband customers
print("churn rate:", round(df.churned.mean(), 3))
X = df.select_dtypes("number").drop(columns=["customer_id", "churned"]); y = df["churned"]
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.25, random_state=0)
for m in [LogisticRegression(max_iter=1000), DummyClassifier()]:      # Dummy: always "stays"
    p = m.fit(Xtr, ytr).predict(Xte)
    print(f"{type(m).__name__:18} accuracy {(p == yte).mean():.3f}   "
          f"churners caught {((p == 1) & (yte == 1)).sum()} of {yte.sum()}")

Output:

churn rate: 0.093
LogisticRegression accuracy 0.903   churners caught 5 of 191
DummyClassifier    accuracy 0.904   churners caught 0 of 191

90.3% accuracy, but only 5 of 191 churners caught, and a model that always says "stays" scores higher (90.4%). This is the accuracy paradox: with 9.3% churn, predicting "stays" for everyone is already about 91% right.

Metric Question it answers Here
Recall (sensitivity) Of the people who really left, what share did we catch? 5 / 191 = 0.03
Precision Of the people we flagged, what share really left?
Accuracy What share of all predictions was right? 0.903 (misleading)

Step 2: metrics for rare events, and a stratified split

  • ROC-AUC: the chance that a random leaver gets a higher score than a random stayer. 0.5 = coin toss, 1 = perfect.
  • PR-AUC (average precision): precision averaged over all levels of recall. A random model scores the churn rate (0.093), so that is the number to beat.
  • Both judge the ranking, without choosing a threshold.
  • Stratified split: keeps the same class share in train and test (stratify=y).
import pandas as pd
from sklearn.model_selection import train_test_split
df = pd.read_csv("/data/churn.csv")
for name, strat in [("plain     ", None), ("stratified", df.churned)]:     # 20 different random splits
    tests = [train_test_split(df, test_size=0.1, stratify=strat, random_state=s)[1] for s in range(20)]
    rates = [t.churned.mean() for t in tests]
    print(f"{name} test churn rate over 20 splits: {min(rates):.3f} to {max(rates):.3f}")
train, test = train_test_split(df, test_size=0.25, stratify=df.churned, random_state=89)
train.to_csv("/data/churn_train.csv", index=False); test.to_csv("/data/churn_test.csv", index=False)
print("train", len(train), "  test", len(test), "  churners in test:", test.churned.sum())

Output:

plain      test churn rate over 20 splits: 0.072 to 0.109
stratified test churn rate over 20 splits: 0.092 to 0.092
train 6000   test 2000   churners in test: 186

With a plain split, the test churn rate swings from 7.2% to 10.9%: your score depends on luck. Stratified: 9.2% every time.

Step 3: pipelines and two models

class_weight="balanced" gives each class a weight inversely proportional to its frequency, so each churner counts about ten times as much as a stayer during training. Gender is left out of the inputs on purpose.

import pandas as pd, joblib
from sklearn.model_selection import cross_validate
from sklearn.compose import make_column_transformer
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import HistGradientBoostingClassifier
tr, cat = pd.read_csv("/data/churn_train.csv"), ["region", "contract", "plan", "autopay"]
X, y = tr.drop(columns=["customer_id", "gender", "churned"]), tr.churned       # gender: not an input
lr = make_pipeline(make_column_transformer((OneHotEncoder(), cat), remainder=StandardScaler()),
                   LogisticRegression(class_weight="balanced"))          # balanced: churners count ~10x
gb = make_pipeline(make_column_transformer((OneHotEncoder(), cat), remainder="passthrough"),
                   HistGradientBoostingClassifier(max_iter=200, learning_rate=0.05, random_state=0))
for name, m in [("logistic (balanced)", lr), ("gradient boosting", gb)]:
    s = cross_validate(m, X, y, cv=5, scoring=["roc_auc", "average_precision", "recall", "precision"])
    s = {k[5:]: v.mean() for k, v in s.items() if k.startswith("test_")}
    print(f"{name:20} ROC-AUC {s['roc_auc']:.3f}  PR-AUC {s['average_precision']:.3f}"
          f"  recall {s['recall']:.2f}  precision {s['precision']:.2f}")
joblib.dump(lr, "/data/churn_recipe.joblib")                  # save the (unfitted) winning pipeline

Output:

logistic (balanced)  ROC-AUC 0.790  PR-AUC 0.325  recall 0.72  precision 0.20
gradient boosting    ROC-AUC 0.732  PR-AUC 0.255  recall 0.10  precision 0.39

The simple logistic model wins on both ranking metrics (PR-AUC 0.325, 3.5× the random 0.093), so it is the one we use. At the default threshold the two models make very different trade-offs (recall 0.72 vs 0.10); the money decides which trade-off is right. We save the winning pipeline, unfitted, as a "recipe" so every later program uses exactly the same steps.

Step 4: a threshold from business cost

Real-life example: every mistake has a price tag.

Outcome Meaning Cost (made up)
True positive (TP) called someone who would leave Rs 500 offer + half of Rs 6,000 still lost = Rs 3,500
False positive (FP) called a happy customer Rs 500 wasted
False negative (FN) didn't call someone who leaves Rs 6,000 lost
True negative (TN) didn't call a happy customer Rs 0

cross_val_predict gives every training customer a probability from a model that never saw them (out-of-fold predictions), so we can choose the threshold without touching the test set.

import numpy as np, pandas as pd, joblib
from sklearn.model_selection import cross_val_predict
tr = pd.read_csv("/data/churn_train.csv")
X, y = tr.drop(columns=["customer_id", "gender", "churned"]), tr["churned"].values
model = joblib.load("/data/churn_recipe.joblib")      # the pipeline saved last time (not fitted)
p = cross_val_predict(model, X, y, cv=5, method="predict_proba")[:, 1]    # out-of-fold probabilities
OFFER, LOST = 500, 6000          # made-up rupees: offer, lost customer; an offer keeps half
def cost(t):
    flag = p >= t
    tp, fp, fn = (flag & (y == 1)).sum(), (flag & (y == 0)).sum(), (~flag & (y == 1)).sum()
    return (tp * (OFFER + LOST / 2) + fp * OFFER + fn * LOST) / 1e5                  # in lakh
print(f"call nobody:      cost Rs {cost(1.01):.1f} lakh")
for t in [0.3, 0.5, 0.7]: print(f"threshold {t}:    cost Rs {cost(t):.1f} lakh  flagged {(p >= t).sum()}")
ts = np.arange(0.05, 0.96, 0.01); best = ts[np.argmin([cost(t) for t in ts])]
print(f"best threshold {best:.2f}: cost Rs {cost(best):.1f} lakh  flagged {(p >= best).sum()} of {len(y)}")

Output:

call nobody:      cost Rs 33.4 lakh
threshold 0.3:    cost Rs 36.0 lakh  flagged 3526
threshold 0.5:    cost Rs 31.3 lakh  flagged 1968
threshold 0.7:    cost Rs 29.7 lakh  flagged 771
best threshold 0.68: cost Rs 29.6 lakh  flagged 875 of 6000

A low threshold (0.3) is worse than calling nobody: 3,526 calls waste too many offers. The cheapest is 0.68: 29.6 lakh, saving 3.8 lakh a quarter versus doing nothing.

Step 5: calibration

A puzzle. Call a customer when p × (loss a call saves) > offer cost: p × 3,000 > 500, so p > 0.167. Our search said 0.68. So the probabilities themselves must be wrong.

  • Calibrated: a model's probabilities mean what they say: of customers given 0.3, about 30% really leave.
  • class_weight="balanced" pretends churners are ~10× more common, so it pushes every probability up.
  • Brier score: the mean of (p − actual outcome)². Lower is better.
  • CalibratedClassifierCV(method="isotonic"): learns, with cross-validation, a step-shaped increasing mapping from the model's scores to real probabilities. (method="sigmoid", also called Platt scaling, fits an S-curve instead.)
import numpy as np, pandas as pd, joblib
from sklearn.model_selection import cross_val_predict
from sklearn.calibration import CalibratedClassifierCV
from sklearn.metrics import brier_score_loss
tr = pd.read_csv("/data/churn_train.csv")
X, y = tr.drop(columns=["customer_id", "gender", "churned"]), tr["churned"].values
raw = joblib.load("/data/churn_recipe.joblib")         # balanced logistic pipeline
cal = CalibratedClassifierCV(raw, method="isotonic", cv=5)        # learns: score -> real probability
for name, m in [("balanced", raw), ("calibrated", cal)]:
    p = cross_val_predict(m, X, y, cv=5, method="predict_proba")[:, 1]
    bins = pd.cut(p, [0, .1, .2, .4, .6, 1])                      # group customers by predicted risk
    t = pd.DataFrame({"said": p, "real": y}).groupby(bins, observed=True).mean().round(2)
    print(f"{name}: average said {p.mean():.3f}, real {y.mean():.3f}  Brier {brier_score_loss(y, p):.4f}")
    print("  said:", t.said.tolist(), "\n  real:", t.real.tolist())

Output:

balanced: average said 0.395, real 0.093  Brier 0.1865
  said: [0.07, 0.15, 0.3, 0.5, 0.74] 
  real: [0.01, 0.02, 0.04, 0.09, 0.25]
calibrated: average said 0.093, real 0.093  Brier 0.0740
  said: [0.04, 0.13, 0.3, 0.46, 0.79] 
  real: [0.04, 0.12, 0.31, 0.41, 0.61]

The balanced model says 0.395 on average when the truth is 0.093; in its riskiest group it says 0.74 and only 25% leave. After calibration: 0.093 vs 0.093, group by group close, Brier 0.1865 → 0.0740. Calibration changes the probabilities but keeps the ranking, so ROC-AUC barely moves.

Step 6: test once

import pandas as pd, joblib
from sklearn.calibration import CalibratedClassifierCV
from sklearn.metrics import average_precision_score, roc_auc_score
tr, te = pd.read_csv("/data/churn_train.csv"), pd.read_csv("/data/churn_test.csv")
drop = ["customer_id", "gender", "churned"]
lr = joblib.load("/data/churn_recipe.joblib")
model = CalibratedClassifierCV(lr, method="isotonic", cv=5).fit(tr.drop(columns=drop), tr.churned)
OFFER, LOST = 500, 6000
T = OFFER / (LOST / 2)                       # flag when p x (loss we save) > offer cost  ->  p > 0.167
p, y = model.predict_proba(te.drop(columns=drop))[:, 1], te.churned.values     # test set, opened ONCE
flag = p >= T; tp, fp, fn = (flag & (y == 1)).sum(), (flag & (y == 0)).sum(), (~flag & (y == 1)).sum()
cost = tp * (OFFER + LOST / 2) + fp * OFFER + fn * LOST
print(f"threshold {T:.3f}   flagged {flag.sum()} of {len(y)}   recall {tp / (tp + fn):.2f}"
      f"   precision {tp / (tp + fp):.2f}")
print(f"ROC-AUC {roc_auc_score(y, p):.3f}   PR-AUC {average_precision_score(y, p):.3f}")
print(f"cost Rs {cost / 1e5:.1f} lakh vs call nobody Rs {y.sum() * LOST / 1e5:.1f} lakh")
joblib.dump(model, "/data/churn_model_v1.joblib")

Output:

threshold 0.167   flagged 291 of 2000   recall 0.45   precision 0.29
ROC-AUC 0.789   PR-AUC 0.321
cost Rs 10.1 lakh vs call nobody Rs 11.2 lakh
Value
Threshold 0.167, straight from the costs
Customers flagged 291 of 2,000 (about 15%)
Recall / precision 0.45 / 0.29
ROC-AUC / PR-AUC 0.789 / 0.321 (matches CV)
Cost Rs 10.1 lakh vs 11.2 lakh calling nobody: about 10% saved, as on the training data

When costs change, only the threshold moves: no retraining needed, because the probabilities are calibrated.

Step 7: SHAP explanations

Real-life example: splitting a restaurant bill fairly. Start from the average meal (the base value, the model's average prediction); each person pays for what they added (each feature's SHAP value: how much it pushed this one prediction up or down); the shares add up exactly to the bill (base value + all SHAP values = this customer's prediction). SHAP comes from Shapley values in game theory, a fair way to split a total among players.

For logistic regression, SHAP values are in log-odds, the score before the sigmoid: log(p / (1 − p)). LinearExplainer uses training customers as the background (the note about 100 samples is SHAP using a sample of them).

import numpy as np, pandas as pd, joblib, shap
tr, te = pd.read_csv("/data/churn_train.csv"), pd.read_csv("/data/churn_test.csv")
drop = ["customer_id", "gender", "churned"]
Xtr, Xte = tr.drop(columns=drop), te.drop(columns=drop)
pipe = joblib.load("/data/churn_recipe.joblib")
pipe.set_params(columntransformer__verbose_feature_names_out=False).fit(Xtr, tr.churned)   # short names
names = pipe[0].get_feature_names_out()
explainer = shap.LinearExplainer(pipe[1], pipe[0].transform(Xtr))       # background: training rows
sv = explainer(pipe[0].transform(Xte)).values           # one push per feature per customer (log-odds)
print("most important overall (mean |SHAP|):")
print(pd.Series(np.abs(sv).mean(0), names).nlargest(5).round(2).to_dict())
i = int(np.argmax(sv.sum(1))); r = te.iloc[i]            # the riskiest test customer
print(f"customer {r.customer_id}: {r.contract}, tenure {r.tenure_months}, complaints {r.complaints_90d},"
      f" outages {r.outages_30d}, autopay {r.autopay}")
top = pd.Series(sv[i], names).sort_values(key=abs, ascending=False).head(4)
print("biggest pushes toward churn:", top.round(2).to_dict())

Output:

most important overall (mean |SHAP|):
{'tenure_months': 0.7, 'complaints_90d': 0.33, 'contract_yearly': 0.27, 'autopay_yes': 0.21, 'contract_monthly': 0.2}
customer 106609: monthly, tenure 26, complaints 5, outages 6, autopay no
biggest pushes toward churn: {'complaints_90d': 2.24, 'outages_30d': 1.04, 'support_calls_90d': 0.46, 'tenure_months': 0.44}

Overall, tenure matters most (new customers leave more), then complaints, contract type and autopay. For customer 106609 the biggest pushes are complaints (+2.24) and outages (+1.04), so the call script should apologise and fix the outages; a discount alone won't help.

Step 8: a fairness check

  • Protected attribute: a characteristic decisions should not be based on, such as gender. We left it out of the inputs.
  • Proxy: another column that carries the same information, so leaving the attribute out is not enough.
  • Demographic parity: are the groups flagged at similar rates?
  • Equal opportunity: among people who really leave, are the groups caught at similar rates (recall per group)? Here being caught means getting a helpful offer.
  • Bootstrap: resample the data with replacement many times to see how much a number moves by chance.
import numpy as np, pandas as pd, joblib
model = joblib.load("/data/churn_model_v1.joblib")          # calibrated model from the test-once program
te = pd.read_csv("/data/churn_test.csv")
te["flag"] = model.predict_proba(te.drop(columns=["customer_id", "gender", "churned"]))[:, 1] >= 0.167
def report(g):
    caught = g[g.churned == 1].flag.mean()                   # recall inside this group
    return pd.Series({"n": len(g), "real churn": g.churned.mean(), "flagged": g.flag.mean(),
                      "recall": caught, "precision": g[g.flag].churned.mean()})
for col in ["gender", "region"]:                             # gender was never an input. Check anyway.
    t = te.groupby(col)[["churned", "flag"]].apply(report).round(2); t["n"] = t.n.astype(int)
    print(t.to_string(), "\n")
rng = np.random.default_rng(0); ch = te[te.churned == 1]       # is the gender gap real, or chance?
gaps = [(s[s.gender == "female"].flag.mean() - s[s.gender == "male"].flag.mean())
        for s in (ch.sample(len(ch), replace=True, random_state=rng) for _ in range(1000))]
lo, hi = np.percentile(gaps, [2.5, 97.5])
print(f"recall gap female - male: {np.mean(gaps):.2f}, 95% interval {lo:.2f} to {hi:.2f}")

Output:

           n  real churn  flagged  recall  precision
gender                                              
female  1001        0.10     0.15    0.51       0.34
male     999        0.09     0.14    0.38       0.24 

           n  real churn  flagged  recall  precision
region                                              
city    1597        0.08     0.11    0.34       0.24
town     403        0.15     0.26    0.68       0.38 

recall gap female - male: 0.13, 95% interval -0.02 to 0.27
  • Gender: flagged at almost the same rate (15% vs 14%), but recall 0.51 vs 0.38. The 95% bootstrap interval for the gap (−0.02 to 0.27) includes zero: with only 186 leavers we cannot tell it from chance. Report it and keep measuring.
  • Region: towns are flagged far more (26% vs 11%), which is not unfair by itself: towns have more outages and really churn more (15% vs 8%). But city recall is only 0.34: city leavers are being missed. That is the next thing to improve.

Step 9: the results write-up

Section Plain words
Goal Choose who gets a Rs 500 retention call
Model Calibrated logistic regression (beat gradient boosting: PR-AUC 0.325 vs 0.255)
Rule Call when risk ≥ 0.167 (from the costs): about 15% of customers
Money (test) Rs 10.1 lakh vs 11.2 lakh with no calls: about 10% saved
Why flagged SHAP: tenure, complaints, contract, autopay; per customer on request
Risks City leavers under-caught; gender recall gap unclear (needs more data); re-check monthly; if the offer price changes, the threshold changes

Common mistakes

Mistake What you see Why Fix
Judging by accuracy 0.903, but 5 of 191 caught rare class recall, precision, PR-AUC
Plain random split test churn 0.072 to 0.109 luck of the split stratify=y
Default threshold 0.5 Rs 31.3 lakh instead of 29.6 ignores the costs choose t from the costs
Low threshold 0.3 Rs 36.0 lakh: worse than calling nobody too many offers count the cost first
Trusting balanced probabilities said 0.395, real 0.093 class weights inflate p CalibratedClassifierCV
"We dropped gender, so it's fair" recall 0.51 vs 0.38 proxies, chance check every group; bootstrap
Choosing the threshold on the test set an optimistic test score test used twice out-of-fold predictions on train

Mini project

Change the business: the offer costs Rs 800 and keeps 60% of leavers. Work out the new threshold by formula, check it with the cost search on calibrated out-of-fold probabilities, and rewrite the results table.

Practice

1. The offer costs Rs 1,000 and a call saves Rs 3,000 from a real leaver. New threshold (calibrated)?

Solution

p × 3,000 > 1,000 → p > 0.333. No retraining needed.

2. A lost customer now costs Rs 9,000 (the offer still keeps half). New threshold?

Solution

Saved per caught leaver = 9,000 / 2 = 4,500. p > 500 / 4,500 = 0.111. A bigger loss means calling more people.

3. Try sigmoid calibration.

Solution
cal = CalibratedClassifierCV(raw, method="sigmoid", cv=5)

Sigmoid (Platt) fits an S-curve: smoother, better with little data; isotonic is more flexible but can overfit small sets. Compare their Brier scores.

4. Why is PR-AUC more informative than ROC-AUC here?

Solution

ROC-AUC counts true negatives through the false positive rate, and with 90% stayers a model can look good while most of its flags are wrong. PR-AUC uses precision, which directly shows how many flags are real leavers; its random baseline is the churn rate.

5. Why must customer_id be dropped from the inputs?

Solution

It's an arbitrary label. A model could learn accidental patterns in it that won't hold for new customers.

Quick quiz

The offer costs Rs 1,000 and a call saves Rs 3,000; probabilities are calibrated. Call above 0.333 (1,000 / 3,000). Only the threshold moves.

Interview questions

  1. Your churn model has 92% accuracy. Is it good? What do you check?
  2. Precision vs recall: which matters more for churn, for fraud, for cancer screening?
  3. ROC-AUC vs PR-AUC on imbalanced data.
  4. How do you choose a classification threshold?
  5. What does class_weight="balanced" do to the predicted probabilities?
  6. What is calibration, how do you measure it, and how do you fix it?
  7. What is a SHAP value? Why do SHAP values add up to the prediction?
  8. Demographic parity vs equal opportunity.
  9. You removed the protected attribute. Is the model fair now?
  10. Explain your model's result to a non-technical manager in three sentences.

Glossary

Term Meaning, with an example
Churn A customer leaving: 9.3% of our customers.
Imbalanced data One class much rarer than the other: 9.3% vs 90.7%.
Accuracy paradox High accuracy from always predicting the common class: 0.904 for "always stays".
DummyClassifier A no-skill model that predicts the most common class.
Confusion matrix The 2×2 table of TP, FP, FN, TN.
True positive (TP) Flagged and really left.
False positive (FP) Flagged but stayed (a false alarm).
False negative (FN) Not flagged but left (a miss).
True negative (TN) Not flagged and stayed.
Recall TP / (TP + FN): share of leavers caught; 0.45 on test.
Precision TP / (TP + FP): share of flags that were right; 0.29 on test.
Threshold Flag if p ≥ t; 0.167 from the costs.
ROC-AUC Chance a random leaver outranks a random stayer: 0.789.
PR-AUC (average precision) Precision averaged over recall levels: 0.321 vs random 0.093.
Stratified split Keeps the class share equal in train and test.
Pipeline Preprocessing and model fitted as one object.
One-hot encoding One 0/1 column per category: contract_monthly.
class_weight="balanced" Weights classes inversely to their frequency: each churner counts ~10×.
Out-of-fold prediction A prediction from a CV model that never saw that row (cross_val_predict).
Business cost The rupee cost of each outcome: FP 500, FN 6,000, TP 3,500.
Calibration Probabilities that match reality: said 0.093, real 0.093.
Brier score Mean of (p − outcome)²: 0.1865 → 0.0740.
Isotonic calibration A step-shaped increasing mapping from scores to probabilities.
Platt (sigmoid) scaling An S-curve mapping from scores to probabilities.
CalibratedClassifierCV scikit-learn wrapper that calibrates a model with CV.
Log-odds log(p / (1 − p)), the score before the sigmoid.
SHAP value A feature's fair share of one prediction: complaints +2.24.
Base value The model's average prediction, where SHAP starts.
Shapley value The game-theory idea SHAP is built on: a fair split of a total.
LinearExplainer SHAP's explainer for linear models.
Protected attribute A characteristic decisions shouldn't depend on: gender.
Proxy A column that carries the same information as a protected one.
Demographic parity Similar flag rates across groups: 15% vs 14% by gender.
Equal opportunity Similar recall across groups: 0.51 vs 0.38 by gender.
Bootstrap Resampling with replacement to measure chance variation: gap interval −0.02 to 0.27.
Results write-up One page for decision makers: goal, rule, money, risks.

The data generator

Run this once to make /data/churn.csv (seed 89; every number is invented):

"""Make the made-up broadband churn dataset for ml_ep89 (customer churn project). Writes /data/churn.csv.
Every number is invented (random, seed 89). 'gender' is NOT used to make churn; it is kept only for the fairness check."""
import numpy as np, pandas as pd, os
rng = np.random.default_rng(89); n = 8000
region = rng.choice(["city", "town"], n, p=[.8, .2])
gender = rng.choice(["female", "male"], n)
contract = rng.choice(["monthly", "yearly"], n, p=[.55, .45])
plan = rng.choice(["basic", "standard", "premium"], n, p=[.4, .4, .2])
tenure = rng.integers(1, 73, n)
bill = np.select([plan == "basic", plan == "standard"], [499, 799], 1199) + rng.normal(0, 40, n).round()
outages = rng.poisson(np.where(region == "town", 1.6, 0.6))
complaints = rng.poisson(0.25 + 0.35 * outages)
calls = rng.poisson(0.5 + 0.6 * complaints)
autopay = rng.random(n) < 0.5
data_gb = rng.gamma(3, np.select([plan == "basic", plan == "standard"], [40, 80], 140))
z = (-2.5 + 1.3 * (contract == "monthly") + 0.55 * complaints + 0.30 * outages - 0.035 * tenure
     + 0.0012 * (bill - 800) - 0.7 * autopay + 0.25 * (calls >= 3) + 0.8 * (tenure <= 3))
churn = rng.random(n) < 1 / (1 + np.exp(-z))
df = pd.DataFrame({"customer_id": np.arange(100001, 100001 + n), "region": region, "gender": gender,
                   "contract": contract, "plan": plan, "tenure_months": tenure, "monthly_bill": bill,
                   "outages_30d": outages, "complaints_90d": complaints, "support_calls_90d": calls,
                   "autopay": np.where(autopay, "yes", "no"), "data_gb": data_gb.round(1), "churned": churn.astype(int)})
os.makedirs("/data", exist_ok=True); df.to_csv("/data/churn.csv", index=False)
print(df.shape, "churn rate", round(df.churned.mean(), 3))

Summary

Accuracy hid a useless model (5 of 191 churners). We switched to recall, precision, ROC-AUC and PR-AUC, split with stratification, and found that a balanced logistic regression beat gradient boosting. We picked the threshold with rupees, discovered the balanced probabilities were inflated (0.395 vs 0.093), calibrated them so the formula threshold 0.167 worked, tested once (10.1 vs 11.2 lakh), explained a customer with SHAP, checked fairness honestly (a gender gap within chance, city leavers under-caught), and wrote it up for the manager.