Machine Learning Full Course · Episode 89. Every program below was run on a Linux machine with Python 3.13, scikit-learn 1.9.1, pandas 3.0.5, NumPy 2.5.3 and SHAP 0.54.1 (one CPU thread), and the output shown is the real output. The broadband customers and the rupee costs are made up: the generator is at the end, so you can make exactly the same file. gender was not used to make churn; it is kept only for the fairness check.
The big idea: when the class you care about is rare, accuracy lies. Judge the model by how well it ranks the rare class (ROC-AUC, PR-AUC), turn its scores into decisions with a threshold chosen from business costs, make sure its probabilities are calibrated so that maths works, explain individual decisions with SHAP, check the results by group before anyone uses them, and write it all up in plain words.
Words you need first
- Churn: a customer leaving. A customer who leaves has churned.
- Classification: predicting a class (leaves / stays), usually via a probability.
- Probability: a number from 0 to 1; 0.3 should mean "3 in 10 such customers leave".
- Threshold: flag the customer if the predicted probability ≥ t. Default t = 0.5.
- Imbalanced data: one class is much rarer than the other (9.3% churn here).
The problem
Real-life example: the broadband company that keeps losing people. 8,000 customers; about 9 in 100 leave each quarter. Each lost customer costs Rs 6,000. A retention call with a Rs 500 offer keeps about half of those who would leave. Calling everyone wastes Rs 500 on every happy customer. Who should get the call?
Step 1: the accuracy trap
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.dummy import DummyClassifier
df = pd.read_csv("/data/churn.csv") # 8,000 made-up broadband customers
print("churn rate:", round(df.churned.mean(), 3))
X = df.select_dtypes("number").drop(columns=["customer_id", "churned"]); y = df["churned"]
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.25, random_state=0)
for m in [LogisticRegression(max_iter=1000), DummyClassifier()]: # Dummy: always "stays"
p = m.fit(Xtr, ytr).predict(Xte)
print(f"{type(m).__name__:18} accuracy {(p == yte).mean():.3f} "
f"churners caught {((p == 1) & (yte == 1)).sum()} of {yte.sum()}")
Output:
churn rate: 0.093
LogisticRegression accuracy 0.903 churners caught 5 of 191
DummyClassifier accuracy 0.904 churners caught 0 of 191
90.3% accuracy, but only 5 of 191 churners caught, and a model that always says "stays" scores higher (90.4%). This is the accuracy paradox: with 9.3% churn, predicting "stays" for everyone is already about 91% right.
| Metric | Question it answers | Here |
|---|---|---|
| Recall (sensitivity) | Of the people who really left, what share did we catch? | 5 / 191 = 0.03 |
| Precision | Of the people we flagged, what share really left? | |
| Accuracy | What share of all predictions was right? | 0.903 (misleading) |
Step 2: metrics for rare events, and a stratified split
- ROC-AUC: the chance that a random leaver gets a higher score than a random stayer. 0.5 = coin toss, 1 = perfect.
- PR-AUC (average precision): precision averaged over all levels of recall. A random model scores the churn rate (0.093), so that is the number to beat.
- Both judge the ranking, without choosing a threshold.
- Stratified split: keeps the same class share in train and test (
stratify=y).
import pandas as pd
from sklearn.model_selection import train_test_split
df = pd.read_csv("/data/churn.csv")
for name, strat in [("plain ", None), ("stratified", df.churned)]: # 20 different random splits
tests = [train_test_split(df, test_size=0.1, stratify=strat, random_state=s)[1] for s in range(20)]
rates = [t.churned.mean() for t in tests]
print(f"{name} test churn rate over 20 splits: {min(rates):.3f} to {max(rates):.3f}")
train, test = train_test_split(df, test_size=0.25, stratify=df.churned, random_state=89)
train.to_csv("/data/churn_train.csv", index=False); test.to_csv("/data/churn_test.csv", index=False)
print("train", len(train), " test", len(test), " churners in test:", test.churned.sum())
Output:
plain test churn rate over 20 splits: 0.072 to 0.109
stratified test churn rate over 20 splits: 0.092 to 0.092
train 6000 test 2000 churners in test: 186
With a plain split, the test churn rate swings from 7.2% to 10.9%: your score depends on luck. Stratified: 9.2% every time.
Step 3: pipelines and two models
class_weight="balanced" gives each class a weight inversely proportional to its frequency, so each churner counts about ten times as much as a stayer during training. Gender is left out of the inputs on purpose.
import pandas as pd, joblib
from sklearn.model_selection import cross_validate
from sklearn.compose import make_column_transformer
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import HistGradientBoostingClassifier
tr, cat = pd.read_csv("/data/churn_train.csv"), ["region", "contract", "plan", "autopay"]
X, y = tr.drop(columns=["customer_id", "gender", "churned"]), tr.churned # gender: not an input
lr = make_pipeline(make_column_transformer((OneHotEncoder(), cat), remainder=StandardScaler()),
LogisticRegression(class_weight="balanced")) # balanced: churners count ~10x
gb = make_pipeline(make_column_transformer((OneHotEncoder(), cat), remainder="passthrough"),
HistGradientBoostingClassifier(max_iter=200, learning_rate=0.05, random_state=0))
for name, m in [("logistic (balanced)", lr), ("gradient boosting", gb)]:
s = cross_validate(m, X, y, cv=5, scoring=["roc_auc", "average_precision", "recall", "precision"])
s = {k[5:]: v.mean() for k, v in s.items() if k.startswith("test_")}
print(f"{name:20} ROC-AUC {s['roc_auc']:.3f} PR-AUC {s['average_precision']:.3f}"
f" recall {s['recall']:.2f} precision {s['precision']:.2f}")
joblib.dump(lr, "/data/churn_recipe.joblib") # save the (unfitted) winning pipeline
Output:
logistic (balanced) ROC-AUC 0.790 PR-AUC 0.325 recall 0.72 precision 0.20
gradient boosting ROC-AUC 0.732 PR-AUC 0.255 recall 0.10 precision 0.39
The simple logistic model wins on both ranking metrics (PR-AUC 0.325, 3.5× the random 0.093), so it is the one we use. At the default threshold the two models make very different trade-offs (recall 0.72 vs 0.10); the money decides which trade-off is right. We save the winning pipeline, unfitted, as a "recipe" so every later program uses exactly the same steps.
Step 4: a threshold from business cost
Real-life example: every mistake has a price tag.
| Outcome | Meaning | Cost (made up) |
|---|---|---|
| True positive (TP) | called someone who would leave | Rs 500 offer + half of Rs 6,000 still lost = Rs 3,500 |
| False positive (FP) | called a happy customer | Rs 500 wasted |
| False negative (FN) | didn't call someone who leaves | Rs 6,000 lost |
| True negative (TN) | didn't call a happy customer | Rs 0 |
cross_val_predict gives every training customer a probability from a model that never saw them (out-of-fold predictions), so we can choose the threshold without touching the test set.
import numpy as np, pandas as pd, joblib
from sklearn.model_selection import cross_val_predict
tr = pd.read_csv("/data/churn_train.csv")
X, y = tr.drop(columns=["customer_id", "gender", "churned"]), tr["churned"].values
model = joblib.load("/data/churn_recipe.joblib") # the pipeline saved last time (not fitted)
p = cross_val_predict(model, X, y, cv=5, method="predict_proba")[:, 1] # out-of-fold probabilities
OFFER, LOST = 500, 6000 # made-up rupees: offer, lost customer; an offer keeps half
def cost(t):
flag = p >= t
tp, fp, fn = (flag & (y == 1)).sum(), (flag & (y == 0)).sum(), (~flag & (y == 1)).sum()
return (tp * (OFFER + LOST / 2) + fp * OFFER + fn * LOST) / 1e5 # in lakh
print(f"call nobody: cost Rs {cost(1.01):.1f} lakh")
for t in [0.3, 0.5, 0.7]: print(f"threshold {t}: cost Rs {cost(t):.1f} lakh flagged {(p >= t).sum()}")
ts = np.arange(0.05, 0.96, 0.01); best = ts[np.argmin([cost(t) for t in ts])]
print(f"best threshold {best:.2f}: cost Rs {cost(best):.1f} lakh flagged {(p >= best).sum()} of {len(y)}")
Output:
call nobody: cost Rs 33.4 lakh
threshold 0.3: cost Rs 36.0 lakh flagged 3526
threshold 0.5: cost Rs 31.3 lakh flagged 1968
threshold 0.7: cost Rs 29.7 lakh flagged 771
best threshold 0.68: cost Rs 29.6 lakh flagged 875 of 6000
A low threshold (0.3) is worse than calling nobody: 3,526 calls waste too many offers. The cheapest is 0.68: 29.6 lakh, saving 3.8 lakh a quarter versus doing nothing.
Step 5: calibration
A puzzle. Call a customer when p × (loss a call saves) > offer cost: p × 3,000 > 500, so p > 0.167. Our search said 0.68. So the probabilities themselves must be wrong.
- Calibrated: a model's probabilities mean what they say: of customers given 0.3, about 30% really leave.
class_weight="balanced"pretends churners are ~10× more common, so it pushes every probability up.- Brier score: the mean of (p − actual outcome)². Lower is better.
CalibratedClassifierCV(method="isotonic"): learns, with cross-validation, a step-shaped increasing mapping from the model's scores to real probabilities. (method="sigmoid", also called Platt scaling, fits an S-curve instead.)
import numpy as np, pandas as pd, joblib
from sklearn.model_selection import cross_val_predict
from sklearn.calibration import CalibratedClassifierCV
from sklearn.metrics import brier_score_loss
tr = pd.read_csv("/data/churn_train.csv")
X, y = tr.drop(columns=["customer_id", "gender", "churned"]), tr["churned"].values
raw = joblib.load("/data/churn_recipe.joblib") # balanced logistic pipeline
cal = CalibratedClassifierCV(raw, method="isotonic", cv=5) # learns: score -> real probability
for name, m in [("balanced", raw), ("calibrated", cal)]:
p = cross_val_predict(m, X, y, cv=5, method="predict_proba")[:, 1]
bins = pd.cut(p, [0, .1, .2, .4, .6, 1]) # group customers by predicted risk
t = pd.DataFrame({"said": p, "real": y}).groupby(bins, observed=True).mean().round(2)
print(f"{name}: average said {p.mean():.3f}, real {y.mean():.3f} Brier {brier_score_loss(y, p):.4f}")
print(" said:", t.said.tolist(), "\n real:", t.real.tolist())
Output:
balanced: average said 0.395, real 0.093 Brier 0.1865
said: [0.07, 0.15, 0.3, 0.5, 0.74]
real: [0.01, 0.02, 0.04, 0.09, 0.25]
calibrated: average said 0.093, real 0.093 Brier 0.0740
said: [0.04, 0.13, 0.3, 0.46, 0.79]
real: [0.04, 0.12, 0.31, 0.41, 0.61]
The balanced model says 0.395 on average when the truth is 0.093; in its riskiest group it says 0.74 and only 25% leave. After calibration: 0.093 vs 0.093, group by group close, Brier 0.1865 → 0.0740. Calibration changes the probabilities but keeps the ranking, so ROC-AUC barely moves.
Step 6: test once
import pandas as pd, joblib
from sklearn.calibration import CalibratedClassifierCV
from sklearn.metrics import average_precision_score, roc_auc_score
tr, te = pd.read_csv("/data/churn_train.csv"), pd.read_csv("/data/churn_test.csv")
drop = ["customer_id", "gender", "churned"]
lr = joblib.load("/data/churn_recipe.joblib")
model = CalibratedClassifierCV(lr, method="isotonic", cv=5).fit(tr.drop(columns=drop), tr.churned)
OFFER, LOST = 500, 6000
T = OFFER / (LOST / 2) # flag when p x (loss we save) > offer cost -> p > 0.167
p, y = model.predict_proba(te.drop(columns=drop))[:, 1], te.churned.values # test set, opened ONCE
flag = p >= T; tp, fp, fn = (flag & (y == 1)).sum(), (flag & (y == 0)).sum(), (~flag & (y == 1)).sum()
cost = tp * (OFFER + LOST / 2) + fp * OFFER + fn * LOST
print(f"threshold {T:.3f} flagged {flag.sum()} of {len(y)} recall {tp / (tp + fn):.2f}"
f" precision {tp / (tp + fp):.2f}")
print(f"ROC-AUC {roc_auc_score(y, p):.3f} PR-AUC {average_precision_score(y, p):.3f}")
print(f"cost Rs {cost / 1e5:.1f} lakh vs call nobody Rs {y.sum() * LOST / 1e5:.1f} lakh")
joblib.dump(model, "/data/churn_model_v1.joblib")
Output:
threshold 0.167 flagged 291 of 2000 recall 0.45 precision 0.29
ROC-AUC 0.789 PR-AUC 0.321
cost Rs 10.1 lakh vs call nobody Rs 11.2 lakh
| Value | |
|---|---|
| Threshold | 0.167, straight from the costs |
| Customers flagged | 291 of 2,000 (about 15%) |
| Recall / precision | 0.45 / 0.29 |
| ROC-AUC / PR-AUC | 0.789 / 0.321 (matches CV) |
| Cost | Rs 10.1 lakh vs 11.2 lakh calling nobody: about 10% saved, as on the training data |
When costs change, only the threshold moves: no retraining needed, because the probabilities are calibrated.
Step 7: SHAP explanations
Real-life example: splitting a restaurant bill fairly. Start from the average meal (the base value, the model's average prediction); each person pays for what they added (each feature's SHAP value: how much it pushed this one prediction up or down); the shares add up exactly to the bill (base value + all SHAP values = this customer's prediction). SHAP comes from Shapley values in game theory, a fair way to split a total among players.
For logistic regression, SHAP values are in log-odds, the score before the sigmoid: log(p / (1 − p)). LinearExplainer uses training customers as the background (the note about 100 samples is SHAP using a sample of them).
import numpy as np, pandas as pd, joblib, shap
tr, te = pd.read_csv("/data/churn_train.csv"), pd.read_csv("/data/churn_test.csv")
drop = ["customer_id", "gender", "churned"]
Xtr, Xte = tr.drop(columns=drop), te.drop(columns=drop)
pipe = joblib.load("/data/churn_recipe.joblib")
pipe.set_params(columntransformer__verbose_feature_names_out=False).fit(Xtr, tr.churned) # short names
names = pipe[0].get_feature_names_out()
explainer = shap.LinearExplainer(pipe[1], pipe[0].transform(Xtr)) # background: training rows
sv = explainer(pipe[0].transform(Xte)).values # one push per feature per customer (log-odds)
print("most important overall (mean |SHAP|):")
print(pd.Series(np.abs(sv).mean(0), names).nlargest(5).round(2).to_dict())
i = int(np.argmax(sv.sum(1))); r = te.iloc[i] # the riskiest test customer
print(f"customer {r.customer_id}: {r.contract}, tenure {r.tenure_months}, complaints {r.complaints_90d},"
f" outages {r.outages_30d}, autopay {r.autopay}")
top = pd.Series(sv[i], names).sort_values(key=abs, ascending=False).head(4)
print("biggest pushes toward churn:", top.round(2).to_dict())
Output:
most important overall (mean |SHAP|):
{'tenure_months': 0.7, 'complaints_90d': 0.33, 'contract_yearly': 0.27, 'autopay_yes': 0.21, 'contract_monthly': 0.2}
customer 106609: monthly, tenure 26, complaints 5, outages 6, autopay no
biggest pushes toward churn: {'complaints_90d': 2.24, 'outages_30d': 1.04, 'support_calls_90d': 0.46, 'tenure_months': 0.44}
Overall, tenure matters most (new customers leave more), then complaints, contract type and autopay. For customer 106609 the biggest pushes are complaints (+2.24) and outages (+1.04), so the call script should apologise and fix the outages; a discount alone won't help.
Step 8: a fairness check
- Protected attribute: a characteristic decisions should not be based on, such as gender. We left it out of the inputs.
- Proxy: another column that carries the same information, so leaving the attribute out is not enough.
- Demographic parity: are the groups flagged at similar rates?
- Equal opportunity: among people who really leave, are the groups caught at similar rates (recall per group)? Here being caught means getting a helpful offer.
- Bootstrap: resample the data with replacement many times to see how much a number moves by chance.
import numpy as np, pandas as pd, joblib
model = joblib.load("/data/churn_model_v1.joblib") # calibrated model from the test-once program
te = pd.read_csv("/data/churn_test.csv")
te["flag"] = model.predict_proba(te.drop(columns=["customer_id", "gender", "churned"]))[:, 1] >= 0.167
def report(g):
caught = g[g.churned == 1].flag.mean() # recall inside this group
return pd.Series({"n": len(g), "real churn": g.churned.mean(), "flagged": g.flag.mean(),
"recall": caught, "precision": g[g.flag].churned.mean()})
for col in ["gender", "region"]: # gender was never an input. Check anyway.
t = te.groupby(col)[["churned", "flag"]].apply(report).round(2); t["n"] = t.n.astype(int)
print(t.to_string(), "\n")
rng = np.random.default_rng(0); ch = te[te.churned == 1] # is the gender gap real, or chance?
gaps = [(s[s.gender == "female"].flag.mean() - s[s.gender == "male"].flag.mean())
for s in (ch.sample(len(ch), replace=True, random_state=rng) for _ in range(1000))]
lo, hi = np.percentile(gaps, [2.5, 97.5])
print(f"recall gap female - male: {np.mean(gaps):.2f}, 95% interval {lo:.2f} to {hi:.2f}")
Output:
n real churn flagged recall precision
gender
female 1001 0.10 0.15 0.51 0.34
male 999 0.09 0.14 0.38 0.24
n real churn flagged recall precision
region
city 1597 0.08 0.11 0.34 0.24
town 403 0.15 0.26 0.68 0.38
recall gap female - male: 0.13, 95% interval -0.02 to 0.27
- Gender: flagged at almost the same rate (15% vs 14%), but recall 0.51 vs 0.38. The 95% bootstrap interval for the gap (−0.02 to 0.27) includes zero: with only 186 leavers we cannot tell it from chance. Report it and keep measuring.
- Region: towns are flagged far more (26% vs 11%), which is not unfair by itself: towns have more outages and really churn more (15% vs 8%). But city recall is only 0.34: city leavers are being missed. That is the next thing to improve.
Step 9: the results write-up
| Section | Plain words |
|---|---|
| Goal | Choose who gets a Rs 500 retention call |
| Model | Calibrated logistic regression (beat gradient boosting: PR-AUC 0.325 vs 0.255) |
| Rule | Call when risk ≥ 0.167 (from the costs): about 15% of customers |
| Money (test) | Rs 10.1 lakh vs 11.2 lakh with no calls: about 10% saved |
| Why flagged | SHAP: tenure, complaints, contract, autopay; per customer on request |
| Risks | City leavers under-caught; gender recall gap unclear (needs more data); re-check monthly; if the offer price changes, the threshold changes |
Common mistakes
| Mistake | What you see | Why | Fix |
|---|---|---|---|
| Judging by accuracy | 0.903, but 5 of 191 caught | rare class | recall, precision, PR-AUC |
| Plain random split | test churn 0.072 to 0.109 | luck of the split | stratify=y |
| Default threshold 0.5 | Rs 31.3 lakh instead of 29.6 | ignores the costs | choose t from the costs |
| Low threshold 0.3 | Rs 36.0 lakh: worse than calling nobody | too many offers | count the cost first |
| Trusting balanced probabilities | said 0.395, real 0.093 | class weights inflate p | CalibratedClassifierCV |
| "We dropped gender, so it's fair" | recall 0.51 vs 0.38 | proxies, chance | check every group; bootstrap |
| Choosing the threshold on the test set | an optimistic test score | test used twice | out-of-fold predictions on train |
Mini project
Change the business: the offer costs Rs 800 and keeps 60% of leavers. Work out the new threshold by formula, check it with the cost search on calibrated out-of-fold probabilities, and rewrite the results table.
Practice
1. The offer costs Rs 1,000 and a call saves Rs 3,000 from a real leaver. New threshold (calibrated)?
Solution
p × 3,000 > 1,000 → p > 0.333. No retraining needed.
2. A lost customer now costs Rs 9,000 (the offer still keeps half). New threshold?
Solution
Saved per caught leaver = 9,000 / 2 = 4,500. p > 500 / 4,500 = 0.111. A bigger loss means calling more people.
3. Try sigmoid calibration.
Solution
cal = CalibratedClassifierCV(raw, method="sigmoid", cv=5)
Sigmoid (Platt) fits an S-curve: smoother, better with little data; isotonic is more flexible but can overfit small sets. Compare their Brier scores.
4. Why is PR-AUC more informative than ROC-AUC here?
Solution
ROC-AUC counts true negatives through the false positive rate, and with 90% stayers a model can look good while most of its flags are wrong. PR-AUC uses precision, which directly shows how many flags are real leavers; its random baseline is the churn rate.
5. Why must customer_id be dropped from the inputs?
Solution
It's an arbitrary label. A model could learn accidental patterns in it that won't hold for new customers.
Quick quiz
The offer costs Rs 1,000 and a call saves Rs 3,000; probabilities are calibrated. Call above 0.333 (1,000 / 3,000). Only the threshold moves.
Interview questions
- Your churn model has 92% accuracy. Is it good? What do you check?
- Precision vs recall: which matters more for churn, for fraud, for cancer screening?
- ROC-AUC vs PR-AUC on imbalanced data.
- How do you choose a classification threshold?
- What does
class_weight="balanced"do to the predicted probabilities? - What is calibration, how do you measure it, and how do you fix it?
- What is a SHAP value? Why do SHAP values add up to the prediction?
- Demographic parity vs equal opportunity.
- You removed the protected attribute. Is the model fair now?
- Explain your model's result to a non-technical manager in three sentences.
Glossary
| Term | Meaning, with an example |
|---|---|
| Churn | A customer leaving: 9.3% of our customers. |
| Imbalanced data | One class much rarer than the other: 9.3% vs 90.7%. |
| Accuracy paradox | High accuracy from always predicting the common class: 0.904 for "always stays". |
| DummyClassifier | A no-skill model that predicts the most common class. |
| Confusion matrix | The 2×2 table of TP, FP, FN, TN. |
| True positive (TP) | Flagged and really left. |
| False positive (FP) | Flagged but stayed (a false alarm). |
| False negative (FN) | Not flagged but left (a miss). |
| True negative (TN) | Not flagged and stayed. |
| Recall | TP / (TP + FN): share of leavers caught; 0.45 on test. |
| Precision | TP / (TP + FP): share of flags that were right; 0.29 on test. |
| Threshold | Flag if p ≥ t; 0.167 from the costs. |
| ROC-AUC | Chance a random leaver outranks a random stayer: 0.789. |
| PR-AUC (average precision) | Precision averaged over recall levels: 0.321 vs random 0.093. |
| Stratified split | Keeps the class share equal in train and test. |
| Pipeline | Preprocessing and model fitted as one object. |
| One-hot encoding | One 0/1 column per category: contract_monthly. |
| class_weight="balanced" | Weights classes inversely to their frequency: each churner counts ~10×. |
| Out-of-fold prediction | A prediction from a CV model that never saw that row (cross_val_predict). |
| Business cost | The rupee cost of each outcome: FP 500, FN 6,000, TP 3,500. |
| Calibration | Probabilities that match reality: said 0.093, real 0.093. |
| Brier score | Mean of (p − outcome)²: 0.1865 → 0.0740. |
| Isotonic calibration | A step-shaped increasing mapping from scores to probabilities. |
| Platt (sigmoid) scaling | An S-curve mapping from scores to probabilities. |
| CalibratedClassifierCV | scikit-learn wrapper that calibrates a model with CV. |
| Log-odds | log(p / (1 − p)), the score before the sigmoid. |
| SHAP value | A feature's fair share of one prediction: complaints +2.24. |
| Base value | The model's average prediction, where SHAP starts. |
| Shapley value | The game-theory idea SHAP is built on: a fair split of a total. |
| LinearExplainer | SHAP's explainer for linear models. |
| Protected attribute | A characteristic decisions shouldn't depend on: gender. |
| Proxy | A column that carries the same information as a protected one. |
| Demographic parity | Similar flag rates across groups: 15% vs 14% by gender. |
| Equal opportunity | Similar recall across groups: 0.51 vs 0.38 by gender. |
| Bootstrap | Resampling with replacement to measure chance variation: gap interval −0.02 to 0.27. |
| Results write-up | One page for decision makers: goal, rule, money, risks. |
The data generator
Run this once to make /data/churn.csv (seed 89; every number is invented):
"""Make the made-up broadband churn dataset for ml_ep89 (customer churn project). Writes /data/churn.csv.
Every number is invented (random, seed 89). 'gender' is NOT used to make churn; it is kept only for the fairness check."""
import numpy as np, pandas as pd, os
rng = np.random.default_rng(89); n = 8000
region = rng.choice(["city", "town"], n, p=[.8, .2])
gender = rng.choice(["female", "male"], n)
contract = rng.choice(["monthly", "yearly"], n, p=[.55, .45])
plan = rng.choice(["basic", "standard", "premium"], n, p=[.4, .4, .2])
tenure = rng.integers(1, 73, n)
bill = np.select([plan == "basic", plan == "standard"], [499, 799], 1199) + rng.normal(0, 40, n).round()
outages = rng.poisson(np.where(region == "town", 1.6, 0.6))
complaints = rng.poisson(0.25 + 0.35 * outages)
calls = rng.poisson(0.5 + 0.6 * complaints)
autopay = rng.random(n) < 0.5
data_gb = rng.gamma(3, np.select([plan == "basic", plan == "standard"], [40, 80], 140))
z = (-2.5 + 1.3 * (contract == "monthly") + 0.55 * complaints + 0.30 * outages - 0.035 * tenure
+ 0.0012 * (bill - 800) - 0.7 * autopay + 0.25 * (calls >= 3) + 0.8 * (tenure <= 3))
churn = rng.random(n) < 1 / (1 + np.exp(-z))
df = pd.DataFrame({"customer_id": np.arange(100001, 100001 + n), "region": region, "gender": gender,
"contract": contract, "plan": plan, "tenure_months": tenure, "monthly_bill": bill,
"outages_30d": outages, "complaints_90d": complaints, "support_calls_90d": calls,
"autopay": np.where(autopay, "yes", "no"), "data_gb": data_gb.round(1), "churned": churn.astype(int)})
os.makedirs("/data", exist_ok=True); df.to_csv("/data/churn.csv", index=False)
print(df.shape, "churn rate", round(df.churned.mean(), 3))
Summary
Accuracy hid a useless model (5 of 191 churners). We switched to recall, precision, ROC-AUC and PR-AUC, split with stratification, and found that a balanced logistic regression beat gradient boosting. We picked the threshold with rupees, discovered the balanced probabilities were inflated (0.395 vs 0.093), calibrated them so the formula threshold 0.167 worked, tested once (10.1 vs 11.2 lakh), explained a customer with SHAP, checked fairness honestly (a gender gap within chance, city leavers under-caught), and wrote it up for the manager.