SyntaxPilot
Courses/Machine Learning/Lesson 83
Video coming soonThis lesson’s video is being made

Start with the notes below; the video appears here as soon as it is on YouTube.

Machine Learning · Lesson 83 of 90

Explainability

interpretability vs explainabilityintrinsically interpretable modelsglobal vs local explanationspermutation importance (revisited)partial dependence plots (PDP)individual conditional expectation (ICE)Shapley valuesSHAPLIMEcounterfactual explanations

Machine Learning Full Course · Episode 83. Every program below was run on a Linux machine with Python 3.13, NumPy 2.5.3, scikit-learn 1.9.1, shap 0.54.1 and lime 0.2.0.1, and the output shown is the real output. The loan data is made up: random numbers with a hidden rule (risk depends on the loan compared with income, plus a penalty for a credit score below 640; age has no effect).

The big idea: an accurate model is often a black box: you see inputs and outputs, not reasons. Some models are interpretable by design (logistic regression, small trees), but they can be less accurate. For black boxes we use explanation methods: global ones describe the model over everyone (permutation importance, PDP, ICE), local ones explain one prediction (Shapley values / SHAP, LIME, counterfactuals). Always check the units and the stability of an explanation.

Sources: Scott Lundberg and Su-In Lee, A Unified Approach to Interpreting Model Predictions (NIPS 2017): "SHAP assigns each feature an importance value for a particular prediction." Marco Tulio Ribeiro, Sameer Singh and Carlos Guestrin, "Why Should I Trust You?": Explaining the Predictions of Any Classifier (2016): LIME "explains the predictions of any classifier in an interpretable and faithful manner, by learning an interpretable model locally around the prediction".

Words you need first

  • Gradient boosting: many small trees added one after another, each fixing the previous ones' mistakes (Episode 36).
  • Logistic regression: a weighted sum of features passed through the sigmoid to give a probability (Episode 22).
  • Log-odds: the raw score before the sigmoid; sigmoid(1.685) = 0.844. Can be any number, positive or negative.
  • Permutation importance: shuffle one column and measure how much accuracy drops (Episode 34).
  • Background data: a sample of ordinary customers used as the "average" reference.

The story: why was Asha rejected?

Asha earns ₹50,000 a month, asks for an ₹8 lakh loan, has credit score 690 and is 29. The bank's most accurate model, 300 gradient-boosted trees, rejects her. She, the bank's staff and the auditors all ask: why? The obvious attempt: open the model and read it.

import numpy as np
rng, n = np.random.default_rng(83), 4000   # made-up loans: income (thousand ₹/month), loan (lakh ₹)
X = np.c_[rng.normal(60, 20, n).clip(10), rng.normal(8, 4, n).clip(1),
          rng.normal(700, 60, n).clip(300, 900), rng.integers(21, 61, n)]   # credit score, age
y = (X[:, 1] * 100 / X[:, 0] + 8 * (X[:, 2] < 640) + rng.normal(0, 2, n) > 13).astype(int)
from sklearn.ensemble import GradientBoostingClassifier
model = GradientBoostingClassifier(n_estimators=300, random_state=0).fit(X, y)
asha = [[50, 8, 690, 29]]                  # income 50, loan 8, credit score 690, age 29
print("Asha's risk:", round(model.predict_proba(asha)[0, 1], 3), "-> REJECTED")
print("Why? The model is", len(model.estimators_), "trees with",
      sum(t[0].tree_.node_count for t in model.estimators_), "decision nodes in total.")
Asha's risk: 0.844 -> REJECTED
Why? The model is 300 trees with 4388 decision nodes in total.

4,388 decision nodes. Reading them gives no reason a person could use. That is a black box.

Real-life example: a recipe card and a master chef

In the kitchen In ML
A recipe card anyone can read: 2 cups rice, 1 spoon salt Interpretable model: read the rule itself
A master chef who cooks by feel; you ask afterwards what made it good Explanation method for a black box (post-hoc)
"Salt matters in all my dishes" vs "this dal needed more ghee" Global vs local explanation

Interpretability vs explainability

  • Interpretability: a person can understand how the model decides just by looking at it.
  • Explainability: producing reasons for what a model does, usually after training (post-hoc, Latin for "after this"). People often use the two words loosely.
  • Intrinsically interpretable models: readable by design: linear and logistic regression, small decision trees, short rule lists.
  • Global explanation: how the model behaves over everyone ("loan size matters most").
  • Local explanation: why this prediction for this person ("for Asha, income pushed hardest").

Glass box vs black box

import numpy as np
rng, n = np.random.default_rng(83), 4000   # made-up loans: income (thousand ₹/month), loan (lakh ₹)
X = np.c_[rng.normal(60, 20, n).clip(10), rng.normal(8, 4, n).clip(1),
          rng.normal(700, 60, n).clip(300, 900), rng.integers(21, 61, n)]   # credit score, age
y = (X[:, 1] * 100 / X[:, 0] + 8 * (X[:, 2] < 640) + rng.normal(0, 2, n) > 13).astype(int)
from sklearn.model_selection import train_test_split
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
Xtr, Xte, ytr, yte = train_test_split(X, y, random_state=0)
glass = make_pipeline(StandardScaler(), LogisticRegression()).fit(Xtr, ytr)
black = GradientBoostingClassifier(n_estimators=300, random_state=0).fit(Xtr, ytr)
print("glass box (logistic) accuracy :", round(glass.score(Xte, yte), 3))
print("black box (300 trees) accuracy:", round(black.score(Xte, yte), 3))
for name, w in zip(["income", "loan", "credit_score", "age"], glass[-1].coef_[0]):
    print(f"  {name:13} weight {w:+.2f}  (per 1 standard deviation)")
glass box (logistic) accuracy : 0.891
black box (300 trees) accuracy: 0.93
  income        weight -2.17  (per 1 standard deviation)
  loan          weight +3.47  (per 1 standard deviation)
  credit_score  weight -0.99  (per 1 standard deviation)
  age           weight +0.09  (per 1 standard deviation)

The logistic regression's weights are readable (per one standard deviation of each feature: loan +3.47, income −2.17, credit score −0.99, age ≈ 0), but its accuracy is 0.891 against 0.93 for the trees: the real rule divides loan by income and has a jump at score 640, which a straight-line model cannot bend to. When the black box is clearly better, we explain it instead.

Global: permutation importance

import numpy as np
rng, n = np.random.default_rng(83), 4000   # made-up loans: income (thousand ₹/month), loan (lakh ₹)
X = np.c_[rng.normal(60, 20, n).clip(10), rng.normal(8, 4, n).clip(1),
          rng.normal(700, 60, n).clip(300, 900), rng.integers(21, 61, n)]   # credit score, age
y = (X[:, 1] * 100 / X[:, 0] + 8 * (X[:, 2] < 640) + rng.normal(0, 2, n) > 13).astype(int)
from sklearn.model_selection import train_test_split
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.inspection import permutation_importance
Xtr, Xte, ytr, yte = train_test_split(X, y, random_state=0)
black = GradientBoostingClassifier(n_estimators=300, random_state=0).fit(Xtr, ytr)
imp = permutation_importance(black, Xte, yte, n_repeats=5, random_state=0)
for name, drop in zip(["income", "loan", "credit_score", "age"], imp.importances_mean):
    print(f"shuffle {name:13} -> accuracy drops by {drop:.3f}")
shuffle income        -> accuracy drops by 0.161
shuffle loan          -> accuracy drops by 0.302
shuffle credit_score  -> accuracy drops by 0.076
shuffle age           -> accuracy drops by 0.001

Shuffling the loan column costs 0.302 accuracy, income 0.161, credit score 0.076, age 0.001. The model relies on the loan most and ignores age. But permutation importance says how much, not how (direction, shape, interactions).

Global: PDP and ICE

Keep every customer exactly as they are, set their loan to 2, 5, 8, 11, 14 lakh, and predict again.

  • ICE (Individual Conditional Expectation): one person's risk curve as the loan changes.
  • PDP (Partial Dependence Plot): the average of all the ICE curves.
import numpy as np
rng, n = np.random.default_rng(83), 4000   # made-up loans: income (thousand ₹/month), loan (lakh ₹)
X = np.c_[rng.normal(60, 20, n).clip(10), rng.normal(8, 4, n).clip(1),
          rng.normal(700, 60, n).clip(300, 900), rng.integers(21, 61, n)]   # credit score, age
y = (X[:, 1] * 100 / X[:, 0] + 8 * (X[:, 2] < 640) + rng.normal(0, 2, n) > 13).astype(int)
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.inspection import partial_dependence
model = GradientBoostingClassifier(n_estimators=300, random_state=0).fit(X, y)
grid = [2, 5, 8, 11, 14]                         # loan sizes to try (lakh ₹)
def risk_if_loan(rows, loan):                    # same people, loan set to one value
    Z = np.array(rows, dtype=float); Z[:, 1] = loan
    return model.predict_proba(Z)[:, 1]
print("loan:          ", grid)
print("PDP (average): ", [round(float(risk_if_loan(X, v).mean()), 2) for v in grid])
for inc in [25, 60, 110]:                        # three people: incomes 25, 60, 110
    print(f"ICE income {inc:<4}", [round(float(risk_if_loan([[inc, 0, 700, 30]], v)[0]), 2) for v in grid])
pd = partial_dependence(model, X, [1], custom_values={1: grid})    # scikit-learn's own
print("sklearn PDP:   ", np.round(pd["average"][0], 2).tolist())
loan:           [2, 5, 8, 11, 14]
PDP (average):  [0.1, 0.23, 0.58, 0.9, 0.98]
ICE income 25   [0.03, 0.92, 0.99, 1.0, 1.0]
ICE income 60   [0.0, 0.0, 0.5, 0.98, 0.99]
ICE income 110  [0.0, 0.0, 0.01, 0.08, 0.77]
sklearn PDP:    [-4.92, -3.55, 0.82, 3.95, 5.41]

The PDP climbs smoothly from 0.10 to 0.98. The ICE curves show three very different people: for income 25 a 5-lakh loan is already 0.92 risk; for income 60 it is 0.0; for income 110 even 11 lakh is only 0.08. That is an interaction: the effect of the loan depends on income, and the average hid it. Plot ICE curves next to every PDP.

Mistake: scikit-learn's PDP gives −4.92

For gradient boosting, partial_dependence uses a fast tree method (method="recursion") that reports log-odds, not probabilities. Asking for probabilities with only response_method="predict_proba" is refused with the real error:

ValueError: With the 'recursion' method, the response_method must be 'decision_function'. Got predict_proba.

The fix is to ask for probabilities and the brute method, which really re-predicts:

import numpy as np
rng, n = np.random.default_rng(83), 4000   # made-up loans: income (thousand ₹/month), loan (lakh ₹)
X = np.c_[rng.normal(60, 20, n).clip(10), rng.normal(8, 4, n).clip(1),
          rng.normal(700, 60, n).clip(300, 900), rng.integers(21, 61, n)]   # credit score, age
y = (X[:, 1] * 100 / X[:, 0] + 8 * (X[:, 2] < 640) + rng.normal(0, 2, n) > 13).astype(int)
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.inspection import partial_dependence
model = GradientBoostingClassifier(n_estimators=300, random_state=0).fit(X, y)
grid = [2, 5, 8, 11, 14]                         # loan sizes to try (lakh ₹)
def risk_if_loan(rows, loan):                    # same people, loan set to one value
    Z = np.array(rows, dtype=float); Z[:, 1] = loan
    return model.predict_proba(Z)[:, 1]
print("loan:          ", grid)
print("PDP (average): ", [round(float(risk_if_loan(X, v).mean()), 2) for v in grid])
for inc in [25, 60, 110]:                        # three people: incomes 25, 60, 110
    print(f"ICE income {inc:<4}", [round(float(risk_if_loan([[inc, 0, 700, 30]], v)[0]), 2) for v in grid])
pd = partial_dependence(model, X, [1], custom_values={1: grid},
                        response_method="predict_proba", method="brute")   # fix: probabilities
print("sklearn PDP:   ", np.round(pd["average"][0], 2).tolist())
loan:           [2, 5, 8, 11, 14]
PDP (average):  [0.1, 0.23, 0.58, 0.9, 0.98]
ICE income 25   [0.03, 0.92, 0.99, 1.0, 1.0]
ICE income 60   [0.0, 0.0, 0.5, 0.98, 0.99]
ICE income 110  [0.0, 0.0, 0.01, 0.08, 0.77]
sklearn PDP:    [0.1, 0.23, 0.58, 0.9, 0.98]

Local: Shapley values

Real-life example: who earned the win?

In cricket In ML
Three batters chase 60 runs together; who deserves the credit? Features are the players
A batter scores more freely with a set partner: credit depends on the order Try every joining order
Average each batter's extra runs over all orders; the shares add up to 60 Shapley value: a fair share of the prediction

Shapley values come from cooperative game theory. For one prediction:

  • Players = features; prize = Asha's risk minus the average customer's risk.
  • A feature "not in the team yet" takes values from the background data (ordinary customers); when it joins, it takes Asha's real value.
  • A feature's Shapley value = its extra push on the risk, averaged over every joining order.
  • Additivity: base value + all Shapley values = this prediction, exactly.
  • Cost: 4 features have 4! = 24 orders; 30 features have about 2.6 × 10³² orders, so libraries use clever shortcuts.
import numpy as np
rng, n = np.random.default_rng(83), 4000   # made-up loans: income (thousand ₹/month), loan (lakh ₹)
X = np.c_[rng.normal(60, 20, n).clip(10), rng.normal(8, 4, n).clip(1),
          rng.normal(700, 60, n).clip(300, 900), rng.integers(21, 61, n)]   # credit score, age
y = (X[:, 1] * 100 / X[:, 0] + 8 * (X[:, 2] < 640) + rng.normal(0, 2, n) > 13).astype(int)
from itertools import permutations
from sklearn.ensemble import GradientBoostingClassifier
model = GradientBoostingClassifier(n_estimators=300, random_state=0).fit(X, y)
asha, bg = np.array([50, 8, 690, 29]), X[:100]        # bg: 100 ordinary customers
def value(team):                    # average risk when only `team` features are Asha's
    Z = bg.copy(); Z[:, list(team)] = asha[list(team)]
    return model.predict_proba(Z)[:, 1].mean()
phi, orders = np.zeros(4), list(permutations(range(4)))
for order in orders:                # every order the features could "join"
    for k, f in enumerate(order):   # f's extra push when it joins, averaged
        phi[f] += (value(order[:k + 1]) - value(order[:k])) / len(orders)
print("average customer risk:", round(value([]), 3))
for name, p in zip(["income", "loan", "credit_score", "age"], phi): print(f"  {name:13} {p:+.3f}")
print("sum:", round(value([]) + phi.sum(), 3), " Asha's risk:", round(value(range(4)), 3))
average customer risk: 0.533
  income        +0.225
  loan          +0.157
  credit_score  -0.044
  age           -0.027
sum: 0.844  Asha's risk: 0.844

The average customer's risk is 0.533; Asha's income adds +0.225, her loan +0.157, her credit score −0.044 and age −0.027 (tiny, model noise), and they add up to exactly 0.844. Note: globally the loan mattered most, but for Asha her income pushed hardest. Global and local answers can differ.

SHAP

SHAP (Shapley Additive Explanations, Lundberg and Lee, 2017) "assigns each feature an importance value for a particular prediction". It has an exact explainer for a few features and a fast TreeExplainer for tree models; computing SHAP values for many customers also gives a global picture.

import numpy as np
rng, n = np.random.default_rng(83), 4000   # made-up loans: income (thousand ₹/month), loan (lakh ₹)
X = np.c_[rng.normal(60, 20, n).clip(10), rng.normal(8, 4, n).clip(1),
          rng.normal(700, 60, n).clip(300, 900), rng.integers(21, 61, n)]   # credit score, age
y = (X[:, 1] * 100 / X[:, 0] + 8 * (X[:, 2] < 640) + rng.normal(0, 2, n) > 13).astype(int)
import shap
from sklearn.ensemble import GradientBoostingClassifier
model = GradientBoostingClassifier(n_estimators=300, random_state=0).fit(X, y)
asha, bg = np.array([[50, 8, 690, 29]]), X[:100]       # the same 100 ordinary customers
risk = lambda Z: model.predict_proba(Z)[:, 1]
explainer = shap.Explainer(risk, shap.maskers.Independent(bg, max_samples=100),
                           algorithm="exact")
e = explainer(asha)
print("base value :", round(float(e.base_values[0]), 3))
for name, v in zip(["income", "loan", "credit_score", "age"], e.values[0]):
    print(f"  {name:13} {v:+.3f}")
base value : 0.533
  income        +0.225
  loan          +0.157
  credit_score  -0.044
  age           -0.027

Exactly our from-scratch numbers.

Mistake: TreeExplainer values don't add up to the risk

import numpy as np
rng, n = np.random.default_rng(83), 4000   # made-up loans: income (thousand ₹/month), loan (lakh ₹)
X = np.c_[rng.normal(60, 20, n).clip(10), rng.normal(8, 4, n).clip(1),
          rng.normal(700, 60, n).clip(300, 900), rng.integers(21, 61, n)]   # credit score, age
y = (X[:, 1] * 100 / X[:, 0] + 8 * (X[:, 2] < 640) + rng.normal(0, 2, n) > 13).astype(int)
import shap
from sklearn.ensemble import GradientBoostingClassifier
model = GradientBoostingClassifier(n_estimators=300, random_state=0).fit(X, y)
asha, bg = np.array([[50, 8, 690, 29]]), X[:100]
ex = shap.TreeExplainer(model, bg)                  # fast SHAP made for tree models
sv = ex.shap_values(asha)[0]
total = float(ex.expected_value + sv.sum())
print("SHAP values:", np.round(sv, 3))
print("base + sum  :", round(total, 3))
print("Asha's risk :", round(model.predict_proba(asha)[0, 1], 3))
SHAP values: [ 1.3    0.97  -0.745 -0.32 ]
base + sum  : 1.685
Asha's risk : 0.844

1.685 ≠ 0.844 because by default TreeExplainer explains the log-odds, not the probability. Pass the total through the sigmoid:

import numpy as np
rng, n = np.random.default_rng(83), 4000   # made-up loans: income (thousand ₹/month), loan (lakh ₹)
X = np.c_[rng.normal(60, 20, n).clip(10), rng.normal(8, 4, n).clip(1),
          rng.normal(700, 60, n).clip(300, 900), rng.integers(21, 61, n)]   # credit score, age
y = (X[:, 1] * 100 / X[:, 0] + 8 * (X[:, 2] < 640) + rng.normal(0, 2, n) > 13).astype(int)
import shap
from sklearn.ensemble import GradientBoostingClassifier
model = GradientBoostingClassifier(n_estimators=300, random_state=0).fit(X, y)
asha, bg = np.array([[50, 8, 690, 29]]), X[:100]
ex = shap.TreeExplainer(model, bg)                  # fast SHAP made for tree models
sv = ex.shap_values(asha)[0]
total = float(ex.expected_value + sv.sum())
print("SHAP values:", np.round(sv, 3))
print("base + sum  :", round(total, 3))
print("Asha's risk :", round(model.predict_proba(asha)[0, 1], 3))
print("sigmoid(base + sum):", round(1 / (1 + np.exp(-total)), 3))   # fix: log-odds -> probability
SHAP values: [ 1.3    0.97  -0.745 -0.32 ]
base + sum  : 1.685
Asha's risk : 0.844
sigmoid(base + sum): 0.844

(Individual TreeExplainer values are in log-odds units too, so don't compare them directly with probability-based SHAP values.)

LIME

LIME (Local Interpretable Model-agnostic Explanations, Ribeiro, Singh and Guestrin, 2016):

  1. Make many slightly changed copies of Asha.
  2. Ask the black box for each copy's prediction.
  3. Weight copies by how close they are to Asha.
  4. Fit a small linear model to them; its weights are the explanation.

Model-agnostic = works with any model. Local = only true near Asha (like a curved road that looks straight when you zoom in).

Mistake: one LIME run with few samples

import numpy as np
rng, n = np.random.default_rng(83), 4000   # made-up loans: income (thousand ₹/month), loan (lakh ₹)
X = np.c_[rng.normal(60, 20, n).clip(10), rng.normal(8, 4, n).clip(1),
          rng.normal(700, 60, n).clip(300, 900), rng.integers(21, 61, n)]   # credit score, age
y = (X[:, 1] * 100 / X[:, 0] + 8 * (X[:, 2] < 640) + rng.normal(0, 2, n) > 13).astype(int)
from lime.lime_tabular import LimeTabularExplainer
from sklearn.ensemble import GradientBoostingClassifier
model = GradientBoostingClassifier(n_estimators=300, random_state=0).fit(X, y)
asha, names = np.array([50, 8, 690, 29]), ["income", "loan", "credit_score", "age"]
for seed in [1, 2]:                        # same person, same model, two random seeds
    lime = LimeTabularExplainer(X, feature_names=names, random_state=seed)
    exp = lime.explain_instance(asha, model.predict_proba, num_features=4, num_samples=500)
    print(f"seed {seed}:", "  ".join(f"{names[i]} {w:+.2f}" for i, w in exp.as_map()[1]))
seed 1: loan +0.18  income +0.11  credit_score -0.02  age +0.01
seed 2: loan +0.24  credit_score -0.12  income +0.10  age +0.04

Same person, same model, two seeds: credit score −0.02 vs −0.12, and even the order changes. LIME is random. The fix is many more samples, and checking that seeds agree:

import numpy as np
rng, n = np.random.default_rng(83), 4000   # made-up loans: income (thousand ₹/month), loan (lakh ₹)
X = np.c_[rng.normal(60, 20, n).clip(10), rng.normal(8, 4, n).clip(1),
          rng.normal(700, 60, n).clip(300, 900), rng.integers(21, 61, n)]   # credit score, age
y = (X[:, 1] * 100 / X[:, 0] + 8 * (X[:, 2] < 640) + rng.normal(0, 2, n) > 13).astype(int)
from lime.lime_tabular import LimeTabularExplainer
from sklearn.ensemble import GradientBoostingClassifier
model = GradientBoostingClassifier(n_estimators=300, random_state=0).fit(X, y)
asha, names = np.array([50, 8, 690, 29]), ["income", "loan", "credit_score", "age"]
for seed in [1, 2]:                        # fix: 20000 samples instead of 500
    lime = LimeTabularExplainer(X, feature_names=names, random_state=seed)
    exp = lime.explain_instance(asha, model.predict_proba, num_features=4, num_samples=20000)
    print(f"seed {seed}:", "  ".join(f"{names[i]} {w:+.2f}" for i, w in exp.as_map()[1]))
seed 1: loan +0.24  income +0.10  credit_score -0.10  age -0.02
seed 2: loan +0.23  income +0.09  credit_score -0.09  age -0.02

Counterfactual explanations

A counterfactual explanation is the smallest change to the inputs that flips the decision: "If your loan were ₹7.1 lakh instead of ₹8 lakh, you would be approved." Only suggest actionable features (loan amount, income, debts), never age, gender or caste.

import numpy as np
rng, n = np.random.default_rng(83), 4000   # made-up loans: income (thousand ₹/month), loan (lakh ₹)
X = np.c_[rng.normal(60, 20, n).clip(10), rng.normal(8, 4, n).clip(1),
          rng.normal(700, 60, n).clip(300, 900), rng.integers(21, 61, n)]   # credit score, age
y = (X[:, 1] * 100 / X[:, 0] + 8 * (X[:, 2] < 640) + rng.normal(0, 2, n) > 13).astype(int)
from sklearn.ensemble import GradientBoostingClassifier
model = GradientBoostingClassifier(n_estimators=300, random_state=0).fit(X, y)
asha, names = np.array([50.0, 8, 690, 29]), ["income", "loan", "credit_score", "age"]
tries = {0: np.arange(50, 151, 1), 1: np.arange(8, 0.9, -0.1),       # raise income, cut loan
         2: np.arange(690, 901, 5), 3: np.arange(29, 61, 1)}          # raise score, get older
for f, values in tries.items():
    for v in values:                       # smallest change first
        z = asha.copy(); z[f] = v
        if model.predict_proba([z])[0, 1] < 0.5:
            print(f"if {names[f]} were {v:g} instead of {asha[f]:g}: APPROVED"); break
    else:
        print(f"no {names[f]} in the range flips the decision")
if income were 57 instead of 50: APPROVED
if loan were 7.1 instead of 8: APPROVED
no credit_score in the range flips the decision
no age in the range flips the decision

Two clear options for Asha: income 57 instead of 50, or a 7.1-lakh loan. No credit score in range and no age flips it, and "become older" would be useless advice anyway.

Which tool when?

Question Tool Scope
Which features does the model rely on overall? Permutation importance Global
How does risk change as one feature changes? PDP (+ ICE) Global (ICE: per person)
Why this prediction, with numbers that add up? Shapley values / SHAP Local
A quick local linear summary for any model LIME Local
What should this person change to get a yes? Counterfactual Local

Common mistakes

Mistake What you see Why Fix
Reading a 300-tree model 4,388 nodes, no reason black box use explanation methods
Trusting only the PDP average hides 0.92 vs 0.0 risk interaction between features plot ICE curves too
PDP of gradient boosting -4.92 (log-odds), or ValueError: With the 'recursion' method, the response_method must be 'decision_function'. Got predict_proba. the fast tree method response_method="predict_proba", method="brute"
Adding TreeExplainer values to get the risk 1.685 ≠ 0.844 values are log-odds sigmoid, or explain the probability
One LIME run with 500 samples −0.02 vs −0.12 random sampling more samples, compare seeds
Counterfactual "be older" useless advice not actionable search actionable features only

Mini project: an explanation report

For 5 rejected customers, print: their risk; their top 3 SHAP features with signs; one or two actionable counterfactuals; and a one-sentence plain-English reason. Then compute mean |SHAP| over 500 customers and compare its ranking with permutation importance.

Practice

1. Draw ICE curves for income (20, 40, 60, 80, 100) for people with loans of 4, 8 and 12 lakh.

Solution

Same risk_if_loan idea with column 0: Z[:, 0] = income. The curves fall as income rises, and the bigger the loan, the higher the income needed before risk drops: the same interaction seen from the other side.

2. Two features: base 0.5, f({A}) = 0.7, f({B}) = 0.6, f({A,B}) = 0.9. Compute the Shapley values.

Solution

Order A→B: A adds 0.2, B adds 0.2. Order B→A: B adds 0.1, A adds 0.3. Average: A = (0.2+0.3)/2 = 0.25, B = (0.2+0.1)/2 = 0.15. Check: 0.5 + 0.25 + 0.15 = 0.9.

3. Why can't permutation importance tell Asha why she was rejected?

Solution

It is global: it measures how much the model relies on a feature over the whole test set, not how each feature pushed one person's prediction.

4. Find a counterfactual that changes two actionable features at once with the smallest total change.

Solution

Loop over pairs (income +i, loan −j) on a small grid, keep the pairs that flip the decision, and pick the one with the smallest normalised change, e.g. |Δincome|/20 + |Δloan|/4 (dividing by each feature's spread).

Quick quiz

Ravi was rejected and asks: "What's the smallest thing I could change to get a yes?" Best answer: a counterfactual explanation. Permutation importance and PDP are global, not about Ravi.

Interview questions

  1. Interpretability vs explainability: what's the difference?
  2. Global vs local explanations: give one method of each.
  3. How is permutation importance computed and what are its limits?
  4. What does an ICE plot show that a PDP can hide?
  5. Explain Shapley values with an analogy, and the additivity property.
  6. Why might SHAP values not add up to the predicted probability?
  7. How does LIME work, and what is its main weakness?
  8. What makes a good counterfactual explanation?

Glossary

Term Meaning, with an example
Black box A model whose inner workings give no usable reason: 300 trees, 4,388 nodes.
Interpretability A person can understand how the model decides by looking at it.
Explainability Producing reasons for a model's behaviour, usually after training.
Post-hoc "After this": an explanation made after the model is trained.
Intrinsically interpretable model Readable by design: logistic regression, small trees, rule lists.
Global explanation About the model over everyone: "loan matters most".
Local explanation About one prediction: "for Asha, income pushed hardest".
Permutation importance Accuracy drop when one column is shuffled: loan 0.302.
PDP (Partial Dependence Plot) Average predicted risk as one feature is set to each value: 0.10 → 0.98.
ICE (Individual Conditional Expectation) One person's risk curve as one feature changes.
Interaction One feature's effect depends on another: the loan matters more for low incomes.
Log-odds The raw score before the sigmoid: 1.685 → sigmoid → 0.844.
Background data Ordinary customers used as the reference "average": 100 rows.
Shapley value A feature's extra push, averaged over every joining order: income +0.225.
Base value The average prediction over the background data: 0.533.
Additivity Base + all Shapley values = the prediction: 0.533 + … = 0.844.
SHAP Shapley Additive Explanations, a library and method (Lundberg and Lee, 2017).
TreeExplainer SHAP's fast explainer for tree models; log-odds by default.
LIME Local Interpretable Model-agnostic Explanations: a small linear model fit around one person.
Model-agnostic Works with any model, because it only uses predictions.
Counterfactual explanation The smallest change that flips the decision: loan 7.1 instead of 8 lakh.
Actionable feature Something a person can change: loan amount, income; not age.

Summary

  • Black boxes can't be read; interpretable models can, but may be less accurate (0.891 vs 0.93).
  • Global: permutation importance (how much), PDP (average shape) and ICE (each person's shape).
  • Local: Shapley values share a prediction fairly among features and add up exactly; SHAP makes it practical.
  • Check units (log-odds vs probability) and stability (LIME seeds).
  • Give people counterfactuals they can act on.

Next episode: Fairness and responsible ML: sources of bias, proxies, fairness metrics, mitigation, model cards, datasheets and privacy.