ML: Predicting Customer Churn

Acquiring a new customer costs far more than keeping an existing one. So ShopKart asks: "Which customers are about to stop buying, so we can win them back before they leave?" That's a churn prediction problem — and it's a textbook binary classification task. We'll reuse the RFM feature table from Lesson 4, attach a churn label, train two models, and — crucially — learn how to tell whether they're any good.

Framing churn as classification

Churn isn't a column in our data; we have to define it. And how you define it is the single most important decision in this lesson — get it wrong and you'll build a model that scores perfectly and predicts nothing.

Here's the trap. The obvious definition is "churned = hasn't ordered in the last 90 days". But recency is one of our features. If the label is computed from a feature, the model just re-derives the rule and every metric comes back at a suspiciously perfect 1.00. That's label leakage, and it is the most common way a churn project fools its own author.

The fix is to split time. Pick a cutoff date. Build features from what happened before it, and take the label from what happened after it. That mirrors reality: on the day you make a prediction, you only know the past.

• Feature window — Jan 1 to Apr 30. Everything the model gets to see.
• Outcome window — May 1 to Jun 30. Did they come back or not?

> Need the data? This lesson loads the ShopKart CSVs. Download orders.csv, customers.csv and products.csv into the same folder as your notebook first — Lesson 1 covers the setup.

import pandas as pd
import numpy as np

orders = pd.read_csv("orders.csv", parse_dates=["order_date"])
customers = pd.read_csv("customers.csv", parse_dates=["signup_date"])
orders["revenue"] = orders["price"] * orders["quantity"]

CUTOFF = pd.Timestamp("2024-05-01")
history = orders[orders["order_date"] < CUTOFF]    # features come from here
future  = orders[orders["order_date"] >= CUTOFF]   # the label comes from here

rfm = history.groupby("customer_id").agg(
    recency=("order_date", lambda d: (CUTOFF - d.max()).days),
    frequency=("order_id", "nunique"),
    monetary=("revenue", "sum"),
).reset_index()
rfm = rfm.merge(customers[["customer_id", "segment"]], on="customer_id", how="left")

# Churned = placed no order at all in the two months after the cutoff
returned = set(future["customer_id"])
rfm["churned"] = (~rfm["customer_id"].isin(returned)).astype(int)

rfm["churned"].value_counts(normalize=True).round(3)

churned
0    0.71
1    0.29
Name: proportion, dtype: float64

About 29% of the 1,198 customers who bought something before May never came back. This class imbalance (more retained than churned) matters for how we evaluate — accuracy alone will mislead us, as we'll see.

The label is now genuinely unknown at prediction time, so the model has to actually learn something. Look at how the two groups differ in the feature window:

rfm.groupby("churned")[["recency", "frequency", "monetary"]].mean().round(1)

         recency  frequency  monetary
churned
0           23.0        5.0   14097.3
1           47.3        2.5    6300.9

Customers who came back had ordered twice as often, spent more than twice as much, and had bought half as long ago. That's real signal — but it's overlapping signal, not a clean rule. Plenty of low-frequency customers come back anyway. That overlap is why our scores below land in the 70s and 80s rather than at 1.00, and that is exactly what an honest churn model looks like.

Preparing features and splitting

We encode the categorical segment, separate features (X) from the label (y), and split into train/test. stratify=y keeps the 71/29 churn ratio in both halves so the test set is representative.

Remember from Lesson 4 that drop_first=True drops "Premium" as the baseline, so the column you get is segment_Regular. Asking for segment_Premium here is a KeyError.

from sklearn.model_selection import train_test_split

data = pd.get_dummies(rfm, columns=["segment"], drop_first=True)
X = data[["recency", "frequency", "monetary", "segment_Regular"]]
y = data["churned"]

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
print("train:", X_train.shape, "  churn rate:", round(y_train.mean(), 3))

train: (958, 4)   churn rate: 0.289

Model 1: Logistic Regression

from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline

logreg = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))
logreg.fit(X_train, y_train)

pred_lr = logreg.predict(X_test)
proba_lr = logreg.predict_proba(X_test)[:, 1]   # P(churn)

The Pipeline scales features and fits the model together, so scaling is learned on training data only — the leakage-safe pattern from Lesson 4.

Model 2: Random Forest

A random forest is an ensemble of decision trees that vote. It captures non-linear patterns and interactions automatically, usually beating logistic regression on tabular data with little tuning.

from sklearn.ensemble import RandomForestClassifier

rf = RandomForestClassifier(n_estimators=200, random_state=42, class_weight="balanced")
rf.fit(X_train, y_train)

pred_rf = rf.predict(X_test)
proba_rf = rf.predict_proba(X_test)[:, 1]

Evaluating: beyond accuracy

Here's the trap. If we predicted "nobody churns" for everyone, we'd be right 70.8% of the time on this test set — high accuracy, useless model. So we need metrics that reveal how well we catch the churners specifically.

• Accuracy — fraction of all predictions correct.
• Precision — of those we flagged as churners, how many really churned. (Avoids wasting win-back offers on loyal customers.)
• Recall — of all actual churners, how many we caught. (Avoids letting churners slip away.)
• ROC-AUC — how well the model ranks churners above non-churners across all thresholds (0.5 = random, 1.0 = perfect).

from sklearn.metrics import (accuracy_score, precision_score,
                             recall_score, f1_score, roc_auc_score)

def report(name, y_true, pred, proba):
    return {
        "model": name,
        "accuracy": round(accuracy_score(y_true, pred), 3),
        "precision": round(precision_score(y_true, pred), 3),
        "recall": round(recall_score(y_true, pred), 3),
        "f1": round(f1_score(y_true, pred), 3),
        "roc_auc": round(roc_auc_score(y_true, proba), 3),
    }

results = pd.DataFrame([
    report("LogReg", y_test, pred_lr, proba_lr),
    report("RandomForest", y_test, pred_rf, proba_rf),
])
results

Read this carefully, because it is more interesting than a clean win.

Neither is "correct". If win-back offers are expensive, take logistic regression's precision. If losing a customer costs far more than a wasted discount, take the forest's recall. This is a business decision, not a modelling one.

Also note accuracy: the "predict nobody churns" baseline scores 0.708 on this test set. The random forest's 0.742 barely clears it — which is precisely why accuracy is the wrong headline metric. ROC-AUC of 0.827 shows logistic regression genuinely ranks churners well above non-churners, and that's the number to trust.

The confusion matrix

A confusion matrix breaks predictions into four buckets — the clearest single view of a classifier's behaviour.

from sklearn.metrics import confusion_matrix
cm = confusion_matrix(y_test, pred_rf)
print(cm)

[[137  33]
 [ 29  41]]

Read the corners: 137 loyal customers correctly left alone, 41 churners correctly caught, 29 churners missed, and 33 loyal customers flagged by mistake. The 29 false negatives are customers who quietly left while the model said they were fine. The 33 false positives cost you a discount on someone who was going to stay anyway.

Compare that with logistic regression's matrix on the same test set:

[[161   9]
 [ 37  33]]

Almost a mirror image. Logistic regression wrongly flags only 9 loyal customers instead of 33 — but it lets 37 churners slip past instead of 29. The forest buys 8 extra saved customers for 24 extra wasted offers. Whether that's a good trade depends entirely on what an offer costs you and what a customer is worth.

You don't have to pick a model to move along this trade-off, either — lowering the 0.5 probability threshold on either model catches more churners at the cost of more false alarms.

Which features matter?

Random forests report feature importance, useful for explaining the model to stakeholders.

importances = pd.Series(rf.feature_importances_, index=X.columns)
importances.sort_values(ascending=False).round(3)

monetary           0.436
recency            0.332
frequency          0.194
segment_Regular    0.038
dtype: float64

The RFM trio takes the top three slots, with segment contributing almost nothing — knowing whether someone is Premium tells you far less than knowing what they actually did.

One caveat worth carrying with you: random forest importances are biased toward high-cardinality features. monetary takes thousands of distinct values while segment_Regular takes two, which gives monetary far more opportunities to be split on. Don't read this ranking as "rupees spent causes retention". For a more trustworthy ranking, reach for permutation importance (sklearn.inspection.permutation_importance), which measures how much performance actually drops when you shuffle a column.

On the job

Churn models are deployed across every subscription and retail business, and this exact workflow — define the label, engineer features, train a baseline plus a stronger model, evaluate with the right metrics — is what you'll do for most classification tasks. The single most important lesson here is don't trust accuracy on imbalanced data. Precision, recall, ROC-AUC and the confusion matrix tell the real story, and choosing which to optimise is a conversation with the business: a retention team with a limited budget wants high precision; a team trying to save every at-risk customer wants high recall. Being able to explain that trade-off is what makes you useful.

Key takeaways

• Churn becomes binary classification once you define a label — and the label must come from a later time window than the features, or you leak.
• Start with logistic regression as a baseline; random forests often do better on tabular data, but here the baseline won — always check rather than assume.
• On imbalanced data, accuracy lies — use precision, recall, F1, and ROC-AUC.
• The confusion matrix exposes false positives vs false negatives, whose costs differ by business.
• Feature importance confirmed RFM (recency, frequency, monetary) drives the prediction.