Feature Engineering

Machine learning models don't learn from raw order logs — they learn from features: numeric columns that summarise each entity you want to predict about. For ShopKart, the entity is the customer, and our goal (next lesson onward) is to recommend products and predict who's about to churn. This lesson turns 12,480 order rows into one clean row per customer, ready to feed a model.

Feature engineering is often the highest-leverage step in the whole pipeline: better features beat fancier models almost every time.

The RFM idea

RFM is a classic, battle-tested way to describe customer value with three numbers:

Recency — how many days since their last order (lower is better; they're still active).
Frequency — how many distinct orders they've placed (higher is better; they're loyal).
Monetary — how much they've spent in total (higher is better; they're valuable).

These three capture most of what matters about a customer's relationship with a store. Let's compute them.

Building the RFM table

We aggregate orders per customer. Recency needs a reference "today" — in a live system that's the current date; here we use the day after the last order so recency is never negative.

> Need the data? This lesson loads the ShopKart CSVs. Download orders.csv, customers.csv and products.csv into the same folder as your notebook first — Lesson 1 covers the setup.

import pandas as pd
import numpy as np

orders = pd.read_csv("orders.csv", parse_dates=["order_date"])
customers = pd.read_csv("customers.csv", parse_dates=["signup_date"])
orders["revenue"] = orders["price"] * orders["quantity"]

snapshot = orders["order_date"].max() + pd.Timedelta(days=1)

rfm = orders.groupby("customer_id").agg(
    recency=("order_date", lambda d: (snapshot - d.max()).days),
    frequency=("order_id", "nunique"),
    monetary=("revenue", "sum"),
).reset_index()

rfm.head()

  customer_id  recency  frequency  monetary
0     CUST001        4          3    1893.0
1     CUST002        5         12   19561.0
2     CUST003        7          8   22474.0
3     CUST004       90          2    4373.0
4     CUST006       37          5   17170.0

Read a couple of rows: CUST003 ordered 7 days ago, has placed 8 orders, and spent ₹22,474 — a star customer. CUST004 hasn't ordered in 90 days and placed just 2 orders — a likely churner. The RFM table makes that contrast a matter of three numbers.

Notice too that this table has fewer rows than customers.csv has customers. groupby only produces a row for customers who actually appear in orders — people who signed up and never bought simply vanish. In a real churn project those are exactly the people you might care about, so decide deliberately whether to merge them back in with zeros.

Adding a few more features

Good feature tables mix behavioural summaries with attributes. Let's bring in customer segment and tenure (days since signup), plus average order value.

rfm = rfm.merge(customers[["customer_id", "segment", "signup_date", "city"]],
                on="customer_id", how="left")

rfm["tenure_days"] = (snapshot - rfm["signup_date"]).dt.days
rfm["avg_order_value"] = rfm["monetary"] / rfm["frequency"]

feat = rfm[["customer_id", "recency", "frequency", "monetary",
            "avg_order_value", "tenure_days", "segment", "city"]]
feat.head()

  customer_id  recency  frequency  monetary  avg_order_value  tenure_days  segment       city
0     CUST001        4          3    1893.0       631.000000          344  Regular  Hyderabad
1     CUST002        5         12   19561.0      1630.083333          726  Regular      Delhi
2     CUST003        7          8   22474.0      2809.250000          640  Regular  Ahmedabad
3     CUST004       90          2    4373.0      2186.500000          654      NaN     Mumbai
4     CUST006       37          5   17170.0      3434.000000          300  Regular    Chennai

Encoding categoricals

Models need numbers, not strings like "Premium" or "Mumbai". One-hot encoding turns each category value into its own 0/1 column. pd.get_dummies does it in one call; drop_first=True avoids redundancy.

encoded = pd.get_dummies(feat, columns=["segment", "city"], drop_first=True)
encoded.filter(like="segment").head(3)
encoded.columns.tolist()

   segment_Regular
0             True
1             True
2             True

['customer_id', 'recency', 'frequency', 'monetary', 'avg_order_value',
 'tenure_days', 'segment_Regular', 'city_Bengaluru', 'city_Chennai',
 'city_Delhi', 'city_Hyderabad', 'city_Kolkata', 'city_Mumbai', 'city_Pune']

Three details are worth pausing on.

Which column survives. drop_first=True drops the alphabetically first value, so "Premium" is the dropped baseline and you get segment_Regular, not segment_Premium. The information is identical — segment_Regular = False means Premium — but you have to read the column name that actually came out, not the one you expected. Guessing wrong here is a KeyError waiting to happen in the next lesson. True/False, not 1/0. Modern pandas returns booleans. scikit-learn treats them as 1 and 0, so nothing breaks; add .astype(int) if you want the classic look. Missing values quietly become the baseline. Remember the 35 customers with no segment. get_dummies gives them segment_Regular = False — indistinguishable from a genuine Premium customer. If that matters, fill or flag the gap before encoding, or pass dummy_na=True.

The 8 ShopKart cities from Lesson 1 become 7 dummy columns (drop_first=True drops one as the baseline), giving 14 columns in total. For high-cardinality columns (hundreds of values) you'd reach for other techniques, but one-hot is perfect for a handful of segments and cities.

Scaling numeric features

Look at the ranges: recency spans ~0–90, but monetary spans hundreds to tens of thousands. Distance- and gradient-based models (logistic regression, k-nearest neighbours) let the big-numbered feature dominate unless you scale. StandardScaler rescales each column to mean 0, standard deviation 1.

from sklearn.preprocessing import StandardScaler

num_cols = ["recency", "frequency", "monetary", "avg_order_value", "tenure_days"]
scaler = StandardScaler()
encoded[num_cols] = scaler.fit_transform(encoded[num_cols])

encoded[num_cols].describe().round(2).loc[["mean", "std", "min", "max"]]

      recency  frequency  monetary  avg_order_value  tenure_days
mean    -0.00      -0.00     -0.00            -0.00          0.0
std      1.00       1.00      1.00             1.00          1.0
min     -0.96      -1.15     -1.00            -1.49         -1.8
max      3.07       4.44      5.25            10.30          1.7

Every numeric feature now sits on the same scale (mean ≈ 0, std ≈ 1), so no single column overpowers the others. The long tails we saw in EDA survive as high maxima — avg_order_value reaches 10.3 standard deviations above the mean, which is one customer whose handful of orders were all premium electronics. That's fine: scaling preserves the shape, it just standardises the units. If a tail that extreme bothers your model, that's an argument for a log transform, not for a different scaler.

Train / test split

The golden rule of ML: evaluate on data the model never saw during training. We hold out a slice as a test set. train_test_split shuffles and divides the rows; random_state makes it reproducible, and stratify keeps class balance consistent (crucial for the churn label we'll add next lesson).

from sklearn.model_selection import train_test_split

X = encoded.drop(columns=["customer_id"])
# (label y comes in the churn lesson; here we just demonstrate the split)
X_train, X_test = train_test_split(X, test_size=0.2, random_state=42)

print("train:", X_train.shape, " test:", X_test.shape)

train: (1056, 13)  test: (264, 13)

One subtlety that trips people up: fit the scaler on the training data only, then apply it to the test data — otherwise information from the test set leaks into training. In a real pipeline you'd wrap the scaler and model in a sklearn Pipeline so this happens automatically. For clarity we scaled up front here, but remember the principle.

On the job

When a data scientist says "the model isn't working," the fix is far more often a feature-engineering change than a model change. RFM in particular is used across retail, SaaS, and banking to segment customers and target campaigns — it's simple, interpretable, and stakeholders trust it. The discipline that separates reliable ML from broken ML is preventing leakage: keeping the test set untouched, and never computing a feature using information that wouldn't be available at prediction time (like a customer's future orders). Get the split and the leakage right and everything downstream is easier.

Key takeaways

Features are per-entity numeric summaries; for ShopKart we build one row per customer.
RFM (recency, frequency, monetary) captures customer value in three interpretable numbers.
One-hot encode categoricals (pd.get_dummies); scale numerics (StandardScaler) so no feature dominates.
Always hold out a test set (train_test_split) and evaluate on unseen data.
Prevent leakage: fit scalers on train only, and never build features from future information.