ML: Building a Product Recommender

"Customers who bought this also bought…" — that little widget drives a huge share of e-commerce revenue. ShopKart wants one. In this lesson we build a working product recommender in scikit-learn using the product catalog we already have. No deep learning, no GPU — just TF-IDF and cosine similarity, the same ideas that power a surprising amount of production recommendation.

Two flavours of recommender

There are two classic approaches, and it helps to know which you're building.

Content-based recommends items similar to what a user already likes, using item attributes. Collaborative filtering recommends based on the behaviour of similar users. Content-based has a big practical advantage: it works for new products that nobody has bought yet (the "cold start" problem), because it only needs the product's own text. We'll build that, then sketch collaborative filtering at the end.

Step 1: Turn product text into vectors with TF-IDF

Each ShopKart product has tags and a description. To compare products by similarity, we first turn their text into numeric vectors. TF-IDF (Term Frequency–Inverse Document Frequency) does this: it gives high weight to words that are frequent in this product but rare across the catalog, so distinctive words ("waterproof", "bluetooth") count more than common ones ("the", "with").

> Need the data? This lesson loads the ShopKart CSVs. Download orders.csv, customers.csv and products.csv into the same folder as your notebook first — Lesson 1 covers the setup.

import pandas as pd
from sklearn.feature_extraction.text import TfidfVectorizer

products = pd.read_csv("products.csv")

# Combine the signal-rich text fields
products["text"] = (products["tags"].fillna("") + " "
                    + products["category"].str.lower() + " "
                    + products["description"].fillna(""))

vectorizer = TfidfVectorizer(stop_words="english", max_features=2000)
tfidf = vectorizer.fit_transform(products["text"])

print("Matrix shape:", tfidf.shape)   # (products, vocabulary terms)

Matrix shape: (600, 764)

Each of the 600 products is now a 764-dimensional vector, where each dimension is a weighted word. Products that share distinctive vocabulary end up with similar vectors.

Step 2: Measure similarity with cosine

Cosine similarity measures the angle between two vectors, ignoring their length — perfect for text, where document length shouldn't matter. It ranges from 0 (nothing in common) to 1 (identical direction). cosine_similarity computes it for every pair of products at once.

from sklearn.metrics.pairwise import cosine_similarity

sim = cosine_similarity(tfidf)      # 600 x 600 matrix
print(sim.shape)
print(round(sim[0, 0], 2), round(sim[0, 1], 2))  # self=1.0, other<1

(600, 600)
1.0 0.15

sim[i, j] is how similar product i is to product j. A product is perfectly similar to itself (1.0), and only mildly similar to an unrelated one (0.15).

Step 3: Recommend similar products

To recommend for a given product, sort its row of the similarity matrix and take the top matches (excluding itself).

# Map product_id to its row index
idx_of = pd.Series(products.index, index=products["product_id"])

def similar_products(product_id, n=5):
    i = idx_of[product_id]
    scores = list(enumerate(sim[i]))
    scores = sorted(scores, key=lambda x: x[1], reverse=True)
    top = [j for j, s in scores if j != i][:n]
    out = products.iloc[top][["product_id", "name", "category", "rating"]].copy()
    out["similarity"] = [round(sim[i, j], 2) for j in top]
    return out

# PRD017 = Boat Rockerz 255 Earphones
similar_products("PRD017")

product_id                       name    category  rating  similarity
    PRD501     Realme Buds Wireless 2 Electronics     3.9        0.65
    PRD412  Boult AirBass Z20 Earbuds Electronics     4.2        0.29
    PRD064 Noise Wired Earphones Lite Electronics     3.4        0.24
    PRD482    Mi Wired Earphones Lite Electronics     4.1        0.24
    PRD148 Realme Wired Earphones Max Electronics     4.2        0.24

For a pair of Boat earphones, the recommender surfaces other wireless earbuds and audio gear — exactly what a shopper viewing that product might want next. Notice it learned "these are audio products" purely from the shared tag/description vocabulary; nobody labelled a category relationship.

Look at the scores, though, not just the order. The top hit scores 0.65 and everything after it drops to 0.29 or below — a cliff, not a gentle slope. Only the first recommendation is a strong match; the rest are "vaguely also audio". A real store would show the top result confidently and think twice about padding the row out to five. Ranking always returns exactly n items whether or not n good matches exist, so check the scores before trusting the tail.

Step 4: Personalise for a customer

To recommend for a customer rather than a product, aggregate similarity across everything they've bought, then rank products they haven't bought yet. This is a simple content-based personalisation.

import numpy as np

orders = pd.read_csv("orders.csv")

def recommend_for_customer(customer_id, n=5):
    bought = orders.loc[orders["customer_id"] == customer_id, "product_id"].unique()
    bought_idx = [idx_of[p] for p in bought if p in idx_of.index]
    if not bought_idx:
        return products.nlargest(n, "rating")[["product_id", "name", "rating"]]

    # Average the similarity profile of everything they bought
    profile = sim[bought_idx].mean(axis=0)
    profile[bought_idx] = -1          # don't re-recommend owned items
    top = np.argsort(profile)[::-1][:n]

    out = products.iloc[top][["product_id", "name", "category"]].copy()
    out["score"] = np.round(profile[top], 2)
    return out

# CUST042 bought Boat earphones + a Milton bottle
recommend_for_customer("CUST042")

product_id                       name       category  score
    PRD501     Realme Buds Wireless 2    Electronics   0.33
    PRD091 Cello Insulated Coffee Mug Home & Kitchen   0.23
    PRD412  Boult AirBass Z20 Earbuds    Electronics   0.14
    PRD129     Pigeon Steel Lunch Box Home & Kitchen   0.13
    PRD222  Cello Steel Lunch Box 2.0 Home & Kitchen   0.13

The recommendations blend both of CUST042's interests — more audio gear and more kitchen products — because we averaged the similarity profiles of everything they bought. The second hit is especially satisfying: CUST042 bought an insulated flask, and the model surfaces an insulated coffee mug, having matched on words like "insulated", "travel" and "hot" rather than on the category label. That's the essence of content-based personalisation.

Note also that the scores here (0.33 at the top) are lower than the per-product scores above. Averaging two unrelated interests pulls every score down, because no single product is close to both earphones and a flask. That's expected — but it means a global score threshold that works for "similar products" will not work for "recommended for you".

How the pieces fit

A note on collaborative filtering

The alternative approach builds a customer-by-product matrix (1 if bought, else 0) and finds similar items by how often they're bought together — item-item collaborative filtering.

# Sketch: build the purchase matrix, then item-item similarity
pivot = (orders.assign(bought=1)
               .pivot_table(index="customer_id", columns="product_id",
                            values="bought", fill_value=0))
item_sim = cosine_similarity(pivot.T)   # products x products, from behaviour

Collaborative filtering captures real buying patterns ("earbuds and phone cases sell together") that text can't, but it fails for brand-new products with no purchases. Production systems often combine both — a hybrid recommender. We keep the content-based version as our working model because it needs nothing but the catalog.

On the job

Recommenders are one of the clearest examples of ML paying for itself — even a modest lift in click-through on a "you may also like" strip moves real revenue. The TF-IDF + cosine pattern you built here is genuinely used in production for content similarity, related-items, and search. Just as important is knowing the trade-offs you'd raise in a design discussion: content-based handles cold-start products; collaborative filtering captures behaviour but needs history; hybrids get the best of both. The next lesson swaps TF-IDF for neural embeddings, which understand meaning far better than word counts — but the cosine-similarity machinery stays identical.

Key takeaways

Content-based recommenders use item attributes; collaborative filtering uses purchase behaviour.
TF-IDF turns product text into weighted vectors; distinctive words carry the most signal.
Cosine similarity ranks products by how alike their vectors are (0 to 1), ignoring length.
Personalise by averaging the similarity profile of a customer's past purchases, then take the top-N unseen items.
Content-based solves cold-start; hybrids combine it with collaborative filtering in production.

Launch Offer