The Project | About Us | Contribute | Donations | License
HOME My courses Completed
◄ Previous
◄

Project 5: Customer segmentation

Next ►
►

Last project, and we come back home. After housing, images, sentiment and fraud, we close the circle where it all began: MercaFresh. The assignment is the classic of unsupervised learning: with no label whatsoever, discover which natural customer segments exist, give them business names, propose an action for each one and leave the system ready for production. You will practice the whole of module 5 end to end — K-means, hierarchical, DBSCAN, PCA and t-SNE — on a synthetic but realistic dataset of ~2,000 customers generated right in the code, building the RFM features and ratios you learned in 03-06 "Feature engineering". And since this is the project that closes the module, we will finish with a recap of all five projects and the bridge to module 10.

Contents

  1. Problem definition: what is "success" without labels?
  2. Generating the customer dataset
  3. From orders to features: RFM and ratios
  4. EDA and scaling
  5. K-means: choosing k with elbow and silhouette
  6. Cross-checking: hierarchical clustering and DBSCAN
  7. Visualization: PCA and t-SNE
  8. Profiling: from centroids to business names
  9. Actions per segment and how to measure them
  10. From segments to production
  11. Module 9 recap

Problem definition: what is "success" without labels?

In projects 1–4 there was a ground truth to measure against. Here there is not: nobody knows how many segments "really" exist. Success is defined differently, as you learned in module 5:

  • Internal validity: compact, well-separated clusters (silhouette, 05-01).
  • Stability: the segments must not change radically with another seed or another sample.
  • Actionability: each segment must be describable in one sentence and mappable to a distinct marketing action. A statistically perfect clustering that cannot be explained to the business is a failure.

Concrete goal: between 3 and 8 segments (fewer does not discriminate, more cannot be managed), profiled and with an action plan.

Generating the customer dataset

We simulate the order history of ~2,000 customers with mixed behavioral archetypes plus noise — so that later we can judge whether the algorithms rediscover them. In reality, this table would come from a SQL query over the order history:

import numpy as np
import pandas as pd

rng = np.random.default_rng(42)
n = 2000

# Latent archetypes (proportions): VIP, regular, occasional, dormant, new
archetype = rng.choice(5, size=n, p=[0.08, 0.32, 0.30, 0.20, 0.10])

# Parameters per archetype: (mean_recency_days, orders_per_year, avg_order)
params = {
    0: (7,  90, 85),    # VIP: weekly shopper, big basket
    1: (12, 45, 45),    # regular: fortnightly, medium basket
    2: (35, 12, 30),    # occasional
    3: (160, 4, 38),    # dormant: has not ordered in months
    4: (10, 3, 25),     # new: recent but little history
}

rows = []
for a in archetype:
    r_mu, f_mu, m_mu = params[a]
    recency = max(1, rng.gamma(2, r_mu / 2))
    orders = max(1, rng.poisson(f_mu))
    avg_order = max(8, rng.normal(m_mu, m_mu * 0.3))
    fresh = np.clip(rng.normal([0.55, 0.45, 0.30, 0.35, 0.40][a], 0.12), 0, 1)
    tenure = (rng.uniform(15, 90) if a == 4
              else rng.uniform(180, 1400))
    rows.append([recency, orders, avg_order, fresh, tenure])

customers = pd.DataFrame(
    rows, columns=["recency_days", "orders_12m", "avg_order_eur",
                   "pct_fresh", "tenure_days"]
)
print(customers.describe().round(1))

An honest note: we know the archetypes because we planted them, but we will not hand them to any algorithm — they are here only so that you, at the very end, can compare what was discovered against what was planted. The clustering will work blind, just like in real life.

From orders to features: RFM and ratios

On top of the base columns we build the segmentation vector, revisiting the RFM from 03-06:

X = pd.DataFrame({
    "recency": customers["recency_days"],              # R
    "frequency": customers["orders_12m"],              # F
    "monetary": customers["orders_12m"] * customers["avg_order_eur"],  # yearly M
    "avg_order": customers["avg_order_eur"],
    "pct_fresh": customers["pct_fresh"],
    "orders_per_active_month": customers["orders_12m"]
                               / (customers["tenure_days"] / 30).clip(lower=1),
})

The last ratio is engineering with intent: a customer with 3 orders and 20 years of tenure is not the same as one with 3 orders and 3 weeks — the intensity ratio pulls them apart, distinguishing promising newcomers from chronic occasionals.

EDA and scaling

import matplotlib.pyplot as plt

X.hist(bins=40, figsize=(12, 6)); plt.tight_layout(); plt.show()
print(X.corr().round(2))

You will see strong skew in recency, frequency and monetary (right tails, the pattern from 03-03 "Data transformation") — we apply a log so that a handful of extremes does not dominate the clusters — and then we standardize, because K-means measures Euclidean distances and, without scaling, the largest-magnitude variable (monetary, in hundreds of euros) would take over, as you learned in 03-05 and suffered in 05-01:

from sklearn.preprocessing import StandardScaler

X_t = X.copy()
for col in ["recency", "frequency", "monetary"]:
    X_t[col] = np.log1p(X_t[col])

scaler = StandardScaler()
X_scaled = scaler.fit_transform(X_t)

K-means: choosing k with elbow and silhouette

The protocol from 05-01 "Clustering: K-means", in full:

from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score

inertias, silhouettes = [], []
ks = range(2, 11)
for k in ks:
    km = KMeans(n_clusters=k, n_init=10, random_state=42).fit(X_scaled)
    inertias.append(km.inertia_)
    silhouettes.append(silhouette_score(X_scaled, km.labels_))

fig, axes = plt.subplots(1, 2, figsize=(12, 4))
axes[0].plot(ks, inertias, "o-"); axes[0].set_title("Elbow (inertia)")
axes[1].plot(ks, silhouettes, "o-"); axes[1].set_title("Mean silhouette")
for ax in axes: ax.set_xlabel("k")
plt.show()

With this data, the elbow bends toward k = 4–5 and the silhouette usually has a local maximum at k = 5 (≈ 0.30–0.40; on real customer data, values like these are normal — human segments are not crisp spheres). We choose k = 5, and we write down the argument: the elbow agrees, the silhouette is reasonable and the number is manageable for marketing. The choice of k is always half metric, half business.

km = KMeans(n_clusters=5, n_init=10, random_state=42).fit(X_scaled)
customers["segment"] = km.labels_
print(customers["segment"].value_counts().sort_index())

Cross-checking: hierarchical clustering and DBSCAN

A single algorithm is a single opinion. The hierarchical clustering from 05-02 contributes the dendrogram — the picture of how customers group together at every scale:

from scipy.cluster.hierarchy import linkage, dendrogram, fcluster

link = linkage(X_scaled, method="ward")
plt.figure(figsize=(11, 4))
dendrogram(link, truncate_mode="lastp", p=30, no_labels=True)
plt.title("Dendrogram (Ward, truncated)"); plt.ylabel("Distance")
plt.show()

hier = fcluster(link, t=5, criterion="maxclust")
print(pd.crosstab(customers["segment"], hier))

If the dendrogram shows 4–5 thick branches and the cross-table reveals that the hierarchical clusters largely coincide with K-means', we have converging evidence: the structure is real, not an artifact of the algorithm.

DBSCAN (05-04) answers a different question: are there customers who belong to no group at all?

from sklearn.cluster import DBSCAN

db = DBSCAN(eps=0.9, min_samples=10).fit(X_scaled)
print(pd.Series(db.labels_).value_counts().head())
print("Noise customers:", (db.labels_ == -1).sum())

The ones labeled −1 are noise customers: odd combinations (a huge basket with minimal frequency, for instance) that fit no segment. K-means would have forced them into the nearest centroid, contaminating its profile. In practice they deserve a separate review: some are data errors (03-01), others are singular opportunities — business accounts, for example. Tune eps with judgment: too small and everything is noise; too large and everything is one single cluster.

Visualization: PCA and t-SNE

Six dimensions cannot be drawn; we project them through the two lenses of module 5 — the linear, global one (PCA, 05-03) and the non-linear, neighborhood-focused one (t-SNE, 05-05):

from sklearn.decomposition import PCA
from sklearn.manifold import TSNE

pca = PCA(n_components=2)
X_pca = pca.fit_transform(X_scaled)
X_tsne = TSNE(n_components=2, perplexity=30,
              random_state=42).fit_transform(X_scaled)

fig, axes = plt.subplots(1, 2, figsize=(13, 5))
for ax, data, title in [(axes[0], X_pca, "PCA"), (axes[1], X_tsne, "t-SNE")]:
    sc = ax.scatter(data[:, 0], data[:, 1], c=customers["segment"],
                    cmap="tab10", s=8, alpha=0.6)
    ax.set_title(title)
plt.colorbar(sc, label="Segment"); plt.show()
print("PCA explained variance:", pca.explained_variance_ratio_.round(2))

Remember the reading rules from 05-05: in PCA the global distances mean something (and pca.components_ tells you which combination of features each axis is); in t-SNE only group membership is interpretable — neither island sizes nor distances between them. If the K-means colors show up as coherent regions in both projections, the clustering earns yet another visual confirmation.

Profiling: from centroids to business names

The phase that turns analysis into product — a table of per-segment means in original units, not scaled ones:

profile = customers.groupby("segment").agg(
    n=("segment", "size"),
    recency=("recency_days", "mean"),
    orders=("orders_12m", "mean"),
    avg_order=("avg_order_eur", "mean"),
    pct_fresh=("pct_fresh", "mean"),
    tenure=("tenure_days", "mean"),
).round(1)
profile["yearly_spend"] = (profile["orders"] * profile["avg_order"]).round(0)
print(profile.to_string())

With these figures each segment gets christened. A typical result (your numbers will vary; the names come from looking at your table, not this one):

Segment n Recency Orders/year Avg. order Yearly spend Business name
0 ≈ 160 8 days 88 €84 ≈ €7,400 VIP
1 ≈ 640 13 days 44 €46 ≈ €2,000 Regulars
2 ≈ 600 36 days 12 €31 ≈ €370 Occasional
3 ≈ 400 165 days 4 €37 ≈ €150 Dormant
4 ≈ 200 11 days 3 €26 ≈ €80 New

And the moment of truth of our experiment: cross customers["segment"] with the planted archetypes (pd.crosstab(customers["segment"], archetype)). You will see a strong but imperfect correspondence — blurry borders between Regulars and Occasional, New customers scattered around — which is exactly what happens with real segments: clustering recovers the structure, not the exact partition.

Actions per segment and how to measure them

A segment without an action is a pretty chart. The table marketing is waiting for:

Segment Proposed action Success metric
VIP Premium loyalty program, free delivery 12-month retention
Regulars Recurring basket subscription Conversion to subscription
Occasional Campaigns in their most-bought categories Order frequency
Dormant Reactivation email with a coupon 30-day reactivation rate
New Onboarding: first 3 purchases guided Survival to the 4th order

And how will we know whether the action works? With the A/B test from 02-04 "Statistical inference": within each segment, one random group receives the action and another does not, and the difference in the success metric is contrasted with its hypothesis test. Without a control group, any improvement is attributable to seasonality or chance — clustering proposes, inference disposes.

From segments to production

We close with the craft of 08-02 "Deploying models to production". The transformations + scaling + K-means pipeline gets serialized and runs as batch scoring: every night (or every week — segments move slowly), each customer's features are recomputed and a segment is assigned with predict, writing the result wherever the CRM reads it:

import joblib
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import FunctionTransformer

def log_rfm(X):
    X = X.copy()
    for col in ["recency", "frequency", "monetary"]:
        X[col] = np.log1p(X[col])
    return X

seg_pipeline = Pipeline([
    ("log", FunctionTransformer(log_rfm)),
    ("scale", StandardScaler()),
    ("km", KMeans(n_clusters=5, n_init=10, random_state=42)),
]).fit(X)

joblib.dump(seg_pipeline, "mercafresh_segmenter.joblib")

# In the nightly job:
model = joblib.load("mercafresh_segmenter.joblib")
customers["segment"] = model.predict(X)

Two watchpoints from the monitoring in 08-03 apply here too: the size of each segment over time (if the Dormant grow by 30%, that is a business alarm, not a model alarm) and the drift of the centroids upon retraining — if the profiles change meaning, the names and the campaigns must be revised along with them. And the reminder from 08-04: segmenting in order to treat customers differently requires checking that no segment is a proxy for a protected group.

Common Mistakes and Tips

  • Forgetting the scaling (or the log). Without standardizing, monetary dictates the clusters single-handedly; without the log, the four extreme customers become four-person "segments". It is the number one mistake in segmentation.
  • Choosing k by the metric alone. A silhouette of 0.41 with k = 2 can be worse business than 0.35 with k = 5: two segments give marketing nothing to play with. Metric and actionability decide together.
  • Profiling in scaled units. A centroid of "−0.3 standard recencies" is not something you can tell marketing. Profiling is always done in original units.
  • Treating segments as eternal truths. They are a snapshot: customers migrate between segments and that migration (how many Regulars fall into Dormant each month?) is often more valuable than the snapshot itself.
  • Launching actions without a control group. Without A/B there is no attribution; you will have spent the budget confirming your own biases.

Challenges to go further

  1. More features, better segments. Add behavioral variables to the vector: percentage of purchases with a coupon, category diversity, return rate, channel (app/web). Hint: rerun elbow + silhouette from scratch — with more dimensions the optimal k can change — and use PCA to watch how much variance the new columns really contribute.
  2. Product clustering. Flip the matrix: instead of customers described by what they buy, products described by who buys them (or by co-occurrence in baskets). Hint: the product × customer matrix is huge and sparse — reduce first with the PCA from 05-03 (or TruncatedSVD, its variant for sparse matrices) and cluster afterwards; the resulting groups feed recommendations of the "people who buy this..." kind.
  3. Tracking dashboard. Build an automatic monthly report with matplotlib: size of each segment over time, average spend per segment and a month-to-month migration matrix. Hint: store each batch scoring with its date; the migration is a pd.crosstab(previous_month_segment, current_segment) — and turn it into the permanent monitoring that 08-03 called for.

Module 9 recap

Five projects, five different problems, one same method. The table summarizing what you have practiced:

Project Problem type Dataset Star techniques Central lesson
1. Housing Regression California Housing Pipeline, Ridge/Lasso, Random Forest, HistGradientBoosting, CV, residuals EDA foreshadows which model will win
2. Images Multiclass classification digits + Fashion-MNIST SVM, PCA, MLP vs. CNN, dropout, early stopping Start simple; let the data justify every layer
3. Sentiment Text classification Reviews (fictional) Cleaning, TF-IDF, n-grams, MultinomialNB, weight interpretation Representation matters as much as the algorithm
4. Fraud Imbalanced classification Synthetic (make_classification) class_weight, resampling, PR curve, AUC-PR, cost-based threshold Separate the model (probabilities) from the decision (euros)
5. Segmentation Unsupervised Synthetic MercaFresh RFM, K-means, hierarchical, DBSCAN, PCA/t-SNE, profiling, A/B Without labels, success is stability + actionability

And cutting across all five: define the problem and the metric before touching data, set the test aside at the start, encapsulate in a Pipeline, compare against a baseline with honest cross-validation, analyze the errors instead of merely counting them, and translate the result into the language of the business.

Conclusion

The circle is closed: MercaFresh opened the course teaching you each piece separately and closes it watching you assemble them all without help — you have discovered its customer segments from scratch, validated them with three algorithms and two projections, given them names, actions and control groups, and left them being served in batch with their monitoring planned. You are no longer the person who started module 1: you have a complete method and five projects that prove it, ready to become the seed of your portfolio. What remains is not learning more lessons, but continuing to learn on your own: in module 10 we leave you the map — books, courses, communities and tools — so that the end of this course is only the beginning of your path in machine learning.

◄ Previous
◄
Next ►
►

Machine Learning Course

Module 1: Introduction to Machine Learning

Module 2: Foundations of Statistics and Probability

Module 3: Data Preprocessing

Module 4: Supervised Machine Learning Algorithms

Module 5: Unsupervised Machine Learning Algorithms

Module 6: Model Evaluation and Validation

Module 7: Advanced Techniques and Optimization

Module 8: Model Implementation and Deployment

Module 9: Hands-On Projects

Module 10: Additional Resources

The Project | About Us | Contribute | Donations | License
© Copyright 2026. All rights reserved