K-means (05-01) forced you to decide K before seeing a single result. Hierarchical clustering reverses the order: it first builds every possible grouping — from each customer standing alone to a single group containing everyone — and then you choose the level at which to cut. The result is a tree structure, the dendrogram, showing which customers resemble each other most, in what order the groups merge, and at what "distance" each merge happens. In this lesson you will learn the agglomerative mechanics step by step (with a worked example by hand), the linkage measures that determine cluster shape, how to build and read a dendrogram with scipy, how to apply AgglomerativeClustering to MercaFresh's customers and compare its segments with K-means', and when the computational price of the hierarchical approach is worth paying.

1. Agglomerative vs. divisive
2. Linkage measures: how distance between groups is defined
3. A worked example by hand: the first merges
4. The dendrogram: construction and reading
5. Cutting the tree: from hierarchy to segments
6. MercaFresh customers with AgglomerativeClustering
7. Comparison with the K-means segments
8. Advantages, computational cost and when to prefer it

There are two ways to build a hierarchy of groups:

We will focus on the agglomerative approach. Its algorithm is remarkably simple:

1. Start with $n$ clusters of one point each.
2. Compute the distance between every pair of clusters.
3. Merge the two closest clusters.
4. Repeat 2-3 until a single cluster remains.

Every merge is recorded along with its distance, and that sequence of merges is the hierarchy. No random initialization, no iterating until convergence: the result is deterministic (given the same data and parameters, it always comes out the same — unlike K-means and its random_state).

Step 2 hides the one important design decision: the distance between two points is the usual Euclidean one (04-05), but what is the distance between two groups of points? Each answer is a linkage measure, and it changes the character of the clustering:

Two practical notes:

• Ward is the sensible default for customer segmentation: at every merge it minimizes the same kind of criterion (within-cluster variance) that K-means minimizes globally with the inertia, so it produces comparable, stable groups.
• Single linkage is the odd one out: where Ward and complete see spheres, single follows chains of neighbors and can recover snaking shapes. That idea of "connecting through local proximity" will reappear, taken to its logical conclusion and done properly, in DBSCAN (05-04).

Since every linkage rests on distances, the rule from 03-05 still stands: scale the features first, or the units will decide for you.

Let's take 5 MercaFresh customers with a single feature, recency_days, and single linkage (the easiest to compute by hand):

Step 1. Pairwise distances: AB=2, AC=5, BC=3, DE=5, CD=32, and the rest larger. The minimum is AB=2 → we merge {A,B} at distance 2.

Step 2. Distances involving the new cluster (single = minimum): d({A,B}, C) = min(5, 3) = 3; d({A,B}, D) = 35; DE = 5. The minimum is 3 → we merge {A,B,C}.

Step 3. d({A,B,C}, D) = 32; d({A,B,C}, E) = 37; DE = 5. The minimum is 5 → we merge {D,E}.

Step 4. Only {A,B,C} and {D,E} remain: they merge at distance min(32, 37) = 32.

The full sequence — (A,B) at 2, (+C) at 3, (D,E) at 5, (everything) at 32 — tells the whole story: there are two natural groups, one of recent customers and one of cold ones, and the enormous distance of the last merge (32 versus 5) is the evidence. That story is exactly what the dendrogram draws.

A dendrogram is the tree of merges: the leaves are the points, each bridge-shaped join represents a merge, and the height of the bridge is the distance at which it happened. In scipy:

What each piece does:

• linkage(X, method=...) runs the complete agglomerative algorithm and returns Z, a matrix with one row per merge: which two clusters joined, at what distance, and how many points the result contains. For our example, its distances are exactly the ones we computed by hand: 2, 3, 5, 32.
• dendrogram(Z) draws the tree. method accepts "single", "complete", "average" and "ward".

How to read a dendrogram (the skill that matters):

• Low bridges = early merges = very similar points. A and B are practically the same customer.
• High bridges = forced merges between groups that barely resemble each other. The jump from 5 to 32 screams "there are two distinct populations here".
• The number of clusters at a given height = the number of vertical lines crossed by a horizontal drawn at that height. At height 10, our horizontal crosses 2 lines: two clusters.

The full hierarchy is informative, but to act you need a concrete partition: you obtain one by cutting the dendrogram at a height. Common criteria:

• Cut where the distance jump is largest: just below the disproportionately tall bridge. It is the hierarchical equivalent of the elbow from 05-01.
• Cut to get a desired K: if the business wants 4 segments, lower the horizontal until it crosses 4 branches.
• The same tree admits several useful cuts: high up, "active vs. dormant" (2 groups, for an executive report); further down, 4-5 operational segments (for campaigns). That multi-scale view is something K-means does not offer: every K requires retraining from scratch.

In scipy, fcluster(Z, t=10, criterion="distance") returns the labels for the cut at height 10.

In scikit-learn the estimator is AgglomerativeClustering. We reuse the X_esc matrix from 05-01 (RFM + inactivity_ratio, scaled with StandardScaler):

Explanation for beginners:

• With hundreds of customers, a full dendrogram is an unreadable tangle of leaves; truncate_mode="lastp", p=20 draws only the 20 final merges, which are the ones that inform the cutting decision. We look for the stretch where the bridges shoot up: if the big jump happens when going from 4 branches to 3, then 4 clusters is a natural cut.
• AgglomerativeClustering asks for either n_clusters (it cuts the tree for you) or distance_threshold (it cuts at a height, leaving K free). linkage="ward" is the default and our recommendation for this case.
• There is no random_state: hierarchical clustering is deterministic.
• An honest detail: although the dendrogram lets you avoid fixing K up front, at the end of the day you cut somewhere — the difference is that you decide seeing the whole structure, not blindly trying Ks.

Do the 4 hierarchical segments match the 4 K-means segments from 05-01? We can cross them with a contingency table (pd.crosstab, which you already used in 03-04):

A typical result:

The reading: each row concentrates its mass in one column — both algorithms have essentially discovered the same four groups (remember that cluster numbers are arbitrary; what matters is the correspondence). No coincidence: Ward and K-means optimize very similar variance criteria. The disagreements (the ~70 customers off the diagonal) are boundary points between segments — precisely the ones that would have silhouettes near 0 in 05-01. When two different algorithms agree like this, confidence that the segments are real structure (and not an artifact of the method) rises sharply; if they disagreed completely, weak structure would be the prime suspect.

The cost deserves a pause: step 2 of the algorithm needs the distance matrix between all pairs of points — with $n$ customers that is on the order of $n^2/2$ distances. With MercaFresh's ~1,000 customers, half a million distances: instantaneous. With the 10 million customers of a large chain, $5 \times 10^{13}$ pairs: simply unworkable, while K-means would keep going.

When to prefer hierarchical:

• A small or medium dataset (up to tens of thousands of points).
• You have no idea how many groups exist and want to see the structure before deciding.
• The hierarchy itself has business value: product taxonomies (drinks > soft drinks > colas), groups within groups, reports at different levels of detail.
• You want a reproducible result with no seeds or initializations.

When K-means: large datasets, a reasonably clear K, or when you need to re-segment often and fast.

• Running linkage on unscaled data. The same cardinal sin as in 05-01: the resulting dendrogram sorts by the highest-magnitude feature. Always scale first.
• Drawing the full dendrogram with thousands of points. Unreadable and slow. Use truncate_mode="lastp" with p between 15 and 30: the final merges are the ones that inform the cut.
• Using Ward with non-Euclidean distances. Ward is defined over variances, which presuppose Euclidean distance. If you need another distance (Manhattan, cosine), switch to average or complete linkage.
• Expecting single linkage to give compact groups. Its specialty is chains; with noisy data it usually produces one mega-cluster and several stray points. For customer segmentation, Ward or complete.
• Tip: validate the cut with the silhouette from 05-01 (silhouette_score(X_esc, labels) works with any clustering, not just K-means) and compare two or three candidate cuts.

Exercise 1. Repeat the lesson's worked example by hand (A=3, B=5, C=8, D=40, E=45) but with complete linkage. Write out the sequence of merges with their distances. Does the merge order change compared with single? Does the final two-group structure change?

Exercise 2. Generate the Ward dendrogram of the MercaFresh customers (or of a synthetic dataset with make_blobs(n_samples=200, centers=4, random_state=7), scaled). Visually locate the largest distance jump and decide on a number of clusters. Then cut with AgglomerativeClustering at that K and compute the silhouette. Does your visual cut match the best silhouette between K=2 and K=6?

Exercise 3. On the same dataset, compare linkage="ward" and linkage="single" with n_clusters=4: print the value_counts() of each one's labels. What size pattern does each linkage produce, and why?

Exercise 1

With complete (maximum instead of minimum):

• Merge 1: the smallest pairwise distance is still AB=2 → {A,B} at 2.
• Merge 2: d({A,B}, C) = max(5, 3) = 5; DE = 5. A tie at 5; scipy merges whichever it finds first — say {A,B,C} at 5 (with complete, the tie order makes no difference to the final result).
• Merge 3: {D,E} at 5.
• Merge 4: d({A,B,C}, {D,E}) = the maximum of all cross distances = d(A,E) = 42 → final merge at 42.

The order is essentially the same and so is the final structure ({A,B,C} vs. {D,E}): with groups this well separated, all linkages agree. The differences between linkages surface with ambiguous data, bridges of intermediate points or outliers — not with clean islands. Notice how the final merge climbs from 32 (single, the distance between the two groups' closest points) to 42 (complete, their farthest ones).

Exercise 2

In the dendrogram, the biggest height jump happens when going from 4 branches to 3 (the four blobs are real), and the maximum silhouette also lands at K=4. When the visual and numeric criteria agree, the decision rests on solid ground; when they disagree, it usually signals clusters of unequal density or size — worth a look at the scatter plot.

Exercise 3

Ward produces 4 groups of comparable size (it distributes variance evenly). Single usually produces a very different pattern: one or two huge clusters and others with a handful of points (even 1), because chaining keeps annexing everything reachable through near neighbors and only leaves out the genuinely isolated points. That behavior, which looks like a flaw here, is almost a form of outlier detection — an intuition that DBSCAN (05-04) will turn into a virtue with its explicit notion of noise.

You now command the second family of clustering: the agglomerative approach that merges the two closest groups at each step, the four linkage measures and their effect on cluster shape (Ward as K-means' close relative, single as a chain tracker), the dendrogram as a multi-scale X-ray of the structure, and the cut that turns it into segments. On MercaFresh you also verified something valuable: hierarchical and K-means agree on the same four segments, a sign that the structure is real. And you know the price: the $O(n^2)$ that makes it unworkable at large scale.

So far we have grouped the customers using their 4 RFM features, and we have been able to picture them two at a time. But the final dataset from 03-06 has many more columns, and back in 04-05 a threat was noted for later: the curse of dimensionality, which degrades distances — the basic ingredient of everything we have done in this module. The next lesson attacks that problem head-on: PCA, the technique that compresses many dimensions into a few while keeping as much information as possible, and which will finally let us draw our segments.