In 05-03 we projected MercaFresh's customers onto the PC1-PC2 plane and got a map that was useful but imperfect: only 73% of the variance fit into the drawing, and two segments could overlap on paper while being separated in reality. The cause is built in: PCA can only rotate, and there are structures no rotation unfolds. This lesson closes the module with the two modern nonlinear visualization techniques — t-SNE and UMAP — which produce 2D maps where the groups separate in often spectacular fashion. You will learn their intuition without heavy mathematics, their key parameters (perplexity, n_neighbors, min_dist), how to apply them to the MercaFresh customers colored by their segments from 05-01 and — just as important as all of the above — the rules of caution for not reading into those maps things they do not say.

1. Why the PCA map sometimes fails to separate
2. t-SNE: preserving local neighborhoods
3. The perplexity parameter and what t-SNE does not preserve
4. UMAP: faster and with better global structure
5. The MercaFresh customers in t-SNE and UMAP
6. Rules of cautious interpretation
7. Comparison table: PCA vs. t-SNE vs. UMAP
8. Closing the module: the map of unsupervised learning

PCA chooses the projection plane that keeps the maximum global variance — but keeping variance is not the same as keeping groups:

• If two segments differ along a low-variance direction, PCA will discard it and on the map they will appear mixed together.
• If the structure is curved (imagine the customers spread over a rolled-up surface, or the half-moons of 05-04), the linear projection flattens it: points far apart along the curve land on top of each other.
• And with many dimensions, one plane is very little room: PCA spends its budget across all the large directions, not across the ones that separate groups.

The idea that changes the game: in order to visualize we do not need to preserve global distances or variances — we need the points that were neighbors in the original space to remain neighbors on paper. Trading global fidelity for local fidelity is the deal t-SNE and UMAP make. They are visualization techniques, not general-purpose compression: they produce maps for human eyes, not features for models (we will come back to this).

t-SNE (t-distributed Stochastic Neighbor Embedding) operates in three conceptual steps:

1. In the original space, it computes for each point which other points are its close neighbors, turning distances into neighborhood probabilities: the very close ones have a high probability of "choosing each other" as neighbors; the distant ones, practically zero. Note what this implies: t-SNE does not even distinguish between far-off points — 20 or 200 units of distance are both "probability ~0".
2. On the 2D plane, it places the points at random to begin with and defines the same neighborhood probabilities over their positions.
3. It moves the points on the plane iteratively (gradient descent, our old acquaintance from 04-01) until the plane's neighborhoods resemble those of the original space as closely as possible: it pulls together those who should be neighbors and pushes apart those who should not.

The result: every little group of neighbors in the original space shows up as a compact islet on the map. Hence t-SNE's fame: clusters that looked blurry in PCA appear as sharply separated islands.

• perplexity controls the effective size of the neighborhood each point considers — intuitively, "how many neighbors I pay attention to". Typical values: 5-50 (default 30). Low perplexity → very local attention: clusters fragment into crumbs. High → wide neighborhoods: more global structures, but small groups may fade away. It must be smaller than the number of points, and it pays to try 2-3 values: if the structure survives several, it is real.
• It is stochastic: the random initialization and the gradient mean two runs give different maps (rotated, mirrored, or with islands relocated). Set random_state for reproducibility.
• It is slow: its exact version is $O(n^2)$; the Barnes-Hut approximation brings it down to $O(n \log n)$, but with hundreds of thousands of points it still costs minutes or hours.

And the crucial part — the price of the local deal:

t-SNE preserves neither distances nor global structure. Two islands ending up far apart on the map does not mean those groups are very different; a big island does not mean that cluster is spread out. t-SNE divides up the paper's space so that everything is visible, expanding dense zones and compressing empty ones.

This is no minor defect: it is the property that defines which questions the map can answer (are there groups? which ones neighbor which?) and which it cannot (how far apart are they? which is more compact?).

UMAP (Uniform Manifold Approximation and Projection) arrived in 2018 with the same philosophy — preserve local neighborhoods — and a different construction: it first weaves a neighbor graph over the data (each point connected to its n_neighbors closest, the machinery of 04-05 once again) and then searches for the 2D layout that best preserves that graph. In practice it brings three advantages:

• Speed: typically 10-100 times faster than t-SNE; scales to millions of points.
• Better global structure: without being fully faithful, the relative positions between clusters tend to be more meaningful than in t-SNE (similar groups tend to land near each other).
• It can transform new points: it has transform() to project unseen data onto an already fitted map, something scikit-learn's t-SNE does not offer.

Its two main dials:

n_neighbors is perplexity's cousin; min_dist is purely cosmetic, it does not change who neighbors whom.

A practical detail: UMAP does not ship with scikit-learn; it is the separate package umap-learn (pip install umap-learn), imported as umap, and it follows the usual fit_transform API.

Let's draw the module's definitive map: the customers with their 7 features (the X_esc matrix from 05-03, scaled — the standing rule), colored by the segments K-means discovered in 05-01.

Points in the code worth commenting on:

• The colors come from rfm["segment"]: we are overlaying the result of one algorithm (K-means) onto the map of another (t-SNE/UMAP). If every island on the map comes out in a nearly pure color, two independent methods agree — the same cross-validation of methods we did with hierarchical clustering in 05-02, now visual.
• We remove the axis ticks on purpose: in t-SNE/UMAP the coordinates mean nothing (they are not "recency" or "PC1"; they are positions optimized for readability).
• A typical result at MercaFresh: the four segments form sharper islands than in the PCA of 05-03; the ~70 boundary customers from the contingency table of 05-02 appear as points of the "wrong" color on the edges of the islands — another way of seeing that they are edge cases.

t-SNE/UMAP maps are so striking that they invite over-interpretation. Rules of caution, all consequences of "local yes, global no":

• The distances between islands are proportional to nothing. Two clusters sitting next to each other are not necessarily similar; two far apart are not necessarily opposites. (UMAP is somewhat more reliable here, but only somewhat.)
• Island sizes and densities deceive. t-SNE expands dense groups and shrinks spread-out ones so everything is visible: a big island is not a heterogeneous segment.
• Clusters can appear where there are none. With low perplexity/n_neighbors, t-SNE fragments even uniform noise into apparent little groups. Verify that the structure survives several parameter values before believing it — and cross-check it with a clustering method, not just with your eye.
• Do not use the coordinates as model features. They are not features: they carry no meaning, they change with the seed, and (in t-SNE) you cannot even compute them for a new data point. For compressing ahead of a model, the tool is PCA (05-03) on variance criteria; t-SNE/UMAP are for communicating and exploring.
• Run it twice before presenting. If a conclusion depends on a detail that changes with the seed, it was not a conclusion.

A recommended and very common workflow: PCA first, t-SNE/UMAP after — reduce from, say, 100 dimensions to 20 with PCA (removes noise and speeds things up) and apply t-SNE/UMAP on those 20 for the final map.

Let's recap module 5, because it forms a whole:

• K-means (05-01): segmentation as a label-free problem; centroids, elbow, silhouette; MercaFresh's four segments with business profiles.
• Hierarchical (05-02): structure at every scale in a dendrogram; linkage measures; the agreement with K-means as validation.
• PCA (05-03): the curse of dimensionality and its linear antidote; explained variance, loadings, the first 2D map.
• DBSCAN (05-04): clusters by density, of any shape, with native noise; MercaFresh's anomalous orders.
• t-SNE and UMAP (05-05): the nonlinear maps that communicate the structure — caution manual included.

The common thread: with no y column, the algorithms have extracted structure — actionable segments, hierarchies, axes of meaning, anomalies — from the geometry of the data alone. All of this converges in project 09-05, where the MercaFresh segmentation is carried out end to end.

• Using t-SNE/UMAP without scaling. They are neighborhood methods, and neighborhoods are distances: without StandardScaler, the map portrays the highest-magnitude feature. The rule has applied to the whole module, and there is no exception here.
• Presenting a map from a single run with a single parameter. Credible structure is the one that survives 2-3 perplexities/n_neighbors and 2 seeds. A single map is an anecdote.
• Measuring distances on the map ("the VIP segment is twice as far from the dormant as from the regulars"). The coordinates are not metric; that sentence means nothing in t-SNE.
• Using the embedding as input to a clustering or classifier to "improve results". It is circular and fragile: the map exaggerates separations and depends on the seed. Cluster on real features (or PCs); use the map for looking only.
• Tip: color the map with business variables in addition to the clusters (c=rfm["avg_order_spend"]): if the color gradient organizes itself spatially, the map is telling you which variable structures each zone — an extremely powerful piece of exploratory analysis with one line of code.

Exercise 1. Without code: for each statement, say whether a t-SNE map supports it, and why. (a) "There are about 4 clearly distinct groups of customers". (b) "The dormant segment is the most heterogeneous, because its island is the biggest". (c) "VIP and regulars are the most similar segments, because their islands sit next to each other". (d) "These 12 customers on the VIP island appear colored as regulars: they are boundary cases worth reviewing".

Exercise 2. Generate a dataset with 4 blobs in 10 dimensions (make_blobs(n_samples=600, centers=4, n_features=10, cluster_std=3.0, random_state=1)), scale it, and compare it across three maps: PCA(2), t-SNE (perplexity 30) and UMAP (if you have it installed; otherwise compare the first two). Color with the true labels returned by make_blobs. Which separates best? Why does PCA struggle here?

Exercise 3. With the dataset from exercise 2, run t-SNE with perplexity 2, 30 and 100 (same seed) and plot the three maps. Describe how the apparent structure changes and extract the practical rule.

Exercise 1

(a) Yes: counting groups and seeing which points share an island is exactly what t-SNE preserves (local neighborhoods). With the caveat of confirming that the 4 islands survive other perplexities. (b) No: t-SNE distorts sizes and densities by design; to measure heterogeneity, use real cluster statistics (deviations, 02-01) in the original space. (c) No: distances between islands are not proportional to similarity between groups; verify with distances between centroids in the scaled space. (d) Yes, as a hypothesis: the local mixing of colors reflects genuine neighborhood between those points and the VIPs; it fits the low silhouettes and the K-means/hierarchical disagreements. "Reviewing" is the right word: the map raises the suspicion, confirmation comes from the original data.

Exercise 2

Expected result: in PCA the 4 colors are visible but overlapping — the PC1-PC2 plane captures only part of the variance, and with cluster_std=3.0 the blobs brush against each other in the projection; the real separation is spread across the 10 dimensions. t-SNE (and UMAP, with an even cleaner drawing in a fraction of the time) shows 4 crisp islands, because it does not project: it repositions the points while preserving who neighbors whom. It is the textbook case of "PCA down to 2D fails to separate even though the groups exist".

Exercise 3

With perplexity 2, the map shatters into dozens of crumbs: each point attends to only 2-3 neighbors and even the inside of a single blob looks like several groups — spurious structure. With 30, the 4 correct islands appear. With 100, the islands remain visible but blurrier and closer together, because each point attends to neighborhoods that already spill beyond its own blob. Practical rule: small perplexity invents clusters, large perplexity blurs them; try several values and keep the structure that persists across a range — that one is real.

You have completed the unsupervised map-making kit: t-SNE repositions the points to preserve local neighborhoods at the price of distorting distances, sizes and global structure; UMAP does the same faster, with better overall geometry and the ability to project new data; and both demand the caution you have learned — cross-checked parameters, repeated seeds, no measuring on the map and no recycling coordinates as features. With the PCA/t-SNE/UMAP table you know how to pick the tool for the goal: compress, explore or communicate.

And with that, module 5 closes: MercaFresh now has customer segments with names and profiles, a hierarchy that validates them, axes that explain them, a watchdog for anomalous orders and maps to tell the whole story — extracted entirely from data without a single label. But a debt to module 4 remains outstanding: there we trained seven supervised models and cheerfully scored them with score() on some held-out data, without asking whether that figure could be trusted, what the hit percentage was hiding when the classes are imbalanced, or how much it would change with a different split of the data. Answering "is my model genuinely good?" with rigor is a whole discipline — data splitting, metrics, cross-validation, ROC curves, overfitting — and it is exactly what module 6 is about.