Logistic regression (04-02) separates the classes with a hyperplane, but it settles for any hyperplane that minimizes the log-loss. Support Vector Machines (SVMs) ask a more demanding question: of all the hyperplanes that separate loyal customers from churners, which one does it with the maximum safety margin? And when no straight cut is enough, they deploy their signature weapon — the kernel trick — to separate in a higher-dimensional space what was inseparable in the original one. In this lesson you'll build that geometric intuition step by step, understand the role of the support vectors and the C parameter, and apply SVC to MercaFresh's churn — with the data scaled, because few model families are as scale-sensitive as this one.

1. The maximum-margin hyperplane
2. Support vectors: the customers who define the boundary
3. Soft margin: the C parameter
4. The kernel trick: separating the inseparable
5. Implementation with scikit-learn: SVC on churn
6. Scale sensitivity and computational cost
7. When to choose (and not to choose) an SVM

Picture MercaFresh's churn with just two features: recency_days and trend. Loyal customers cluster bottom-right (low recency, high trend); churners, top-left. Between the two groups fit infinitely many separating lines: one hugging the loyal customers, another hugging the leavers, and everything in between.

Which to prefer? The SVM's intuition: the line that passes as far as possible from the closest points of both classes. That distance — from the boundary to the nearest point on each side — is the margin, and maximizing it is the algorithm's entire objective.

Why does the margin matter? Because future data won't land exactly where the training data did. A boundary hugging the loyal customers will misclassify the first new loyal customer who lands a millimeter beyond it. The maximum-margin boundary leaves the largest possible "buffer zone" on each side: it's the geometric bet on generalization — the same goal we pursued when watching the tree's overfitting in 04-03, now baked into the training criterion itself.

Here comes the model's most elegant property. The position of the maximum-margin hyperplane depends only on the points that touch the margin — the ones closest to the boundary. Those points are the support vectors (they give the algorithm its name: they literally support the boundary).

Practical consequences:

• Far-away points don't matter: the ultra-loyal customer with a recency of 2 days could move or vanish and the boundary wouldn't shift an inch. Compare with the linear regression of 04-01, where all points pull on the line (and an outlier hijacks it).
• The model is "sparse": out of 800 training customers, perhaps only 60 are support vectors. The rest are irrelevant for prediction.
• The support vectors are the business's borderline cases: the ambiguous customers, neither clearly loyal nor clearly lost. Inspecting them (model.support_vectors_) is looking at exactly the gray zone where the retention campaign's budget is won or lost.

With real data, a hyperplane that separates the classes perfectly almost never exists: there is always some loyal customer with a churner's profile and vice versa (noise, dubious labels, genuine overlap). Demanding perfect separation — the hard margin — is either impossible or produces boundaries contorted to please exceptions.

The solution is the soft margin: allow some points to violate the margin (or even land on the wrong side), paying a penalty for each violation. The C parameter sets the price:

C is the dial for the trade-off between fitting the training set and keeping margin — the same tension as max_depth in the tree, in a different disguise. Finding its optimal value systematically (together with the kernel's) is the subject of 07-05; in this lesson we'll turn it by hand to see its effect.

The case that breaks everything above: data where no straight line works. A classic example with a MercaFresh flavor: the customers who churn are those with very low spend (they're not interested) and those with very high spend (the competition snatches them with aggressive offers), while mid-spend customers stay. On the spend axis, the "churn" class occupies both extremes: impossible to separate with a single cut.

The saving idea: add a dimension. If we give each customer the extra feature spend² (distance to the center), the churners at both extremes end up high (large spend²) and the loyal ones low — and in that expanded space, a plane separates them cleanly. The linear boundary of the expanded space, seen from the original space, is a curve.

The kernel trick does this without building the new features: the SVM's mathematics only needs dot products between points, and a kernel function directly computes "the dot product the points would have in the expanded space" without ever visiting it. It's like getting the benefit of PolynomialFeatures (03-06) with infinitely many features at the cost of a few.

The RBF kernel deserves one more sentence: it classifies each new point by its similarity (a Gaussian bell, a relative of the normal from 02-02) to the support vectors — it can draw arbitrarily curved boundaries, even islands. Its gamma parameter controls the reach of that similarity: large gamma = narrow bells = a very flexible boundary (and prone to overfitting); small gamma = wide bells = a smooth boundary.

The SVM computes distances between points, so scaling is not optional: without it, avg_order_spend (tens of euros) would crush inactivity_ratio (0–1) in any distance. We bring back the full preprocessor from 03-06 — with its RobustScaler — inside the Pipeline, just as in 04-02:

Reading keys:

• gamma="scale": an sklearn heuristic that adapts gamma to the data's variance; a good starting point.
• probability=True: the native SVM returns distances to the boundary, not probabilities; this option adds an internal (costly) calibration so you can use predict_proba as in 04-02. If you only need the label or the ranking from decision_function, leave it out.
• In the comparison, notice the pattern: with C = 100, train accuracy rises and test accuracy usually drops — overfitting peeking through — while the number of support vectors falls (strict margin, fewer points inside it). With C = 0.1, a wide margin, more support vectors and smoother boundaries.
• If linear performs about as well as rbf, keep linear: faster, more interpretable (it has coefficients like 04-02) and fewer hyperparameters.

Two engineering warnings before falling in love with the model:

Scale. The demonstration of why the RobustScaler from 03-05 is not decorative:

Cost. Training a kernel SVM grows roughly between O(n²) and O(n³) with the number of samples — doubling the customers multiplies the time by 4-8. Practical orders of magnitude:

Moreover, predicting requires comparing each new point against all the support vectors: a model with thousands of them is slow in production too (08-02).

Rule of the trade: on a tabular problem of MercaFresh's size (thousands of customers), try logistic regression as a baseline, the tree for interpretability, and the RBF SVM when you suspect curved boundaries the others are missing. If the SVM wins clearly, it usually signals nonlinear structure that the ensembles (07-02, 07-03) will exploit as well.

• Training without scaling. The number one SVM mistake, and a silent one: the model trains, predicts and performs badly without saying why. The Pipeline with the 03-06 preprocessor makes it structurally impossible.
• Jumping straight to the RBF kernel with random C and gamma. Two sensitive dials at once produce anything from severe underfitting to total memorization. Start with the defaults (C=1, gamma="scale") and compare with the linear kernel before getting fancy; systematic fine-tuning arrives in 07-05.
• Using predict_proba without knowing its cost. probability=True trains an additional calibration with internal validation: it multiplies training time. For rankings (whom to call first), decision_function is enough, and free.
• Ignoring class imbalance. With 90% loyal customers, the optimal margin may sacrifice the minority class entirely. SVC(class_weight="balanced") rebalances the penalties; the fine-grained diagnosis, in 06-02.
• Tip: always look at n_support_. If most of the train set are support vectors, the model is memorizing (C or gamma too high) or the classes genuinely overlap — either way, it's an alarm no other number gives you this cheaply.

Exercise 1. Conceptual: in the churn dataset there is an unmistakably loyal customer (recency 1 day, trend 2.0) far from the boundary, and an ambiguous customer (recency 55, trend 0.8) right against it. Which of the two affects the SVM's boundary if you remove them from training? And in a logistic regression? Justify your answer.

Exercise 2. With the lesson's Pipeline, train an RBF SVM with gamma in [0.01, 0.1, 1, 10] (with C=1 fixed) and show train and test accuracy for each value. Interpret the pattern in terms of "the reach of the similarity bell".

Exercise 3. MercaFresh opens a wholesale line and its dataset grows from 5,000 to 2,000,000 labeled orders. The team proposes retraining the RBF SVM as is. Give two reasons to advise against it and two concrete alternatives from the course.

Exercise 1

The far-away customer is not a support vector: removing them doesn't change the boundary at all — the SVM depends only on the points that hold up the margin. The ambiguous one almost certainly is: removing them may shift the boundary visibly. In logistic regression both count (every point contributes to the log-loss), although the far one contributes a minuscule gradient: the boundary would move, slightly but measurably, in both cases. Moral: the SVM focuses its attention exactly on the gray zone; logistic regression listens to everyone.

Exercise 2

Expected pattern: with gamma = 0.01 the bells are so wide that everything resembles everything — a nearly flat boundary, mediocre performance on both sets (underfitting). As gamma rises, test improves up to a maximum. With gamma = 10 the bells are so narrow that each support vector only "sees" its immediate neighborhood: train shoots toward 100% and test drops — the model has drawn an island around every train customer. It's the 06-05 curve again, now with gamma as the complexity dial.

Exercise 3

Reasons against: (1) training cost of ~O(n²)-O(n³): going from 5,000 to 2,000,000 rows multiplies the time by hundreds of thousands — from minutes to months; (2) the resulting model would have an enormous number of support vectors, making prediction slow too and the model heavy in production (08-03 covers this maintenance). Alternatives from the course: logistic regression (04-02), which scales linearly and gives probabilities; trees (04-03) and their ensembles (07-02, 07-03), which handle millions of rows and nonlinearity; or at least LinearSVC, the kernel-free linear variant, if you want to keep the margin philosophy. With very large data, simple models catch up with complex ones: the kernel SVM's advantage lives in the scarce-data regime.

SVMs have given you the module's third geometry: where logistic regression fits a probabilistic hyperplane and the tree carves the space into rectangles, the SVM seeks the maximum-margin cut held up only by the borderline points, tolerates exceptions at a price set by C, and — with the kernel trick — separates in implicit spaces what was inseparable in the original. In exchange it demands respect: always scale, be careful with C and gamma, and stay aware of its cost as the data grows.

Notice the ingredient the RBF kernel put on the table: classifying by similarity to other points. The next lesson takes that idea to its purest, most radical expression: a model with no training, no equation and no explicit boundary, which classifies each new customer simply by asking its nearest neighbors what they are. It's K-Nearest Neighbors, the most intuitive algorithm in the whole module.