We now know what type of problem we have (04-02) and how to get the data ready (04-03). What remains is to open the black box of fit: what exactly does each algorithm do when it "learns"? In this lesson we will go through the classical algorithms that solve the vast majority of business problems: linear regression, logistic regression, k nearest neighbours, decision trees, random forests (and the idea of ensembles), support vector machines, Naive Bayes and k-means. For each one we will give the intuition, the pinch of maths needed to understand what is being optimised (remembering that "learning is optimising", 03-04) and code on NovaMarket's data: NovaClean demand for regression, returned orders for classification and customers for clustering. We will finish with a comparison table for choosing an algorithm and a code comparison of several classifiers on the same orders. It matters because choosing and understanding the algorithm is what allows you to interpret its results, explain them to Diego, know when to be suspicious and, later, tune it (04-06). A warning: the final comparison uses only accuracy, a provisional measure that in 04-05 we will replace with better ones.

1. Linear regression: the line that gets it least wrong
2. Logistic regression: from weighted sum to probability
3. k nearest neighbours: predicting by similarity
4. Decision trees: chained questions
5. Random forests and the idea of ensembles
6. Support vector machines: the widest margin
7. Naive Bayes: probability with independence
8. k-means from the inside: centroids and the elbow method
9. Comparison table and how to choose
10. Code comparison of several classifiers
11. Common Mistakes and Tips
12. Exercises
13. Conclusion

Linear regression assumes that the label is a weighted sum of the features plus a constant: with a single feature, a straight line y = a + b·x; with several, y = a + b₁·x₁ + b₂·x₂ + .... The parameters are a (intercept) and the coefficients b. Which line to choose? The one that minimises the sum of squared errors (least squares): for each example the difference between the actual value and the predicted one is computed, squared (so that errors above and below do not cancel out and so that large ones are penalised more) and summed.

A small numerical example: four weeks with sales 350, 340, 380, 390. We compare three candidate lines:

LinearRegression().fit finds the last one without trying candidates: for this problem there is a closed-form formula (the so-called normal equations) that gives the optimal coefficients directly. It is one of the few algorithms with an exact solution; most of the following ones optimise by iterative search.

The great virtue of linear regression is interpretability: each coefficient says how much the prediction changes per unit of the feature, with the others held fixed.

In 04-02 a straight line on week gave a mean error of 47 units because it ignored seasonality and Black Friday. We add features that represent them (as announced in 04-03) and the same algorithm improves a lot:

Output:

Explanation: the line now has four coefficients. week (2.47) recovers the true trend of 2.5 units/week; sin and cos capture the yearly wave (the combination of the two is equivalent to a wave of amplitude √(10² + 61²) ≈ 62, against the true 60); black_friday adds 255 units that week (250 in truth). The mean error drops from 47 to 17 units. The algorithm is the same; what has changed are the features. It is the lesson of 04-03 confirmed with numbers.

Despite the name, it is a classification algorithm. It computes the same weighted sum as linear regression, z = a + b₁·x₁ + ... + bₙ·xₙ, but instead of using z as the prediction it passes it through the sigmoid function, σ(z) = 1 / (1 + e^(−z)), which turns any number into a value between 0 and 1 interpretable as the probability of the positive class. If z = 0, the probability is 0.5; very positive values give a probability close to 1 and very negative ones, close to 0. It is exactly the formula we used in the generator of 04-01 to create the "hidden truth"; that is why logistic regression is a natural candidate for these data.

Then a threshold is applied: by default, probability ≥ 0.5 → class 1. That threshold is not sacred: in 04-05 we will see that it is worth moving it according to the cost of each error.

The parameters (the coefficients) are learned by maximising the likelihood of the data, or equivalently by minimising the log loss (04-01), which heavily punishes assigning a low probability to what actually happened. There is no closed-form formula: it is optimised iteratively by following the gradient, the idea we announced when closing 03-04 and that we will develop in 05-03.

Output:

Reading: each euro of amount adds 0.012 to z; each delivery day, 0.31; being a new customer, 1.88; each additional item subtracts 0.20. For the €250 order (1 item, 5 days, new customer), z = −4.906 + 0.012·250 − 0.201·1 + 0.307·5 + 1.882·1 = 1.30, and the sigmoid gives 0.786: a return is predicted. The same order from a regular customer would have z = −0.58 and probability 0.36: it would not be flagged. Diego can follow the calculation with a calculator, and that is the strength of this model. Note: without scaling, the coefficients are not comparable across columns (0.012 per euro versus 1.88 for being new); to compare importance you have to standardise first (04-03) or look at the range of each variable.

k-NN learns no formula: it memorises the training set and, to predict a new order, looks for the k most similar orders (the closest in feature space, with the usual Euclidean distance) and votes: the majority class among those neighbours (or the mean of their values, in regression). It is the most intuitive algorithm ("orders like this one were returned") and it has two practical consequences:

• It needs scaling (04-03): without it, amount (tens or hundreds) dominates the distance and new_customer (0/1) does not count.
• The hyperparameter k controls smoothness: k = 1 follows the nearest neighbour (memorises the noise), a large k averages a lot (loses detail). In 04-06 we will see that it is the textbook example of overfitting versus underfitting.

Output:

With k = 1 the model gets 100 % of the training set right (each order is its own neighbour) and only 81.5 % of the test set: pure memory. With k = 15 it stabilises at 87.5 %. And without scaling it drops to 83.7 %, almost the level of "never returned" (83.6 %): the distance was dominated by the amount.

A decision tree does what learn_threshold did in 01-02, but repeatedly: it chooses the column and threshold that best separate the classes, splits the data in two, and repeats in each half until the leaves are pure enough or the maximum depth is reached. The result is a set of readable rules.

What does "best separate" mean? The impurity of a group is measured: if all its orders belong to the same class, impurity 0; if they are 50/50, maximum impurity. The two usual measures are Gini impurity, 1 − p² − (1−p)² for two classes with proportion p of positives, and entropy, −p·log₂p − (1−p)·log₂(1−p). Both behave the same in practice. The tree tries every threshold of every column and chooses the one that most reduces the weighted impurity of the two children compared with the parent.

With numbers from our data: at the root there are 2,250 orders with 16.4 % returned, Gini = 1 − 0.164² − 0.836² = 0.275. The split amount ≤ 195.65 leaves 1,895 orders on the left with 11.5 % returned (Gini 0.204) and 355 on the right with 42.8 % (Gini 0.490); the weighted impurity is (1,895·0.204 + 355·0.490)/2,250 = 0.249, a gain of 0.026. The alternative split on new_customer would give 0.252 (less gain); that is why the tree asks about the amount first. That is all the "learning" there is: a greedy search (03-02) for the best question at each node.

We use the four numeric columns plus the category as one-hot (unscaled: trees do not need it):

Output (abridged):

Each leaf shows how many training orders fell into it [not returned, returned] and the majority class. It reads like a rulebook: "orders above €195 from new customers: returned (56 of 72; and 37 of 37 if it is electronics)"; "regular customers only above €358". Diego recognises his intuition corrected by the data: his €300 threshold was reasonable for regular customers and too high for new ones. The leaf [64, 60] (new customers between €142 and €196) is almost 50/50: it is the grey zone of 02-04.

If we remove max_depth, the tree keeps splitting until the leaves are pure: on these data it reaches depth 21 with 414 leaves, gets 100 % right on training and 82.5 % on test, worse than depth 3. This is the overfitting we will study in 04-06; for now, hold on to the fact that depth is a hyperparameter that must be limited.

A single tree is unstable: it changes a lot if the data change a little. The solution is an ensemble: train many models and combine their votes. The random forest trains hundreds of trees, each on a random sample with replacement of the data (bagging, from bootstrap aggregating) and considering at each node only a random subset of columns, and predicts by majority (or average in regression). Each tree gets things wrong differently, and the errors cancel out when voting: it is the wisdom of crowds applied to models.

Output:

n_estimators is the number of trees and min_samples_leaf the minimum number of orders per leaf (it avoids single-example leaves). You lose the rule reading of a single tree, but you gain robustness and the feature importances we used in 04-03 (here, the amount explains more than half of the separation).

The other great family of ensembles is boosting (gradient boosting: GradientBoostingClassifier, and the XGBoost, LightGBM and CatBoost libraries): instead of independent trees in parallel, they are trained in sequence, and each new tree concentrates on correcting the errors of the previous ensemble. It is, today, the algorithm that wins the most competitions and projects on tabular data; it is presented in 07-03 and for now it is enough to know that it exists and when to try it (when the forest works and you want to squeeze out more).

The SVM (support vector machine) looks for the boundary that separates the two classes leaving the widest possible margin on both sides; the examples that touch the margin are the support vectors, and only they determine the boundary. When the classes cannot be separated by a straight line (or a plane), the kernel trick projects the data into a higher-dimensional space where they can be, without computing that projection explicitly; the RBF (Gaussian) kernel is the most widely used and allows curved boundaries. The main hyperparameters are C (how much misclassified points are penalised) and gamma (how local the boundary is with RBF). It needs scaling and scales poorly to very large datasets (hundreds of thousands of rows), but it is very solid at medium sizes.

Intuition only: the SVM does not return probabilities naturally (there is a probability=True option, slower) and its boundaries cannot be read as rules.

Naive Bayes applies Bayes' theorem (which we will study in detail in 06-03) to compute the probability of each class given the features, with the "naive" assumption that the features are independent of each other given the class. That assumption is almost never true, but the algorithm works surprisingly well, is extremely fast and needs little data; it is the classic of spam filtering and a good baseline for classifying reviews by their words (case 4). On our orders, GaussianNB().fit(X_train, y_train) gets 86.0 % right on test.

In 04-02 we used k-means to segment customers. Internally it is an iterative two-step algorithm (Lloyd's algorithm):

1. Choose k initial centroids (points in feature space; KMeans uses a smart initialisation called k-means++).
2. Assign each customer to the nearest centroid.
3. Recompute each centroid as the mean of the customers assigned to it.
4. Repeat 2-3 until the assignments no longer change.

It is a hill climb (03-04) on the inertia, the sum of squared distances of each point to its centroid: each iteration reduces it, and it stops at a local optimum; that is why n_init=10 runs 10 different starts and keeps the best, exactly like the random restarts of 03-04. On our customers it converges in 4 iterations.

The number of groups k is a hyperparameter that has to be decided. The elbow method trains k-means for several values of k and plots the inertia: it always drops as k grows (with k = n it would be 0), but a point comes where adding groups contributes little; that "elbow" is a good k.

Output:

From 1 to 2 groups the inertia drops by 1,000; from 2 to 3, another 400; from 3 to 4, only 100, and after that less than 50 per group: the elbow is at k = 3, which matches the three kinds of customer. If you plot it with Matplotlib (plt.plot(range(1, 9), inertias, marker="o")) you will see it as a bent arm. The elbow is not always so clear; then it is complemented with the silhouette coefficient (silhouette_score) and, above all, with the business interpretability of the groups.

Marta's practical rule: start with logistic (or linear) regression as an interpretable baseline, try a random forest, and only if it pays off move on to boosting or networks. And choose according to the whole problem, not just accuracy: if Diego needs to justify every refund denial, the tree or the logistic regression win even if the forest is slightly more accurate.

We reuse the preprocessing from 04-03 as we packaged it in its section 9.1 (prepare_orders and build_preprocessing in novamarket_ml.py) and change only the last step of the pipeline:

Output:

Observations (provisional, because accuracy misleads with imbalanced classes, as we will see in 04-05):

• Logistic regression is the best here, which is no surprise: the generator's "hidden truth" is a sigmoid of a weighted sum, precisely its family. On real data, with interactions and non-linearities, forests usually overtake it.
• k nearest neighbours does worse than in section 3 (0.841 versus 0.875): with the 21 columns of the pipeline, many of them noise, distances lose their meaning (the "curse of dimensionality"). It is the algorithm most sensitive to feature selection.
• The forest has the largest gap between train (0.908) and test (0.859): the beginnings of overfitting, which 04-06 will handle with hyperparameters.
• No model gets past 87 % and the baseline is at 83.6 %. By this measure it looks like little gain; in 04-05 we will see that, in what matters (detecting returns without bothering honest customers), the difference is much larger.

One important algorithm is missing from this list: the perceptron and neural networks, which are "just another algorithm" for classification and regression (scikit-learn has MLPClassifier), but whose architecture and training deserve a whole module, module 5.

• Using k-NN or SVM without scaling. Results drop to baseline level, as we saw with k-NN without StandardScaler.
• Leaving a tree with no depth limit. It memorises (100 % on train, 82.5 % on test). Set max_depth or min_samples_leaf, or use a forest.
• Interpreting logistic regression coefficients without looking at the units. 0.012 per euro can weigh more than 1.88 for being a new customer when the amount varies by hundreds of euros. Standardise before comparing.
• Choosing the algorithm by fashion. Boosting or networks are not better by definition; for 3,000 orders with few columns, logistic regression wins. Always try the simple baseline.
• Trusting k-means without the elbow or interpretation. It always finds k groups; the elbow, the silhouette and business sense say whether they are worth anything.
• Comparing models with a single accuracy figure. That is what we have just done, and that is why we have insisted it is provisional. The next lesson gives the right tools.

Exercise 1. Add to the demand regression (section 1.2) the feature units_previous_week (d["units"].shift(1), dropping the first row with dropna()). Does the mean error improve? Look at the coefficient it receives. Why does it contribute little when trend, seasonality and Black Friday are already there, and in what real situation would it be more valuable?

Exercise 2. Compute by hand (or with two lines of pandas) the Gini impurity of the split delivery_days <= 4 at the root of the training data of section 4, and compare it with that of amount <= 195.65 (0.249). Which one would the tree prefer? Then check what happens to the depth and the number of leaves if you train with min_samples_leaf=50 instead of max_depth=3.

Exercise 3. With the customers of section 8, compute the silhouette coefficient (from sklearn.metrics import silhouette_score; silhouette_score(Xc, km.labels_)) for k from 2 to 6. Does the maximum coincide with the elbow? Then change the seed of generate_customers_ml to 7 and repeat: does the conclusion hold?

Solution 1. The mean error stays practically the same (17.0 versus 16.8 units) and the lag coefficient is almost zero (0.04): the other four features already explain the series, so the previous week's sales add no new information. It would be valuable on a real series, where demand has inertia that neither the fixed trend nor the seasonality capture (a campaign starting, a product becoming fashionable, a stock-out): there the lag is usually the most important feature. It is also the only feature that requires care with leakage (04-03): always shift(1).

Solution 2. For delivery_days <= 4: the left half (days 1-4, 1,293 orders) has 11.7 % returned and the right (5-7, 957 orders) 22.9 %; the Ginis are 0.206 and 0.353 and the weighted impurity 0.269, a gain of only 0.006 against the 0.026 of the split on amount. The tree prefers the amount. With min_samples_leaf=50 (no max_depth) the tree grows more than with depth 3 (depth 8 and 32 leaves), but no leaf has fewer than 50 orders, which limits memorisation: it gets 0.876 on training and 0.864 on test, similar to the depth-3 tree. They are two different ways of curbing the same problem, and in 04-06 we will compare them methodically.

Solution 3. With seed 42, the silhouette is maximal at k = 3 (0.57, versus 0.55 with k = 2 and 0.53 with k = 4) and drops to 0.42 with k = 6: it coincides with the elbow and with the three kinds of customer. With seed 7 the values change slightly (0.58 at k = 3) but the maximum stays at k = 3, because the hidden structure (three groups) is the same; only the specific customers change. If on a real dataset the silhouette were low for every k (below 0.25) or the elbow did not appear, the honest conclusion would be that there are no clear groups, and forcing a segmentation would be inventing structure.

In this lesson we have opened the box of fit for the classical algorithms: linear regression minimises squared errors and, with the right features, reduced the NovaClean demand error from 47 to 17 units; logistic regression passes a weighted sum through the sigmoid to give readable probabilities; k nearest neighbours predicts by similarity and demands scaling; trees chain questions choosing at each node the one that most reduces Gini impurity, and produce rules Diego can read; random forests average many trees (bagging) and boosting chains them, correcting errors; SVMs look for the widest margin with the help of kernels; Naive Bayes applies Bayes with independence; and k-means alternates assigning and recomputing centroids, with the elbow method to choose k. The comparison table and the code comparison have left us with a first ranking of models on NovaMarket's orders, measured only by accuracy.

And that is the pending problem: accuracy says logistic regression gets 87.1 % right and the "never returned" baseline 83.6 %, a difference that looks small to Diego. In the next lesson, Model Evaluation and Validation, we will see why that figure misleads with imbalanced classes, we will learn to read the confusion matrix and translate each type of error into euros, we will measure precision, recall, F1 and AUC, choose the threshold according to cost, use regression metrics for demand and validate with methods that are not fooled by a single lucky split, including temporal validation for series.