The two previous models share one deep limitation: their decision boundary is a hyperplane — a straight cut through feature space. But MercaFresh's churn doesn't always yield to a straight cut: a customer churns if they've gone a long time without buying and their trend collapses, or if they're on the basic plan and their spend drops, and those rules built from "and" and "or" draw stepped boundaries. Decision trees attack the problem the way a human analyst would: by chaining questions. In this lesson you'll see how an algorithm decides what to ask and in what order (Gini impurity and entropy, with the arithmetic worked by hand), you'll train and draw a tree on MercaFresh's churn, and you'll understand why this highly interpretable model is also a compulsive memorizer if you don't rein it in.

1. The intuition: classifying by asking
2. How each split is chosen: Gini impurity
3. Entropy and information gain
4. Regression trees (briefly)
5. Implementation with scikit-learn: MercaFresh churn
6. Interpretability: readable rules and feature importance
7. Key hyperparameters and the tendency to overfit
8. Strengths and limitations

A decision tree is a cascade of binary questions about the features. Each customer enters at the root, answers questions and descends to a leaf, which issues the prediction:

Notice three things:

• Every root-to-leaf path is a readable business rule: "if the customer has gone more than 45 days without buying and places ≤1.5 orders/month, predict churn (91% of the historical customers in that leaf churned)".
• Leaves store proportions, not just labels: the tree also gives probabilities (the churn fraction in the leaf).
• The resulting boundary is stepped: each question cuts the space with a plane perpendicular to an axis; the combination forms rectangular regions. It is a genuinely nonlinear model without any feature transformation.

The million-euro question: given the dataset, which question goes at the root? recency <= 45 or spend <= 20? And why 45 rather than 60? The algorithm needs a numeric criterion to compare candidate questions.

The idea: a good question separates customers into groups that are as pure as possible — groups where almost everyone is churn or almost everyone is loyal. Gini impurity measures how mixed a group is:

$$Gini = 1 - \sum_{k} p_k^2$$

where $p_k$ is the proportion of each class in the group. For two classes:

Full hand calculation. A node with 10 MercaFresh customers: 4 churn, 6 loyal. Initial Gini: $1 - 0.4^2 - 0.6^2 = 1 - 0.16 - 0.36 = 0.48$. Candidate: recency_days <= 60.

Gini after splitting = size-weighted average: $\frac{6}{10} \cdot 0.278 + \frac{4}{10} \cdot 0.375 = 0.167 + 0.150 = 0.317$.

The split reduces the impurity from 0.48 to 0.317: a gain of 0.163. The algorithm repeats this calculation for every feature and every possible cut point, picks the split with the highest gain, and recurses on each child group until the nodes are pure or a limit is reached. This greedy procedure (always take the locally best option, never look back) is called CART and is what scikit-learn implements.

The classic alternative criterion comes from information theory. Entropy measures a group's uncertainty:

$$H = -\sum_{k} p_k \log_2 p_k$$

With the same node as before (4 churn, 6 loyal): $H = -0.4 \log_2 0.4 - 0.6 \log_2 0.6 = 0.529 + 0.442 = 0.971$ bits — nearly maximum uncertainty (1 bit, that of a 50/50). A pure group has entropy 0: there is nothing left to guess. The information gain of a split is the reduction in entropy, computed with the same weighted average we used for Gini.

Gini or entropy? In practice they yield nearly identical trees:

Hold on to the shared idea: splitting is buying purity, and the tree always buys where the gain is greatest.

The same mechanism predicts numbers: for the monthly spend of 04-01, each leaf predicts the mean of the target across its customers, and the "impurity" to reduce is the variance (the MSE within the node). The question orders_per_month <= 3.2 is good if it separates customers into two groups with internally similar spend. DecisionTreeRegressor implements this with the same interface. The resulting prediction is a step function — constant within each rectangular region — which makes these trees poor extrapolators but good at capturing jumps and thresholds.

We reuse the churn dataset from the previous lesson (the df from 04-02). A liberating detail that picks up the table in 03-05: trees compare each feature against itself (recency <= 45 doesn't change whether recency is in days or scaled), so they need neither scaling nor unskewing. Imputing nulls and encoding categoricals is enough:

Reading the code:

• max_depth=3: at most 3 chained questions — a tree that fits on one slide. In a moment we'll see why limiting depth is not optional.
• min_samples_leaf=20: no leaf may end up with fewer than 20 customers; it forbids rules built on anecdotes.
• plot_tree paints each node with its question, its Gini, how many samples it holds and its class split — the theory of sections 2 and 3, drawn. More intense colors = purer nodes.
• random_state=42: ties between equally good splits are broken at random; fixing the seed makes the tree reproducible.

Two outputs of the tree are worth gold in a business meeting. The first, the rules in plain text:

Any retention manager understands that without knowing what a Gini is. The second, the importance of each feature: how much total impurity reduction the splits using it contributed:

If recency_days and trend dominate the ranking, the model confirms the business thesis of 03-06: churn announces itself through silence and cooling off. Caution: importances sum to 1 and get shared out among correlated features somewhat arbitrarily (the same warning as with the coefficients in 04-01/04-02), and this method tends to favor features with many distinct values.

Here is the dark side. Without limits, the algorithm keeps splitting until every leaf is pure — even if that takes one leaf per customer. That tree knows the training set by heart: it scores 100% on train and collapses on test, because its final splits capture not patterns but individual noise. This is overfitting, which we'll diagnose rigorously in 06-05; trees are its textbook example.

The brakes (pre-pruning hyperparameters):

Choosing these values systematically is the subject of hyperparameter optimization (07-05). And a preview that explains half the ML industry: the best cure for a tree's overfitting is not pruning it more fiercely, but averaging many different trees — Random Forest-style ensembles (07-02) and gradient boosting (07-03) are born exactly there. In this course, the individual tree is the building block; there you'll see the building.

• Training without limits and boasting about 100% on train. That number measures memory, not learning. Always compare train against test; a large gap is the overfitting alarm (06-05).
• Scaling the features "just in case" with a shared Pipeline. It breaks nothing, but it wrecks interpretability: nobody understands the rule recency <= 0.83 (in robust units). For trees, leave features in their natural units.
• Taking the drawn tree as stable truth. Retraining with 5% more data can reorganize entire branches. The importances tend to be more stable than the structure; base the business conclusions on them.
• Reading feature_importances_ as causality. It is impurity reduction, not causal effect — the same 02-03 caution we've carried since the coefficients.
• Tip: always start with a small tree (max_depth=3) and draw it. Even if the final model is something else, that drawing is the project's best exploration and communication tool: it tells you which features cut, and where.

Exercise 1. By hand: a node has 8 customers (4 churn, 4 loyal). Split A separates them into (3 churn, 1 loyal) and (1 churn, 3 loyal); split B separates them into (4 churn, 2 loyal) and (0 churn, 2 loyal). Compute the weighted Gini after each split and decide which one the algorithm would pick.

Exercise 2. Train the churn tree with max_depth from 1 to 12 and plot accuracy on train and on test against depth. Describe the three zones of the curve and locate the reasonable depth.

Exercise 3. Use export_text to extract the complete rule along the path leading to the leaf with the highest churn proportion in the max_depth=3 tree, and translate it into a sentence that could appear in a report for MercaFresh's retention team.

Exercise 1

Initial Gini: $1 - 0.5^2 - 0.5^2 = 0.5$.

• Split A: each child has Gini $1 - (3/4)^2 - (1/4)^2 = 0.375$. Weighted: $\frac{4}{8}(0.375) + \frac{4}{8}(0.375) = 0.375$.
• Split B: left child $1 - (4/6)^2 - (2/6)^2 = 0.444$; right child $1 - 0 - 1 = 0$ (pure). Weighted: $\frac{6}{8}(0.444) + \frac{2}{8}(0) = 0.333$.

B wins (0.333 < 0.375): even though it leaves one fairly mixed child, it manufactures one completely pure node, and the criterion finds that worthwhile. Lesson: the algorithm values total weighted purity, not a balanced split.

Exercise 2

Three zones: (1) depths 1-2, both curves low — the tree is too simple for the pattern (underfitting); (2) an intermediate zone (typically 3-5), where test reaches its maximum; (3) beyond that, train keeps climbing toward 100% while test stalls or drops — the tree memorizes noise (overfitting). The reasonable depth is the one at the test maximum. This inverted-U curve is the portrait of the bias-variance trade-off we'll formalize in 06-05.

Exercise 3

Find the leaf with class: churn and the highest purity and chain its conditions together. With the simulated data, a typical result: recency_days > 52.5 and trend <= 0.74 and orders_per_month <= 2.1. Report translation: "The highest-risk segment is customers who have gone more than 52 days without buying, whose recent activity is less than three quarters of their usual level, and who place two orders a month or fewer: historically, the vast majority of these customers end up churning. We recommend prioritizing them in the retention campaign." The rule is actionable precisely because the features (03-06) were designed with a business reading in mind.

You've added the first nonlinear model to your toolbox: the tree classifies by chaining questions, chooses each one by buying maximum purity (Gini or entropy — you can now do the arithmetic by hand), reads as business rules and ranks features by importance. You've also seen its Achilles' heel: without max_depth and min_samples_leaf, it memorizes instead of learning — a preview of overfitting (06-05) and the motivation for ensembles (07-02). And one new convenience: it is the first model in the course that needs no scaling.

The tree draws stepped boundaries, perpendicular to the axes. The next lesson attacks the geometry from the opposite angle: instead of carving up the space with questions, it searches directly for the best possible cut — the hyperplane that separates the classes with the maximum safety margin — and, when no straight cut suffices, it projects the data into a space where one exists. These are support vector machines.