In the previous lesson we previewed boosting as the sequential family of ensembles: models trained one after another, each correcting the errors of the last. This lesson is devoted entirely to its star representative, gradient boosting, probably the winningest algorithm of the last decade on tabular data: it dominates Kaggle competitions and solves in production problems exactly like MercaFresh's churn and demand forecasting. You'll understand its mechanics through a worked numerical example, learn to handle its two main knobs (learning rate and number of estimators), meet the implementations that matter — sklearn, HistGradientBoosting, XGBoost, LightGBM — and pit it against the Random Forest from the previous lesson.

1. The idea: learning from the residuals
2. A numerical example by hand: 3 iterations
3. Learning rate and n_estimators: the balance
4. Gradient boosting in scikit-learn
5. HistGradientBoosting: the modern version
6. XGBoost and LightGBM: the board's dominators
7. Boosting on MercaFresh's churn: duel with Random Forest
8. The Achilles' heel: overfitting and its brakes
9. Interpretation: feature importance and SHAP

Random Forest trains independent trees and averages. Gradient boosting does the opposite: it trains small, weak trees in a chain, where each one has a single mission: predicting the error left behind by the previous ones.

In regression, with squared error, the process is very intuitive:

1. Start with a trivial prediction: the mean of y (our old friend the DummyRegressor from 06-02).
2. Compute the residuals: residual = actual_y - current_prediction.
3. Train a small tree to predict those residuals from the features.
4. Update: new_prediction = current_prediction + η · tree_prediction, where η (eta) is the learning rate, a factor that shrinks each correction.
5. Go back to step 2, as many times as n_estimators says.

The name "gradient" comes from the fact that the residuals are, mathematically, the gradient (with flipped sign) of the squared loss function: each tree takes a step of gradient descent — the same one we used in 04-01 to fit linear regression — but in prediction space rather than coefficient space. With other losses (e.g. logistic for classification) the same scheme applies, replacing the residual with the corresponding "pseudo-residual"; that's why the method generalizes to classification without changing philosophy.

Take 4 MercaFresh customers and their monthly spend (in €). To keep the arithmetic easy to follow, we'll use learning rate η = 1 (no shrinkage) and simplified "trees" that predict the mean residual of their group (high- vs. low-frequency customers):

Iteration 0. Initial prediction = global mean = (120+100+40+60)/4 = 80 for everyone.

Iteration 1. Tree 1 splits by frequency and predicts the mean residual of each group: +30 for high frequency, −30 for low. We update:

The error has dropped dramatically: from residuals of ±40/±20 to ±10.

Iteration 2. Within each group the residuals now sum to zero, so a tree that only saw frequency would predict 0; suppose tree 2 finds another feature (e.g. tenure) that separates Ana from Bruno and David from Carla, and predicts +10/−10 accordingly:

Three steps (base prediction + 2 trees) and the train set is nailed. And here lies the double moral: the sum of weak correctors reaches an accuracy none of them could achieve alone… but we've also just seen how quickly this method can memorize the training set — with zero residual on train, whatever is left is fitting to noise. In practice, η = 1 is almost never used: shrinking each step is the first brake against overfitting.

The two main knobs are coupled:

• learning_rate (η): how much each tree contributes. Typical values: 0.01–0.3.
• n_estimators: how many trees are chained.

The relationship is one of compensation: if you halve η, you need roughly twice as many trees to reach the same point. Why bother lowering η, then? Because many small steps generalize better than few large ones: each gentle correction leaves room for the following trees to adjust course, while a large step can chase noise irreversibly.

Unlike Random Forest — where more trees only stabilized —, in boosting n_estimators is genuinely a complexity knob: each additional tree keeps lowering the training error, and past a certain point the validation error starts climbing. It's exactly the silhouette of the curves in 06-05, and its natural antidote will be the early stopping we'll see shortly.

The classic implementation: GradientBoostingRegressor and GradientBoostingClassifier.

Note max_depth=3: in boosting, the trees are dwarfs on purpose (1–4 levels). The weak learner is a design decision: the power comes from the sequence, not the individual. subsample < 1 adds bagging-style randomness ("stochastic gradient boosting"), which usually improves generalization.

Since scikit-learn 0.21 there is a much faster reimplementation: HistGradientBoostingClassifier/Regressor. Its trick is binning each feature into a histogram of at most 255 buckets, so finding the best split no longer requires sorting continuous values: you sweep the buckets. On datasets of tens of thousands of rows it is orders of magnitude faster, and in addition:

• It handles missing values natively (it learns which branch to send NaNs to, with no prior imputation — though what you learned in 03-02 is still needed for the other models).
• It ships with built-in early stopping (early_stopping=True) using an internal validation fraction.
• It supports native categorical features (categorical_features).

Current rule of thumb: for gradient boosting within the sklearn ecosystem, use the Hist version by default; the classic one remains for small datasets or for compatibility.

Outside sklearn live the two libraries that popularized modern gradient boosting (installed separately: pip install xgboost lightgbm):

Why does this family dominate on tabular data (tables of customers, transactions, sensors…)? Because trees capture interactions and non-linearities effortlessly, ignore feature scales, digest mixes of numeric and categorical variables, and boosting squeezes out every last drop of signal — all with training times of seconds or minutes. On this turf they usually beat even deep networks, as we'll discuss honestly in 07-04.

Both libraries offer early stopping against an explicit validation set, the professional way of setting n_estimators:

You request a deliberately high n_estimators and let the validation set decide where to stop: if after 50 rounds the metric hasn't improved, it halts and keeps the best iteration. Notice that the discipline of 06-01 reappears: the validation set used for stopping cannot be the final test set.

Let's pit the previous lesson's champion against the new contender, on the same churn dataset with RFM features and the same stratified cross-validation from 06-03.

On small datasets with simple signal like this one, the two usually finish neck and neck (sometimes the forest wins). The general pattern in practice is:

• Random Forest: excellent results with zero tuning; hard to break. The strong baseline.
• Gradient boosting: a higher ceiling — with its hyperparameters well chosen it usually scrapes out a few extra metric points, especially on large datasets with complex interactions — but it demands that tuning (which we'll automate in 07-05) and vigilance against overfitting.

For MercaFresh the business reading is concrete: if the forest delivers an F1 of 0.82 effortlessly and well-tuned boosting delivers 0.85, those three points are dozens of recoverable customers each month correctly prioritized by the retention campaign — usually worth the effort.

Boosting chases residuals, and residuals eventually become noise: if you let it run, it memorizes the train set (we saw it in the hand-worked example, which reached zero residual). It is the star pupil of the pathologies of 06-05, and that's why it comes equipped with more brakes than any other algorithm:

Look at the last row: the regularization of 07-01 reappears inside the trees. The techniques in this module aren't sealed compartments; they're combinable pieces. The standard defensive recipe: low learning rate + early stopping + shallow trees, and always validate with module 6's CV.

Like Random Forest, boosting models offer feature_importances_ (in classic sklearn and in XGBoost/LightGBM) with the same virtues and the same biases we discussed in 07-02: useful as a global ranking, dangerous with correlated features.

For serious explanations, the current standard is SHAP (SHapley Additive exPlanations, the shap library): it assigns to each feature of each individual prediction a signed contribution — "the model gives this customer a 78% churn probability: the 45-day recency contributes +0.20, their 2 incidents +0.12 and their 5 years of tenure −0.08". That is gold for MercaFresh's retention team, which needs to know not just who is leaving but why. We won't develop it here — it belongs to advanced interpretability and will resurface in the ethical context of 08-04 —, but you should know it exists and that it pairs especially well with tree models.

• Raising n_estimators "because more is better". True in Random Forest, false in boosting: here every extra tree adds complexity. Set a high cap and let early stopping decide.
• Tuning the learning rate without touching the number of trees. They're coupled: if you drop η from 0.1 to 0.01, multiply the tree budget by ~10 or the model will be undertrained.
• Using deep trees as the base. max_depth=10 in boosting is usually a recipe for overfitting; the strength lies in the sequence of weak correctors, not in strong correctors.
• Doing early stopping against the test set. The set that decides when to stop takes part in training for all practical purposes: it's one of the leaks from 06-01. Use a validation split or the algorithm's own internal validation, and keep the test set untouched.
• Scaling the features "just in case". It does no harm, but it's unnecessary: like all tree-based methods, boosting is insensitive to scale. Save that pipeline step (and remember it is mandatory for SVM, K-NN or the regularized models of 07-01).
• Tip: in real projects, train a Random Forest first (strong baseline, 07-02) and then a HistGradientBoosting/LightGBM with early stopping. If the boosting doesn't clearly beat the forest, keep the forest: fewer hyperparameters to maintain.

1. Boosting by hand, one more iteration. Repeat the lesson's numerical example but with learning rate η = 0.5: recompute the predictions and residuals of iterations 1 and 2 and add an iteration 3 (assume each tree still predicts the mean residual of its group, alternating the frequency split and the within-group split). Do you reach zero residual? What does this illustrate about η?
2. The η–n_estimators coupling. On MercaFresh's churn, train GradientBoostingClassifier(max_depth=3, subsample=0.8) with the combinations (η=0.3, 50 trees), (η=0.1, 150), (η=0.03, 500) and (η=0.3, 500). Evaluate F1 with the stratified CV. Do the results confirm the table in section 3?
3. Early-stopping curve. With a (stratified) train_test_split, set aside 20% of the churn data for validation. Train GradientBoostingClassifier(n_estimators=400, learning_rate=0.1, max_depth=3) and use staged_predict_proba to compute the log-loss on train and on validation after each tree. Plot both curves: at which iteration would early stopping with patience 20 have halted? Which shape from 06-05 do you recognize?

1. With η = 0.5, iteration 1 adds half the correction: predictions 95/95/65/65 and residuals +25/+5/−25/−5. Iteration 2 (correction by frequency groups: mean residuals ±15, applied at 50%) leaves predictions 102.5/102.5/57.5/57.5 and residuals +17.5/−2.5/−17.5/+2.5; iteration 3 keeps reducing but doesn't reach zero. The illustration: with η < 1 the model approaches the perfect train fit asymptotically — it never nails it in one blow —, and that leaves room to halt the sequence (early stopping) at the point of best generalization before it memorizes the noise.
2. The first three combinations (balanced budget η·n_estimators ≈ 15) yield similar F1 scores, with the usual slight edge for low η + many trees and a smaller spread between folds. The fourth (η=0.3 with 500 trees, budget 150) performs worse on validation despite having the lowest training error: pure overfitting. It confirms the table: what matters is the balanced product, and overspending the budget takes its toll.
3. The training log-loss descends monotonically (boosting can always keep filing down residuals). The validation one drops, hits a minimum — typically between iterations 80 and 200 with these data — and then climbs gently: the classic silhouette of the validation curve from 06-05, now with "number of trees" on the x-axis. Early stopping with patience 20 would have cut off about 20 iterations after the minimum, keeping the model from the best iteration. This exercise is exactly what early_stopping=True automates under the hood.

Gradient boosting delivers on boosting's promise: a trivial initial prediction plus a chain of dwarf trees, each trained on the residuals of the ensemble so far, with the learning rate shrinking every step — gradient descent in prediction space. You've seen its two coupled knobs (η and n_estimators), its implementations in order of modernity (classic sklearn, HistGradientBoosting, XGBoost and LightGBM), its duel with Random Forest on MercaFresh's churn and, above all, its character: the most powerful algorithm on tabular data is also the most prone to memorizing, which is why it travels surrounded by brakes — low learning rate, early stopping, small trees, subsampling and the regularization of 07-01 applied to its leaves. With trees and ensembles we've come a long way from simple models; the next lesson explores the other great road to power: stacking layers of neurons until you build deep networks, the deep learning that the MLP of 04-07 left foretold.