In the previous lesson we improved a model by restraining it with regularization. This lesson takes the opposite route: improving by combining many models so that their individual errors cancel each other out. It is one of the most profitable ideas in all of machine learning: ensemble methods dominate competitions on tabular data and are the de facto standard in production for problems like MercaFresh's churn. Here you'll understand why combining works, meet the three big families — bagging, boosting and stacking —, study Random Forest in detail and pit it against the lone decision tree of 04-03 on our churn dataset, with the same cross-validation from module 6.

1. Why combining models works
2. Bagging: bootstrap + aggregation
3. Random Forest in detail
4. Random Forest on MercaFresh's churn
5. Voting classifiers: hard and soft
6. Stacking: a model that learns to combine
7. Boosting: the sequential family (a preview)
8. Comparison table of the three families
9. The price: interpretability

Imagine you ask 1,000 people how many oranges fit on a MercaFresh pallet. Each individual estimate will be poor, but the average of all of them is usually surprisingly good: those who overshoot compensate for those who undershoot. It's the "wisdom of crowds", and it works under two conditions:

1. Each estimator is better than chance (even if only slightly).
2. The errors are independent (or at least weakly correlated): they don't all miss in the same direction.

With models it's exactly the same. If you have 100 classifiers that are right 70% of the time and their errors were independent, a majority vote would be right far more than 70% of the time: for the majority to fail, more than 50 of them would have to be wrong at once, which is unlikely if each one fails on different customers.

The connection with 06-05 is direct: averaging models reduces variance. A deep decision tree has low bias but sky-high variance (it changes completely with a different training sample); if we average many different trees, each one's random component cancels out and the shared signal remains. The critical condition — and the hardest one — is decorrelation: a hundred copies of the same model add nothing, just like asking the same person a thousand times. All the ingenuity of ensemble methods lies in manufacturing diversity:

Bagging = Bootstrap AGGregatING. The recipe:

1. Bootstrap: generate B samples of the training set, each the same size as the original but sampled with replacement (a customer can appear several times; others, not at all). We already used this idea in statistical inference (02-04); here it serves to manufacture B "alternative versions" of the dataset.
2. Training: train a model (typically a deep, unpruned tree) on each sample.
3. Aggregation: to predict, classification votes; regression averages.

Each bootstrap sample leaves out, on average, 37% of the observations (the probability that a given example never comes up in n draws with replacement is $(1-1/n)^n \approx e^{-1} \approx 0.37$). That excluded 37% will play a starring role shortly: out-of-bag evaluation.

Random Forest is bagging of trees plus one extra trick that multiplies diversity: feature bagging. At every split of every tree, instead of evaluating all the features (as the tree in 04-03 did), only a random subset is considered (by default, $\sqrt{p}$ features in classification, controlled by max_features).

Why does it matter so much? Without it, if one feature is very dominant — in our churn problem, recency is —, every tree would pick it for its first split and they'd all end up looking too much alike: correlated errors, little gain from averaging. By randomly vetoing features at each node, the trees are forced to explore different paths and their errors decorrelate.

Three additional gifts from Random Forest:

• OOB score (out-of-bag): each tree is evaluated on the 37% of observations it never saw in its bootstrap. Aggregating those predictions yields an estimate of performance on unseen data for free, with no separate validation split. It's a complement to (not a substitute for) the cross-validation of 06-03.
• Feature importance: how much each feature reduces impurity, averaged over all trees and all splits. An instant relevance ranking (with caveats we'll cover in Common Mistakes).
• Robustness with barely any tuning: more trees (n_estimators) never causes overfitting by itself — it only stabilizes the average; the cost is compute time. The knobs that do control each tree's complexity are the ones you already know from 04-03 (max_depth, min_samples_leaf).

Let's rerun module 6's flagship experiment: predicting churn with the RFM features we built in module 3, comparing the lone tree from 04-03 against the forest, with the same StratifiedKFold from 06-03.

Typical observations from the result:

• The forest improves the mean F1 of the tree and, above all, reduces the spread between folds: less variance, exactly as the theory promised. We don't need to fine-tune max_depth with validation curves as in 06-05: the forest's trees grow deep and the averaging absorbs their individual overfitting.
• oob_score_ yields a figure consistent with the CV without spending a single fold.
• feature_importances_ points to recency, frequency and incidents as the dominant variables — consistent with what MercaFresh's retention team already suspected and with how we generated the data.

Practical note: trees don't need scaling (04-03), so here the pipeline can skip the StandardScaler; if the ensemble included scale-sensitive models (SVM, K-NN, logistic regression), the Pipeline from module 3 becomes mandatory again.

Bagging manufactures diversity by resampling data; another route is to combine algorithms of a different nature. In 06-03 we ran a tournament among the classifiers of module 4; the VotingClassifier turns them from rivals into a team:

• Hard voting: each model casts its class and the majority wins.
• Soft voting: the probabilities (predict_proba) are averaged and the class with the highest mean wins. It's usually better because it incorporates each model's confidence, but it requires all the members to produce reasonably calibrated probabilities.

Each scale-sensitive member carries its own pipeline with scaling: the anti-leakage discipline of 06-01 holds inside the ensemble. Voting shines when the members perform similarly but fail on different customers; if one model is clearly worse than the rest, it can drag the vote down (better to remove it or weight it with weights).

Voting combines with a fixed rule (majority or mean). Stacking goes one step further: it trains a meta-model that learns from the data how to combine the base models' predictions. For instance, it might learn that "when K-NN and the logistic model disagree, the logistic model tends to be right on long-standing customers".

The fine print is that the meta-model must be trained on out-of-fold predictions (obtained via internal cross-validation), not on predictions over the same data the base models were trained on — otherwise it would inherit their overfitting, a leak of the kind we learned to fear in 06-01. StackingClassifier handles this automatically:

Stacking is the endgame weapon of competitions (often with several layers of meta-models), but in real projects its gain over a good voting ensemble or a good forest is usually small and its maintenance cost, high. Use it when every tenth of a metric point is worth money.

The techniques above train models in parallel and independently. Boosting trains them in sequence: each new model concentrates on the examples (or errors) where the previous ones failed. Instead of reducing variance by averaging strong models, boosting reduces bias by summing many deliberately weak models (tiny trees) that correct one another.

That inversion of philosophy has important consequences (more power, more overfitting risk, a sequence that can't be parallelized) and has produced the most dominant algorithms on tabular data: Gradient Boosting, XGBoost, LightGBM. It's the subject of the next lesson (07-03); here it's enough to place it on the map.

The single tree of 04-03 had a virtue no ensemble keeps: it could be drawn and explained — "if recency > 30 days and frequency < 3 orders, high risk". A forest of 300 trees is a far more opaque box: nobody can follow 300 votes at once.

Palliatives exist — feature_importances_ gives a global ranking, and in 07-03 we'll mention SHAP for per-customer explanations —, but the underlying decision is a business one: if MercaFresh's retention team needs to justify to management why each customer gets called, an interpretable tree with an F1 of 0.78 may be worth more than an opaque forest with 0.82. If all that matters is being right, the ensemble wins. This performance-interpretability trade-off will resurface in the ethical considerations of 08-04.

• Combining nearly identical models. Three logistic regressions with different seeds are no crowd: without diversity there is no error cancellation. Mix algorithm families or use resampling/feature bagging.
• Treating n_estimators as a complexity knob. In Random Forest, more trees don't overfit: they only stabilize (and cost more). The complexity brakes are each tree's (max_depth, min_samples_leaf). In boosting, watch out: there n_estimators does increase the overfitting risk, as we'll see in 07-03.
• Trusting feature_importances_ blindly. Impurity-based importance favors continuous or high-cardinality features and is split arbitrarily among correlated features. Cross-check with permutation_importance on validation data before drawing business conclusions.
• Using soft voting with models that lack reliable probabilities. An SVM without probability=True doesn't provide predict_proba, and some models produce poorly calibrated probabilities that distort the average. Vet the members before voting.
• Evaluating the stacking with the same CV that generates its meta-features, carelessly. Let StackingClassifier manage its internal CV and evaluate the whole ensemble with an external CV (as we did here): it's the clean way to avoid leaks.
• Tip: always start with RandomForestClassifier on default parameters as a "strong baseline". It's hard to break and quickly tells you how much signal is in the data; then decide whether anything fancier is worth it.

1. The value of decorrelation. Simulate 500 binary predictions from a "committee" of 25 classifiers that are individually right 65% of the time (use rng.random((500, 25)) < 0.65 as a matrix of independent hits) and compute the majority vote's accuracy. Repeat making 20 of the 25 exact copies of the same classifier. Compare both results.
2. The forest's stabilization curve. On MercaFresh's churn, evaluate RandomForestClassifier with n_estimators in [1, 5, 10, 25, 50, 100, 200, 400] using the lesson's stratified CV. Plot mean F1 and standard deviation against the number of trees. Beyond how many trees does adding more stop paying off? Does this curve look like an overfitting curve like those in 06-05?
3. Ensemble tournament. Compare with the same CV: (a) the best individual model from module 4 you obtained in 06-03, (b) a soft VotingClassifier with three heterogeneous models, (c) RandomForestClassifier(n_estimators=300). Present the mean F1 ± deviation table and argue which one MercaFresh should deploy, taking interpretability into account as well.

1. With 25 independent classifiers at 65%, the majority (≥13 hits out of 25) is right around 93-94% of the time: the crowd pulverizes the individual. With 20 identical copies, the "vote" is decided in practice by that one repeated classifier and the accuracy drops back to ~65%. The numerical moral of the whole lesson: without decorrelated errors, the ensemble is theater.
2. Mean F1 climbs sharply up to ~50-100 trees and then flattens; the spread between folds shrinks steadily. Past ~200 trees the improvements are noise and you're only paying compute. It is not an overfitting curve: unlike max_depth in 06-05, there is no descending stretch no matter how many trees you add — the curve converges, it doesn't turn around.
3. Typical result: the forest tops the table, the voting ensemble comes close (sometimes tying if its members are good and diverse) and the best individual model trails with a larger spread between folds. The deployment decision isn't automatic: with differences of a few hundredths of F1, the interpretability argument can tip the scales toward the simple model; with clear differences, the forest wins and the feature importances serve as a global explanation for the business.

You've seen why combining models works — decorrelated errors that cancel out, variance that collapses when averaging — and the three ways of manufacturing it: bagging (bootstrap + aggregation, culminating in Random Forest with its feature bagging, its OOB score and its importances), voting and stacking (diversity through algorithmic heterogeneity), and boosting, the sequential family we left previewed. On MercaFresh's churn, the forest beat the lone tree of 04-03 with less variance between folds and barely any tuning, at the cost of sacrificing the drawable tree's interpretability. The most powerful member of the family is still pending: boosting, where each model is born to correct the previous one's errors and which today dominates machine learning on tabular data. It's the subject of the next lesson: Gradient Boosting.