The Project | About Us | Contribute | Donations | License

HOME

The previous lesson ended on a worry: all our metrics were computed on a single train/test split, and that split depends on the luck of a random_state. What if, with a different seed, the F1 had been noticeably different? In this lesson we will first show that this variability is real, and then solve it with cross-validation (CV): instead of evaluating once, evaluate K times while rotating the data, and report a mean and a standard deviation. It is the standard technique for comparing models honestly — we will use it to finally pit the module 4 algorithms against each other on MercaFresh's churn — and the foundation on which the hyperparameter search of 07-05 is built.

The problem: one split, one number, lots of variance
K-fold step by step
cross_val_score and cross_validate in practice
StratifiedKFold for churn
Interpreting mean ± deviation: is the difference significant?
An honest tournament: the module 4 models head to head
Pipeline inside the CV: leak-free evaluation
Variants and computational cost

The problem: one split, one number, lots of variance

Let's verify it empirically: same model, same data, different split seed.

from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import f1_score
import numpy as np

results = []
for seed in range(20):                         # 20 different splits
    X_tr, X_te, y_tr, y_te = train_test_split(
        X, y, test_size=0.2, random_state=seed, stratify=y
    )
    model = LogisticRegression(max_iter=1000).fit(X_tr, y_tr)
    results.append(f1_score(y_te, model.predict(X_te)))

results = np.array(results)
print(f"Min F1  : {results.min():.3f}")
print(f"Max F1  : {results.max():.3f}")
print(f"Mean F1 : {results.mean():.3f} ± {results.std():.3f}")

A typical output with a medium-sized dataset: minimum 0.58, maximum 0.69, mean 0.63 ± 0.03. Eleven percentage points between the worst and the best seed, without changing absolutely anything about the model. Two consequences:

A single number ("my F1 is 0.66") hides a lottery: maybe you drew a lucky seed.
Comparing two models on different splits (or even on the same one, if the difference is small) can lead to wrong conclusions: the observed difference may be pure partition luck.

The loop above already points at the solution: evaluate several times and summarize with mean and deviation. Cross-validation does exactly this, but more efficiently: without wasting data, and guaranteeing that every row is used exactly once as test.

K-fold step by step

K-fold cross-validation works like this:

The dataset (the training data; the final test set from 06-01 stays locked in its safe) is split into K blocks ("folds") of equal size. Common values: K = 5 or K = 10.
K rounds are run. In round i, fold i acts as the evaluation set and the remaining K−1 as training data.
K metrics are obtained, one per round, and their mean and standard deviation are reported.

flowchart TD
    D["Training data<br/>split into 5 folds: F1 F2 F3 F4 F5"] --> R1
    subgraph Rounds["5 rounds of training and evaluation"]
        R1["Round 1: train F2-F5, evaluate F1 → metric m1"]
        R2["Round 2: train F1,F3-F5, evaluate F2 → metric m2"]
        R3["Round 3: train F1,F2,F4,F5, evaluate F3 → metric m3"]
        R4["Round 4: train F1-F3,F5, evaluate F4 → metric m4"]
        R5["Round 5: train F1-F4, evaluate F5 → metric m5"]
    end
    R1 --> M["Result: mean(m1..m5) ± deviation(m1..m5)"]
    R2 --> M
    R3 --> M
    R4 --> M
    R5 --> M

Advantages over the single split:

Every row is evaluated exactly once (and trained on K−1 times): no data is "wasted" as permanent test.
The mean of K measurements is far more stable than a lone measurement.
The standard deviation quantifies the uncertainty: we know how much to trust the number.

Important: CV produces K disposable models whose only purpose is estimating performance. The final model that would go to production is trained afterwards on all the training data.

`cross_val_score` and `cross_validate` in practice

scikit-learn automates the whole process:

from sklearn.model_selection import cross_val_score
from sklearn.linear_model import LogisticRegression

model = LogisticRegression(max_iter=1000)

# 5-fold CV measuring F1; cv=5 uses StratifiedKFold automatically for classification
scores = cross_val_score(model, X_train, y_train, cv=5, scoring="f1")
print("F1 per fold:", np.round(scores, 3))       # e.g. [0.61 0.65 0.63 0.60 0.66]
print(f"F1: {scores.mean():.3f} ± {scores.std():.3f}")

Key details:

scoring accepts the metrics from 06-02 by name: "accuracy", "precision", "recall", "f1", "roc_auc" (we'll see it in 06-04), "neg_mean_absolute_error", "r2"... The error metrics come negated (neg_) because scikit-learn always maximizes: an MAE of 18 shows up as −18.
cross_val_score receives the model untrained: it takes care of cloning, training and evaluating on each fold.

When you want several metrics at once (and timings), use cross_validate:

from sklearn.model_selection import cross_validate

res = cross_validate(
    model, X_train, y_train, cv=5,
    scoring=["f1", "recall", "precision", "accuracy"],
    return_train_score=True,        # useful for diagnosing overfitting (06-05)
)
print(f"Val F1   : {res['test_f1'].mean():.3f} ± {res['test_f1'].std():.3f}")
print(f"Recall   : {res['test_recall'].mean():.3f}")
print(f"Train F1 : {res['train_f1'].mean():.3f}")   # train-val gap → 06-05
print(f"Time/fold: {res['fit_time'].mean():.2f} s")

`StratifiedKFold` for churn

With churn at 20%, the folds must preserve that proportion, for the same reason we stratified in 06-01: a fold that by chance had 12% churn would yield a metric that isn't comparable. StratifiedKFold stratifies each fold:

from sklearn.model_selection import StratifiedKFold

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(model, X_train, y_train, cv=cv, scoring="f1")

Two notes:

For classification, passing cv=5 (an integer) already uses StratifiedKFold under the hood; building the explicit object lets you set shuffle=True and random_state (recommended: it shuffles before cutting the folds and makes the experiment reproducible).
For regression, cv=5 uses a non-stratified KFold, which is the appropriate choice.

Interpreting mean ± deviation: is the difference significant?

Suppose two churn models with 5-fold CV:

Logistic regression: F1 = 0.63 ± 0.02
Decision tree (max_depth=5): F1 = 0.61 ± 0.04

Is the logistic model "better"? Here lesson 02-04 resurfaces: a sample mean carries uncertainty, and before declaring a winner you must ask whether the difference is significant or compatible with chance. The practical intuition, in the spirit of confidence intervals:

If the mean ± deviation ranges overlap widely (as here: 0.61–0.65 versus 0.57–0.65), the evidence that one model is better is weak.
If one model sits clearly above even after accounting for the deviation (e.g., 0.70 ± 0.02 versus 0.61 ± 0.03), the difference is hard to attribute to chance.
For a finer analysis you can compare the results fold by fold (both models evaluated on the same folds form pairs — the same idea as the paired t-test of 02-04), although with only 5 values a formal test has little power; in professional practice the overlap rule, used prudently, settles most decisions.

The moral: always report mean ± deviation, and be suspicious of rankings decided by differences in the third decimal place.

An honest tournament: the module 4 models head to head

In module 4 we presented each algorithm separately, with instrumental splits. Now we can compare them rigorously: same CV, same folds, same metric.

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.svm import SVC
from sklearn.neighbors import KNeighborsClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn.neural_network import MLPClassifier
from sklearn.model_selection import StratifiedKFold, cross_validate

models = {
    "Logistic regression": LogisticRegression(max_iter=1000),
    "Tree (max_depth=5)":  DecisionTreeClassifier(max_depth=5, random_state=42),
    "SVM (RBF)":           SVC(),
    "K-NN (k=15)":         KNeighborsClassifier(n_neighbors=15),
    "Naive Bayes":         GaussianNB(),
    "MLP":                 MLPClassifier(hidden_layer_sizes=(32,), max_iter=1000,
                                         random_state=42),
}

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)  # same folds for everyone

for name, clf in models.items():
    pipe = Pipeline([("scaler", StandardScaler()), ("clf", clf)])  # see next section
    res = cross_validate(pipe, X_train, y_train, cv=cv, scoring=["f1", "recall"])
    print(f"{name:22s} F1 = {res['test_f1'].mean():.3f} ± {res['test_f1'].std():.3f}"
          f"   Recall = {res['test_recall'].mean():.3f}")

A typical result on MercaFresh's churn:

Model	F1 (mean ± std)	Recall	Comment
Logistic regression	0.63 ± 0.02	0.60	Solid, stable, interpretable
SVM (RBF)	0.63 ± 0.03	0.58	Ties, at a higher compute cost
MLP	0.62 ± 0.04	0.59	Similar and more variable
Tree (max_depth=5)	0.61 ± 0.04	0.62	Competitive; sensitive to the partition
K-NN (k=15)	0.59 ± 0.03	0.55	Somewhat behind
Naive Bayes	0.56 ± 0.03	0.66	Better recall, worse precision

(The exact numbers will depend on your data; what transfers is the method.) The reading: given the overlapping deviations, the logistic regression, the SVM and the MLP are effectively tied — and then other criteria decide: interpretability, speed, simplicity (the logistic model wins all three at MercaFresh). Also note that each model uses its "reasonable" hyperparameters here, untuned; the systematic search that could reshuffle this ranking builds on this very CV and is the topic of 07-05 (GridSearchCV).

Pipeline inside the CV: leak-free evaluation

Pay attention to a detail in the code above: the StandardScaler goes inside the Pipeline that is passed to cross_validate. That is not aesthetics — it is the anti-leak rule of 06-01 applied to CV:

# ❌ WRONG: scaling before the CV
X_scaled = StandardScaler().fit_transform(X_train)   # sees ALL folds at once
cross_val_score(LogisticRegression(max_iter=1000), X_scaled, y_train, cv=5)

# ✅ CORRECT: the Pipeline refits the scaler in every round,
# using only that round's training folds
pipe = Pipeline([("scaler", StandardScaler()),
                 ("clf", LogisticRegression(max_iter=1000))])
cross_val_score(pipe, X_train, y_train, cv=5)

In the wrong version, the scaler is fitted with data that in each round will act as "test", contaminating all K evaluations. With imputation, feature selection or target-dependent encodings, the leak can seriously inflate the metrics. Golden rule: every step that learns from the data goes inside the Pipeline, and the entire Pipeline goes inside the CV. This is the payoff of module 3's work: our preprocessing was already packaged in Pipelines/ColumnTransformers, so it slots in here unchanged.

Variants and computational cost

Two variants worth knowing:

LeaveOneOut (LOO): the extreme case K = n (each row is a fold). It squeezes the most out of very small datasets, but requires training n models and its estimate has high variance. It rarely pays off compared to a 5-fold or repeated 10-fold.
TimeSeriesSplit: for temporal data like MercaFresh's demand, where classic K-fold is invalid (it would train on the future to evaluate the past — the temporal leak of 06-01). It generates growing cuts that always train on the past and evaluate on the next block: train [1], evaluate [2]; train [1,2], evaluate [3]; train [1,2,3], evaluate [4]...

from sklearn.model_selection import TimeSeriesSplit
cv_temporal = TimeSeriesSplit(n_splits=5)
scores = cross_val_score(demand_model, X_demand, y_demand,
                         cv=cv_temporal, scoring="neg_mean_absolute_error")

On cost: CV multiplies training time by K (and by the number of models compared, and later by each hyperparameter combination in 07-05). A practical guide:

Situation	Reasonable strategy
Small/medium dataset, fast models	5-fold or 10-fold without hesitation
Expensive models (large SVMs/MLPs, lots of data)	5-fold, or 3-fold during initial exploration
Huge dataset (millions of rows)	A single well-made split is already stable
Time series	TimeSeriesSplit
Tiny dataset (< 200 rows)	10-fold or LOO, accepting the variance

cross_validate(..., n_jobs=-1) parallelizes the folds across CPU cores — an almost free speedup.

Common Mistakes and Tips

Running CV on the whole dataset, test included. CV replaces the validation set, not the test set: it is applied to the training data, and the test set from 06-01 stays reserved for the final verdict.
Preprocessing before the CV. The most frequent silent leak in real notebooks. Pipeline inside the CV, always.
Comparing models on different folds. Use the same StratifiedKFold object (same seed) for all contenders; otherwise, part of the difference will be partition noise.
Reporting only the mean. Without the deviation you cannot judge whether a difference between models matters.
Using shuffled K-fold on time series. That is evaluating by predicting the past with the future; use TimeSeriesSplit.
Forgetting to retrain at the end. After choosing a model with CV, the definitive model is trained on all the training data before the final test evaluation (and before production).
Tip: fix an "evaluation protocol" at the start of the project (CV, metric, seed) and don't change it midway; changing the rules of the game once you have seen results is an invitation to self-deception.

Exercises

Exercise 1

Explain why this code overestimates performance and fix it:

from sklearn.impute import SimpleImputer
X_imp = SimpleImputer(strategy="mean").fit_transform(X_train)
X_scaled = StandardScaler().fit_transform(X_imp)
scores = cross_val_score(LogisticRegression(max_iter=1000), X_scaled, y_train,
                         cv=5, scoring="f1")

Exercise 2

With 5-fold CV you get: model A, recall = 0.64 ± 0.05; model B, recall = 0.66 ± 0.06. Your boss at MercaFresh wants to announce that "B is better". What would you tell them, and what would you do to increase confidence in the comparison?

Exercise 3

Write the code that compares DecisionTreeClassifier with max_depth 3, 5 and 10 on churn using the same StratifiedKFold(5, shuffle=True, random_state=42) and scoring="f1", printing each one's mean ± deviation. (You are doing by hand, in miniature, what GridSearchCV will automate in 07-05.)

Solutions

Solution 1. The imputer and the scaler are fitted on the whole of X_train before the CV, so in every round the evaluation fold has already influenced the imputation mean and the scaling parameters: data leakage that inflates the metrics. The fix — put both steps in a Pipeline that gets refitted inside each round:

from sklearn.pipeline import Pipeline
pipe = Pipeline([
    ("imputer", SimpleImputer(strategy="mean")),
    ("scaler", StandardScaler()),
    ("clf", LogisticRegression(max_iter=1000)),
])
scores = cross_val_score(pipe, X_train, y_train, cv=5, scoring="f1")

Solution 2. The difference (0.02) is much smaller than the deviations (0.05–0.06): the ranges 0.59–0.69 and 0.60–0.72 overlap almost completely, so B's edge is perfectly compatible with partition luck — as we saw in 02-04, a difference is not a conclusion until you have ruled out noise. To gain confidence: (a) evaluate both on the same folds and compare fold by fold (a paired comparison); (b) repeat the CV with several seeds (e.g., RepeatedStratifiedKFold) to collect more measurements; (c) if they are still tied after that, choose on other criteria (cost, interpretability, guaranteed minimum recall).

Solution 3.

from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import StratifiedKFold, cross_val_score

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

for depth in [3, 5, 10]:
    tree = DecisionTreeClassifier(max_depth=depth, random_state=42)
    scores = cross_val_score(tree, X_train, y_train, cv=cv, scoring="f1")
    print(f"max_depth={depth:2d} → F1 = {scores.mean():.3f} ± {scores.std():.3f}")

Because the same cv object is used, the three trees are evaluated on identical folds and the comparison is fair. A typical pattern: 3 falls short, 10 overfits and 5 wins — the reason behind that "inverted U" shape is what validation curves will show us in 06-05, and the automation of this search comes in 07-05.

Conclusion

Cross-validation fixes the fragility of the single split: K-fold rotates the data so every row is evaluated once, cross_val_score/cross_validate automate it, StratifiedKFold preserves the churn proportion in each fold, and the result is read as mean ± deviation, applying the statistical skepticism of 02-04 before declaring winners. With it we have been able to hold the first fair tournament among the module 4 models, with the Pipeline inside the CV as the anti-leak guarantee, and we have noted the variants (LeaveOneOut, TimeSeriesSplit) and the computational cost. But throughout this tournament the classifiers decided with their default 0.5 threshold, and we have known since 04-02 that this threshold is a business lever. The next lesson exploits it to the full: we will sweep every possible threshold at once with the ROC curve and summarize the result in a single number, the AUC.

Cross-validation

Contents

The problem: one split, one number, lots of variance

K-fold step by step

`cross_val_score` and `cross_validate` in practice

`StratifiedKFold` for churn

Interpreting mean ± deviation: is the difference significant?

An honest tournament: the module 4 models head to head

Pipeline inside the CV: leak-free evaluation

Variants and computational cost

Common Mistakes and Tips

Exercises

Exercise 1

Exercise 2

Exercise 3

Solutions

Conclusion

Machine Learning Course

Module 1: Introduction to Machine Learning

Module 2: Foundations of Statistics and Probability

Module 3: Data Preprocessing

Module 4: Supervised Machine Learning Algorithms

Module 5: Unsupervised Machine Learning Algorithms

Module 6: Model Evaluation and Validation

Module 7: Advanced Techniques and Optimization

Module 8: Model Implementation and Deployment

Module 9: Hands-On Projects

Module 10: Additional Resources

Cross-validation

Contents

The problem: one split, one number, lots of variance

K-fold step by step

cross_val_score and cross_validate in practice

StratifiedKFold for churn

Interpreting mean ± deviation: is the difference significant?

An honest tournament: the module 4 models head to head

Pipeline inside the CV: leak-free evaluation

Variants and computational cost

Common Mistakes and Tips

Exercises

Exercise 1

Exercise 2

Exercise 3

Solutions

Conclusion

Machine Learning Course

Module 1: Introduction to Machine Learning

Module 2: Foundations of Statistics and Probability

Module 3: Data Preprocessing

Module 4: Supervised Machine Learning Algorithms

Module 5: Unsupervised Machine Learning Algorithms

Module 6: Model Evaluation and Validation

Module 7: Advanced Techniques and Optimization

Module 8: Model Implementation and Deployment

Module 9: Hands-On Projects

Module 10: Additional Resources

`cross_val_score` and `cross_validate` in practice

`StratifiedKFold` for churn