The Project | About Us | Contribute | Donations | License
HOME My courses Completed
◄ Previous
◄

Data splitting: training, validation and test

Next ►
►

At the end of the previous module we left a question hanging: "is my model actually any good?". Answering it rigorously is a whole discipline in itself, and it starts with something deceptively simple: deciding which data the model trains on and which data it is evaluated on. In modules 4 and 5 we used train_test_split instrumentally, without justifying it; in this lesson we will understand why splitting the data is essential, what role each set plays (training, validation and test), how to stratify when classes are imbalanced — like MercaFresh's churn — and, above all, how to avoid data leakage, that silent mistake we already hinted at in lessons 03-02 and 03-05, which invalidates entire evaluations without ever raising an error.

Contents

  1. Memorizing is not learning: why you never evaluate on the training data
  2. The train/test split with train_test_split
  3. The validation set: the three-way split
  4. Stratification with imbalanced classes
  5. Data leakage: the silent enemy
  6. Temporal splits: when time matters
  7. The complete map of the three sets

Memorizing is not learning: why you never evaluate on the training data

In lesson 01-01 we defined Machine Learning as the ability to generalize from examples: we don't want a model that remembers MercaFresh's past orders, but one that gets the orders that haven't happened yet right.

The classic analogy is the exam. If a teacher hands students the exact exam questions a week in advance, everyone will score full marks by memorizing the answers. That perfect score doesn't measure whether they have learned the subject; it measures their memory. To know whether they really know the material, you have to test them with questions they haven't seen.

Exactly the same thing happens with models:

  • Evaluating on the training data measures how much the model has memorized. A decision tree with no depth limit (we saw this in 04-03) can reach 100% accuracy on training simply by creating one leaf per customer.
  • Evaluating on data it has never seen measures how much it has generalized, which is the only thing that matters in production.

A minimal example proves it:

from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

# X, y: MercaFresh churn dataset built in module 3
# (RFM features: recency, frequency, average spend, etc.)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

tree = DecisionTreeClassifier(random_state=42)  # no max_depth: grows freely
tree.fit(X_train, y_train)

print("Accuracy on train:", accuracy_score(y_train, tree.predict(X_train)))
print("Accuracy on test: ", accuracy_score(y_test, tree.predict(X_test)))

A typical result would be 1.00 on training and 0.78 on test. The first number is an illusion; the second is the honest estimate of how the model will behave with the next real customer. We will give that gap between the two numbers a name and a cure in lesson 06-05 (overfitting).

The train/test split with train_test_split

The standard tool is train_test_split, which you already know instrumentally. Let's now break down its parameters:

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(
    X, y,
    test_size=0.2,      # fraction reserved for test
    random_state=42,    # seed: makes the split reproducible
    shuffle=True,       # shuffles the rows before splitting (True by default)
)
  • test_size: typical sizes are 20%–30% for test. With large datasets (hundreds of thousands of rows) a smaller fraction is enough, because in absolute terms there are still plenty of test examples.
  • random_state: the split is random; fixing the seed guarantees that you, your colleague and your "future self" all get exactly the same partition. Without it, every run gives slightly different results (we will exploit this in 06-03 to motivate cross-validation).
  • shuffle: shuffles the rows before cutting. It is essential if the CSV comes sorted (by sign-up date, by region...): without shuffling, the test set could contain only recent customers or customers from a single region. The big exception is time series data, which we will cover below: there, shuffling is precisely the mistake.
Scenario Suggested test_size Comment
Small dataset (< 1,000 rows) 0.2–0.3 Every row counts; cross-validation (06-03) will help
Medium dataset (thousands–tens of thousands) 0.2 The de facto standard
Very large dataset (> 100,000) 0.1 or less 10% is already thousands of test examples
Temporal data cut by date Never random; see the section on temporal splits

The validation set: the three-way split

Train and test seem enough... until we start making decisions while looking at the test set. Imagine this workflow at MercaFresh:

  1. You train a tree with max_depth=3 → test accuracy: 0.81.
  2. You try max_depth=5 → 0.84.
  3. You try max_depth=8 → 0.83.
  4. You keep max_depth=5 and report "my model has 84% accuracy".

That 84% is no longer an honest estimate: you used the test set to choose the hyperparameter, so the test set has (indirectly) taken part in building the model. It is like letting a student retake the exam ten times and keeping the best grade: the test set has been "spent".

The solution is to split into three sets:

  • Training (train): fits the model's internal parameters (fit).
  • Validation: compares alternatives — hyperparameters, algorithms, feature sets — and picks the best one. It is consulted many times.
  • Test: touched exactly once, at the end, to estimate the real performance of the model you have already chosen.

train_test_split has no three-way mode, but chaining two calls is all it takes:

# First split: set aside the test set (20%) and never touch it again
X_temp, X_test, y_temp, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

# Second split: from the remaining 80%, take 25% for validation
# (0.25 x 0.8 = 0.2 → final 60/20/20 breakdown)
X_train, X_val, y_train, y_val = train_test_split(
    X_temp, y_temp, test_size=0.25, random_state=42, stratify=y_temp
)

print(len(X_train), len(X_val), len(X_test))  # e.g. 6000, 2000, 2000

Common breakdowns are 60/20/20 or 70/15/15. Two previews: cross-validation (06-03) lets you do without a fixed validation set by reusing the training data intelligently, and the tools that automate hyperparameter search over validation (GridSearchCV and friends) are the topic of 07-05.

Stratification with imbalanced classes

MercaFresh's churn hovers around 20%: out of every 100 customers, about 20 cancel. With a purely random split, the test set could end up with 14% or 26% churn by sheer luck, and any metrics computed on it would no longer be comparable with reality.

The stratify=y parameter forces each set to preserve the class proportions of the original dataset:

import numpy as np

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

print("Overall churn :", np.mean(y))        # e.g. 0.200
print("Churn in train:", np.mean(y_train))  # ≈ 0.200
print("Churn in test :", np.mean(y_test))   # ≈ 0.200

Practical rules:

  • With classification, use stratify=y almost always; it never hurts and it is critical with imbalance.
  • The smaller the dataset or the rarer the minority class, the more important stratification becomes.
  • Regression has no direct stratify (there are no classes), although you can stratify by bins of the target variable if needed.

Data leakage: the silent enemy

Data leakage happens when information that would not be available at prediction time sneaks into training or evaluation. The model looks excellent on test and fails in production. In 03-02 and 03-05 we anticipated it with a rule: fit on train, transform on test. Now we look at it in depth through three subtle leaks.

Leak 1: scaling (or imputing) before splitting

# ❌ WRONG: the scaler "sees" the test set
from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)            # mean and std of the ENTIRE dataset
X_train, X_test, ... = train_test_split(X_scaled, y, ...)

# ✅ CORRECT: split first, then fit using train only
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2,
                                                    random_state=42, stratify=y)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)  # learns from train ONLY
X_test_scaled = scaler.transform(X_test)        # applies what it learned

In the wrong version, the mean and standard deviation used for scaling incorporate information from the test rows. The leak is small with StandardScaler, but with mean imputation, correlation-based feature selection or target-based encodings it can be huge. The robust way to armor yourself is the Pipeline from module 3, which applies fit-on-train/transform-on-test automatically; in 06-03 we will see that inside cross-validation the Pipeline becomes outright mandatory.

Leak 2: duplicates spread across train and test

If the MercaFresh dataset has duplicate rows (the same customer exported twice, or the same customer with two accounts), shuffling can leave one copy in train and another in test. The model "gets it right" on test because it already saw that exact row during training: memory disguised as generalization.

# Before splitting: detect and remove duplicates
print("Exact duplicates:", df.duplicated().sum())
df = df.drop_duplicates()

# A subtler variant: several rows for the SAME customer (monthly snapshots).
# The fix is to split BY CUSTOMER, not by row:
from sklearn.model_selection import GroupShuffleSplit

gss = GroupShuffleSplit(n_splits=1, test_size=0.2, random_state=42)
train_idx, test_idx = next(gss.split(df, groups=df["customer_id"]))

The general rule: if several rows share an entity (customer, store, session), that entity must fall entirely into train or entirely into test.

Leak 3: information from the future

This is the most treacherous one. Suppose that to predict March churn you build the feature "average spend over the last quarter"... computed with April data. Or that a column like churn_reason only gets filled in after the customer cancels: that is a perfect leak, because it predicts churn using a consequence of churn. Warning signs:

  • A feature with suspiciously high correlation with the target (99% accuracy is grounds for suspicion, not celebration!).
  • Columns that, in the real system, are filled in after the event you want to predict.
  • Aggregates (means, counters) computed over windows that include the future.

The control question is always the same: "would I have this piece of data available at the exact moment of making the prediction?". If the answer is no, out it goes.

Temporal splits: when time matters

For MercaFresh's demand forecasting (how many units of fruit will we sell next week?) the data has a temporal order, and the model will always be used that way: trained on the past, predicting the future. The evaluation must mimic that use:

# ❌ WRONG for time series: mixes past and future
train_test_split(X, y, test_size=0.2, shuffle=True)

# ✅ CORRECT: chronological cut
df = df.sort_values("date")
cutoff = "2025-10-01"
train = df[df["date"] < cutoff]   # e.g. January 2023 – September 2025
test  = df[df["date"] >= cutoff]  # October – December 2025

If we shuffle, the model trains on December sales to "predict" the previous June: in practice it is seeing the future, and the resulting metric is pure fantasy (it is leak 3 at split scale). With three sets, the order is preserved: the oldest data for train, the middle stretch for validation and the most recent for test. There is also a cross-validation variant designed specifically for time series (TimeSeriesSplit) that we will look at briefly in 06-03.

The complete map of the three sets

flowchart TD
    D["Full MercaFresh dataset<br/>(after cleaning, no duplicates)"] -->|"train_test_split<br/>stratify=y"| T["Training (60%)"]
    D -->|"train_test_split<br/>stratify=y"| V["Validation (20%)"]
    D -->|"set aside from the start"| P["Test (20%)"]

    T -->|"fit()"| M["Candidate model(s)"]
    V -->|"compare and choose<br/>hyperparameters / algorithm"| M
    M -->|"final chosen model"| F["Final evaluation"]
    P -->|"exactly once"| F
    F --> R["Honest estimate of<br/>production performance"]

Three roles, three rules:

Set Who uses it How many times Question it answers
Training The model's and the transformers' fit() Many What patterns are in the data?
Validation The human (or the hyperparameter search) Many Which alternative do I pick?
Test The final evaluation One How will it perform in production?

Common Mistakes and Tips

  • Evaluating on train and reporting that number. The number one beginner mistake. Training accuracy is only useful for diagnosing overfitting (06-05), never as a quality metric.
  • "Spending" the test set. Every time you look at the test result and change something in the model, the test set loses value. Decide with validation; save the test set for the final verdict.
  • Forgetting random_state. Without a fixed seed, you won't be able to reproduce your results or compare experiments fairly.
  • Not stratifying with imbalanced classes. With 20% churn and a small dataset, the proportion in test can drift a lot by chance.
  • Scaling/imputing before splitting. The classic leak. Use a Pipeline and you won't have to remember.
  • Shuffling temporal data. If the model will predict the future, evaluate it predicting the future.
  • Tip: do the split as early as possible in your workflow, right after basic cleaning, and treat X_test/y_test as if they were locked in a safe.

Exercises

Exercise 1

A colleague at MercaFresh shows you this code and boasts about 97% accuracy. Identify two methodological problems.

scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)
X_train, X_test, y_train, y_test = train_test_split(X_scaled, y, test_size=0.2)
model.fit(X_train, y_train)
print(accuracy_score(y_train, model.predict(X_train)))

Exercise 2

Create a stratified 70/15/15 split (train/validation/test) for a churn dataset X, y, with random_state=7, and verify that the churn proportion is similar in all three sets. Hint: the second call to train_test_split must divide the remaining 30% into equal halves.

Exercise 3

MercaFresh wants to predict the daily demand for each product using data from 2023–2025. Propose (no code) how you would split the data into train/validation/test, and justify why shuffle=True would be a mistake here.

Solutions

Solution 1. (a) Data leakage: the StandardScaler is fitted on the entire dataset before splitting, so the mean and standard deviation incorporate information from the test set; the correct approach is to split first, then fit_transform on train only and transform on test. (b) Evaluating on training data: the reported accuracy is computed with y_train and predictions on X_train — it measures memorization, not generalization; it should be computed on the test set (and on top of that, random_state is missing for reproducibility, and stratify=y if there is imbalance, which would be additional improvements).

Solution 2.

# Step 1: set aside the 30% that will later be split between val and test
X_train, X_rest, y_train, y_rest = train_test_split(
    X, y, test_size=0.30, random_state=7, stratify=y
)
# Step 2: split that 30% in half → 15% and 15%
X_val, X_test, y_val, y_test = train_test_split(
    X_rest, y_rest, test_size=0.50, random_state=7, stratify=y_rest
)

import numpy as np
for name, vec in [("train", y_train), ("val", y_val), ("test", y_test)]:
    print(f"{name}: {len(vec)} rows, churn = {np.mean(vec):.3f}")

Thanks to stratify, the three churn percentages should be practically identical to the overall one (~0.20).

Solution 3. A chronological split: for example, train on 2023 and 2024, validation on the first half of 2025 and test on the second half of 2025 (the exact cutoffs depend on the data volume, but the order past → validation → test is non-negotiable). shuffle=True would be a mistake because it would mix rows from future dates into training: the model would learn, for instance, from the 2025 Christmas campaign sales to "predict" earlier days — a temporal information leak that inflates the metrics and does not reflect real usage (training on the past to predict the future).

Conclusion

In this lesson we have turned the mechanical train_test_split of earlier modules into a methodology: evaluate on unseen data because generalizing is not memorizing; add a validation set to choose hyperparameters without contaminating the test set; stratify when classes are imbalanced, like MercaFresh's churn; armor yourself against data leakage (scaling before splitting, spread-out duplicates, information from the future); and respect chronological order when the data is temporal. We now know which data to evaluate on; the next question is which number to use: the accuracy we have been relying on has a serious trap with imbalanced classes, and in the next lesson we will roll out the full arsenal of metrics — confusion matrix, precision, recall, F1, MAE, RMSE, R² — to measure exactly what the business cares about.

◄ Previous
◄
Next ►
►

Machine Learning Course

Module 1: Introduction to Machine Learning

Module 2: Foundations of Statistics and Probability

Module 3: Data Preprocessing

Module 4: Supervised Machine Learning Algorithms

Module 5: Unsupervised Machine Learning Algorithms

Module 6: Model Evaluation and Validation

Module 7: Advanced Techniques and Optimization

Module 8: Model Implementation and Deployment

Module 9: Hands-On Projects

Module 10: Additional Resources

The Project | About Us | Contribute | Donations | License
© Copyright 2026. All rights reserved