The Project | About Us | Contribute | Donations | License

HOME

The previous lesson ended with several NaN values planted on purpose: we turned impossible ages and future dates into missing values because that was more honest than inventing a number. Now it's time to decide what to do with those gaps, and with all the ones that came in the MercaFresh data from the start. Missing data is unavoidable in any real project, and how you handle it can completely change a model's conclusions: dropping rows can bias the sample, imputing badly can fabricate false patterns, and doing it at the wrong moment can leak information the model should never have. In this lesson you'll learn to diagnose why data is missing, visualize its patterns and choose between dropping and imputing with good judgment.

Why data goes missing: MCAR, MAR and MNAR
Detecting and visualizing nulls
Dropping: rows and columns
Simple imputation: mean, median, mode and constants
scikit-learn's SimpleImputer
Advanced imputation: KNNImputer and missingness indicators
The silent risk: data leakage

Why data goes missing: MCAR, MAR and MNAR

Before filling a gap you have to ask why it's empty. Statistics distinguishes three missingness mechanisms, and the difference is not academic: it determines which strategies are valid.

MCAR: Missing Completely At Random

The absence depends on nothing: it's pure chance. At MercaFresh, an intermittent server failure lost the delivery_time field for 2% of orders chosen at random. No type of order was more likely to lose it than any other.

Consequence: the remaining data is still representative. Dropping those rows doesn't bias anything; it only shrinks the sample.

MAR: Missing At Random (conditional on what we observe)

The absence depends on other columns we do have. At MercaFresh, age is missing far more often among customers who signed up through the mobile app (where the field was optional) than among those from the web form (where it was mandatory). Once you know the signup channel, the absence carries no extra information.

Consequence: dropping rows biases the sample (we'd lose app users, who may have a different churn profile), but you can impute well using the related columns.

MNAR: Missing Not At Random

The absence depends on the missing value itself. A classic example at MercaFresh: the post-delivery rating (satisfaction) is missing mostly among unhappy customers, who don't bother answering the survey. The gap is information: those who don't answer tend to be dissatisfied.

Consequence: this is the most dangerous case. Imputing the mean would fabricate "moderately satisfied" customers who were actually angry — precisely the ones about to churn. Here, missingness indicators (we'll see them below) are almost mandatory.

Mechanism	What does the absence depend on?	MercaFresh example	Reasonable strategy
MCAR	Nothing (pure chance)	Random server failure	Drop or impute: both safe
MAR	Other observed columns	Age missing by signup channel	Impute using those columns
MNAR	The missing value itself	The dissatisfied don't answer	Missingness indicator + caution

In practice there is no definitive test to tell them apart: you reason with business knowledge. The useful question is always: who or what decided not to fill in this field, and does that have anything to do with what I want to predict?

Detecting and visualizing nulls

Let's rebuild an extract of MercaFresh customers, already free of duplicates and impossible values (inherited from 03-01), but with its nulls:

import pandas as pd
import numpy as np

df = pd.DataFrame({
    "customer_id": [101, 104, 105, 106, 107, 108, 109, 110, 111, 112],
    "age": [34, np.nan, 41, 38, np.nan, 45, 31, 27, np.nan, 52],
    "signup_channel": ["web", "app", "web", "app", "app", "web", "app", "web", "app", "web"],
    "total_spend": [1250.5, 0.0, 2100.75, 310.2, 15400.0, 670.4, 0.0, 980.1, 45.9, 1530.0],
    "satisfaction": [4.5, np.nan, 4.8, np.nan, 4.2, 3.9, np.nan, 4.6, np.nan, 4.1],
    "delivery_time_min": [42, 55, np.nan, 61, 38, 47, 52, np.nan, 66, 44],
})

The basic count:

# Nulls per column, in absolute terms and as a percentage
print(df.isna().sum())
print((df.isna().mean() * 100).round(1))
# age                  30.0 %
# satisfaction         40.0 %
# delivery_time_min    20.0 %

isna() returns a DataFrame of booleans (True where there's a null); sum() counts them and mean() gives the proportion directly. But totals don't tell the whole story: the pattern matters. Are values always missing in the same rows? Is satisfaction missing for the same customers as age?

# How many columns are missing per row?
print(df.isna().sum(axis=1).value_counts())

# Do the nulls of two columns coincide?
print(pd.crosstab(df["age"].isna(), df["satisfaction"].isna()))

# A clue about the mechanism: does missing age depend on the channel?
print(df.groupby("signup_channel")["age"].apply(lambda s: s.isna().mean()))
# app    0.6   <- 60% missing in app and 0% in web: smells like MAR!
# web    0.0

That last groupby is the detective tool: it crosses missingness with other columns to intuit the mechanism. For large datasets, a visual map helps:

import matplotlib.pyplot as plt

plt.imshow(df.isna(), aspect="auto", cmap="gray_r")
plt.xticks(range(len(df.columns)), df.columns, rotation=45)
plt.ylabel("Row")
plt.title("Null map (black = missing)")
plt.tight_layout()
plt.show()

Each black cell is a gap: vertical bands point to problem columns; aligned patterns across columns, to correlated absences. (The missingno library automates these plots, but with imshow the tools you already have are enough.)

Dropping: rows and columns

The simplest option is to delete. dropna implements it with several nuances:

# Drop every row with AT LEAST one null (aggressive: here we'd lose 7 out of 10)
df_no_nulls = df.dropna()

# Drop only rows missing a specific critical column
df_age = df.dropna(subset=["age"])

# Drop rows with more than 2 nulls (thresh = minimum number of NON-null values required)
df_thresh = df.dropna(thresh=len(df.columns) - 2)

# Drop an entire COLUMN
df_no_satisfaction = df.drop(columns=["satisfaction"])

When is dropping acceptable?

Rows: when nulls are few (rule of thumb: < 5% of rows) and the mechanism is MCAR. If it's MAR or MNAR, dropping rows distorts the sample: in our example, deleting the rows without age would mostly remove app customers.
Columns: when a huge proportion is missing (> 50-60%) and the column isn't critical for the business. Beware the MNAR exception: satisfaction is 40% missing, but the fact that it's missing may predict churn — before throwing it away, consider keeping its missingness indicator.
Never drop a row because the null is in the target variable of a production record you want to predict; discarding unlabeled rows only makes sense in training.

Simple imputation: mean, median, mode and constants

Imputing means filling the gap with an estimated value. The basic recipes with pandas:

# Mean: for symmetric numeric columns
df["age"] = df["age"].fillna(df["age"].mean())

# Median: for numeric columns with outliers or skew (remember 02-01!)
df["delivery_time_min"] = df["delivery_time_min"].fillna(
    df["delivery_time_min"].median())

# Mode: for categorical columns ([0] because mode() can return several ties)
df["signup_channel"] = df["signup_channel"].fillna(
    df["signup_channel"].mode()[0])

# A constant with business meaning
df["satisfaction"] = df["satisfaction"].fillna(-1)   # -1 = "did not answer"

Mean or median? The criterion comes straight from module 2: the mean is sensitive to outliers, the median isn't. With the €15,400 restaurant customer in the dataset, imputing total_spend with the mean would inflate every gap; the median is the robust default choice.

Group-wise imputation exploits MAR structure: if age is missing depending on the channel, impute with the median of each channel:

df["age"] = df["age"].fillna(
    df.groupby("signup_channel")["age"].transform("median"))

The hidden cost of all simple imputation: it artificially reduces variance. Every gap filled with the median is a point nailed to the center of the distribution; if you impute 30% of a column, its histogram will grow an artificial spike and its correlations (02-03) will weaken. Heavy imputation is not free.

scikit-learn's `SimpleImputer`

Why use scikit-learn if fillna already works? Because SimpleImputer memorizes the value it imputes with (it learns it in fit) and reapplies it identically to new data (transform). That fit/transform separation is the key to the final section of this lesson, and the general pattern of the entire library.

from sklearn.impute import SimpleImputer

# Imputer for numeric columns, with the median strategy
imp_num = SimpleImputer(strategy="median")

num_cols = ["age", "delivery_time_min"]
df[num_cols] = imp_num.fit_transform(df[num_cols])

# The learned value is stored, ready to be applied to future data:
print(imp_num.statistics_)    # [37.0, 49.5] e.g.: the medians of each column

Available strategies: "mean", "median", "most_frequent" (works for categoricals) and "constant" (with fill_value=). The same object, once fitted, is later applied with imp_num.transform(new_data) — without recomputing anything.

Advanced imputation: KNNImputer and missingness indicators

KNNImputer

Simple imputation ignores the row's context: a young high-spending customer and an occasional-shopper retiree get assigned the same median age. KNNImputer is finer-grained: for each gap it finds the k most similar rows in the other columns (its "neighbors") and averages their values.

from sklearn.impute import KNNImputer

imp_knn = KNNImputer(n_neighbors=3)
df[["age", "total_spend", "delivery_time_min"]] = imp_knn.fit_transform(
    df[["age", "total_spend", "delivery_time_min"]])

The internal "nearest neighbors" mechanics are exactly the K-NN algorithm we'll study in depth in lesson 04-05; for now the intuition is enough: it fills each gap by copying from the rows that look most alike. Two practical warnings: it only works with numeric columns, and since it measures distances, columns with large scales dominate the computation — a problem we'll solve in lesson 03-05.

Missingness indicators

When the absence is informative (MNAR), imputing and staying quiet destroys the signal. The solution: add a boolean column that remembers where the gap was, and then impute with a clear conscience.

# 1) Record the footprint of the absence BEFORE imputing
df["satisfaction_missing"] = df["satisfaction"].isna().astype(int)

# 2) Now, yes, impute the original column
df["satisfaction"] = df["satisfaction"].fillna(df["satisfaction"].median())

This way the churn model can learn the real pattern: "customers who don't answer the survey cancel more". SimpleImputer(add_indicator=True) generates these indicators automatically. It's a cheap and surprisingly powerful technique: in many datasets, the indicator ends up being a better predictor than the imputed column.

The silent risk: data leakage

One last warning remains — the most important of the lesson. In module 6 (lesson 06-01) we'll see that data is split into a training set and a test set, and that the latter must simulate future data the model has never seen. Well then: if you compute the median for imputation using the whole dataset and then split, the test data will have influenced that value. The model will have "peeked" at the future. This is called data leakage.

# BAD: the median is computed on ALL the data, test rows included
df["age"] = df["age"].fillna(df["age"].median())
# ... and only afterwards split into train/test

# GOOD (conceptual outline, developed in 06-01):
# 1. split into train and test first
# 2. imp = SimpleImputer(strategy="median"); imp.fit(X_train)  <- learns ONLY from train
# 3. X_train = imp.transform(X_train)
# 4. X_test  = imp.transform(X_test)   <- applies the value learned on train

This is where scikit-learn's fit/transform pair stops being a stylistic quirk and becomes the barrier against leakage: learn from train, apply to everything. The effect of getting it wrong is treacherous because it raises no error: your evaluation simply comes out better than the model deserves, and the disappointment arrives in production. Hold on to this idea: it will resurface with scaling (03-05), with target encoding (03-04) and in general with any step that "learns" something from the data.

Common Mistakes and Tips

Imputing without asking about the mechanism. Filling satisfaction (MNAR) with the mean fabricates happy customers who don't exist. First diagnose (MCAR/MAR/MNAR), then act.
Using the mean on columns with outliers. With the €15,400 customer included, the mean spend represents nobody. Median by default; mean only for reasonably symmetric distributions.
Dropping rows carelessly. A bare dropna() can empty half your dataset and bias what's left. Always check how many rows you lose and who they are.
Imputing before the train/test split. Data leakage gives no warning: it silently inflates your metrics. Learn imputation values from training data only.
Forgetting the missingness indicator. If you suspect MNAR, create the _missing column before imputing; afterwards you'll never be able to reconstruct it.
Confusing nulls with sentinels. A 999 or -1 inherited from the source system is not NaN to pandas: convert them first (df.replace(999, np.nan)) or your null counts will lie.
Tip: record in the code how many nulls there were, which strategy you applied to each column and why. Imputation is a modeling decision, not a plumbing chore.

Exercises

Exercise 1

For each MercaFresh scenario, classify the mechanism (MCAR, MAR or MNAR) and justify it in one sentence: (a) the order weight is missing for the records of a week when the warehouse scale was broken; (b) the postal code is missing more often in orders placed by phone, because the operator sometimes skips it; (c) declared income is missing mostly among high-income customers, who prefer not to disclose it.

Exercise 2

With the lesson's df DataFrame (the version with nulls), impute delivery_time_min with the median per city, given that a city column exists. Write the code with groupby + transform and explain why it can be better than the global median.

Exercise 3

Create a SimpleImputer with the median strategy and add_indicator=True, fit it on the columns ["age", "satisfaction"] and print the shape (shape) of the result. Explain why more columns come out than went in.

Solutions

Exercise 1

(a) MCAR: the breakdown has no relation to the type of order or the weight's value; it's pure (temporal) chance.
(b) MAR: the absence depends on the channel (an observed column), not on the value of the postal code itself; knowing the channel, you can impute without bias.
(c) MNAR: the absence depends on the missing value itself (high earners hide it more); imputing the mean would underestimate the missing incomes, and a missingness indicator is advisable.

Exercise 2

df["delivery_time_min"] = df["delivery_time_min"].fillna(
    df.groupby("city")["delivery_time_min"].transform("median"))

transform("median") returns a Series the same size as df with each row's group median, so fillna fills every gap with the median of its city. It beats the global median if delivery time varies systematically by city (delivering in Barcelona ≠ delivering in Sevilla): we're exploiting MAR structure, just like with age by channel.

Exercise 3

from sklearn.impute import SimpleImputer

imp = SimpleImputer(strategy="median", add_indicator=True)
result = imp.fit_transform(df[["age", "satisfaction"]])
print(result.shape)   # (10, 4)

Two columns went in and four come out: the two imputed originals plus one binary indicator for each column that had nulls (1 = the value was missing). This preserves the missingness information, essential if the mechanism is MNAR as we suspect for satisfaction.

Conclusion

You now know how to handle gaps with judgment: diagnose the mechanism (MCAR, MAR, MNAR) before acting, quantify and visualize null patterns, drop only when it's safe, impute with mean/median/mode or by groups, scale up to SimpleImputer and KNNImputer when needed, preserve the missingness signal with indicators, and — above all — learn imputation values only from the training data to avoid leaking information. The MercaFresh churn dataset is now complete: no duplicates, no impossible values and no gaps.

But complete doesn't mean ready: several of its variables have heavily skewed distributions (spend, with its long tail of restaurant customers), the dates are still not very useful as they are, and the orders haven't yet been summarized per customer. In the next lesson, data transformation, we'll reshape those variables — logarithms, discretization, date components, aggregations — and meet scikit-learn pipelines to chain all these steps together in an orderly way.

Handling missing data

Contents

Why data goes missing: MCAR, MAR and MNAR

MCAR: Missing Completely At Random

MAR: Missing At Random (conditional on what we observe)

MNAR: Missing Not At Random

Detecting and visualizing nulls

Dropping: rows and columns

Simple imputation: mean, median, mode and constants

scikit-learn's `SimpleImputer`

Advanced imputation: KNNImputer and missingness indicators

KNNImputer

Missingness indicators

The silent risk: data leakage

Common Mistakes and Tips

Exercises

Exercise 1

Exercise 2

Exercise 3

Solutions

Conclusion

Machine Learning Course

Module 1: Introduction to Machine Learning

Module 2: Foundations of Statistics and Probability

Module 3: Data Preprocessing

Module 4: Supervised Machine Learning Algorithms

Module 5: Unsupervised Machine Learning Algorithms

Module 6: Model Evaluation and Validation

Module 7: Advanced Techniques and Optimization

Module 8: Model Implementation and Deployment

Module 9: Hands-On Projects

Module 10: Additional Resources

Handling missing data

Contents

Why data goes missing: MCAR, MAR and MNAR

MCAR: Missing Completely At Random

MAR: Missing At Random (conditional on what we observe)

MNAR: Missing Not At Random

Detecting and visualizing nulls

Dropping: rows and columns

Simple imputation: mean, median, mode and constants

scikit-learn's SimpleImputer

Advanced imputation: KNNImputer and missingness indicators

KNNImputer

Missingness indicators

The silent risk: data leakage

Common Mistakes and Tips

Exercises

Exercise 1

Exercise 2

Exercise 3

Solutions

Conclusion

Machine Learning Course

Module 1: Introduction to Machine Learning

Module 2: Foundations of Statistics and Probability

Module 3: Data Preprocessing

Module 4: Supervised Machine Learning Algorithms

Module 5: Unsupervised Machine Learning Algorithms

Module 6: Model Evaluation and Validation

Module 7: Advanced Techniques and Optimization

Module 8: Model Implementation and Deployment

Module 9: Hands-On Projects

Module 10: Additional Resources

scikit-learn's `SimpleImputer`