The Project | About Us | Contribute | Donations | License

HOME

Clues have been piling up throughout the module: the depth-unlimited tree that nailed 100% on training and sank on test (06-01), the return_train_score in cross_validate "useful for diagnosing overfitting" (06-03), the suspiciously perfect AUC that invited a hunt for leaks (06-04). This lesson pulls all those clues together into the central diagnosis of Machine Learning: does my model memorize (overfitting) or not learn enough (underfitting)? We will learn to diagnose it by comparing the training error with the validation error, to understand the bias-variance trade-off behind it, and to use two graphical tools — validation curves and learning curves — that answer the two quintessential practical questions: how much complexity is right? and would more data help? We will close the module with an honest-evaluation checklist that summarizes everything learned.

Definitions and analogy: memorizing the exam vs. not studying
Diagnosis: training error versus validation error
The bias-variance trade-off
Validation curves: how much complexity?
Learning curves: would more data help?
Remedies for overfitting and underfitting
Closing the module: the honest-evaluation checklist

Definitions and analogy: memorizing the exam vs. not studying

Let's return to the exam analogy from 06-01, now with three students:

The memorizer learns the textbook exercise solutions by heart, comma by comma. On those exercises they score a perfect 10; on the exam (new questions), they fail: they never understood the concepts, only the specific cases, noise included. This is overfitting: the model fits the training data so tightly that it captures its quirks and its noise, losing the ability to generalize.
The one who didn't study barely skimmed the syllabus. They fail the textbook exercises and the exam alike: they didn't even capture the basic patterns. This is underfitting: the model is too simple (or the features too poor) for the real structure of the problem.
The one who understood the material does well on the exercises (without acing every one: they didn't memorize the typos) and performs similarly on the exam. This is the sweet spot: learning the pattern, ignoring the noise.

At MercaFresh we have already met all three: the unpruned tree of 04-03 was the memorizer (one leaf per customer); a logistic regression on a single poor feature would be the one who didn't study; the max_depth=5 tree from the 06-03 tournament came close to the sweet spot.

Diagnosis: training error versus validation error

The diagnosis is made with two numbers, not one: the error (or metric) on training and on validation. The four possible scenarios:

Train error	Validation error	Diagnosis	MercaFresh example
Low	Low (similar)	Sweet spot — good fit	Logistic with good RFM features: F1 0.66 / 0.63
Low	High (wide gap)	Overfitting	Depth-unlimited tree: F1 1.00 / 0.62
High	High (similar)	Underfitting	Logistic with a single feature: F1 0.35 / 0.34
High	Low	Suspected bug	Data leakage, badly built sets, chance in small samples

Reading rules:

The gap train − validation measures overfitting: the wider it is, the more the model is memorizing.
The level of the validation error measures real quality. A model can have zero gap and be terrible (underfitting), or a huge gap with decent validation (usable but improvable overfitting).
The fourth scenario shouldn't exist: if validation clearly beats training, review the protocol (a leak in reverse? a validation set too small and lucky?).

In code, it is 06-03's return_train_score=True:

from sklearn.model_selection import cross_validate
from sklearn.tree import DecisionTreeClassifier

for depth in [None, 5]:                      # None = no limit
    tree = DecisionTreeClassifier(max_depth=depth, random_state=42)
    res = cross_validate(tree, X_train, y_train, cv=5,
                         scoring="f1", return_train_score=True)
    print(f"max_depth={depth}: train F1 = {res['train_score'].mean():.2f} | "
          f"val F1 = {res['test_score'].mean():.2f}")
# max_depth=None: train F1 = 1.00 | val F1 = 0.58   ← textbook overfitting
# max_depth=5   : train F1 = 0.70 | val F1 = 0.63   ← reasonable gap

The bias-variance trade-off

Behind the two phenomena lies a fundamental trade-off. Intuitively, a model's error on new data has two controllable components:

Bias: error from overly rigid assumptions. A high-bias model (a straight line for a curved phenomenon) fails systematically in the same way, no matter what data it is trained on. It is the signature of underfitting.
Variance: error from excessive sensitivity to the specific training data. A high-variance model (an extremely deep tree) changes drastically if you change a few rows: it learns the noise of that sample. It is the signature of overfitting. (In fact, the deviation across folds in 06-03 was already giving us an empirical measure of this sensitivity.)

As model complexity grows, bias goes down (richer patterns can be represented) but variance goes up (there is more freedom to chase the noise). The total error draws a U:

flowchart LR
    subgraph Axis["Model complexity →"]
        A["Low complexity<br/>─────────<br/>HIGH bias<br/>Low variance<br/>= UNDERFITTING<br/>(train bad, val bad)"]
        B["Right complexity<br/>─────────<br/>Moderate bias<br/>Moderate variance<br/>= SWEET SPOT<br/>(minimum validation error)"]
        C["High complexity<br/>─────────<br/>Low bias<br/>HIGH variance<br/>= OVERFITTING<br/>(train perfect, val bad)"]
    end
    A --> B --> C

Every module 4 algorithm has its complexity dials: max_depth and friends in trees, the number of neighbors in K-NN (small k = more variance), the layer sizes in the MLP, C and the kernel in SVM... Finding the sweet spot of those dials is precisely what the validation set of 06-01 is for — and what we are now going to visualize.

Validation curves: how much complexity?

The validation curve plots the metric on train and on validation as a function of a complexity hyperparameter. With validation_curve over the max_depth of the churn tree from 04-03:

from sklearn.model_selection import validation_curve
from sklearn.tree import DecisionTreeClassifier
import matplotlib.pyplot as plt
import numpy as np

depths = range(1, 16)
train_scores, val_scores = validation_curve(
    DecisionTreeClassifier(random_state=42),
    X_train, y_train,
    param_name="max_depth", param_range=depths,
    cv=5, scoring="f1",
)

tr_m, va_m = train_scores.mean(axis=1), val_scores.mean(axis=1)
va_s = val_scores.std(axis=1)

plt.plot(depths, tr_m, "o-", label="Training F1")
plt.plot(depths, va_m, "o-", label="Validation F1")
plt.fill_between(depths, va_m - va_s, va_m + va_s, alpha=0.2)  # ± std (06-03)
plt.axvline(depths[np.argmax(va_m)], ls="--", color="gray",
            label=f"Optimum: max_depth={depths[np.argmax(va_m)]}")
plt.xlabel("max_depth"); plt.ylabel("F1"); plt.legend(); plt.show()

How to read the resulting chart (the pattern is always the same):

Left zone (max_depth 1–3): both curves low and close together → underfitting; the tree is too simple even for the training data.
Right zone (max_depth > 8): the training F1 keeps climbing towards 1.0 while the validation F1 drops → overfitting; each extra level of depth memorizes noise.
The peak of the validation curve (typically around max_depth 4–6 on churn) is the sweet spot: just the right complexity. It is the bias-variance "U", seen on real data.

This explains the result of exercise 3 in lesson 06-03 (3 too shallow, 10 too deep, 5 the winner). The validation curve explores one hyperparameter with the others fixed; the systematic exploration of several at once is the hyperparameter optimization of 07-05.

Learning curves: would more data help?

The other practical question: if MercaFresh invested in collecting more customer history, would the model improve? The learning curve plots the metric on train and validation as a function of the number of training examples:

from sklearn.model_selection import learning_curve
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

pipe = Pipeline([("sc", StandardScaler()),
                 ("clf", LogisticRegression(max_iter=1000))])

sizes, train_scores, val_scores = learning_curve(
    pipe, X_train, y_train,
    train_sizes=np.linspace(0.1, 1.0, 8),   # from 10% to 100% of the training data
    cv=5, scoring="f1",
)

plt.plot(sizes, train_scores.mean(axis=1), "o-", label="Training F1")
plt.plot(sizes, val_scores.mean(axis=1), "o-", label="Validation F1")
plt.xlabel("Number of training examples"); plt.ylabel("F1")
plt.legend(); plt.show()

Interpretation by curve shape:

Shape at 100% of the data	Diagnosis	More data?
Curves apart (train high, val low) and the validation one still rising	Overfitting with headroom	Yes: more data would close the gap
Curves already converged (together and flat) at a high level	Stable sweet spot	No: more data barely adds anything
Curves converged at a low level	Underfitting	No: the problem is the model or the features, not the volume

It is an invaluable chart for investment decisions: collecting data costs money and months, and the learning curve tells you in advance whether it will be worth anything.

Remedies for overfitting and underfitting

Diagnosis done; the prescription list:

Against overfitting (wide train-validation gap)

More training data — the most reliable remedy when the learning curve shows headroom: with more examples, individual noise weighs less.
Reduce model complexity: less tree depth, more neighbors in K-NN, fewer neurons in the MLP, or simply a simpler algorithm. The validation curve tells you how much to trim.
Fewer features / better features: removing noisy or redundant variables reduces the opportunities to memorize (PCA, from 05-03, can help as a prior reduction).
Regularization: penalizing the model's large weights to force smoother solutions. It is the star remedy in linear models and networks, and has its own lesson — Ridge, Lasso and Elastic Net in 07-01 — so here we only leave it noted.
Early stopping (in passing): in models that train by iterations (the MLP, and the gradient boosting of 07-03), stopping training when the validation error stops improving, even if the training error keeps falling.
Ensembles: combining many high-variance models averages out their individual errors (the variance cancels). It is the idea behind Random Forest and company — 07-02.

Against underfitting (train and validation equally bad)

Increase complexity: more depth, more neurons, a non-linear kernel in the SVM, a more expressive algorithm.
Better features: often the bottleneck is not the model but the information it receives. The feature engineering of 03-06 (ratios, RFM aggregates, interactions) usually moves the needle more than any change of algorithm.
Less regularization (if there is any) and, in iterative models, training longer.
What does not work: adding more rows. A model that fails to capture the pattern with 10,000 examples won't do it with 100,000 either — confirmed by a learning curve converged at a low level.

Diagnosis	Main levers	Where they are developed
Overfitting	More data, less complexity, regularization, early stopping, ensembles	07-01, 07-02, 07-03
Underfitting	More complexity, better features, less regularization	03-06, module 7

Closing the module: the honest-evaluation checklist

This module opened with the question module 5 left hanging: "is my model actually any good?". We can now answer it methodically. The checklist, lesson by lesson:

Clean split (06-01): test set set aside from the start and touched only once; stratify with imbalanced classes; chronological split when time is involved; no spread-out duplicates; no feature that wouldn't exist at prediction time.
Metric agreed in advance (06-02): chosen with the business according to FP and FN costs (recall for MercaFresh's churn), never after the fact; confusion matrix always in view.
Baseline (06-02, picking up 01-04): DummyClassifier/DummyRegressor evaluated with the same metric; the model must beat them clearly or there is no story to tell.
Cross-validation (06-03): mean ± deviation, same folds for all candidates, Pipeline inside the CV; differences between models judged with the skepticism of 02-04.
Operating threshold (06-04): chosen on validation according to costs and capacity, not inherited from 0.5; AUC to compare, the curve to decide.
Train vs. validation diagnosis (06-05): gap monitored, validation curve for complexity, learning curve before asking for more data.
Final skepticism: a result that is too good (99% accuracy, AUC 0.99) is investigated as a bug — it almost always is one (data leakage).

A model that passes this checklist has credible performance. The next frontier is making it better, and that is what module 7 is about.

Common Mistakes and Tips

Looking only at the validation error and ignoring the training one. Without both numbers there is no diagnosis: you won't know whether to attack bias or variance, and the remedies are opposites.
Treating every bad result as overfitting. Underfitting is just as common and is cured with the opposite (more complexity, not less). Diagnose before medicating.
Chasing a zero gap. A small gap is normal and healthy; a model with exactly equal train and validation is usually underfitted. What you optimize is the validation error, not the gap.
Adding data without looking at the learning curve. If the curves have already converged, those months of collection won't move the metric.
Tuning complexity by looking at the test set. Validation curves are built with CV on the training data; the test set stays in its safe until the very end (06-01).
Tip: always print the pair (train metric, validation metric) in every experiment, from day one. It is a one-line habit that catches 90% of modeling problems on the first pass.

Exercises

Exercise 1

Diagnose each scenario (overfitting, underfitting, sweet spot or suspected bug) and propose the first corrective action:

(a) K-NN with k=1 on churn: train F1 = 0.99, validation F1 = 0.55.
(b) Linear regression of monthly spend with age as the only feature: train R² = 0.12, validation R² = 0.11.
(c) Logistic with RFM features: train F1 = 0.68, validation F1 = 0.64.
(d) Tree on a dataset including the churn_reason column: train F1 = 0.99, validation F1 = 0.99.

Exercise 2

Write the code that computes the validation curve of the n_neighbors hyperparameter (values 1 to 30) for a KNeighborsClassifier (with scaling in a Pipeline) on churn, with 5-fold CV and scoring="f1", and prints the optimal k. State which end of the range you expect to overfit and why.

Exercise 3

The churn model's learning curve shows: at 100% of the data, train F1 = 0.91, validation F1 = 0.63, and the validation curve has been rising since 40% of the data without flattening out. MercaFresh can (a) buy 6 more months of history or (b) pay a consultancy to try more algorithms. Which option does the chart support, and why?

Solutions

Solution 1.

(a) Textbook overfitting: k=1 memorizes each customer (maximum variance). First action: increase k (the validation curve over n_neighbors would say by how much).
(b) Underfitting: train and validation equally low; age alone does not explain spend. First action: better features (the RFM ones from 03-06), not more complexity on top of a poor feature.
(c) Sweet spot: good validation level and a small gap (0.04). Action: nothing urgent; fine-tune with the module 7 techniques if you want to squeeze out more.
(d) Suspected bug: near-perfect performance on validation too. churn_reason only gets filled in after the cancellation: it is the future-information leak of 06-01. First action: drop the column and re-evaluate.

Solution 2.

from sklearn.model_selection import validation_curve
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
import numpy as np

pipe = Pipeline([("sc", StandardScaler()), ("knn", KNeighborsClassifier())])
ks = range(1, 31)

train_sc, val_sc = validation_curve(
    pipe, X_train, y_train,
    param_name="knn__n_neighbors",   # the Pipeline's step__parameter syntax
    param_range=ks, cv=5, scoring="f1",
)
va_m = val_sc.mean(axis=1)
print(f"Optimal k: {list(ks)[np.argmax(va_m)]}  (val F1 = {va_m.max():.3f})")

The small-k end overfits (k=1 in particular): each prediction depends on a single neighbor, so the model reproduces the individual noise of the training data (high variance). With very large k the opposite happens: the prediction increasingly resembles "the majority class of the broad neighborhood" and underfitting appears.

Solution 3. The chart supports option (a), more data: there is a wide gap between train (0.91) and validation (0.63) — a sign of overfitting — and the validation curve keeps rising as data is added, meaning it has not yet converged: each additional example is still improving generalization. Trying more algorithms (b) without attacking the variance would probably reproduce the same pattern. If the validation curve had been flat since 40%, the answer would be the opposite: more history would add nothing, and it would be time to change the model or the features.

Conclusion

Overfitting and underfitting are the two ways a model can fail — memorizing the exam or not studying for it — and we now know how to tell them apart with the train/validation error pair, understand them through the bias-variance trade-off, locate the complexity sweet spot with validation_curve, decide whether more data would help with learning_curve, and apply the right remedy to each diagnosis. This lesson closes module 6 and its honest-evaluation checklist: clean split, agreed metric, baseline, cross-validation, business-driven threshold and fit diagnosis. The question "is my model actually any good?" now has a methodological answer; the next one is "can I make it even better?", and that is what module 7 is devoted to: regularization to tame overfitting with surgical precision, ensembles and gradient boosting to combine models, deep networks, and the systematic hyperparameter optimization that automates the searches we did here by hand.

Overfitting and underfitting

Contents

Definitions and analogy: memorizing the exam vs. not studying

Diagnosis: training error versus validation error

The bias-variance trade-off

Validation curves: how much complexity?

Learning curves: would more data help?

Remedies for overfitting and underfitting

Against overfitting (wide train-validation gap)

Against underfitting (train and validation equally bad)

Closing the module: the honest-evaluation checklist

Common Mistakes and Tips

Exercises

Exercise 1

Exercise 2

Exercise 3

Solutions

Conclusion

Machine Learning Course

Module 1: Introduction to Machine Learning

Module 2: Foundations of Statistics and Probability

Module 3: Data Preprocessing

Module 4: Supervised Machine Learning Algorithms

Module 5: Unsupervised Machine Learning Algorithms

Module 6: Model Evaluation and Validation

Module 7: Advanced Techniques and Optimization

Module 8: Model Implementation and Deployment

Module 9: Hands-On Projects

Module 10: Additional Resources