There is an enemy we have been dodging since module 2: when we trained the MNIST dense network in 02-05, the curves showed mild overfitting — the training loss kept dropping while the validation loss stalled and began to rise. We noted it then; now we solve it. This lesson presents the full arsenal of regularization: the set of techniques that make a model generalize to new data instead of memorizing the training set. They cut across everything — they improve the catalog CNN, the review classifier, the sales LSTM, the transfer learning from the previous lesson and even the GAN's discriminator — which is why they are, probably, the techniques you will apply most often in your professional life.

1. Overfitting in depth: diagnosis with curves
2. Bias vs. variance in practical terms
3. L1 and L2 regularization (weight decay)
4. Dropout
5. Batch Normalization
6. Early Stopping
7. Data augmentation for images
8. Summary table of the arsenal
9. Integrative example: the MNIST network from 02-05, improved
10. A mention: learning rate schedules

Overfitting is memorizing instead of learning. An overfitted model performs very well on the training data and poorly on new data: it has learned the set's accidental quirks (the noise) in addition to — or instead of — the real patterns.

You already know the diagnostic instrument: the train/val curves that fit() returns in history. The three clinical pictures:

The unmistakable signal of overfitting is the divergence of the curves: the gap between train and val grows with the epochs. In 02-05 that gap was small; in the transfer learning of 05-03, with only 1,600 images, it can become a chasm if nothing is done.

Theory calls this the bias-variance trade-off. Without formalisms, the working version:

• High bias = the model is too simple or rigid for the real pattern → underfitting. You can tell because not even training goes well.
• High variance = the model is so flexible that it changes drastically with each specific sample of data: it fits the noise → overfitting. You can tell by the train/val gap.

The practical decision recipe, in order:

1. Is the training loss bad? → a bias problem: bigger model, train longer, better features. (Regularization doesn't help here; it even hurts.)
2. Training goes well but validation diverges? → a variance problem: regularization (this lesson), more data, or a smaller model.
3. Both going well? → touch nothing; move on to evaluating on test.

The entire arsenal that follows attacks case 2: variance.

The first family of techniques acts on the loss: a penalty for large weights is added.

• L2 (the usual one, also called weight decay): adds to the loss the sum of the squared weights, multiplied by a small factor λ. total_loss = data_loss + λ·Σw²
• L1: the same but with absolute values (λ·Σ|w|); its side effect is pushing many weights exactly to zero (producing "sparse" networks).

Why do small weights generalize better? The intuition: a huge weight means the network leans enormously on one specific detail of the input — the kind of fragile dependency that characterizes memorization. With the penalty, each weight must "earn its salary": it stays large only if it reduces the data loss by more than it costs in penalty. The result is smoother functions, less able to contort themselves to fit individual noise points. Remember the "purchasing committee" from 01-04: L2 stops any single member from shouting so loudly that they decide alone.

In Keras it is applied per layer:

Typical values of λ: between 1e-5 and 1e-2. Too high and the network becomes unable to learn (bias); too low and it does nothing.

Dropout is deep learning's most emblematic regularization technique: during training, on each pass, a random fraction of the layer's neurons is switched off (for example 30%, rate=0.3). Different neurons are switched off in each batch.

Why does something so apparently destructive work?

• It prevents co-adaptation: without dropout, a neuron can specialize in correcting the errors of one specific colleague, creating fragile dependencies and circuits that memorize. If your colleague can vanish at any moment, you cannot build your function on top of them: each neuron must learn features that are useful on their own.
• It trains an implicit committee: each switch-off pattern defines a different sub-network; the final model behaves like the average of an enormous ensemble of smaller networks — and averaging models reduces variance (the same purchasing-committee logic from 01-04, now inside the network).

Operational details:

• Different behavior in training and inference: during training it switches neurons off; during prediction (predict/evaluate) it switches nothing off and compensates the magnitudes automatically. Keras handles this by itself — but it explains why the training loss reported by fit can look worse than the validation loss at the start: the train loss is measured with dropout active.
• Where to place it: after large dense layers (the classic spot) and after the global pooling in CNN heads, as we did in 05-03. In intermediate convolutions it is less common; in RNNs specific variants are used (the LSTM layer's dropout/recurrent_dropout arguments).
• Typical rate: 0.2–0.5. Start with 0.2–0.3; 0.5 is aggressive and belongs with very large dense layers.

Batch Normalization (BN) normalizes a layer's activations within each batch: it subtracts the batch mean and divides by its deviation, leaving them centered and with controlled scale (and then applies a learnable scale and shift, so as not to lose expressiveness).

Why it helps:

• It speeds up and stabilizes training: without BN, each layer receives inputs whose distribution shifts constantly as the earlier layers learn — like aiming at a moving target. BN keeps those distributions stable, which allows larger learning rates and makes the network far less sensitive to initialization. It also mitigates the vanishing/exploding gradient from 02-03, because it stops activations from drifting into the flat regions of the activation functions (the "tap" closing shut).
• A secondary regularizing effect: since the mean and deviation are computed over the batch, each example is normalized slightly differently depending on who shares its batch. That small noise acts as gentle regularization — a side benefit, not its main purpose.

Where it goes: between a layer's linear part and its activation is the classic recipe (Dense/Conv2D → BatchNormalization → Activation), although placing it after the activation is also common and works; in practice you will see both. You have already used it twice without knowing: in the DCGAN generator (05-01) and inside MobileNetV2 (05-03).

Like Dropout, BN behaves differently at inference: in predict it uses means and deviations accumulated during training (not those of the current batch, which might be a single example). Hence the training=False we passed to the frozen base in 05-03.

The simplest technique and perhaps the most profitable: stop training when validation stops improving. If overfitting kicks in from a certain epoch onwards, don't train past it — the best model is the one from before the curves diverge.

In Keras it is a callback:

Three details that make the difference:

• patience: validation oscillates naturally; without patience, you would stop at the first dip. 5–10 epochs is typical.
• restore_best_weights=True: without this, you keep the weights from the last epoch (already worse), not the best one. Always turn it on.
• With early stopping you can set epochs high without fear: the number of epochs stops being a critical hyperparameter to guess.

Against variance, nothing works better than more data. When there isn't any, you manufacture plausible variations of what you have: this is the data augmentation we promised in 03-04 when discussing the product-photo classifier.

The idea: a toaster is still a toaster if the photo is mirrored, slightly rotated, zoomed in a bit or lit differently. By applying those transformations randomly every epoch, the network never sees exactly the same image twice — memorizing becomes impossible and the network learns real invariances (the "toaster-ness" that survives the changes).

In modern Keras, augmentation lives as layers inside the model, active only during training:

A sanity rule: transformations must preserve the label and be plausible in production. Mirroring a coffee maker, fine; mirroring an MNIST digit turns some into garbage (a mirrored 2 is not a 2), and flipping a toaster upside down doesn't help if no real photo will ever arrive that way.

Recommended strategy: early stopping always; data augmentation if you work with images; then dropout or L2 if the train/val gap persists; BN if training is also slow or unstable.

Let's settle the outstanding debt. This is the 02-05 dense network (~97.7% and mild overfitting), reinforced with BN + Dropout + Early Stopping:

And the curve comparison, the visual verdict:

What to expect when you run it:

• Original: the curves from 02-05 — train drops without pause, val peels away and rises. The gap grows with the epochs.
• Regularized: the curves travel together; the validation curve stays level with or even below the training curve at first (remember: the train loss is measured with dropout active). Training stops on its own once it stops improving, and restore_best_weights hands you back the best point.
• On test, the regularized version usually scrapes out a few tenths (towards ~98%+) — but that isn't the important gain: it is that validation performance now predicts test performance, because the model generalizes. On problems with less data than MNIST (like TecnoMarket's catalog), the difference isn't tenths of a point: it is many points.

One last piece, mentioned only in passing (the optimizers themselves were covered in 02-04): instead of a fixed learning rate, it is common to reduce it during training — large steps at first to make progress, small steps at the end to fine-tune. Two frequent forms in Keras: keras.callbacks.ReduceLROnPlateau (divides the lr when validation stalls — pairs beautifully with early stopping) and the predefined schedules (exponential decay, cosine). It is not regularization in the strict sense, but it is among the most profitable "improvement techniques" per line of code.

• Regularizing a model that underfits. If the training loss is already bad, dropout/L2 will make it worse. Diagnose first (bias vs. variance); regularize variance only.
• Misreading the curves with dropout. Validation loss sitting below training loss is not a Keras bug: the train loss is computed with neurons switched off and the val loss with the full network.
• Augmenting the validation/test data. Augmentation layers must act only in training (Keras's Random* layers already do this automatically); if you augment validation, your metrics stop being comparable across epochs.
• Stacking everything at once. If you add L2 + dropout 0.5 + BN + aggressive augmentation in one go and the network doesn't learn, you won't know what to remove. Add techniques one at a time, measuring their effect on the curves.
• Choosing augmentations that change the label. The horizontal mirror wrecks digits and text; a 45° rotation never occurs in real catalog photos. Always ask "could a real image arrive like this, and would it still be the same class?".
• Forgetting restore_best_weights=True. Early stopping without it halts training but leaves you with the weights of an already-degraded epoch.

Exercise 1. For each scenario, say whether the problem is bias or variance and which two measures from the arsenal you would apply first: (a) the review classifier from 04-03 scores 99.5% on train and 78% on validation; (b) the sales LSTM from 04-04 gives high error on both train and validation; (c) the transfer learning from 05-03 with a frozen base gives good, tightly matched train/val curves, but after fine-tuning the validation curve worsens epoch after epoch.

Exercise 2. Design the Sequential data-augmentation layer for TecnoMarket's product photos (catalog shots: product centered, light background, always upright). Justify each transformation you include and name one transformation you would not include and why.

Exercise 3. A colleague trains with EarlyStopping(monitor="val_loss", patience=0) and complains that training "stops almost as soon as it starts", at epoch 4, even though the overall validation trend was downward. Explain what is happening and how to fix it.

Solution 1.

• (a) A huge train/val gap → variance (clear overfitting). First measures: dropout in the classifier's dense/recurrent layers and early stopping; if it persists, shrink the model or obtain more labeled reviews.
• (b) Bad on both curves → bias (underfitting). Regularization doesn't help here: increase capacity (more units/layers), train more epochs, or improve the input features (more informative windows, calendar variables such as Black Friday).
• (c) Fine-tuning has introduced variance: unfreezing made the effective trainable capacity shoot up with the same data. Measures: early stopping with restore_best_weights (return to the best point) and data augmentation; reducing how many layers are unfrozen or lowering the learning rate further also works.

Solution 2.

I would not include large rotations (e.g. RandomRotation(0.25), up to 90°): catalog photos always show the product upright, so the network would spend capacity on an invariance it will never need in production — and some categories could become confusable when rotated. (Excluding the flip would also be defensible for products with visible text on the front, such as boxes: mirrored text is not plausible.)

Solution 3.

With patience=0, training stops as soon as val_loss worsens for a single epoch relative to the best seen so far. But validation loss oscillates naturally (batches, dropout, chance), so a small blip at epoch 4 — even with an underlying downward trend — triggers the stop. The fix: allow slack with patience=5 (or 10 for long training runs) and keep restore_best_weights=True, so several epochs beyond each dip are explored and, when training truly stops, the best epoch's weights are restored.

You have settled the debt opened in 02-05: you now know how to diagnose overfitting in the curves (a growing train/val gap), tell it apart from underfitting through the bias/variance lens, and attack it with an orderly arsenal — L2 for restrained weights, dropout as an internal committee, Batch Normalization for fast and stable training, early stopping as a universal safety net and data augmentation as a factory of plausible examples. These techniques cut across everything: apply them retroactively to any model in the course and, very especially, to the projects in module 7.

One last piece of the module remains, and it is the most influential of the decade: in 04-03 we flagged the single-vector bottleneck in sequence-to-sequence models. The solution — attention — didn't just solve that problem: it reorganized the whole of deep learning around a new architecture, the Transformer. That is the next lesson.