In 05-03 we trained an MLP and it tied with logistic regression on the returns predictor. That is not a defeat for networks: it is the sign that their home ground is elsewhere. This lesson explains what exactly deep learning is, why it took off in 2012 and not before, and which architectures build in the structure of the data and can therefore see, hear and read where the MLP and the classical models cannot reach: convolutional networks for images (with a convolution computed by hand and a mini-example in PyTorch on synthetic photos of damaged and intact packages, the NovaMarket incidents case), recurrent networks for sequences, autoencoders for detecting anomalies and transfer learning, the idea that has changed professional practice: not training from scratch. We will close with the landscape of applications and with their costs and limits. It matters because this is where networks justify their complexity, and where Marta will find the tools for the photos in incidents.csv and, in the next lesson, for the reviews.

1. What makes deep learning "deep" and why it took off
2. Learning hierarchical representations
3. Convolutional networks: convolution, filters, pooling and feature maps
4. Code: a CNN for NovaMarket's incident photos
5. Recurrent networks: sequences, hidden state, LSTM and GRU
6. Autoencoders and anomaly detection
7. Transfer learning and pretrained models
8. Landscape of applications
9. Costs and limits
10. Common Mistakes and Tips
11. Exercises
12. Conclusion

Deep learning is machine learning with neural networks of many layers (from a few to hundreds). The "deep" is not just a number: it means the network learns a hierarchy of representations (section 2) instead of receiving hand-designed features. In module 4 Marta created amount_per_item or previous_return_rate with her business knowledge; a deep network invents its own "columns" from pixels or words.

The theory had existed since the 1980s (backpropagation, 05-03) and convolutional networks since 1989 (LeCun's LeNet, for reading postcodes). Why the take-off in 2012? Four things came together:

The symbolic moment is ImageNet 2012 (01-01): AlexNet, an 8-layer convolutional network with 60 million parameters trained on two GPUs, cut the classification error from 26 % to 16 % in one go, when the yearly improvements had been of one point. From then on, every domain (vision, speech, text) fell to the same recipe: deep network, lots of data, GPU.

The central idea: each layer transforms the output of the previous one into something more abstract. In a network that recognises photos of packages:

Nobody programs a "crack detector": it appears because it is useful for minimising the loss. The same happens with text (letters → words → phrases → meaning) and with audio (waves → phonemes → words). This is the answer to the question of 05-01: why networks can read and look. They learn the conversion from raw data to features, the part that in module 4 Marta did by hand and that for pixels and words nobody knows how to do well by hand. And it also explains why depth pays off more than width (05-02): composing simple transformations is more efficient than approximating the function in one go.

An MLP on a 16×16 image flattens it into 256 numbers without knowing that pixel 17 is below pixel 1. Convolutional networks (CNNs) build in two assumptions that hold for images: patterns are local (a crack occupies a few neighbouring pixels) and can occur at any position (the crack may be at the top or at the bottom). They achieve it with three pieces:

A filter (or kernel) is a small matrix of weights, typically 3×3. The convolution slides the filter over the image and, at each position, multiplies element by element and sums. The result is a feature map that says "how much this patch of the image resembles the pattern of the filter". Example by hand: a 5×5 image with a bright square (value 9) on a 0 background, and a filter that detects vertical edges (left column −1, centre 0, right +1):

We place the filter in the top-left corner (rows 0-2, columns 0-2 of the image): (0·−1 + 0·0 + 0·1) + (0·−1 + 9·0 + 9·1) + (0·−1 + 9·0 + 9·1) = 18. We shift it one column to the right (columns 1-3): (0 + 0 + 0) + (−9 + 0 + 9) + (−9 + 0 + 9) = 0. One more (columns 2-4): (0 + 0 + 0) + (−9 + 0 + 0) + (−9 + 0 + 0) = −18. Repeating over the 3 × 3 possible positions:

Resulting feature map (3×3): the left column "lights up" (left edge of the square, dark→bright transition), the right one lights up negatively (bright→dark) and the centre, with no horizontal changes, stays at 0. The transposed filter (rows −1, 0, +1) detects horizontal edges and gives [[18, 27, 18], [0, 0, 0], [−18, −27, −18]]. Details: the output shrinks (5×5 → 3×3) unless a border of zeros is added (padding), and the filter can jump 2 at a time (stride). The essential point: the 9 weights of the filter are learned by backpropagation, just like those of a dense layer, and a convolutional layer has many filters (8, 16, 64...) that produce as many maps. With 9 weights per filter and the same weights across the whole image, a convolutional layer has vastly fewer parameters than an equivalent dense one and detects the pattern wherever it is.

Pooling reduces the size of the maps by keeping a summary of each block, almost always the maximum (max pooling) of 2×2 blocks. On [[1, 3, 2, 0], [4, 6, 1, 1], [0, 2, 7, 5], [3, 1, 8, 2]] it gives [[6, 2], [3, 8]]. It provides invariance to small shifts (if the crack moves by one pixel, the block maximum does not change) and divides the computation by four at each level.

Repeated convolution → ReLU → pooling blocks (each with more filters and smaller maps: the hierarchy of section 2), followed by Flatten and one or two dense layers that classify. It is the structure of LeNet (1989), AlexNet (2012) and, with residual shortcuts to reach hundreds of layers, of the ResNets (2015).

We do not have the real photos of incidents.csv, so we generate synthetic 16×16 greyscale images with numpy: a bright box on a dark background, and in half of them some damage (a diagonal crack or a dark dent). The goal is to see the CNN at work and compare it with an MLP; in 09-03 a full project will be done with real images.

Output (truncated):

You can see the box (values 5-7) and the diagonal crack of zeros running down from row 7 to row 11. Now the CNN, an MLP for comparison, and the training (the loop is that of 05-03, with CrossEntropyLoss because we have 2 output logits):

Output (varies with the seed and the version):

Reading, line by line of the model: nn.Conv2d(1, 8, kernel_size=3, padding=1) learns 8 filters of 3×3 over 1 channel (72 weights + 8 biases) and, thanks to the padding, keeps the 16×16 size; MaxPool2d(2) brings it down to 8×8; the second convolution produces 16 maps of 8×8 with filters that look at the 8 previous maps (16 × 8 × 3 × 3 = 1,152 weights); after the second pooling there remain 16 maps of 4×4, i.e. 256 values that Flatten turns into a vector for the dense layers. The result is emphatic: the CNN gets 98.7 % of the test photos right (confusion matrix: 75 intact ones correctly classified, 73 damaged ones detected, 2 damaged ones that slip through) with fewer parameters (9,538) than the MLP (16,578), which stays at 57 %, barely better than chance, because for it every pixel is an independent column and the same crack in another position is a new pattern. The maps of the first layer (cnn[1](cnn[0](Xte_t[:1])) has shape [1, 8, 16, 16]) are 8 "filtered" versions of the photo; the learned weights (cnn[0].weight[0, 0]) form edge detectors similar to the one in section 3.1, without anyone having designed them. For Diego: with 450 labelled photos and one minute of CPU we have a damaged-package classifier that works; with real photos it would take more data, data augmentation (exercise 2) and probably transfer learning (section 7).

Time series (the weekly demand of the NovaClean, 04-04) and text are sequences in which order matters and length varies. A recurrent network (RNN) processes the sequence element by element while maintaining a hidden state h, a vector that summarises what has been seen so far: at each step, h_t = f(W·x_t + U·h_{t−1} + b), that is, the same neuron as always but also fed with its own previous state. The same weights W and U are applied at every step (as the CNN filter is applied at every position), and the final state, or the sequence of states, is passed to an output layer.

The problem with the simple RNN is that, when training it, the gradient travels through as many steps as the sequence has elements, being multiplied at each one: the vanishing gradient (05-02) in its temporal version. In practice, it forgets what happened more than a few dozen steps ago. LSTMs (long short-term memory, 1997) and GRUs (gated recurrent unit, 2014) solve it with gates: small sigmoid neurons that decide, at each step, which part of the state is forgotten, which part of the input is written and what is read for the output. A memory "conveyor belt" along which information and gradient travel without degrading. There is no need to memorise the equations; you do need to know that PyTorch's nn.LSTM and nn.GRU (for instance, nn.GRU(input_size=1, hidden_size=8, batch_first=True), with 264 parameters, returns for a sequence of 12 weeks 12 states of 8 values) are the tool when the dependency is long. Applications: demand forecasting with the previous 8-12 weeks (unlike the MLP of solution 3 in 05-02, the GRU respects the order), word-by-word text classification, speech recognition. In text, since 2018 transformers (05-05) have replaced recurrent networks; in time series they remain competitive. 09-03 includes a guided project with a recurrent network on the NovaClean demand.

An autoencoder is a network trained to reproduce its input: an encoder compresses the data into a small vector (the "bottleneck" or latent representation) and a decoder tries to reconstruct the original; the loss is the reconstruction error (MSE). It seems useless (why copy?), but by passing through the bottleneck the network is forced to learn what is essential in the data, without labels: it is unsupervised learning (04-02).

Star use at NovaMarket: anomaly detection (case 3, fraud). The autoencoder is trained only on normal orders; unusual orders, resembling nothing it has learned, are reconstructed poorly. With a minimal autoencoder (nn.Sequential(nn.Linear(4, 2), nn.Tanh(), nn.Linear(2, 4)), bottleneck of 2) on the standardised columns amount, num_items, delivery_days and new_customer of the 3,000 orders, 200 epochs of Adam leave the mean reconstruction error at 0.51. If we generate 30 anomalous orders (huge amount, many items, one-day delivery, new customer), their mean error is 8.0; setting the threshold at the 99th percentile of the normal orders (2.61) detects all 30 anomalies with 1 % false alarms. Other applications: dimensionality reduction to visualise customers (a non-linear alternative to PCA), noise removal in images, and the variational variant (VAE), the first modern generative image model.

The idea that has changed practice the most: do not train from scratch. A CNN trained on ImageNet has learned in its first layers detectors of edges, textures and shapes that work for any image, including photos of packages. Transfer learning consists of taking that pretrained model, removing its output layer (which classified ImageNet's 1,000 classes), putting a new one for our problem (2 classes) and training only that layer (feature extraction) or the whole network with a small learning rate (fine-tuning). Result: good models with hundreds of examples instead of millions, and in minutes instead of days.

Illustrative example, not executable in this environment (it requires torchvision and downloading the weights), of the pattern with a ResNet-18:

The same exists for text (BERT, GPT and company, 05-05), audio (Whisper) and almost any domain, distributed in repositories such as Hugging Face. Marta sums up the strategic consequence for NovaMarket: "we are not going to train a vision network from scratch; we are going to adapt one that already sees".

Recommendation deserves a pause: each customer and each product receives an embedding (05-02) and a network predicts the affinity from both; NovaMarket's 12,000 products end up in a space where the NovaClean is close to vacuum bags and far from coffee makers, learned only from what is bought together. And in deep reinforcement learning, the network approximates the value of each action (03-03, 04-02); AlphaGo combined a CNN that evaluated boards with tree search.

Diego, having seen the 98.7 % on the photos, asks how much it costs and what can go wrong. Honest answer:

• Labelled data: the CNN needed 450 labelled photos; a real one, thousands, or transfer learning. Labelling costs people's time and carries labelling biases (02-03).
• Compute and energy: training large models costs from thousands to millions of euros in GPUs and a considerable energy footprint; inference counts too if it runs millions of times a day. Choosing the smallest model that solves the problem is an economic and environmental decision.
• Opacity: we cannot read the 9,538 weights of the CNN the way we read the coefficients of logistic regression. There are explanation techniques (saliency maps that highlight which pixels weighed most, SHAP), but they are approximate. Under the AI Act (02-04), a system that decides about people needs explainability and human oversight; for classifying photos of packages the risk is low, for denying a return it is not.
• Fragility: networks make confident mistakes on data unlike the training data (a photo with different lighting, another kind of box) and are vulnerable to adversarial examples (imperceptible perturbations that change the prediction). Monitoring in production is required (08-01).
• Bias: they learn what is in the data, including what is undesirable (02-04). A "product in good condition" CNN trained only on brown boxes will fail on white boxes.
• Maintenance debt: framework versions, GPUs, retraining. Every model in production is a system that ages.

• Flattening images for an MLP. The spatial structure is lost and the model needs far more data and parameters for a worse result, as we have just seen (57 % against 98.7 %). For images, convolutions (or a pretrained model).
• Training from scratch by default. With little data, transfer learning almost always wins. Before designing a network, look for a pretrained model of the domain.
• Forgetting the order of the dimensions. PyTorch expects (batch, channels, height, width) in CNNs and (batch, time, features) in recurrent networks with batch_first=True. A mistake here raises no exception: it gives absurd results.
• Getting the shape of the CNN output wrong. Before the first dense layer you need to know how many values come out of the last pooling (here 16 × 4 × 4 = 256); compute it or print it with a test tensor.
• No data augmentation on images. Rotating, cropping and changing the brightness is the cheapest regularisation in vision (exercise 2).
• Selling deep learning as magic. It costs data, GPUs and maintenance, and it is opaque. For small tables, the logistic regression or the forest of module 4 remain the first option, as we checked in 05-03.

Exercise 1. Compute by hand the convolution of the 5×5 image of section 3.1 with the filter [[1, 1, 1], [0, 0, 0], [−1, −1, −1]] (horizontal edge detector). Check the result with a function conv2d(img, k) that walks through the positions with two loops and (img[i:i+3, j:j+3] * k).sum(). Which edge of the box lights up positively and which negatively?

Exercise 2. Data augmentation. With the CNN already trained in section 4, evaluate the accuracy on the horizontally mirrored test set (torch.flip(Xte_t, dims=[3])), where the diagonal cracks take the opposite orientation. Then retrain a new CNN adding the mirrored versions to the training set (torch.cat([Xtr_t, torch.flip(Xtr_t, dims=[3])]) and duplicating the labels) and measure again on the normal and the mirrored test sets. Explain the result.

Exercise 3. In the autoencoder of section 6, what happens to the false alarms and to the detection of the 30 anomalies if the threshold is set at the 95th percentile instead of the 99th? And if the autoencoder is trained with the anomalous orders mixed in among the normal ones? Reason without running it and, if you wish, check it by replicating the code described.

Solution 1. The result is [[−18, −27, −18], [0, 0, 0], [18, 27, 18]]. In the top output row the filter lands with its +1 weights on the background (0) and with the −1s on the box (9): the top edge of the box lights up negatively; the middle row gives 0 (inside the box there is no vertical change); in the bottom row the +1s land on the box and the −1s on the background: the bottom edge lights up positively. It is the map of the transposed filter of section 3.1 with the sign flipped (that one had the −1s at the top). Each filter "sees" one orientation and one direction, and that is why a layer has many filters.

Solution 2. In our run, the CNN trained without mirrors drops from 98.7 % to 96.0 % on the mirrored test set (6 damaged ones slip through instead of 2): it has partly learned "crack = diagonal going down to the right" and the opposite orientation is less familiar to it. With the mirrors added to training (900 images), the new CNN reaches 99.3 % on the normal test set and 98.7 % on the mirrored one: data augmentation has taught it the right invariance (a crack is a crack in any orientation) without labelling a single extra photo. In a real case you would add rotations, crops and brightness changes, and apply them on the fly in every epoch instead of duplicating the dataset.

Solution 3. With the 95th percentile, the threshold drops (by construction, 5 % of the normal orders sit above it), so the false alarms rise from 1 % to 5 % (about 150 orders out of 3,000) and the 30 anomalies are still detected (their error, around 8, is far above either threshold). It is the precision-recall trade-off of 04-05: the threshold is chosen by cost, and here 5 % manual reviews would probably be excessive for no benefit. If the anomalies are mixed into training, the autoencoder also learns to reconstruct them (they are only 30 among 3,000, so it will reconstruct them somewhat worse than the normal ones, but better than if it had never seen them): their error drops, some fall below the threshold and detection worsens. That is why anomaly autoencoders are trained on data as "clean" as possible, or one accepts that they only detect the very unusual.

We have defined deep learning as many-layer networks that learn hierarchical representations (edges → parts → objects), and explained its take-off in 2012 by the confluence of data (ImageNet), GPUs, ReLU and frameworks. We opened up convolutional networks: the convolution as a sliding filter (computed by hand: the vertical edge detector on the 5×5 box), pooling and feature maps, and we saw them win on NovaMarket's synthetic incident photos (98.7 % accuracy with 9,538 parameters, against 57 % for an MLP with more parameters). We introduced recurrent networks with their hidden state and the LSTM/GRU gates for long sequences, autoencoders that detect anomalous orders by their reconstruction error, and transfer learning, the practice of adapting pretrained models instead of training from scratch. The landscape of applications and the list of costs and limits (labelled data, compute, opacity, fragility, bias) complete the picture Marta needs to decide with Diego where a deep network is worth it.

The most important data for NovaMarket remains: text: the reviews in reviews.csv and the customer-service assistant. Recurrent networks read it word by word and forgot; since 2017 a different architecture does it, based on attention, which has ended up dominating images and audio too and has given rise to large language models. It is the topic of the last lesson of the module: Transformers, Large Language Models and Generative AI.