The module's final lesson, and the model family that dominates the headlines. Neural networks bring no new geometric idea like the SVM's margin or the tree's questions: they bring a principle of composition — stacking many tiny computing units, each suspiciously similar to the logistic regression of 04-02, until practically any function can be approximated. In this lesson you'll take the artificial neuron apart piece by piece (picking up the perceptron that the history of 01-02 left standing in 1958), run a forward pass with real numbers, understand backpropagation as the gradient descent of 04-01 applied in a chain, and train an MLPClassifier on MercaFresh's churn. We'll close with the comparison of the module's seven algorithms — your map for choosing a model — and the question that opens module 5.

1. The artificial neuron: weights, bias and activation
2. Activation functions: sigmoid, tanh and ReLU
3. The layered architecture
4. Forward pass: a complete numeric example
5. Backpropagation and gradient descent (conceptual)
6. Implementation with scikit-learn: MLPClassifier on churn
7. When a network adds value over the previous models
8. Final comparison: the module's 7 algorithms

An artificial neuron does three things, in a row:

1. Combines its inputs linearly: $z = w_1 x_1 + w_2 x_2 + \dots + w_n x_n + b$ (the weights $w_i$ weigh each input; the bias $b$ shifts the result — it's the intercept from 04-01 under another name).
2. Activates: passes $z$ through a nonlinear function $f$.
3. Emits the output $a = f(z)$ to the next neurons.

Sound familiar? A neuron with a sigmoid activation is exactly the logistic regression of 04-02: linear combination + sigmoid. Rosenblatt's perceptron (1958), which you met in the history of 01-02, was this very thing with a step activation (0 or 1, no nuance) — and its famous limitation was the one shared by all our linear models: straight boundaries only. The revolution isn't in the neuron, but in what happens when you connect them: the output of some becomes the input of others, and the composition of many simple nonlinear transformations can bend the boundary as much as needed. Without the activation's nonlinearity, by the way, stacking layers would be useless: a chain of linear functions collapses into a single linear function.

ReLU (Rectified Linear Unit) deserves an explanation given its apparent crudeness: if the input is negative, it emits 0 (the neuron "stays silent"); if positive, it lets it through unchanged. That simplicity is its virtue — its slope is 1 across the entire active zone, so the gradient flows without vanishing no matter how many layers it crosses, the problem that hobbled sigmoid and tanh for decades. For the output layer, sigmoid (binary) or softmax (multiclass, 04-02) remain the choices.

Neurons are organized in layers: an input layer (one neuron per feature — it computes nothing, only distributes), one or more hidden layers (they extract intermediate representations) and an output layer (it emits the prediction). Every neuron in a layer connects to all the neurons in the next — hence the technical name multilayer perceptron (MLP), or dense network:

An intuition for what each layer learns: the hidden neurons specialize in detecting intermediate patterns — one might fire at the "growing silence" profile (high recency + low trend), another at "premium customer cooling off" — and the output layer combines those detectors like a logistic regression over learned features. That is the model's deep promise: the network does its own feature engineering (03-06), learning the combinations we used to design by hand. The number of layers and neurons per layer are hyperparameters: more = more expressive capacity and more overfitting risk (06-05).

The forward pass is a data point's journey from input to prediction. A minimal network: 2 inputs, 2 hidden neurons with ReLU, 1 sigmoid output. A customer with scaled features: recency = 1.0 (high), trend = −0.5 (falling).

Hidden layer (made-up but plausible weights):

• Neuron h1 (weights 1.2 and −1.0, bias −0.2), the "customer fading out" detector: $z_1 = 1.2(1.0) + (-1.0)(-0.5) - 0.2 = 1.2 + 0.5 - 0.2 = 1.5 \Rightarrow a_1 = \text{ReLU}(1.5) = 1.5$
• Neuron h2 (weights −0.8 and 1.5, bias 0.1), the "customer reactivating" detector: $z_2 = -0.8(1.0) + 1.5(-0.5) + 0.1 = -0.8 - 0.75 + 0.1 = -1.45 \Rightarrow a_2 = \text{ReLU}(-1.45) = 0$

Output layer (weights 1.1 and −1.3, bias −0.4):

$z_s = 1.1(1.5) + (-1.3)(0) - 0.4 = 1.25 \Rightarrow \sigma(1.25) = \frac{1}{1+e^{-1.25}} \approx 0.78$

Prediction: 78% probability of churn. Read the story the numbers tell: the fade-out detector fired strongly (1.5), the reactivation detector went completely silent (the ReLU muted it), and the output translated that balance into a probability with the sigmoid of 04-02. Eleven parameter values, and you can already see the mechanics that, at the scale of thousands of neurons, recognizes faces or translates languages.

Where do the weights come from? From the same loop as 04-01: define a cost function (log-loss, as in 04-02) and go down its slope. The novelty is computing that slope when several layers stand between each weight and the error. Enter backpropagation:

1. Forward pass: with the current weights, compute the prediction and its error.
2. Backward pass: propagate the error backwards, layer by layer, assigning each weight its share of responsibility (calculus's chain rule, applied systematically): "the output overshot by 0.3; h1 contributed this much, so h1's weights must be corrected by that much...".
3. Update: each weight takes a small step against its gradient, with the learning rate from 04-01.
4. Repeat with more data, over many epochs (complete passes through the dataset), until convergence.

A crucial difference from 04-01 and 04-02: a network's error landscape is not a bowl — it has multiple valleys, plateaus and ravines. Gradient descent no longer guarantees the global minimum: training twice with different seeds gives different networks (that's why weights are initialized at random and random_state matters). In practice it works remarkably well, with descent variants (Adam, sklearn's default) that adapt the step size along the way.

Networks demand scaled features (gradient descent struggles with mismatched scales — the same reason as in 04-02, aggravated): once again, the 03-06 preprocessor to the rescue:

Reading the code:

• hidden_layer_sizes=(16, 8): two hidden layers of 16 and 8 neurons. For small tabular data, start modestly: one or two layers of 8–32 neurons. Every neuron adds weights to fit and capacity to memorize.
• early_stopping=True: internally sets aside 10% of the train set as validation and stops training when that validation stops improving — the simplest antidote against overfitting from too many epochs (the full logic, in 06-05).
• max_iter=2000: the epoch cap. If a non-convergence warning appears, raise it or check the scaling.
• The parameter counter puts things in perspective: our small network already handles several hundred weights, against the dozen or so of logistic regression. More capacity, more appetite for data and more risk of memorizing — with 800 customers, the edge over simple models may be zero or negative, and that too is a lesson.

On this very churn problem, the network commonly ties with or narrowly beats logistic regression: our 03-06 features already hand-encode much of the useful nonlinearity (ratios, trend, interaction). Networks shine when that engineering is not done or not possible.

• It adds value: complex nonlinear relationships with abundant data (tens of thousands of rows and up); problems where the useful features are combinations nobody knows how to design; non-tabular data — images, audio, sequential text — where the specialized architectures have no rival.
• It doesn't (or subtracts): small or medium tabular datasets with good features, where logistic regression, trees or their ensembles (07-02, 07-03) perform as well or better at a fraction of the cost and the mystery; contexts that demand explaining every decision (an MLP produces neither readable rules nor odds ratios).

What this lesson has shown is the minimal network. The architectures that justify the hype — deep networks with dozens of layers, convolutional (CNN) for images, recurrent (RNN) for sequences, transformers for language — along with the techniques that make them trainable, are covered in the deep learning lesson (07-04). Everything essential, however, you already have: they are neurons, layers, forward pass and backpropagation, at another scale.

The decision map for the whole module:

How to use the table in a real MercaFresh-style project: start with the interpretable baseline (logistic for classification, linear for regression); put a tree next to it for its rules and its tolerance of minimal preprocessing; if you suspect curved boundaries and the dataset is manageable, try an RBF SVM and a small network; use K-NN as a probe of your features' quality and Naive Bayes when speed or data scarcity rule. And remember that the two most powerful cells in tabular ML — tree ensembles and boosting — arrive in 07-02 and 07-03, built on what you already know.

• Starting the project with the neural network. It's the model with the most dials, the biggest appetite for data and the fewest explanations. It earns its right to exist only if it beats logistic regression and the tree in validation — most of the time, on tabular data, it doesn't.
• Training without scaling. Gradient descent zigzags and fails to converge; the max_iter warning is usually a symptom of this, not of too few epochs. The Pipeline with 03-06 prevents it.
• Ignoring the randomness of initialization. Two training runs with different seeds give different results; fix random_state to reproduce, and if the result varies a lot between seeds, be suspicious: the model is at the limit of its data.
• Interpreting hidden neurons as guaranteed concepts. "h1 detects the customer fading out" is a plausible after-the-fact reading, not a verified property; the internal representations of a real network are distributed and entangled.
• Tip: always time it and compare against the module's table. One extra point of accuracy can cost 100 times more training, memory and opacity; make the decision a conscious one — and with the right metrics, which is exactly what module 6 brings.

Exercise 1. Repeat the forward pass of the numeric example for a healthy customer: recency = −0.8, trend = 1.2 (scaled). Use the same weights. Which hidden neuron fires now, and what churn probability comes out?

Exercise 2. With the lesson's Pipeline, train three networks with hidden_layer_sizes = (2,), (16, 8) and (128, 64, 32), and compare accuracy on train and test. Relate the pattern to the number of parameters and to what you saw in exercise 2 of 04-03.

Exercise 3. No code: the output layer of our binary MLP is one neuron with a sigmoid over the hidden activations. Explain in what exact sense the MLP "is a logistic regression over learned features", and what that implies about the network's decision boundary in the space of hidden activations versus the space of the original features.

Exercise 1

• h1: $z_1 = 1.2(-0.8) + (-1.0)(1.2) - 0.2 = -0.96 - 1.2 - 0.2 = -2.36 \Rightarrow a_1 = 0$ (the ReLU mutes it).
• h2: $z_2 = -0.8(-0.8) + 1.5(1.2) + 0.1 = 0.64 + 1.8 + 0.1 = 2.54 \Rightarrow a_2 = 2.54$.
• Output: $z_s = 1.1(0) + (-1.3)(2.54) - 0.4 = -3.70 \Rightarrow \sigma(-3.70) \approx 0.024$.

A 2.4% churn probability: the detectors have swapped roles — now the "fading out" one is silent and the "reactivating" one shouts, and its negative output weight (−1.3) pushes the probability down. The same network, without changing a single weight, responds coherently to opposite profiles: that is what the layers buy you.

Exercise 2

Expected pattern: (2,) falls short (few parameters, mediocre train and test — underfitting); (16, 8) gives the best test score; (128, 64, 32) — thousands of parameters for 640 train rows — sends train soaring while test stalls or worsens: memorization. It's exactly the curve of exercise 2 in 04-03 with max_depth, and gamma's in 04-04: every model family has its complexity dial (depth, gamma, K, layers) and they all draw the same U. That universal pattern has a name and a treatment: overfitting/underfitting, in 06-05.

Exercise 3

The output neuron computes $\sigma(w \cdot \mathbf{a} + b)$ where $\mathbf{a}$ are the hidden activations: that is literally the logistic regression of 04-02, with $\mathbf{a}$ playing the role of the features. The difference is that $\mathbf{a}$ doesn't come from the ColumnTransformer: the hidden layers learned it during training. Geometric implication: in activation space, the network's boundary is a hyperplane (logistic regression guarantees it); but since the features→activations transformation is nonlinear, that hyperplane, seen from the original space, is an arbitrary curved surface. It's the same move as the SVM's kernel (04-04) — separating linearly in a transformed space — with one capital difference: the kernel fixes the transformation in advance; the network learns it from the data.

You've taken the neural network apart down to its pieces — neurons that combine, bias and activate; layers that compose detectors; a forward pass you can compute by hand; backpropagation assigning blame backwards so that the gradient descent of 04-01 can adjust each weight — and you've trained it on churn with the same Pipeline as always. You know where it belongs: maximum flexibility, maximum appetite for data, minimum transparency, and its deep version (CNN, RNN, transformers) waiting for you in 07-04.

With it the module closes: seven algorithms, seven central ideas, and a comparison table that is your map for choosing. But notice what all seven demanded without exception: a y column — the label saying which customer churned, how much they spent. And when it doesn't exist? MercaFresh has no column saying which segment each customer belongs to, because the segments are precisely what it wants to discover. Learning structure from unlabeled data is unsupervised learning, and module 5 opens it with its emblematic algorithm: K-means — whose name, at last, you will no longer confuse with this week's K-NN.