The Project | About Us | Contribute | Donations | License
HOME My courses Completed
◄ Previous
◄

Neural Networks and Deep Learning

Next ►
►

We closed 05-03 with a confessed limitation: to capture Rutalia's nonlinear truth we had to manufacture the features by hand (polynomials, interactions). Neural networks eliminate that work: they also learn the transformations. The good news is that you already have the two pieces they are made of: the linear combination + sigmoid of logistic regression (05-02) is exactly a neuron, and gradient descent (05-03) is exactly their training algorithm. The only genuinely new thing is backpropagation: how to compute the gradient when there are stacked layers — the chain rule turned into an algorithm. In this lesson we will build a network from scratch: a forward pass computed by hand with concrete numbers, a guided derivation of backprop on a minimal network, and a complete implementation in pure numpy trained on the canonical dataset to predict delays.

Contents

  1. From the neuron to the multilayer network
  2. Why nonlinear activations are necessary
  3. Activation functions
  4. Forward pass by hand: one concrete delivery
  5. Backpropagation: the chain rule turned into an algorithm
  6. Complete implementation in numpy
  7. Training: epochs, batches and loss curves
  8. What deep learning is and when it pays off
  9. Frameworks: what PyTorch and Keras automate

From the neuron to the multilayer network

An artificial neuron is the logistic regression of 05-02: it receives inputs, computes a linear combination and applies an activation function.

output = g(w·x + b)          — with g = sigmoid, it IS logistic regression

The structural novelty: stacking neurons in layers. Each neuron in a layer receives the outputs of all the neurons in the previous layer:

flowchart LR
    subgraph E["Input (features)"]
    x1[distance_km]
    x2[departure_hour]
    end
    subgraph O["Hidden layer"]
    h1((h1))
    h2((h2))
    end
    subgraph S["Output"]
    o((p_delayed))
    end
    x1 --> h1 & h2
    x2 --> h1 & h2
    h1 --> o
    h2 --> o

The key interpretation: the hidden layer learns intermediate features. In 05-03, we invented the distance × rush_hour term ourselves; here, a hidden neuron can learn to activate precisely for "long delivery at rush hour" — the network manufactures its own interactions. The weights of each layer are grouped into a matrix, and the whole forward pass is matrix multiplications: that is why numpy (and GPUs) are the natural habitat of networks.

Why nonlinear activations are necessary

Exam question: what happens if we remove the activations (g = identity)? Let's chain two linear layers:

h = W1·x + b1
y = W2·h + b2 = W2·(W1·x + b1) + b2 = (W2·W1)·x + (W2·b1 + b2)

The result is W'·x + b': another linear function. No matter how many layers you stack, a network without activations collapses exactly to the linear regression of 05-03 — hundreds of parameters to express the same thing as a straight line. The nonlinearity between layers is what prevents the collapse and gives the network its power: with enough hidden neurons and a nonlinear activation, a network can approximate any continuous function (the universal approximation theorem). Being able to represent it does not guarantee being able to learn it, but it opens the door.

Activation functions

Function Formula Range Derivative Typical use
Sigmoid 1/(1+e⁻ᶻ) (0, 1) σ(z)·(1−σ(z)) Output layer in binary classification (it's a probability)
Tanh tanh(z) (−1, 1) 1−tanh²(z) Hidden layers (historical); zero-centered
ReLU max(0, z) [0, ∞) 1 if z>0, else 0 Hidden layers (the modern standard)

Why did ReLU dethrone the sigmoid in the hidden layers? Because of their derivatives. The sigmoid's derivative peaks at 0.25 and drops to nearly 0 when |z| is large (the neuron saturates): when chaining layers, the gradients multiply and vanish — the early layers stop learning. ReLU's derivative is 1 across its entire active zone: the gradient traverses the layers without shrinking, and it also costs a comparison instead of an exponential. Rule of thumb: ReLU in the hidden layers, sigmoid at the binary output.

Forward pass by hand: one concrete delivery

Nothing replaces doing the arithmetic once. Minimal network: 2 inputs (standardized), 2 hidden neurons with ReLU, 1 sigmoid output. Take a delivery like C-1042: far away and at rush hour → after standardizing, x = (1.0, 1.5) (distance, hour). Current weights (as a half-finished training run might have left them):

W1 = [[ 0.8, -0.4],      b1 = [0.1, 0.2]
      [ 0.5,  0.9]]
W2 = [ 1.2, -0.7]        b2 = -0.3

Step by step:

Hidden layer (pre-activation):
  z1 = 0.8·1.0 + 0.5·1.5 + 0.1 = 1.65     → h1 = ReLU(1.65) = 1.65
  z2 = -0.4·1.0 + 0.9·1.5 + 0.2 = 1.15    → h2 = ReLU(1.15) = 1.15

Output:
  z3 = 1.2·1.65 + (-0.7)·1.15 + (-0.3) = 0.875
  p  = σ(0.875) = 1/(1+e^-0.875) ≈ 0.706

The network estimates a 70.6% probability of delay for this delivery. In code, the whole computation is:

relu = lambda z: np.maximum(0, z)
x = np.array([1.0, 1.5])
h = relu(W1.T @ x + b1)          # hidden layer: (2,)
p = sigmoid(W2 @ h + b2)         # output: scalar, 0.706

If the delivery really did arrive late (y = 1), the error is p − y = −0.294: the network fell short. The million-dollar question: how much should each one of the 9 parameters (4+2 in the hidden layer, 2+1 at the output) change to reduce that error? Answering it is backpropagation.

Backpropagation: the chain rule turned into an algorithm

In 05-03, with a single level (ŷ = w·x + b), differentiating the cost with respect to each weight was direct. Now W1 affects the cost through h, which affects it through z3, which affects it through p. The tool is the chain rule: the derivative of a composition is the product of the derivatives of the links. Backpropagation is simply organizing it so as not to repeat computations: you compute the gradient of the last layer and propagate it backwards, reusing at each layer what was already computed in the next one.

Let's derive it, guided, on the minimal network, with the cross-entropy cost L = −[y·ln p + (1−y)·ln(1−p)] (the standard in classification; its gradient is cleaner than the MSE's here).

Link 1 — from the loss to z3. Composing the derivative of L with respect to p with that of the sigmoid (σ' = σ(1−σ)) produces a memorable cancellation:

δ3 = ∂L/∂z3 = p − y = 0.706 − 1 = −0.294

The output's "error message" is, literally, the prediction error. (The same form the gradient had in linear regression: no coincidence — cross-entropy is chosen precisely for that.)

Link 2 — output layer gradients. z3 = W2·h + b2, so each weight of W2 receives the error scaled by the activation that flowed through it:

∂L/∂W2 = δ3 · h = (−0.294·1.65, −0.294·1.15) = (−0.485, −0.338)
∂L/∂b2 = δ3 = −0.294

Link 3 — propagating the error to the hidden layer. How much of the blame does each hidden neuron carry? The share transmitted by its weight toward the output, filtered by the derivative of its ReLU (1 here, because both z values were positive; a "switched off" neuron would receive no blame):

δ1 = δ3 · W2[0] · ReLU'(z1) = −0.294 · 1.2 · 1 = −0.353
δ2 = δ3 · W2[1] · ReLU'(z2) = −0.294 · (−0.7) · 1 = +0.206

Note the sign of δ2: since its output weight is negative (−0.7), h2 is better off going down so that p goes up. The network distributes blame with a sign.

Link 4 — hidden layer gradients. Same pattern as link 2, one level down: ∂L/∂W1[i,j] = δj · xi, ∂L/∂b1 = δ.

And with all the gradients, one descent step from 05-03 as-is: W ← W − α·∂L/∂W. That is all of backpropagation: forward while storing the activations, backward multiplying errors by weights and activation derivatives, layer by layer going backwards. Computationally it is dynamic programming over the computation graph (01-03): each δ is computed once and reused for all the gradients in its layer; the backward pass costs the same as the forward, O(number of weights), where a naive application of the chain rule would cost far more.

Complete implementation in numpy

Let's generalize to matrices and the full canonical dataset (13 features, hidden layer of 16, binary output):

class NeuralNetwork:
    """2-layer network (ReLU hidden + sigmoid output) in pure numpy."""

    def __init__(self, d_input, d_hidden, seed=42):
        rng = np.random.default_rng(seed)
        # Small random initialization: breaking symmetry is essential
        # (with equal weights, all the neurons would learn the same thing)
        self.W1 = rng.normal(0, np.sqrt(2 / d_input), (d_input, d_hidden))
        self.b1 = np.zeros(d_hidden)
        self.W2 = rng.normal(0, np.sqrt(2 / d_hidden), (d_hidden, 1))
        self.b2 = np.zeros(1)

    def forward(self, X):
        """Predicts and STORES the activations (backprop will need them)."""
        self.Z1 = X @ self.W1 + self.b1        # (n, 16) pre-activation
        self.H = np.maximum(0, self.Z1)        # (n, 16) ReLU
        self.P = sigmoid(self.H @ self.W2 + self.b2)  # (n, 1) probability
        return self.P

    def backward(self, X, y, alpha=0.1):
        """Backpropagation + one gradient descent step."""
        n = len(y)
        # Link 1: error at the output (cross-entropy + sigmoid)
        d3 = self.P - y.reshape(-1, 1)                   # (n, 1)
        # Link 2: output layer gradients
        gW2 = self.H.T @ d3 / n                          # (16, 1)
        gb2 = d3.mean(axis=0)
        # Link 3: propagate the error to the hidden layer
        d1 = (d3 @ self.W2.T) * (self.Z1 > 0)            # (n, 16); ReLU' = mask
        # Link 4: hidden layer gradients
        gW1 = X.T @ d1 / n                               # (13, 16)
        gb1 = d1.mean(axis=0)
        # Descent step (05-03, unchanged)
        self.W2 -= alpha * gW2; self.b2 -= alpha * gb2
        self.W1 -= alpha * gW1; self.b1 -= alpha * gb1

    def loss(self, X, y):
        p = np.clip(self.forward(X), 1e-9, 1 - 1e-9)     # avoids log(0)
        y = y.reshape(-1, 1)
        return float(-(y * np.log(p) + (1 - y) * np.log(1 - p)).mean())

Every line of backward is one of the links we derived by hand, vectorized for n examples at once. The mask (self.Z1 > 0) is ReLU's derivative applied in bulk: switched-off neurons propagate no blame.

Training: epochs, batches and loss curves

The training loop is the mini-batch loop of 05-03, literally:

# Canonical dataset, delayed target, 05-01 pipeline
net = NeuralNetwork(d_input=X_tr_s.shape[1], d_hidden=16)
rng = np.random.default_rng(0)
history = {"train": [], "test": []}

for epoch in range(300):
    order = rng.permutation(len(y_tr))                 # shuffle every epoch
    for i in range(0, len(y_tr), 64):                  # mini-batches of 64
        chunk = order[i:i + 64]
        net.forward(X_tr_s[chunk])
        net.backward(X_tr_s[chunk], y_tr[chunk], alpha=0.1)
    history["train"].append(net.loss(X_tr_s, y_tr))
    history["test"].append(net.loss(X_te_s, y_te))

pred = (net.forward(X_te_s) > 0.5).astype(int).ravel()
print(f"numpy network accuracy: {(pred == y_te).mean():.3f}")

Vocabulary you already know from 05-03, now in its definitive habitat:

  • Epoch: one complete pass through the training set. Here 300; networks need many.
  • Mini-batch: 64 deliveries per gradient step — the noise/cost balance from 05-03.
  • Loss curve: history["train"] and history["test"] are your electrocardiogram. If both go down, all is well; if train goes down and test goes up, the network has started memorizing — that is the moment to stop (early stopping: the temporal version of the pruning from 05-02).

On this problem the network should match or slightly beat the random forest of 05-02, capturing the multiplicative structure without feature engineering. And an important warning: unlike the linear MSE of 05-03, a network's loss surface is not convex — there are local minima and plateaus, initialization matters (that is why we break symmetry with random values) and two training runs with different seeds may yield different networks. Mini-batch noise, curiously, helps escape the bad corners.

What deep learning is and when it pays off

Deep learning = networks with many hidden layers, where each layer learns representations on top of the previous layer's (pixels → edges → shapes → objects). Depth composes transformations, and that is exponentially expressive. But it is not free, and for data like Rutalia's it is rarely the first option:

Scenario Data Recommended model Why
Small/medium tabular (our dataset) 10³-10⁵ rows Random forest / gradient boosting / small network Trees perform as well or better, without fine-tuning and cheaper
Massive tabular with complex interactions >10⁶ rows Medium networks compete The volume feeds the capacity
Images (damaged package in the photo?) Many CNN Convolutions: they exploit spatial structure
Sequences/time series (hourly demand) Many RNN / transformers They exploit temporal order
Text (courier incident notes) Many Transformers Attention over context; the basis of LLMs

CNNs, RNNs and transformers are specialized architectures: the same machinery (neurons, activations, backprop, gradient descent) with connections structured for each data type. We do not develop them here — with what you have learned, you have the exact foundation they are built on.

A practical cost warning: training deep learning seriously is computationally expensive — hours or days of GPU time, a hyperparameter search (architecture, α, batch, epochs...) that multiplies the training runs, and a real energy and financial bill. Our 16-neuron network trains in seconds on a CPU; a vision model is orders of magnitude more. Before deploying deep learning at Rutalia, the mandatory question is whether a random forest, trained in minutes and interpretable, doesn't already deliver 95% of the benefit.

Frameworks: what PyTorch and Keras automate

In professional practice nobody writes the backward pass by hand: you use PyTorch or Keras/TensorFlow. What they automate, in order of importance:

  • Autograd (automatic differentiation): you define only the forward; the framework records the operation graph and generates the backward on its own. Autograd does for you, for any architecture, exactly the backprop you just wrote — which is why it was worth writing once: now you know what is inside the box.
  • GPU: the matrix multiplications are dispatched to massively parallel hardware by changing one line (.to("cuda")).
  • Advanced optimizers: Adam, momentum... — gradient descent with a per-parameter adaptive learning rate; the loop is the same.
  • Prefabricated layers: convolutions, attention, normalization, dropout — the bricks of the architectures in the table above.

Our complete network, in Keras, so you can see the correspondence:

# model = keras.Sequential([
#     keras.layers.Dense(16, activation="relu"),     # our W1, b1 + ReLU
#     keras.layers.Dense(1, activation="sigmoid"),   # our W2, b2 + sigmoid
# ])
# model.compile(optimizer="adam", loss="binary_crossentropy")
# model.fit(X_tr_s, y_tr, epochs=300, batch_size=64)   # our loop

Four lines that encapsulate everything we have built. Use them — but now knowing what they execute.

Common Mistakes and Tips

  • Forgetting the nonlinear activations. A deep network without them is an expensive linear regression (we proved it algebraically). If your network doesn't beat the linear model, check that the activations are there.
  • Initializing the weights to zero. Every neuron in a layer then receives the same gradient and learns the same thing forever (unbroken symmetry). Small random initialization, always.
  • Not storing the forward activations. The backward pass needs H and Z1; recomputing them doubles the cost, not having them makes backprop impossible. It is the same principle as the memoization from 01-03.
  • log(0) in the cross-entropy. When the sigmoid saturates, p reaches 0.0 or 1.0 in floating point and the loss returns NaN. np.clip(p, 1e-9, 1-1e-9) saves your training run.
  • Unstandardized data. Everything said in 05-03 about gradient descent applies, multiplied: large inputs saturate the sigmoids and unbalance the gradients from the very first epoch.
  • Tip: always monitor BOTH loss curves (train and test). Train going down + test going up = overfitting happening live: stop right there (early stopping). It is the cheapest and most reliable diagnostic there is.

Exercises

  1. The linear collapse, verified. Modify NeuralNetwork so the hidden layer uses the identity instead of ReLU (self.H = self.Z1, and in the backward remove the mask). Train both versions for 300 epochs and compare test accuracy with sklearn's logistic regression. Do you confirm that the "linear" network doesn't beat logistic regression while the ReLU one does?

  2. One backprop step by hand. With the minimal network from the manual forward pass (given weights, x=(1.0, 1.5), y=1) and α=0.5, compute the 9 gradients by hand, update the parameters and run the forward again. Check that the new p is greater than 0.706 (the network corrects toward the right class).

  3. Loss curves and induced overfitting. Train the network with d_hidden=128 on only the first 200 training deliveries for 2000 epochs, logging both losses every 50. Locate the epoch from which the test loss starts rising while the train loss keeps falling, and apply early stopping: what accuracy does the network stopped at that point achieve versus the network trained to the end?

Solutions

Exercise 1:

# In forward:  self.H = self.Z1
# In backward: d1 = d3 @ self.W2.T        (without the (Z1 > 0) mask)

The identity version stalls at an accuracy indistinguishable from LogisticRegression() — as we proved, W2·(W1·x+b1)+b2 is a linear function of x, so its decision boundary is a hyperplane, just like logistic regression's. The ReLU version beats it by capturing the interactions (distance×hour, congested zone) that the hyperplane cannot express.

Exercise 2:

δ3 = p − y = −0.294
∂L/∂W2 = δ3·h = (−0.485, −0.338)      ∂L/∂b2 = −0.294
δ1 = δ3·1.2·1 = −0.353                δ2 = δ3·(−0.7)·1 = 0.206
∂L/∂W1 = [[δ1·x1, δ2·x1], [δ1·x2, δ2·x2]]
       = [[−0.353, 0.206], [−0.529, 0.309]]
∂L/∂b1 = (−0.353, 0.206)

Update (θ ← θ − 0.5·g):
W2 → (1.443, −0.531)   b2 → −0.153
W1 → [[0.976, −0.503], [0.765, 0.745]]   b1 → (0.276, 0.097)

New forward: z1 = 0.976 + 0.765·1.5 + 0.276 = 2.400 → h1 = 2.400
             z2 = −0.503 + 0.745·1.5 + 0.097 = 0.712 → h2 = 0.712
             z3 = 1.443·2.400 − 0.531·0.712 − 0.153 = 2.933
             p = σ(2.933) ≈ 0.949 > 0.706 ✓

A single step and the probability of delay climbs from 0.706 to ≈0.95: every parameter moved in the exact direction the chain rule assigned it. (With α=0.5 the step is large and didactic; in real training, smaller steps and many examples at once.)

Exercise 3:

net = NeuralNetwork(X_tr_s.shape[1], d_hidden=128)
Xp, yp = X_tr_s[:200], y_tr[:200]
for epoch in range(2000):
    net.forward(Xp); net.backward(Xp, yp, alpha=0.1)
    if epoch % 50 == 0:
        print(epoch, round(net.loss(Xp, yp), 3),
              round(net.loss(X_te_s, y_te), 3))

With 128 neurons and only 200 examples, the network has capacity to spare for memorizing: the train loss falls toward 0 indefinitely, while the test loss reaches its minimum relatively early and then climbs back up — the U-shaped curve from 05-01, now drawn along the training time axis. The network stopped at the test minimum (early stopping) yields better accuracy than the one trained for 2000 epochs: training more is not training better.

Conclusion

There is no black box anymore: a neural network is stacked logistic regressions with nonlinear activations (without them, an algebraic collapse to the linear model of 05-03), trained with the gradient descent of 05-03 and with the gradients computed by backpropagation — the chain rule organized as dynamic programming over the computation graph, propagating blame layer by layer backwards. You computed it by hand, vectorized it in numpy, trained it with mini-batches on the canonical dataset and watched overfitting draw itself on the loss curves. You also know when not to use it: on tabular data like Rutalia's, a random forest usually puts up a fight at a fraction of the cost, and deep architectures (CNNs, RNNs, transformers) shine on images, sequences and text, where PyTorch/Keras autograd will do for you the backward pass you now understand from the inside. This closes supervised learning. The module's last frontier remains: what if there is no label? Nobody has marked which zones of the city "behave alike" — it has to be discovered. In 05-05, clustering: unsupervised learning, where we will also close a circle we left open in module 3 with Kruskal's MST.

◄ Previous
◄
Next ►
►
The Project | About Us | Contribute | Donations | License
© Copyright 2026. All rights reserved