The Project | About Us | Contribute | Donations | License

HOME

In the previous lesson you opened the hood of TensorFlow and discovered that model.fit() is, deep down, a packaged GradientTape loop. Today you will meet the framework that chose not to package that loop: PyTorch. Created by Meta (Facebook) and released in 2017, PyTorch is now the de facto standard in research — the vast majority of the papers you will read publish their code in PyTorch — and a first-class option in industry. Its philosophy is to give you explicit control over every step. To prove that you already know almost everything you need, in this lesson we will reproduce the same gradient example from 06-01 and rewrite the 784-128-64-10 MNIST network from 02-05 in PyTorch, training it until we can compare results with the Keras version. The TecnoMarket team wants to speak both languages; so should you.

What PyTorch is and its philosophy
PyTorch tensors: creation, operations and devices
Autograd: PyTorch's GradientTape
Building models with nn.Module
The explicit training loop, line by line
Evaluation with torch.no_grad()
Training the MNIST network and comparing with Keras

What PyTorch is and its philosophy

PyTorch was born in Meta's research labs as the "Pythonic" successor to Torch (a framework written in Lua). Three ideas define its character:

Define-by-run (dynamic definition): the operation graph is built while the Python code runs, not beforehand. Every forward pass can be different: you can use if statements, for loops, print intermediate tensors with a simple print. It feels like numpy with gradients.
Explicit control: no fit(). You write the training loop yourself. More lines of code, yes, but zero magic: when something breaks, you see exactly where.
De facto standard in research: if module 5 made you curious about transformers and LLMs, almost all their open-source code (including Hugging Face's transformers library) is built on PyTorch.

Philosophy	Keras/TensorFlow	PyTorch
Implicit motto	"The common case should be easy"	"Everything should be explicit"
Training loop	Packaged in `fit()`	You write it
How it feels	Driving an automatic	Driving a manual

Installation (you already did it in 01-05 with pip install torch; add torchvision for the vision datasets):

import torch
print(torch.__version__)              # e.g. 2.x
print(torch.cuda.is_available())      # True if an NVIDIA GPU is set up

PyTorch tensors: creation, operations and devices

PyTorch tensors are conceptually identical to TensorFlow's. The syntax changes, not the idea:

import torch

scalar = torch.tensor(4.99)                        # price of a USB cable
vector = torch.tensor([120.0, 15.5, 899.0])        # prices of 3 products
matrix = torch.rand(2, 3)                          # uniform random, shape (2,3)
zeros  = torch.zeros(3, 3)
print(vector.shape)    # torch.Size([3])
print(vector.dtype)    # torch.float32  (same default as TF)

Operations, with their equivalences:

Operation	TensorFlow (06-01)	PyTorch
Matrix product	`tf.matmul(a, b)`	`torch.matmul(a, b)` or `a @ b`
Mean	`tf.reduce_mean(a)`	`a.mean()`
Reshape	`tf.reshape(a, (4,1))`	`a.reshape(4, 1)` or `a.view(4, 1)`
Type conversion	`tf.cast(a, tf.float32)`	`a.float()` or `a.to(torch.float32)`
To numpy	`a.numpy()`	`a.numpy()` (identical)

Broadcasting works with the same rules you saw in 06-01: equal dimensions or size-1 dimensions are compatible.

The bridge to numpy

import numpy as np

arr = np.array([1.0, 2.0, 3.0])
t = torch.from_numpy(arr)     # numpy -> tensor (they share memory!)
arr2 = t.numpy()              # tensor -> numpy (also shares memory)

t[0] = 99.0
print(arr[0])                 # 99.0 -> changing one changes the other

Sharing memory is efficient (no copy involved) but it can surprise you: if you do not want the link, use torch.tensor(arr) (a copy) instead of from_numpy.

Devices: .to(device)

In PyTorch moving data between CPU and GPU is explicit (in TensorFlow it was automatic). The universal pattern you will see in all PyTorch code:

device = "cuda" if torch.cuda.is_available() else "cpu"

x = torch.rand(32, 784)
x = x.to(device)              # move the tensor to the GPU (or stay on CPU)
# model.to(device)            # models are moved the same way

Golden rule: model and data must be on the same device, or PyTorch will throw an error like Expected all tensors to be on the same device. It is one of the most common beginner mistakes.

Autograd: PyTorch's GradientTape

PyTorch also records operations to differentiate automatically, but with no explicit tape: it is enough to flag a tensor with requires_grad=True and all operations on it get recorded. Let's reproduce the same example from 06-01 — minimizing L(w) = (w - 4)² — to compare the two styles side by side:

import torch

w = torch.tensor(0.0, requires_grad=True)   # ~ tf.Variable(0.0)

loss = (w - 4.0) ** 2       # the graph builds itself, no "with tape"
loss.backward()             # ~ tape.gradient(loss, w): differentiate backwards
print(w.grad)               # tensor(-8.) -> the same -8.0 as in 06-01

Key differences from GradientTape:

There is no with context: any operation on a tensor with requires_grad=True is always recorded.
backward() is called on the loss, and gradients appear in the .grad attribute of each leaf tensor.
Gradients accumulate: if you call backward() twice without clearing, .grad holds the sum. That is why every PyTorch loop clears gradients at each step.

The complete gradient descent, twin of the one in 06-01:

w = torch.tensor(0.0, requires_grad=True)
lr = 0.1

for step in range(30):
    loss = (w - 4.0) ** 2
    loss.backward()                  # computes dL/dw and leaves it in w.grad
    with torch.no_grad():            # update WITHOUT recording this operation
        w -= lr * w.grad             # w = w - lr * gradient
    w.grad.zero_()                   # clear! otherwise it would accumulate

print(w.item())   # ~3.995: converges to 4, just like numpy (02-03) and TF (06-01)

Three versions of the same algorithm, three levels of automation:

Version	Derivative	Update	Loop
numpy (02-03)	By hand	By hand	By hand
TensorFlow (06-01)	Automatic (tape)	By hand (`assign_sub`)	By hand
PyTorch (today)	Automatic (autograd)	By hand (later handled by `optimizer.step()`)	By hand
Keras `fit()`	Automatic	Automatic	Automatic

Building models with nn.Module

In PyTorch models are always built with subclassing — the third API you saw in 06-01 is the standard here. You inherit from nn.Module, declare the layers in __init__ and connect them in forward(). Let's rewrite the 784-128-64-10 MNIST network from 02-05:

import torch
from torch import nn

class MNISTNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.layer1 = nn.Linear(784, 128)        # ~ Dense(128) on a 784 input
        self.layer2 = nn.Linear(128, 64)         # ~ Dense(64)
        self.output_layer = nn.Linear(64, 10)    # ~ Dense(10)
        self.relu = nn.ReLU()

    def forward(self, x):
        x = self.relu(self.layer1(x))
        x = self.relu(self.layer2(x))
        return self.output_layer(x)              # logits WITHOUT softmax (see note)

model = MNISTNet()
print(model)   # prints the structure, like summary() in Keras

Important observations:

nn.Linear(784, 128) requires explicit input and output sizes; Keras inferred the input from the data. More typing, fewer surprises.
There is no softmax on the output. In PyTorch, the loss function nn.CrossEntropyLoss expects raw logits (it applies log-softmax internally for numerical stability). It is the exact equivalent of compiling in Keras with SparseCategoricalCrossentropy(from_logits=True). Pairing softmax with CrossEntropyLoss is a classic mistake that degrades training.
You never call forward() directly: you call the model, model(x), and PyTorch invokes forward with the necessary hook machinery.
Counting parameters: sum(p.numel() for p in model.parameters()) → 109,386, exactly the same as the Keras version from 02-05 (784·128+128 + 128·64+64 + 64·10+10).

The explicit training loop, line by line

Here is the big difference from Keras. First, the data with DataLoader (the twin of tf.data):

from torch.utils.data import DataLoader, TensorDataset
from torchvision import datasets, transforms

transform = transforms.ToTensor()   # PIL -> float32 tensor in [0,1]
train_ds = datasets.MNIST(root="data", train=True, download=True, transform=transform)
test_ds  = datasets.MNIST(root="data", train=False, download=True, transform=transform)

train_loader = DataLoader(train_ds, batch_size=32, shuffle=True)   # ~ tf.data's shuffle+batch
test_loader  = DataLoader(test_ds, batch_size=32)

And now the complete loop, commented line by line:

device = "cuda" if torch.cuda.is_available() else "cpu"
model = MNISTNet().to(device)                          # model onto the device

loss_fn = nn.CrossEntropyLoss()                        # ~ sparse_categorical_crossentropy
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)  # ~ optimizer="adam"

for epoch in range(5):                                 # ~ epochs=5 in fit()
    model.train()                                      # training mode (affects dropout/BN)
    for x_batch, y_batch in train_loader:              # ~ fit() iterates batches for you
        x_batch = x_batch.view(x_batch.size(0), -1)    # (32,1,28,28) -> (32,784), the flatten
        x_batch, y_batch = x_batch.to(device), y_batch.to(device)

        logits = model(x_batch)                        # 1. FORWARD: predictions
        loss = loss_fn(logits, y_batch)                # 2. LOSS: how wrong we are

        optimizer.zero_grad()                          # 3. CLEAR accumulated gradients
        loss.backward()                                # 4. BACKWARD: distribute the blame (02-03)
        optimizer.step()                               # 5. UPDATE weights with Adam

    print(f"Epoch {epoch+1}: last batch loss = {loss.item():.4f}")

Breakdown of the five steps at the heart of the loop:

Forward (model(x_batch)): the data flows through the network and the logits come out. It is the forward propagation from 02-03.
Loss (loss_fn(...)): a single number measuring the batch's error, as in 02-04.
optimizer.zero_grad(): zeroes the .grad of every parameter. Remember: autograd accumulates; without this line, each step would add up the gradients of all previous steps and training would diverge.
loss.backward(): backpropagation. Computes the gradient of the loss with respect to each of the 109,386 parameters.
optimizer.step(): Adam updates each parameter using its .grad. It is the w.assign_sub(lr * grad) from 06-01, with Adam's sophistication (02-04).

What Keras automates and what PyTorch exposes

Task	Keras `fit()`	PyTorch
Iterating epochs and batches	Automatic	`for epoch...` / `for x, y in loader`
Moving data to GPU	Automatic	Manual `.to(device)`
Forward	Automatic	`model(x)`
Loss computation	Automatic (you pass it to `compile`)	`loss_fn(logits, y)`
Clearing gradients	Automatic	`optimizer.zero_grad()`
Backward	Automatic	`loss.backward()`
Updating weights	Automatic	`optimizer.step()`
Metrics and progress bar	Automatic	You compute/print them
Train/eval mode (dropout, BN)	Automatic	`model.train()` / `model.eval()`

Neither column is "better": Keras gives you speed for the standard; PyTorch gives you transparency for the non-standard. When in 05-01 you saw the skeleton of GAN training (two networks, two optimizers alternating), that kind of custom loop is where PyTorch's explicit style shines.

Evaluation with torch.no_grad()

When evaluating we are not training: no gradients are needed. torch.no_grad() turns off autograd's recording, saving memory and time:

model.eval()                              # evaluation mode (dropout off, BN frozen)
correct, total = 0, 0

with torch.no_grad():                     # nothing inside gets recorded
    for x_batch, y_batch in test_loader:
        x_batch = x_batch.view(x_batch.size(0), -1).to(device)
        y_batch = y_batch.to(device)
        logits = model(x_batch)
        preds = logits.argmax(dim=1)      # class with the highest logit
        correct += (preds == y_batch).sum().item()
        total += y_batch.size(0)

print(f"Test accuracy: {correct / total:.4f}")

Two different things that often get confused:

model.eval() changes the behavior of certain layers (dropout stops switching neurons off, batch norm uses its running statistics — you saw this in 05-04).
torch.no_grad() turns off gradient computation.

For evaluation you use both. It is the equivalent of what Keras's model.evaluate() does on its own.

Training the MNIST network and comparing with Keras

Running the loop above for 5 epochs (the same budget as in 02-05), a typical result:

	Keras (02-05)	PyTorch (today)
Architecture	784-128-64-10, ReLU	Identical
Parameters	109,386	109,386
Optimizer / loss	Adam / crossentropy	Adam / CrossEntropyLoss
Test accuracy (~5 epochs)	~97.7%	~97.5 – 97.9%
Lines of training code	~5 (`compile` + `fit`)	~25 (explicit loop)

The conclusion the TecnoMarket team writes on their whiteboard: same network, same data, same result. The small decimal differences come from random initialization and batch ordering, not from the framework. What changes is the development experience, and that is exactly what we will compare systematically in the next lesson.

Common Mistakes and Tips

Forgetting optimizer.zero_grad(): the number one mistake. Gradients accumulate, the loss wobbles or explodes, and no error message warns you. If your PyTorch training "does not converge", check this first.
Softmax + CrossEntropyLoss: nn.CrossEntropyLoss already includes the (log) softmax. Adding nn.Softmax at the output silently trains badly. Output = raw logits.
Model and data on different devices: Expected all tensors to be on the same device. Move both with .to(device); the model once, the data on every batch.
Forgetting model.eval() when evaluating: with dropout active at test time, the measured accuracy will be worse than the real one. And the reverse: forgetting model.train() when going back to training.
.item() vs. the tensor: loss is a tensor connected to the graph; to log its value use loss.item(). Accumulating tensors in a list over many epochs retains the graph and exhausts memory.
Tip: when a shape does not add up, take advantage of define-by-run: put print(x.shape) inside forward(). That immediate debugging is one of PyTorch's great pleasures.

Exercises

Exercise 1: gradient by hand and with autograd

Use autograd to compute the gradient of L(w, b) = (3*w + b - 10)**2 at w=1.0, b=0.0 (the same as exercise 2 in 06-01). Verify that you get -42 and -14, identical to the GradientTape version. Then run 200 descent steps with lr 0.01, using torch.no_grad() for the update.

Exercise 2: the review network as an nn.Module

TecnoMarket wants the binary review classifier (positive/negative) as a PyTorch module: an input of 200 features (the review already vectorized, as in 04-03), hidden layers of 64 and 32 with ReLU, and an output of 1 logit. Write the class ReviewClassifier(nn.Module) and state which loss function you would use and why the output has no sigmoid.

Exercise 3: reading the loop

Without running it, explain what would go wrong (and with what symptom) in this loop:

for x, y in train_loader:
    logits = model(x)
    loss = loss_fn(logits, y)
    loss.backward()
    optimizer.step()

Solutions

Solution 1:

import torch

w = torch.tensor(1.0, requires_grad=True)
b = torch.tensor(0.0, requires_grad=True)

loss = (3.0 * w + b - 10.0) ** 2
loss.backward()
print(w.grad.item(), b.grad.item())   # -42.0, -14.0 (identical to 06-01)

lr = 0.01
for _ in range(200):
    loss = (3.0 * w + b - 10.0) ** 2
    if w.grad is not None:
        w.grad.zero_(); b.grad.zero_()   # clear before recomputing
    loss.backward()
    with torch.no_grad():
        w -= lr * w.grad
        b -= lr * b.grad

print(loss.item())        # ~0.0

Same result as with GradientTape: both autograds compute the same math from 02-03.

Solution 2:

from torch import nn

class ReviewClassifier(nn.Module):
    def __init__(self):
        super().__init__()
        self.layer1 = nn.Linear(200, 64)
        self.layer2 = nn.Linear(64, 32)
        self.output_layer = nn.Linear(32, 1)   # 1 logit for binary classification
        self.relu = nn.ReLU()

    def forward(self, x):
        x = self.relu(self.layer1(x))
        x = self.relu(self.layer2(x))
        return self.output_layer(x)

Loss: nn.BCEWithLogitsLoss. Just like CrossEntropyLoss with the softmax, this loss includes the sigmoid internally (with better numerical stability), so the network must return the raw logit. It is the equivalent of BinaryCrossentropy(from_logits=True) in Keras.

Solution 3: Three things are missing:

optimizer.zero_grad(): gradients accumulate step after step; symptom: the loss stops decreasing or diverges with no visible errors.
Flattening and device: if x comes from MNIST with shape (32, 1, 28, 28) and the model expects (32, 784), there will be a shape error in nn.Linear; and if the model is on the GPU and the data is not, a device error.
model.train() before the loop (relevant if the model has dropout/BN).

The correct order of the core is: forward → loss → zero_grad() → backward() → step().

Conclusion

You now speak both languages. PyTorch has shown you the other style: tensors nearly identical to TensorFlow's (with explicit .to(device) movement and a direct bridge to numpy), autograd solving the same gradient as GradientTape via requires_grad + backward(), models as nn.Module classes (the subclassing of 06-01 as the norm, not an option) and, above all, the explicit training loop: forward, loss, zero_grad, backward, step — the five steps Keras packages inside fit(). The MNIST network from 02-05 rewritten in PyTorch reaches the same accuracy: the framework does not change the math, it changes who writes each piece.

The inevitable question is: which one should you use? In the next lesson we will make the honest comparison — philosophy, debugging, ecosystems, industry vs. research — and see what the TecnoMarket team would choose and why, with a reassuring conclusion: the concepts you have learned travel with you, whatever the tool.

Introduction to PyTorch

Contents

What PyTorch is and its philosophy

PyTorch tensors: creation, operations and devices

The bridge to numpy

Devices: .to(device)

Autograd: PyTorch's GradientTape

Building models with nn.Module

The explicit training loop, line by line

What Keras automates and what PyTorch exposes

Evaluation with torch.no_grad()

Training the MNIST network and comparing with Keras

Common Mistakes and Tips

Exercises

Exercise 1: gradient by hand and with autograd

Exercise 2: the review network as an nn.Module

Exercise 3: reading the loop

Solutions

Conclusion

Deep Learning Course

Module 1: Introduction to Deep Learning

Module 2: Neural Network Fundamentals

Module 3: Convolutional Neural Networks (CNN)

Module 4: Recurrent Neural Networks (RNN)

Module 5: Advanced Deep Learning Techniques

Module 6: Tools and Frameworks

Module 7: Hands-On Projects

Module 8: Ethical Considerations and the Future of Deep Learning