The Project | About Us | Contribute | Donations | License

HOME

In 04-03 we flagged a problem by name: in sequence-to-sequence models, the whole input sentence had to be squeezed into a single vector before generating the output — the bottleneck that degrades long sequences, like summarizing a 300-word review on a sticky note and translating from the note. This lesson presents the solution, attention, and the architecture born from taking it to the extreme: the Transformer, which since 2017 has dethroned RNNs and is the foundation of BERT, GPT and today's LLMs — closing the historical arc we opened in 01-02. For TecnoMarket, understanding this piece explains why the practical route for analyzing its reviews is no longer training LSTMs from scratch, but leaning on pretrained language models: the transfer learning of 05-03 applied to text.

The bottleneck, revisited
Attention: look at the whole sequence and weight it
Query, key and value: the catalog analogy
Self-attention step by step with numbers
Multi-head: several spotlights at once
The Transformer architecture at a high level
Why it dethroned RNNs
From BERT and GPT to LLMs
What it means for TecnoMarket
MultiHeadAttention in Keras: a review classifier

The bottleneck, revisited

Recall the seq2seq scheme from 04-03: an RNN encoder reads the sentence word by word and its final hidden state — a single vector — is all the decoder receives to produce the output. The problems:

Forced compression: it doesn't matter whether the input has 5 words or 300; everything must fit in the same fixed-size vector.
Recency bias: what was read at the end is "fresh" in the state; what came at the beginning has been diluted step by step (a relative of the vanishing gradient from 02-03, which the LSTM alleviates with its conveyor belt but does not eliminate over long sequences).
Sequential processing: to represent word 300 you must have processed the previous 299, one by one. Nothing can be parallelized.

The question that led to attention: what if the decoder, instead of depending on a single summary, could look back at the entire input sequence whenever it needs to?

Attention: look at the whole sequence and weight it

That is exactly the idea. With attention, the model keeps one vector for every input position (not just the last one) and, each time it needs to produce or represent something, it computes how much each position matters for what it is doing right now and builds a weighted average.

Intuitively: when translating "not" in a sentence, the model puts almost all its weight on the input's negation word and ignores the rest; when deciding the sentiment of "the battery is terrible but the screen is excellent", it can split its focus between "terrible" and "excellent" without the distance between them mattering. The attention weights are learned — they are the result of training, not hand-written rules — and they are also inspectable: you can visualize what the model attended to, an extra dose of interpretability RNNs never offered.

Query, key and value: the catalog analogy

The modern formulation of attention uses three roles: query (Q), key (K) and value (V). The natural analogy at TecnoMarket is searching its catalog:

You type "cheap espresso coffee maker" into the search box — that is your query (Q).
Every product in the catalog has an indexed record: title, category, tags — that is its key (K).
The search engine compares your query against all the keys and assigns each product a relevance score.
What you take away from each product is not its key but its content: price, description, image — its value (V).
The "result" is a blend dominated by the most relevant products.

Attention does exactly this with vectors: the relevance score is a dot product Q·K (sound familiar? it is the same operation as the cosine similarity we used to compare embeddings in 03-04 and 04-03 — without the normalization), the scores go through a softmax (02-02) that turns them into weights summing to 1, and the output is the weighted sum of the values.

In self-attention, the Transformer's key twist, every word in a sentence plays all three roles at once: each word generates its Q, its K and its V (through three learned weight matrices), fires its query against the keys of every word in the sentence — including itself — and is re-represented as a blend of the values. The result: each word's representation comes to incorporate its context. The word "bank" ends up represented differently in "river bank" and in "online bank", because it attended to different neighbors.

Self-attention step by step with numbers

Let's see it with a minimal three-token phrase from a review: "not working well". We will use toy vectors of dimension 2 and focus on how the word "working" gets re-represented. Suppose that, after applying the learned matrices, we have:

Token	Q	K	V
not	(1, 0)	(1, 0)	(−1, 0)
working	(1, 1)	(0, 1)	(0, 1)
well	(0, 1)	(0, 1)	(1, 1)

Step 1 — scores: the query of "working", Q = (1, 1), is multiplied (dot product) with each token's key:

with "not": (1, 1)·(1, 0) = 1
with "working": (1, 1)·(0, 1) = 1
with "well": (1, 1)·(0, 1) = 1

(In practice you divide by the square root of the dimension for stability — with dim 2, by √2 ≈ 1.41: all end up at ≈ 0.71.)

Step 2 — softmax: the three scores are equal, so softmax spreads the weight evenly: (0.33, 0.33, 0.33). If the score with "not" had been 2 instead of 1 (after scaling, 1.41 versus 0.71), softmax would give ≈ (0.50, 0.25, 0.25): the negation would dominate the blend — this is how "working" finds out that it is being negated.

Step 3 — blending the values: the new representation of "working" is the weighted sum of the Vs:

0.33·(−1, 0) + 0.33·(0, 1) + 0.33·(1, 1) = (0, 0.67)

That is the entire mechanism: dot product → softmax → weighted average. Three operations you already knew separately, computed for all words at once as matrix products — hence its efficiency on GPU. And notice the crucial detail: the distance between words appears nowhere. "not" influences "working" just the same whether it sits next door or 200 words away: the bottleneck and the recency bias have disappeared.

Multi-head: several spotlights at once

A single attention blends all relevance into one distribution of weights. But a sentence hosts different kinds of relationships at once: syntax ("working" relates to its subject), negation ("not" modifies "working"), coreference ("it" refers to "the battery")... The Transformer's solution is multi-head attention: run several independent attentions in parallel — each with its own Q/K/V matrices, each a different "spotlight" — and concatenate their results.

The analogy with CNNs is direct: just as in 03-02 each filter of a convolutional layer specialized in one pattern (vertical edges, a certain texture), each attention head tends to specialize in one type of relationship. Eight or twelve spotlights looking at the same sentence from different angles.

The Transformer architecture at a high level

The paper "Attention Is All You Need" (2017) made the radical bet its title announces: eliminate recurrence entirely and build everything out of attention. The architecture, from a bird's-eye view:

flowchart TB
    T[Sentence tokens] --> E[Embeddings<br/>one per token]
    P[Positional encoding<br/>marks the ORDER] --> S
    E --> S((sum))
    S --> B1[Transformer block 1]
    B1 --> B2[Transformer block 2]
    B2 --> BN[... block N]
    BN --> OUT[Contextual representations<br/>one per token]

    subgraph Block[Each Transformer block]
        A[Multi-head self-attention<br/>each token looks at all] --> FF[Feed-forward network<br/>processes each position]
    end

The pieces, and why almost all of them ring a bell:

Embeddings: the same layer from 04-03 — each token becomes a dense vector.
Positional encoding: removing recurrence loses the notion of order (attention treats the sentence as a set: "dog bites man" = "man bites dog"). The solution is to add to each embedding a vector encoding its position, restoring order to the network.
Stacked blocks (6, 12, 96...): each combines multi-head self-attention (communication between positions) with a small feed-forward network (per-position processing), plus residual connections — the ResNet skip connections we saw in 03-03, the "gradient highway" essential for stacking so many blocks — and layer normalization (a close cousin of the BatchNormalization from 05-04).
No recurrence: nothing waits for anything. All positions are processed in parallel, and that is the property that changed history: it allows training with volumes of text and model sizes that an RNN, condemned to read word by word, could never digest.

Why it dethroned RNNs

Aspect	RNN / LSTM (module 4)	Transformer
Sequence processing	Sequential: token by token	Parallel: all tokens at once
Long-range dependencies	Degrade with distance (despite the LSTM's conveyor belt)	Distance irrelevant: any token attends to any other in one step
Bottleneck	Yes: the hidden state summarizes everything	No: one representation per token, always accessible
Training speed on GPU	Limited (does not parallelize over time)	Excellent (matrix products)
Scalability to huge models/data	Poor	The reason for the LLM era
Cost with very long sequences	Linear in length	Quadratic (every token with every token): its Achilles heel
Interpretability	Opaque hidden state	Visualizable attention weights
Still worth using?	Yes: time series, modest devices, very long sequences on a budget	The standard in NLP and expanding (vision, audio)

The LSTM in your review classifier wasn't bad; it's just that the Transformer eats better with the same resources, and above all it scales where the RNN gets stuck.

From BERT and GPT to LLMs

On top of the Transformer architecture were built the two families that define modern NLP. Both exploit the same idea as the transfer learning of 05-03: pretrain at great expense once, adapt cheaply many times.

BERT (2018) — understanding. It is pretrained by masking words at random across billions of sentences and asking the network to guess them by looking at the context in both directions (recall Bidirectional from 04-02, cubed). The result is a model that "understands" the language and can be fine-tuned in hours for sentiment, classification, entity extraction or semantic search. It is MobileNetV2's frozen base, but for text.
GPT — generation. It is pretrained on the simplest task imaginable: predicting the next word, over and over, on enormous amounts of text, attending only to the preceding context. From that humble task, at sufficient scale, emerges the ability to write, summarize, translate and converse.
Today's LLMs (GPT-4 and successors, Claude, Llama, Gemini...) are, in essence, the same recipe at brutal scale — more blocks, more heads, more data, more GPU — plus later tuning techniques for following instructions. Here closes the arc we opened in the history of 01-02: from the perceptron to AlexNet took half a century; from "Attention Is All You Need" to conversational LLMs, five years. We cover the trends and what comes next in 08-03.

What it means for TecnoMarket

The practical consequence is the moral of 05-03 applied to language: TecnoMarket should not train language models from scratch. Its sensible roadmap for the reviews:

Short term: keep the LSTM classifier from 04-03 where it already works (it is cheap and in production with its confidence threshold).
Next step: replace it with a pretrained English BERT-style model, fine-tuned on TecnoMarket's labeled reviews — same data, several points more accuracy, and better handling of negations and long sentences ("I wouldn't say it's bad, but...").
Beyond: use LLMs via API for generative tasks — summarizing a product's 400 reviews into three sentences for its page, drafting reply suggestions for customer service (with human review: the review-queue pattern from 03-04, once again).

MultiHeadAttention in Keras: a review classifier

Keras ships multi-head attention as a layer (layers.MultiHeadAttention). To get our hands on the mechanism, let's build a mini sentiment classifier that replaces the LSTM from 04-03 with a self-attention block:

from tensorflow import keras
from tensorflow.keras import layers

VOCAB_SIZE = 10000    # distinct tokens, as in 04-03
SEQ_LEN = 200         # reviews truncated/padded to 200 tokens
EMB_DIM = 64

inputs = keras.Input(shape=(SEQ_LEN,))               # token ids

# 1. Token embeddings (04-03) + POSITION embeddings (learned)
token_emb = layers.Embedding(VOCAB_SIZE, EMB_DIM)(inputs)
positions = keras.ops.arange(SEQ_LEN)                # [0, 1, ..., 199]
pos_emb = layers.Embedding(SEQ_LEN, EMB_DIM)(positions)
x = token_emb + pos_emb   # without this, attention would ignore ORDER

# 2. Multi-head self-attention: the sentence attends TO ITSELF
#    (query=x, value=x, key=x -> that's why it's "self")
attention = layers.MultiHeadAttention(
    num_heads=4,       # 4 spotlights in parallel
    key_dim=16,        # Q/K dimension per head (4 x 16 = 64)
)(query=x, value=x, key=x)

# 3. Residual connection + normalization: the Transformer block recipe
#    (the gradient highway from 03-03, in its NLP version)
x = layers.LayerNormalization()(x + attention)

# 4. From one representation PER TOKEN to one vector per review, then classify
x = layers.GlobalAveragePooling1D()(x)   # averages the 200 vectors
x = layers.Dropout(0.3)(x)               # regularization (05-04)
outputs = layers.Dense(1, activation="sigmoid")(x)   # sentiment +/-

model = keras.Model(inputs, outputs)
model.compile(optimizer="adam", loss="binary_crossentropy",
              metrics=["accuracy"])
model.summary()

The fine points:

query=x, value=x, key=x: all three roles come from the same sequence — that is exactly what makes it self-attention. (In a seq2seq with cross-attention, the query would come from the decoder and keys/values from the encoder.)
Position embeddings added in: the learned version of positional encoding. Comment out that line and watch the accuracy fall: without position, "not working well" and "working well... not" would be indistinguishable.
Residual + LayerNormalization (x + attention): the same residual-block pattern you implemented in 03-03, which lets you stack blocks without choking the gradient. For a "real" Transformer you would add the small feed-forward network and repeat the block N times.
Trained on IMDB (the dataset from 04-03), this model reaches figures similar to the LSTM (~86-88%) while training faster per epoch on GPU. The Transformer's decisive advantage isn't visible at this toy scale — it appears when scaling to enormous models and corpora, which is exactly what the pretrained models have already done for you.

We will not train a full Transformer here (nor is there any need: the lesson of 05-03 is that this pretraining is already paid for); the goal was for the mechanism to stop being a black box.

Common Mistakes and Tips

Forgetting the positional encoding. Conceptual mistake number one: attention on its own is blind to order. If your model with MultiHeadAttention performs suspiciously poorly, check that you are adding position embeddings.
Believing that attention "understands". Attention weights are learned similarities useful for the task, not human comprehension. Visualizing them helps with debugging, but over-interpreting them leads to wrong conclusions.
Discarding RNNs out of fashion. For the sales series from 04-04, with a univariate sequence and modest hardware, the LSTM remains an excellent choice. Attention's quadratic cost with long sequences is real.
Confusing the Q/K/V roles. The catalog mnemonic: the query is what you are looking for, the key is the index it is compared against, the value is what you take away. In self-attention all three come from the same place, but through different projection matrices.
Training a Transformer from scratch with little data. With IMDB's ~25,000 examples, a large Transformer overfits (apply what you learned in 05-04). The right route with little data is to start from a pretrained model — transfer learning, 05-03.

Exercises

Exercise 1. Using the 04-03 bottleneck and the RNN vs. Transformer table, explain why an LSTM tends to fail on the review "At first I thought it was the best speaker I had ever owned, with spectacular sound and a gorgeous design, but after two weeks it stopped working" and why self-attention has an easier time.

Exercise 2. Using the Q/K/V table from the numerical example, compute the new representation of the token "not" (query Q = (1, 0)): scores against the three keys, softmax (you may approximate; without scaling, for simplicity) and the blend of values. The values were: not → (−1, 0), working → (0, 1), well → (1, 1).

Exercise 3. The TecnoMarket team is debating two proposals: (A) training a Transformer from scratch on its 30,000 labeled reviews; (B) fine-tuning a pretrained English BERT on those same reviews. Argue which is preferable, connecting it to two specific lessons of this module.

Solutions

Solution 1.

The review is long and its first half is intensely positive ("the best", "spectacular", "gorgeous"); the decisive turn ("stopped working") arrives at the end, but the nuance that governs it ("but after two weeks") requires relating the ending to everything that came before. In an LSTM, information travels step by step through the hidden state: the positive sentiment from the beginning arrives diluted at the end (recency bias) and everything must fit in a single vector (bottleneck), so a mix of contradictory signals far apart is its worst-case scenario. In self-attention, the tokens "but" and "stopped working" can attend directly, with high weight and in a single step, to the praise at the beginning — distance carries no penalty — and the final representation captures the full contrast: the key to classifying it correctly as negative.

Solution 2.

Scores of Q_not = (1, 0) against each key:

with K_not = (1, 0): 1·1 + 0·0 = 1
with K_working = (0, 1): 1·0 + 0·1 = 0
with K_well = (0, 1): 0

Softmax of (1, 0, 0): e¹ ≈ 2.72 and e⁰ = 1 twice → weights ≈ (2.72, 1, 1)/4.72 ≈ (0.58, 0.21, 0.21).

Blend of values: 0.58·(−1, 0) + 0.21·(0, 1) + 0.21·(1, 1) = (−0.58 + 0.21, 0.21 + 0.21) ≈ (−0.37, 0.42). The token "not" is re-represented attending mostly to itself, but incorporating some context from "working" and "well" — with genuinely trained matrices, those distributions adjust to whatever the task needs.

Solution 3.

Option B is preferable, through two direct connections with this module:

05-03 (transfer learning): pretraining a language model demands corpora of billions of words and industrial GPU — the "luxury" someone already paid for. Fine-tuning a pretrained BERT is the same move as MobileNetV2 with the product photos: generic knowledge for free, cheap specific adaptation. 30,000 reviews are excellent for fine-tuning but laughable for pretraining.
05-04 (regularization): a Transformer with enough capacity to learn the language from scratch has enormous variance; with 30,000 examples it would memorize the training set (a giant train/val gap) no matter how much regularization you stack on. The pretrained model already brings the "features of the language" and only adjusts the last mile, with far lower overfitting risk.

(Besides, option B reaches production sooner and with better metrics — engineering counts too.)

Conclusion

The bottleneck from 04-03 had a solution: instead of compressing the sequence into one vector, attention keeps everything and weights what to look at in each moment — a three-step mechanism (Q·K, softmax, blend of V) built from pieces you already mastered. The Transformer took the idea to the limit: attention only, no recurrence, with positional encoding so as not to lose order and stacked blocks over residual connections — and its parallelism unleashed the scale that produced BERT, GPT and the LLMs, closing the story we began telling in 01-02. For TecnoMarket, the operational lesson is that analyzing its reviews means fine-tuning pretrained models, not competing with them.

This concludes the advanced-techniques module: you now know how to generate (GANs), compress and detect anomalies (autoencoders), reuse knowledge (transfer learning), generalize (regularization) and attend (Transformers). In module 6 we head down to the workshop: TensorFlow and PyTorch in depth, how to choose a framework, and how to save, load and deploy the models you already know how to build.

Attention Mechanisms and Transformers

Contents

The bottleneck, revisited

Attention: look at the whole sequence and weight it

Query, key and value: the catalog analogy

Self-attention step by step with numbers

Multi-head: several spotlights at once

The Transformer architecture at a high level

Why it dethroned RNNs

From BERT and GPT to LLMs

What it means for TecnoMarket

MultiHeadAttention in Keras: a review classifier

Common Mistakes and Tips

Exercises

Solutions

Conclusion

Deep Learning Course

Module 1: Introduction to Deep Learning

Module 2: Neural Network Fundamentals

Module 3: Convolutional Neural Networks (CNN)

Module 4: Recurrent Neural Networks (RNN)

Module 5: Advanced Deep Learning Techniques

Module 6: Tools and Frameworks

Module 7: Hands-On Projects

Module 8: Ethical Considerations and the Future of Deep Learning