In the previous lesson we treated the LLM as a black box that receives text and returns text. In this lesson we're going to open that box just enough for you to work with it effectively. You don't need to master the matrix math happening inside a transformer, just as you don't need to know B+ trees to use PostgreSQL; but you do need to understand tokens (because they determine the limits and cost of every call), embeddings (because they're the foundation of the semantic search DocuBot will use), the idea of attention (because it explains why the model handles context so well) and the fact that everything boils down to predicting the next token (because it explains both the model's capabilities and its failures). This practical view will let you reason about the model's behavior instead of treating it as magic.

1. The central idea: predicting the next token
2. Tokenization: how an LLM sees text
3. Embeddings: representing meaning with numbers
4. The transformer architecture and the attention mechanism
5. From prediction to response: sampling, temperature and top_p
6. How an LLM is trained: pre-training, fine-tuning and RLHF

Surprising as it may seem, everything an LLM does boils down to a single operation repeated many times:

Given an input text, compute which fragment of text (token) is most likely to come next.

When you ask DocuBot "What is Nubelia?", the model doesn't look the answer up in a database or run any symbolic reasoning. It generates the answer token by token: it predicts the first token of the answer, appends it to the text, predicts the next one taking everything before it into account, and so on until it predicts a special "end" token.

In pseudocode:

Note two direct consequences of this design, which will resurface throughout the course:

• The model always generates something. There's no built-in "I don't know" mechanism; if the most probable text following a question is an answer that looks certain, that's what it produces, true or not. Here lies the seed of the hallucinations we'll analyze in lesson 01-04.
• The input text is everything. The only way to influence the prediction is to change what the model receives. That's why prompting (module 2) and context management (modules 3 and 4) matter so much.

That such a simple mechanism produces translations, working code or reasoning is a matter of scale: to predict the next token well for any text, the model is forced to learn the grammar, facts, styles and reasoning patterns present in its training data.

We've said "token" several times. Let's define it properly, because it's the most practical concept in the whole lesson.

An LLM doesn't process characters or words: it processes tokens, variable-length fragments of text produced by splitting the input against a fixed dictionary (the vocabulary, typically 50,000 to 200,000 tokens). The splitting process is called tokenization, and it's performed by a deterministic component called the tokenizer before the text ever reaches the neural network.

• Frequent words are usually a single token: " the", " of", " project".
• Rare or long words get split into several: "Nubelia" might tokenize as ["Nub", "elia"].
• Spaces and punctuation marks are part of the tokens (often the leading space is attached to the word).
• Code gets tokenized too: def, parentheses, indentation... it all consumes tokens.

An illustrative example (the exact splits depend on each tokenizer):

As a quick mental rule for estimating: in English, 1 token ≈ 4 characters ≈ 0.75 words. English is where most models are at their most token-efficient, because the vocabularies of most tokenizers were built on predominantly English text; other languages typically consume noticeably more tokens per word (often 10–30% more, sometimes far more for languages with different scripts).

A classic example: if a model fails to count the "r"s in a word, it's not because it "can't count"; it's because it never sees the individual letters — it sees one or two opaque tokens representing the whole word.

For DocuBot this has a direct consequence we'll come back to in module 4: Nubelia's documentation takes up far more tokens than fit in a single call, so we won't be able to "hand the whole wiki" to the model; we'll have to select the fragments relevant to each question.

A neural network only operates on numbers, so the first step after tokenizing is to convert each token into a vector: a list of hundreds or thousands of decimal numbers. That vector is the token's embedding.

The fascinating thing about embeddings is that they're not arbitrary codes: they capture meaning. During training, the model learns to place tokens (and, by extension, phrases) that appear in similar contexts at nearby positions in the vector space. The result is a kind of "map of meaning":

"Closeness" is measured with simple geometric operations (like cosine similarity, the angle between two vectors). And the concept scales beyond individual tokens: there are specialized models that produce the embedding of an entire sentence or document, so that "how do I restart the reporting service?" and "restarting the reporting module" end up with very close vectors even though they share almost no words.

Can you see where this is heading for DocuBot? If we can turn each fragment of Nubelia's documentation into a vector, and the user's question too, then finding the relevant documentation becomes searching for the nearest vectors. That is semantic search, the centerpiece of RAG, and we'll develop it in lesson 04-01. For now, hold on to the idea: embedding = meaning turned into coordinates.

Inside the LLM itself, the embeddings of the input tokens are the starting point: the first layer of the network takes them and, layer by layer, refines them with information from their context. What performs that refinement is, precisely, attention.

As we saw in the previous lesson, the transformer architecture (2017) is the milestone that made LLMs possible. Its central innovation is the attention mechanism. Let's look at it conceptually, without math.

The problem it solves: the meaning of a word depends on its context, and that context can be far away. Look:

"The support team checked the test bench before the deployment." "She sat on the park bench to read the documentation."

To represent "bench" properly, the model needs to look at other words in the sentence — and decide which ones matter. That's exactly what attention is: a mechanism by which each token computes how much importance it should give to every other token in the context, and updates its representation (its embedding) by blending in information from the relevant tokens.

A useful analogy for developers: attention works like a weighted query. Each token issues a "query" to the rest of the tokens ("which of you are relevant to me?"), each token responds with what it offers, and the result is a weighted average where the relevant tokens weigh more.

On top of this idea, the full architecture is organized as follows:

• A transformer stacks dozens of layers, each with multiple attention "heads" running in parallel (each head learns to focus on different kinds of relationships: syntax, coreference, code patterns...), alternated with small neural networks that transform each position.
• Layer after layer, each token's representation is enriched with more and more context: from "which token am I" to "what exactly do I mean in this sentence, this paragraph and this document".
• At the output of the last layer, the representation of the final token is used to compute the probability of every possible next token — and we're back to the lesson's central idea.

Two advantages of the transformer that explain its success over the earlier recurrent networks:

1. Parallelism: attention processes all tokens at once (not one after another), which makes it possible to train on gigantic volumes of data.
2. Reach: any token can attend directly to any other, no matter how far away, so long-distance dependencies don't get "diluted".

A model's famous parameters (when people talk about models "with billions of parameters") are precisely the learned numbers that govern these layers: the weights of the attention and of the transformations. More parameters usually means more capacity to capture patterns — and a higher cost to run.

With this you have the right level of detail: enough to reason about the model, without getting lost in the internal mechanics.

The model produces a probability distribution over the next token. But which one gets chosen? This is where sampling comes in, and with it the two parameters you'll see in every LLM API: temperature and top_p.

The most obvious option would be to always pick the most probable token (greedy sampling). It works, but it produces repetitive, predictable text, and for creative questions it always gives the same output. So in practice the token is chosen at random, weighted by its probability — and the sampling parameters control how much randomness.

• temperature (typically between 0 and 1, sometimes up to 2): rescales the probability distribution.
  • Near 0: sharpens the differences — probable tokens become near-certain. More deterministic, conservative, repeatable outputs.
  • Near 1: respects the original distribution — more variety and "creativity", but also more risk of going off track.
• top_p (nucleus sampling): instead of considering all tokens, it considers only the smallest set of tokens whose cumulative probability reaches p (for example, 0.9), and samples only among them. It trims off the "tail" of improbable tokens.

An initial practical guide (we'll refine it when we integrate the API in module 3):

An important warning we'll return to in 01-04: not even with temperature = 0 is the output 100% guaranteed to be identical across calls. Treat LLMs as non-deterministic components by design.

The last piece of the puzzle: how does the model get so good at predicting? Training a modern LLM has three main phases. You won't train a model in this course, but knowing the phases explains behaviors you'll see every day.

The model is trained on enormous amounts of text (websites, books, public code...) on a single task: predicting the next token. Trillions of tokens, weeks or months of compute on thousands of GPUs. This is where the model's general knowledge comes from: languages, facts, styles, programming languages.

Two practical consequences:

• The model's knowledge is frozen at the date of its data (the knowledge cutoff, which we'll analyze in 01-04).
• The model only knows what was in that data. Nubelia's internal wiki, obviously, wasn't.

The result of this phase (the base model) is a very powerful text completer, but not an assistant: if you type it a question, it may well continue with more similar questions, because that's a statistically plausible text.

To turn the completer into an assistant, the model is retrained on a much smaller, curated dataset of conversation examples: user instruction → desired response. The model learns the format of behaving like an assistant: answering what's asked, following instructions, maintaining a dialogue.

Finally, the quality and style of the responses are tuned: people rate and compare model responses ("this one is better than that one"), a reward model is trained on those preferences, and the LLM is adjusted through reinforcement learning to produce better-rated responses: more helpful, safer, better explained.

A subtle consequence of RLHF worth knowing: by optimizing for responses people "like", models develop a certain tendency to sound confident and agreeable, even when they shouldn't. Yet another piece of the hallucination puzzle we'll assemble in lesson 01-04.

• Confusing words with tokens. Model limits and prices are expressed in tokens. If you estimate "this document is 2,000 words, plenty of room", you may be in for a surprise — and in languages other than English the token count climbs even faster. Always use the tokenizer (or token counter) of the specific model to estimate.
• Believing the model "looks up" information when it answers. A pure LLM doesn't query any database at response time: it generates from its parameters and from the text you've passed it. If you need current or private data, you'll have to put it in the context yourself (modules 3 and 4).
• Blaming "lack of intelligence" for errors that are really about tokenization. Counting letters, reversing strings or doing arithmetic with very long numbers are tasks where the token representation works against the model. For those tasks, use code, not an LLM.
• Raising the temperature to "improve" factual answers. It's the other way around: for DocuBot-style tasks (answers grounded in documentation) you want a low temperature. High temperature is for tasks where variety adds value.
• Expecting absolute determinism. Design your application assuming two identical calls can return different text. Validate outputs instead of comparing them against an exact string (this will be key when testing, in module 6).
• Tip: when a model's behavior baffles you, mentally walk back through this lesson's pipeline — what tokens is it seeing? what was in its context? what text would be statistically plausible next? — and most "mysteries" dissolve.

The Nubelia team wants to pass DocuBot, on each call, a documentation fragment of about 1,200 words in English along with the user's question (about 30 words). Using the rule of thumb that in English 1 token ≈ 0.75 words (i.e. 1 word ≈ 1.33 tokens):

1. Estimate how many input tokens each call will have (ignore additional instructions).
2. If the chosen model had a context window of 8,000 tokens, what percentage of the window would the input consume?
3. Why is this estimate only indicative, and what would you do to get the real figure?

For each of these three uses of the LLM at Nubelia, indicate whether you'd use a low (≈0–0.3), medium (≈0.5–0.7) or high (≈0.8–1.0) temperature, and justify:

1. DocuBot answering "which ports does the reporting service use?" from a documentation fragment.
2. Generating 10 varied name proposals for a new platform feature.
3. Summarizing the notes from a technical meeting in one paragraph.

A colleague asks you: "Why do people say attention was the key to LLMs? What does it actually do?". Write a 4–6 line explanation, without math, using the sentence "The test bench failed" versus "She sat on the bench" as your example.

Solution 1:

1. Document: 1,200 × 1.33 ≈ 1,600 tokens. Question: 30 × 1.33 ≈ 40 tokens. Total ≈ 1,640 input tokens.
2. 1,640 / 8,000 ≈ 21% of the window. There's room to spare, but remember the window must also hold the system instructions, the conversation history and the generated response.
3. It's indicative because the words-to-tokens ratio depends on each model's tokenizer and on the type of text (technical text with proper nouns like "Nubelia" or code identifiers tends to fragment into more tokens). For the real figure, I'd tokenize the text with the specific model's token-counting tool or endpoint.

Solution 2:

1. Low. It's a factual answer grounded in documentation: we want accuracy and consistency; creativity here is a defect.
2. High. The goal is variety: with a low temperature the 10 proposals would look too similar to each other (and across runs).
3. Low or medium-low. A summary must be faithful to the content; some flexibility of wording is acceptable, but we don't want it to "embellish" or introduce ideas that weren't in the notes.

Solution 3 (example answer):

"Before transformers, models processed text word by word and struggled to relate distant words. Attention lets the model, when processing each word, look at all the other words in the text and decide which ones are relevant to interpreting it. For example, 'bench' means nothing specific on its own: in 'the test bench failed', attention gives weight to 'test' and 'failed' and the model interprets it as technical infrastructure; in 'she sat on the bench', the weight goes to 'sat' and it's interpreted as furniture. And since this is computed for all words in parallel, it became possible to train on enormous volumes of data — and out of that scale came LLMs."

Now you know what's inside the box: an LLM tokenizes the text (which is why limits and costs are measured in tokens), represents each token as an embedding that captures meaning (an idea that will be the foundation of DocuBot's semantic search in module 4), refines those representations with layers of attention that decide which parts of the context matter, and with all of that predicts the next token, over and over, with a degree of randomness controlled by temperature and top_p. You also know the three training phases — pre-training, fine-tuning and RLHF — and their consequences: broad but time-frozen knowledge, assistant behavior, and a certain tendency to sound more confident than it should.

With this mental model you can now critically evaluate what the market offers. In the next lesson, "Model and Provider Landscape", we'll look at which model families exist, which criteria to use to compare them (capability, context, cost, latency, modalities) and how to decide between proprietary models via API and self-hosted open models — decisions Nubelia will have to make before writing the first line of DocuBot.