At the end of 04-04 we left three uncomfortable questions unanswered: does the pipeline retrieve the right chunks? Is the 0.45 threshold wise, or a coincidence? Are the answers faithful to what was retrieved? In 02-04 we learned to evaluate prompts with a dataset of 30-50 questions, a binary rubric and a CSV — but a RAG system adds a new difficulty: it has two engines that can fail independently. If DocuBot answers badly, was it because retrieval brought the wrong chunks, or because generation ignored the right ones? Without separating those two questions, every bad answer is a mystery and every "improvement" is a blind bet. This lesson extends our evaluation infrastructure to the RAG world: we will annotate which chunks each question in the dataset should retrieve, implement the classic retrieval metrics (recall@k, precision@k, MRR) in plain Python, extend the 02-04 rubric with the criteria specific to context-grounded generation, and build the diagnostic table and the experiments (k, chunk size, overlap) that turn the design decisions of 04-03 and 04-04 into decisions backed by data.

1. Two engines, two evaluations
2. Extending the 02-04 dataset: annotating the expected chunks
3. Retrieval metrics: recall@k, precision@k and MRR
4. Generation metrics in RAG: the extended rubric
5. Component-level diagnosis: the symptom table
6. Experiments: k, chunk size and overlap

Let's follow a bad answer back to its origin. A user asks "How many members does the Pro plan allow?" and DocuBot answers something vague about invitations. There are two possible stories:

• Retrieval failed: the top-4 didn't include the "Plans and billing" chunk with the 25-member limit. The LLM did the right thing with what it had — the problem lies in the chunking, the embeddings model, k or the filters.
• Generation failed: the right chunk was in the <documentation> block, but the model ignored it, misread it or embellished it. The problem lies in the prompt, the model or the block's format.

The cures are disjoint: redoing the chunking doesn't fix a prompt problem, and polishing the prompt doesn't fix a top-k that never brings the answer. Hence this lesson's principle:

Evaluate retrieval without the LLM, and generation with known retrieval. Only then evaluate end to end.

The good news is that half the work is already done: the 02-04 dataset (30-50 questions with their 4 case types), the binary rubric, the CSV logging and the discipline of 3 runs per case remain valid. They only need to be extended.

To measure retrieval we need the ground truth: for each question, which chunks in the corpus contain its answer. It is manual work, and it is the most profitable investment in this lesson — done once, it sustains every experiment afterwards.

The practical process: for each question in the dataset, find (by hand, reading the docs, using retrieve() as a hint but verifying by eye) the ids of the chunks that answer it. The deterministic ids from 04-03 make this possible: they are stable across re-ingestions.

Three annotation notes that prevent grief:

• The "no answer" cases (~20% of the dataset) carry an empty list — and they are what calibrates 04-04's RELEVANCE_THRESHOLD: for them, the best chunk's similarity should land below the threshold.
• Ambiguous cases usually have several expected chunks: any reasonable subset may suffice to answer; the metrics will take that into account.
• Annotations expire with the chunking: if you change chunking.py (and therefore the ids), the annotations must be reviewed. One more reason to version the chunking strategy as we did with PROMPT_VERSION.

With the ground truth annotated, evaluating retrieval needs no LLM: run retrieve() and compare ids. The three standard metrics, each answering a different question:

Implementation in simple Python, no dependencies:

How to read those example numbers:

• recall@4 = 0.91: in 91% of cases (weighted by chunks) what was needed reached the context. The failing cases are the work list: examining them one by one usually reveals patterns (all from the same page? all with user vocabulary far removed from the docs?).
• precision@4 = 0.47 looks low, but it is normal: if a question needs 1 chunk and we retrieve 4, the maximum precision is 0.25. Precision is compared across configurations (does it rise when k drops?), not against an ideal of 1.0.
• MRR = 0.86: the first relevant chunk usually comes 1st or 2nd — good for our "best first" ordering.
• And the no-answer cases have their own measurement: what fraction of them lands below the RELEVANCE_THRESHOLD (should be: all) versus what fraction of the easy cases lands above it (should be: all). Those two figures are the threshold calibration that 04-04 left pending.

Suppose recall is high: the right material reaches the context. What remains is evaluating what the model does with it. The three criteria specific to generation with retrieved context:

1. Faithfulness to the context (groundedness): is everything asserted in the answer backed by the chunks in the <documentation> block? An answer can be true and still unfaithful (the model knew it "from memory" — unacceptable for DocuBot, rule 2 of SYSTEM_DOCUBOT).
2. Answer relevance: does it answer this question, or does it correctly summarize the chunks without addressing what was asked?
3. Correct use of sources: does the sources field point to the chunks actually used, and does the answer avoid attributing to a source something it doesn't say? (Since 04-04 the URLs are set by the system, so what is monitored here is content attribution, not the URL's existence.)

Instead of inventing a new rubric, we extend the 02-04 one: its 6 binary criteria gain these 3, and the evaluation follows the same protocol — human judgment, binary (pass/fail, no half measures), 3 runs per case due to non-determinism (01-04), logged to CSV:

An evaluation trick that saves hours: since the pipeline logs which chunks it retrieved (the closing tip from 04-04), the human evaluator can judge faithfulness by reading only those chunks, not the whole documentation. Evaluating criterion 7 for a case goes from minutes to seconds.

This process is manual and doesn't scale to hundreds of cases — we know. In 06-03 we will see how to automate it using an LLM as judge (LLM-as-judge) with this very rubric as its foundation; today's manual version is not throwaway work but the gold standard against which that automated judge will be validated.

With retrieval and generation metrics separated, each pathology points at its component. This table is the daily working tool:

The discipline the table imposes: when facing a bad answer, look at that case's retrieval metrics first. It is one glance at the CSV (were the expected chunks in the top-k?) and it forks the diagnosis into two paths that never mix. Without this discipline, the instinct is always "tweak the prompt" — which only cures half the diseases.

The decisions from 04-03 and 04-04 (~450-token chunks, 300-character overlap, k=4, threshold 0.45) were reasoned starting points. Now they can become conclusions. The experimental method is the one from 02-04 — change one variable at a time, measure over the full dataset, log to CSV:

Typical results of a k sweep (your numbers will vary):

The pattern is almost universal: recall grows with k but flattens (the "elbow"), precision falls monotonically, and the cost per question grows linearly with k. The decision is an informed compromise — and with this table, k=4 stops being "what the course said" and becomes "the elbow of our curve".

The other two canonical experiments follow the same mechanics, with one important operational difference:

• Chunk size (MAX_SIZE: try e.g. 1000 / 1800 / 3000) and with/without overlap (OVERLAP: 0 / 300): each variation demands re-running ingestion.py (new chunking → new embeddings → repopulate the collection) and, if the ids change, reviewing the annotations. These are expensive experiments — which is why they are run less often and recorded with CHUNKING_VERSION, so they aren't repeated out of forgetfulness.
• The generation variable (prompt, temperature) is experimented on with the extended rubric, not with recall/precision — each engine with its own metrics.

• Evaluating only end to end. "The answer is bad" without knowing whether retrieval or generation failed leads to fixing the wrong component. The separate metrics exist to fork the diagnosis.
• Optimizing precision@k in the abstract. With 1 relevant chunk and k=4, the maximum precision is 0.25 and that doesn't mean anything is wrong. Compare it across configurations; recall is what rules in RAG.
• Annotating the expected chunks using only retrieve() as the oracle. If you annotate what the system already returns, you will measure that the system returns what it returns: a guaranteed — and false — recall of 1.0. Annotation requires reading the documentation.
• Changing two variables at once (k and the prompt, or chunk size and threshold): the improvement — or the regression — will have no assignable cause. One variable, one measurement, one CSV row. We learned this lesson in 02-04, and in RAG it hurts twice as much because there are twice as many knobs.
• Forgetting the no_answer cases when evaluating. A system that answers something to 100% of questions has splendid retrieval metrics and a serious honesty problem. The two threshold-calibration figures are part of the dashboard, not an extra.
• Tip: set alarm thresholds over the dataset (e.g. "recall@4 ≥ 0.85 and zero no_answer cases answered with content") and re-run the evaluation after every re-ingestion or version change. It is the RAG version of the budget-with-alert from 03-04 — and in 06-04 we will see how to watch these signals continuously in production.

1. Complete the threshold measurement. Write evaluate_threshold(threshold), which walks the dataset and returns two figures: the fraction of no_answer cases correctly rejected (best chunk below the threshold) and the fraction of easy cases correctly accepted (above it). Run it for thresholds 0.30, 0.45 and 0.60 and choose with data.
2. MRR by hand. Without running any code: three questions whose relevant chunks appear at positions 1, 3 and (doesn't appear) of the top-4. Compute the MRR and explain which action from the diagnostic table you would trigger if that were your overall value.
3. The deceptive case. The question "How many tasks does the Pro plan allow?" gets: recall@4 = 1.0 (the limits chunk is in the top-4, position 2) but the answer says "up to 500 tasks and 50 members" when the docs say 25 members. Which rubric criteria fail, what is the diagnosis according to the table, and which two concrete actions would you try first?

1. Structure of the function:

Typical result: at 0.30 all the easy cases are accepted but several no_answer cases "sneak through"; at 0.60 everything dubious is rejected but legitimate easy cases start falling too; 0.45 usually keeps both figures high. If no threshold separates the two populations well, the problem isn't the threshold: it is that some chunks are deceptively similar to unanswerable questions — go back to the chunking, or add an explicit page to the docs ("Supported payment methods") that gives the system something correct to retrieve.

2. MRR = (1/1 + 1/3 + 0) / 3 = 0.444. It is low, but the table demands looking at the why before acting: the third case is not an ordering problem but a recall one (the chunk doesn't appear at all), so the right action starts at the "low recall@k" row (chunking, k, embeddings) for that case; the second case (position 3) is indeed about ordering and would point to reranking only if the pattern repeats. Moral: MRR is never diagnosed alone — always alongside recall.

3. Criterion 7 fails (faithfulness: "50 members" is not backed by the chunk, which says 25) and possibly criterion 9 (it attributes to the source something it doesn't say); criterion 8 passes (it answers the question). With perfect recall and broken faithfulness, the table points at generation: the context arrived and the model altered it. Two first actions: (a) reinforce in answer_prompt() the instruction to copy figures and limits verbatim from the documentation (and re-evaluate over the full dataset, not just this case — the 02-04 rule); (b) check the call's temperature (the 01-03 decision: low for DocuBot) in case some change raised it. If it persisted, check whether the chunk mixes several plans' limits in a table the model misreads — which would, curiously, hand the problem back to the chunking: diagnoses are probable, not infallible, and that is why everything is logged.

With this lesson, module 4 is complete — and DocuBot stops being a pipeline that works in demos to become a RAG system we know how to measure and diagnose: the 02-04 dataset now has the expected chunks annotated for each question (with the no_answer cases as threshold calibrators), eval_rag.py measures retrieval without spending a single LLM token (recall@k as the reigning metric, precision@k and MRR as companions that are read together), the 6-criterion rubric grew to 9 with faithfulness to the context, relevance and source usage, the symptom table forks every bad answer toward its true culprit — retrieval or generation, each with its own cures — and the CSV-logged experiments turned k=4, chunk size and overlap from inherited values into the measured elbow of our own curve. Looking back, module 4 delivered on the promise that closed module 3: embeddings went from intuition to tool (04-01, with the semantic cache as a bonus), the docs_nubelia collection gave the search persistent, filterable memory (04-02), ingestion.py turned hundreds of pages into well-crafted chunks with traceable sources (04-03), and answer_with_rag() finally filled, automatically, that <documentation> block which SYSTEM_DOCUBOT had demanded since 02-01 — burying the naive DocuBot that invented PDF exports. But note the verb that defines everything we built: DocuBot knows. It knows what the documentation says and reports it with sources. What it still cannot do is act: if a user asks "How many tasks does my Atlas project have right now?", the answer is in no wiki — it is in the Nubelia API, one call away, a call DocuBot cannot make. Giving it hands as well as memory is the leap of module 5: starting with function calling (05-01) — the mechanism by which the LLM stops merely writing answers and starts invoking our functions — and continuing until DocuBot becomes an agent that reasons, acts and observes in a loop. It has mastered the documentation; now it is going to learn to use the product.