Throughout the module we've written DocuBot's system prompt, a few-shot classifier, and a JSON output contract. Each piece seems reasonable — but "seems reasonable" is not an engineering criterion. When in a few weeks someone proposes "add a rule to make DocuBot friendlier", how will you know whether that rule improves the assistant or silently breaks the anti-hallucination behavior that took so much effort to tune? Any engineer's answer to a question like that is the same: tests. This lesson turns prompting into a disciplined process: the prompt as a versioned artifact, an iteration cycle with hypothesis and measurement, the construction of the set of 30-50 real questions we announced in 01-03 when choosing DocuBot's model, simple rubrics for scoring responses, and a fair way to compare variants despite the non-determinism we've known about since 01-04. All manual and with minimal tools (a table, a CSV): automation arrives in 06-03, and here we build exactly what that automation will need.

1. The prompt as an engineering artifact
2. The iteration cycle: hypothesis → change → test
3. DocuBot's evaluation set: 30-50 real questions
4. Quality criteria and simple rubrics
5. Comparing variants fairly
6. Recording results: table and CSV

A production prompt is not a sentence you write once: it's an artifact that evolves, that user-visible behavior depends on, and that several people will touch. It deserves the same treatment as code:

• It lives in the repository. We already set this in motion in 02-02: the templates are in prompts.py, under version control. No loose prompts in shared documents or in a provider's web console, where nobody knows who changed what.
• It has an explicit version. Nothing sophisticated is needed: a constant and a changelog in the module itself are enough to start.

• Changes get reviewed. A prompt change is a change in the system's behavior: it goes through review just like a code change. The difference — and hence this lesson — is that the human reviewer can't "read" the effect of a prompt the way they read an if: they need test results.
• Every version can be reproduced. If DocuBot answered well on Monday and badly on Thursday, you want to be able to answer "which exact version of the prompt was running on Monday?". Versioning is what makes that question possible. (In module 6 we'll see that the model and its parameters are also part of that reproducible snapshot.)

The mental shift is this: in traditional software, behavior is guaranteed by the compiler and verified by tests; with LLMs, the "source code" includes natural-language text whose effect is only known empirically. That's why the rest of the lesson is about building that empirical verification.

The natural (and wrong) way to iterate on prompts is: I try one question, I don't like the response, I tweak the prompt, I try that same question, now it comes out fine, I declare the change good. The problem has a name in engineering: regression. The tweak that fixes question A can break questions B, C, and D — and with a system prompt of interdependent rules like DocuBot's, it happens constantly. For example: you add "be warmer and friendlier" to improve the tone, and the model — pushed by the RLHF sycophancy we saw in 01-02 — starts softening the "I can't find that information…" with helpful additions that border on invention. You fixed the tone, you broke rule 3.

The disciplined cycle has four steps:

1. Hypothesis. State the problem in an observable way: not "the responses are weak", but "in 4 of the 10 billing cases, DocuBot doesn't cite the source". A good hypothesis includes the suspected cause: "rule 4 is far from the end of the prompt and gets lost with long contexts (lost in the middle)".
2. Change: just one. If you touch three things at once and the result improves, you don't know which of the three acted (or whether one of them made something worse that another compensated for). One change per iteration — the old rule of changing one variable at a time.
3. Test with the WHOLE set of cases, not just the case that motivated the change. This is where regressions get caught.
4. Decision by comparing versions. With the results of both versions on the same cases (section 5), you decide: adopt (and bump PROMPT_VERSION, noting why) or discard and go back to the hypothesis.

This loop is handcrafted on purpose: doing it by hand with 30-50 cases teaches you what to look at, and that judgment is exactly what you'll automate in 06-03.

In 01-03, when choosing a model for DocuBot, we left an announcement pending: an "in-house evaluation of 30-50 real questions". The time has come to build it. An evaluation set (eval set) is a fixed list of input cases, each with its expected result or its correctness criterion. The keys:

Where the questions come from. From reality, not from your imagination: Nubelia's historical support tickets, the wiki searches, the questions from the internal Q&A channel. Questions invented by whoever wrote the prompt suffer the obvious bias: you unconsciously write questions your prompt answers well. If the product has no traffic yet, ask colleagues who haven't seen the prompt to write questions.

Coverage by case type. A good set is not a collection of easy questions: it's a map of everything that can arrive. For DocuBot, four types with indicative proportions:

Two types deserve comment:

• The "no answer" cases are the most valuable in the set. They are the direct proof that the naive DocuBot of 01-04 has been left behind. And they're easy to evaluate: the expected response is literally the fixed phrase of rule 3 with confidence: "low" — that's why we pinned an exact phrase in 02-01: so we could check it.
• The mildly adversarial ones are not a full security test (that's 06-01): they are the everyday clumsiness and mischief. The false premise is especially treacherous: "where is the clone-project button?" presupposes it exists, and a sycophantic model tends to "find" it.

Every case carries its expected result. The minimum format per case: id, question, type, expected_category (for the classifier), and criterion (what the response must satisfy). For the easy ones, the criterion can include the source it must cite; for the "no answer" ones, the fixed phrase; for the ambiguous ones, sometimes more than one category is acceptable — note it down (permissions|integrations).

Why 30-50 and not 500? Because you'll evaluate them by hand on every iteration and it has to be doable in an afternoon. 30-50 well-chosen cases detect most regressions; the set will grow over time (every real production failure is a new case — the same idea as the broken outputs of 02-03), and when it grows too big to evaluate by hand, that will be the signal to automate (06-03).

"Is this a good response?" is too vague to score consistently — the same person scores differently on a Monday and on a Friday. A rubric decomposes "good" into binary or nearly binary criteria, each verifiable by looking at the response:

Design notes:

• Binary criteria whenever possible. "Does it cite the correct source? yes/no" is scored the same today and tomorrow; "quality of the citation, from 1 to 10" is not. Only tone allows an intermediate point, and even that is bounded.
• Separate criteria, not one global grade. A response can be correct and not cite a source. If you only note "7/10", you lose the information about what failed, which is exactly what feeds the next hypothesis of the cycle.
• Some criteria are automatable already. "Valid JSON?" is parse_docubot_response from 02-03; "expected category?" is a string comparison; "does it say the I-don't-know phrase?" is an in. Even though this lesson is manual, write these checks as Python functions right away: they are the first pieces of 06-03.
• Correctness is judged by whoever knows the docs. For DocuBot, the reference is the Nubelia documentation included in the case's prompt — the evaluator checks against it, not against their memory.

With the rubric, each evaluated case produces a row of numbers instead of an impression. And 50 rows of numbers can be compared; 50 impressions cannot.

Suppose you have prompt v1.2 and a variant v1.3 (hypothesis: repeating the anti-hallucination rule at the end will reduce inventions in long contexts). For the comparison to be fair, three conditions are needed:

1. Same cases. Both versions are run over the complete evaluation set, no exceptions. Comparing v1.2 on some cases and v1.3 on others says nothing.
2. Everything else equal. Same model, same temperature, same documentation in the context. If you change prompt and model at once, the double-change problem returns: you won't know what to attribute the difference to.
3. Several runs per case. Here the non-determinism of 01-04 reappears: the same version, with the same case, can succeed sometimes and fail others — even at low temperature. A single run per case can give you a difference between versions that is pure chance. The practical mitigation: run each case 3 times per version and work with the proportion of passes (a criterion that passes 3/3 is solid; 2/3 is a gray zone; and a case that oscillates between runs is, in itself, a finding: it points to an ambiguous instruction).

With that, the comparison boils down to a per-criterion table:

Reading: the hypothesis is confirmed (groundedness +15 points, precisely in the "no answer" and false-premise cases) and there are no relevant regressions (variations of ±1-2 points with 3 runs per case are expected noise). Decision: adopt v1.3. If instead groundedness had gone up at the cost of, say, DocuBot answering "I don't know" on easy cases too (correctness collapsing), the table would have shown it — that balance between "don't invent" and "don't become uselessly cautious" is THE central trade-off of a documentation assistant, and it can only be managed by measuring both sides.

Two warnings of statistical honesty: with 30-50 cases, small differences (±3-5 points) are not conclusive — beware the temptation to adopt changes on marginal improvements; and don't iterate so much against the same set that you end up fitting the prompt to those 50 cases (the equivalent of overfitting): refresh and expand the set periodically.

None of the above is any use if the results live in your memory. The minimum viable record is one CSV per evaluation run — readable with any spreadsheet and processable with Python:

And a small helper to aggregate (again: Python for tools, not for calling any API):

Good record-keeping practices:

• One row per case and run, not per case: this preserves the variability between runs (the run column), which as we saw is information, not noise to hide.
• The prompt version on every row. It's what lets you compare v1.2 against v1.3 months later, or detect that a regression appeared exactly between two versions.
• Also save the complete responses (in separate files, referenced by case_id): the table says that case N-031 failed on format; the saved response says how, and that how is the next hypothesis.
• The CSV goes into the repository next to the prompt. Evaluation and evaluated artifact share history: whoever reviews the v1.3 prompt change sees, in the same commit, the evidence that it improves things.

This handcrafted record is deliberately identical in structure to what the automated evaluation of 06-03 will produce: same cases, same rubric, same columns — except a script will run it (and, for criteria like tone, another LLM as judge, with the caveats we'll see there). Nothing built here gets thrown away.

• Testing only the case that motivated the change. It's THE mistake. Every iteration is validated against the complete set; otherwise, every fix is a potential regression.
• Changing several things at once. Whether it improves or worsens, you won't know why. One change per version, and the version noted in the history.
• An evaluation set made only of easy cases. It will always give 95% pass rates and detect nothing. The value is in the ambiguous ones, the "no answer" ones, and the adversarial ones — the cases that make you uncomfortable are the ones doing the work.
• Having the same person who wrote the prompt write the evaluation questions. Guaranteed bias. Use real questions or ask for cases from someone who hasn't seen the prompt.
• Scoring with a global "gut feeling" grade. Without a rubric, the evaluation measures your mood. Separate binary criteria, recorded per column.
• Drawing conclusions from a single run. Non-determinism turns single-run comparisons into coin tosses. Three runs per case, minimum, to decide between versions.
• Recording nothing. Without a CSV or versions, a month from now you'll repeat the same discussions ("hadn't we already tried putting the rule at the end?") without data.
• Tip: turn every production failure into an evaluation-set case before fixing it. It's the prompt version of "write the test that reproduces the bug before fixing it", and it guarantees the set evolves toward real failures, not imagined ones.

Exercise 1. Write 8 cases for DocuBot's evaluation set respecting the lesson's proportions (3 easy, 2 ambiguous, 2 with no answer in the docs, 1 mildly adversarial). For each case, give: id, question, type, expected_category, and correctness criterion. Context: the available documentation covers roles and permissions, project archiving, basic billing, and the Slack integration. (Invent plausible Nubelia-employee questions; don't reuse the ones from the lesson.)

Exercise 2. Your colleague proposes moving from v1.3 to v1.4 with this justification: "I added 'be more brief' to the prompt and also switched to a cheaper model; I tried it with 5 questions I made up and all 5 came out fine, so it's better and cheaper". Point out all the methodological problems in this claim and describe the correct experiment to evaluate the proposal.

Exercise 3. These are the aggregated results (proportion of passes, 3 runs × 40 cases) of an iteration whose change was adding to the system prompt: "If you are not completely sure, answer that you can't find the information":

Would you adopt v1.4? Interpret what has happened and propose a better hypothesis for the next iteration.

Solution 1. One possible solution (yours will differ; what's gradable is the structure):

Notice N-107: the question is legitimate and clearly categorized, but the available documentation doesn't cover it — the "no answer" type depends on the provided context, not on the question.

Solution 2. Problems: (1) two changes at once (prompt and model): any difference is unattributable; (2) cases invented by the proposer and not from the evaluation set: self-confirmation bias; (3) 5 cases don't cover the types (where are the "no answer" and adversarial ones?); (4) a single run per case: with non-determinism, 5/5 can be luck; (5) no comparison with the previous version on the same cases: "they came out fine" is not "they came out better"; (6) nothing recorded or versioned. Correct experiment: split it into two iterations (first the prompt change with the current model; then, if warranted, the model change with the winning prompt); each iteration runs the complete set of 30-50 cases, 3 times per case, scored with the rubric, recorded in a CSV with its prompt_version, and the decision is made by comparing the criteria table between versions — explicitly watching that "more brief" doesn't degrade correctness, sources, or the anti-hallucination phrase.

Solution 3. I would not adopt v1.4 as-is. What has happened: the instruction "if you are not completely sure…" has made DocuBot hyper-cautious — groundedness and the "no answer" cases improve (0.98, 0.99), but at the cost of collapsing correctness on easy cases (0.93 → 0.71): it now answers "I can't find the information" to questions that are in the documentation. It's the central trade-off seen in the lesson: caution has been bought by paying with usefulness, and an assistant that doesn't answer what it does know fails at its job. A better hypothesis for the next iteration: replace the subjective criterion ("completely sure") with the verifiable criterion that rules 2-3 of 02-01 already used, reinforcing it: "Answer with confidence when the information is in <documentation>; use the 'I can't find…' phrase only when it is not" — and check in the next run that the easy cases recover to ≥0.90 without groundedness dropping below 0.95.

With this lesson, the prompt engineering module is complete — and complete means something very concrete: DocuBot is no longer an idea, it's a versioned artifact with its evaluation. You have the system prompt with identity, anti-hallucination rules, and tone (02-01); the techniques for demanding tasks — few-shot for the support classifier, chain of thought for the ambiguous cases, task decomposition — and the Python templates that package it all (02-02); the JSON contract category/answer/sources/confidence with its defensive validation (02-03); and now the discipline that holds it all together: versions with a changelog, the hypothesis → change → test cycle, the set of 30-50 real questions with its four case types, rubrics of binary criteria, fair comparisons with multiple runs, and a CSV record that turns every prompting decision into a decision backed by data. But everything we've "run" in this module has been on paper: written prompts and expected responses. In module 3 the system comes to life: we'll make the first real call to an LLM API from Python, finally send the system/user roles we designed in 02-01, connect the templates in prompts.py and the parser from 02-03 to real responses, and face what paper doesn't show: streaming, errors, retries, and the real cost of every token.